mgajewskik/openshift

OpenShift / OKD

Production-first guidance for OpenShift Container Platform, OKD, and related OpenShift-family distributions. Optimize for mechanism-level diagnosis, operator ownership, version-aware procedures, safe day-2 operations, and public-safe examples. Skip generic Kubernetes tutorials unless the user asks.

This skill distills a senior OpenShift operator corpus into progressive references. Load only the nearest reference for the task.

Start Here

Classify the request first, then read the smallest useful reference.

• Core thesis, Kubernetes-to-OpenShift translation, senior heuristics, shallow-understanding checks -> read references/mental-models.md
• Cluster anatomy, request flow, control/data plane, ClusterOperators, CR ownership, stateful vs regenerable components -> read references/architecture-lifecycle.md
• OCP vs OKD vs managed OpenShift vs MicroShift, support boundaries, lifecycle caveats -> read references/distributions-versions.md
• IPI/UPI/Agent-based/Assisted/SNO/compact/HCP installs, disconnected and air-gapped mirroring -> read references/install-airgap.md
• RHCOS/SCOS, MachineConfig, MCP, MCO render-drain-reboot, CVO, updates, EUS upgrades -> read references/cvo-mco-rhcos.md
• OLM, OperatorHub, CatalogSource, Subscription, InstallPlan, CSV, operator upgrades -> read references/olm-operators.md
• OVN-Kubernetes, Routes vs Ingress, DNS, MetalLB, EgressIP/EgressFirewall, NetworkPolicy/ANP -> read references/networking-routes.md
• SCC/PSA, OAuth, identity providers, kubeadmin, ServiceAccount tokens, auth incidents -> read references/security-identity-scc.md
• LSO, LVMS, ODF, StorageClasses, image registry storage, OpenShift Virtualization/KubeVirt -> read references/storage-virtualization.md
• Monitoring, user workload monitoring, logging/Loki/Vector, audit forwarding, Pipelines, GitOps, Service Mesh -> read references/platform-addons.md
• Incidents, upgrades, etcd, certs, must-gather, inspect, node debug, famous caveats, quick probes -> read references/day2-runbooks.md

Use This Skill For

• designing, installing, reviewing, or troubleshooting OpenShift/OKD clusters
• operating CVO, MCO, OLM, ingress, network, storage, security, monitoring, and logging planes
• disconnected or air-gapped install and upgrade planning
• day-2 incidents: degraded ClusterOperators, stuck MCPs, SCC denials, image pulls, registry, certs, etcd, route failures, OLM failures
• translating vanilla Kubernetes patterns into OpenShift-native patterns
• building public-safe runbooks and procedures without local/customer-specific details

Do Not Use This Skill For

• generic Kubernetes advice when no OpenShift-specific behavior matters
• blindly executing live-cluster mutation commands
• managed-service-specific console workflows unless the user names ROSA, ARO, or OSD
• subscription/legal/compliance conclusions beyond operational support-boundary caveats
• exposing customer names, local paths, internal hostnames, credentials, tokens, pull secrets, or raw incident artifacts

Default Operating Stance

• Start with oc get co, oc get clusterversion, oc get mcp, events, and relevant operator status before workload-level guessing.
• Find the owning operator and source CR. Do not patch generated Deployments, DaemonSets, static-pod resources, or rendered MachineConfigs when a source CR owns the behavior.
• Treat OpenShift behavior as version-sensitive. Ask for or inspect OCP/OKD minor, z-stream, platform, topology, and installed operator versions when relevant.
• Prefer read-only probes first. Mutation requires preflight, blast-radius statement, rollback or restore path, and post-checks.
• For production upgrades, storage mutation, network policy, SCC grants, kubeadmin removal, global pull-secret changes, etcd work, or force-upgrades, require explicit user approval.
• Distinguish OCP, OKD, managed OpenShift, and MicroShift support and lifecycle boundaries.
• Keep examples generic and public-safe: use example.com, registry.internal.example.com, idp.example.com, and placeholder names.

Core Mental Models

1. OpenShift is Kubernetes plus opinionated platform userland. Identity, ingress, monitoring, builds, registry, pod security, OS lifecycle, and upgrades are load-bearing defaults.
2. The cluster manages itself through operators. Find the ClusterOperator, find the CR it watches, change the source of truth.
3. The OS is part of the cluster. RHCOS/SCOS node state is rendered by MCO, applied by MCD, and normally changed through MachineConfig-family CRs.
4. SCCs are the Kubernetes-to-OpenShift footgun. Pod admission is service-account/SCC driven, priority-sensitive, and stricter than many upstream Helm charts expect.
5. CVO owns platform version; OLM owns add-on operators. Do not conflate platform upgrades with OperatorHub-installed operator upgrades.
6. Etcd is the cluster. Most platform components can regenerate from CRs; etcd state and workload PV data are the real durability boundaries.
7. Disconnected OpenShift is a coupled supply-chain system. Images, mirror substitution, signatures, catalogs, update graph, pull secrets, trust bundles, and telemetry assumptions must align.
8. Routes are native; Ingress is a compatibility surface. For OCP-native HTTP routing, reason through Route -> HAProxy router -> Service endpoints -> OVN.
9. Node changes are reboot-shaped. MachineConfig, KubeletConfig, ContainerRuntimeConfig, pull-secret and mirror changes can roll MachineConfigPools.
10. Supportability is an architecture constraint. OKD, community operators, force upgrades, and manual node drift may be technically possible but operationally different.

Interview Triggers

Ask focused questions before final guidance when any are true:

• the request mutates production cluster state, node OS, networking, storage, SCC, identity, ingress certs, pull secrets, upgrade state, or etcd
• OpenShift minor/z-stream, OKD/OCP distinction, platform topology, or disconnected status affects correctness
• the user asks for “best practices” without topology, workload, support, or failure-domain context
• the request involves force-upgrade, Upgradeable=False, EUS, paused MCPs, quorum, restore, or node replacement
• public documentation or reusable skill content could leak local/customer specifics

High-value questions:

1. Which distribution and version: OCP, OKD, MicroShift, ROSA/ARO/OSD? Which minor/z-stream?
2. What topology: SNO, compact 3-node, HA control plane + workers, HCP, bare metal, vSphere, cloud?
3. Connected, proxy-only, or disconnected/air-gapped? Is a local mirror/OSUS in use?
4. What is the first platform evidence: oc get co, oc get clusterversion, oc get mcp, relevant events/logs?
5. Is the operation read-only, reversible, disruptive, or destructive?
6. What is the backup/rollback/restore path, especially etcd snapshot and workload PV backup?

Mode Router

Choose one primary mode and at most one secondary mode.

| Mode | Use when | Load | |---|---|---| | model | learning, architecture, Kubernetes-to-OpenShift translation, mental-model checks | references/mental-models.md | | architecture | control/data plane, request flow, operator ownership, API groups, state boundaries | references/architecture-lifecycle.md | | distribution | OCP vs OKD vs managed OpenShift vs MicroShift, support/lifecycle decisions | references/distributions-versions.md | | install | IPI, UPI, ABI, Assisted, SNO, compact, HCP, install-config risk | references/install-airgap.md | | airgap | oc-mirror, IDMS/ITMS, signatures, CatalogSource, OSUS, mirror registry | references/install-airgap.md | | upgrade | CVO, channels, EUS, Upgradeable=False, conditional updates, MCO rollout | references/cvo-mco-rhcos.md + references/day2-runbooks.md for runbook detail | | nodes | RHCOS/SCOS, MachineConfig, MCP, KubeletConfig, ContainerRuntimeConfig, node drift | references/cvo-mco-rhcos.md | | operators | OLM, OperatorHub, Subscription, InstallPlan, CSV, catalog issues | references/olm-operators.md | | network | OVN-K, DNS, NetworkPolicy, ANP/BANP, EgressIP, MetalLB | references/networking-routes.md | | ingress | Routes, Ingress, router, wildcard certs, route 503s, Gateway API | references/networking-routes.md | | security | SCC, PSA, OAuth, IdP, kubeadmin, service-account tokens, RBAC | references/security-identity-scc.md | | storage | StorageClasses, LSO, LVMS, ODF, registry storage, snapshots | references/storage-virtualization.md | | virtualization | OpenShift Virtualization, KubeVirt, VM storage/networking/live migration | references/storage-virtualization.md | | addons | monitoring, logging, audit forwarding, Pipelines, GitOps, Service Mesh | references/platform-addons.md | | incident | degraded CO, stuck MCP, image pull, etcd, certs, must-gather, node debug | references/day2-runbooks.md |

Common combinations:

• incident + suspected subsystem (security, network, operators, nodes, storage)
• install + airgap
• upgrade + airgap
• security + operators for operator pod admission failures
• network + ingress for route/LB/VIP/DNS failures
• storage + virtualization for VM live migration and PV design

Core Workflow

1. Identify distribution, version, topology, connection mode, support boundary, and blast radius.
2. Load the nearest reference only.
3. Establish owner: CVO, MCO, CNO, Ingress Operator, Authentication Operator, OLM, storage/logging/operator-specific controller, or workload owner.
4. Gather read-only evidence before mutation.
5. Recommend the smallest supported change at the source CR or documented procedure.
6. Include preflight, action, post-check, rollback/restore, and stop condition for risky work.
7. Separate official/version-sensitive facts, practitioner heuristics, and local assumptions.
8. For public artifacts, run anti-leak checks before presenting final prose.

Output Contract

Default response shape:

1. Verdict - likely layer or recommended path
2. Why - mechanism-level OpenShift reason
3. Smallest safe path - probes first, then minimal change if warranted
4. Risks / edge cases - version, support, security, data, network, reboot, and operator ownership caveats
5. Validation - exact observations that prove convergence
6. Rollback / next step - how to revert or the next probe

Mode-specific additions:

• incident: add Likely owner, Read-only probes, Do not do yet, Stop condition
• upgrade: add Preflight, Go/no-go gates, Operator compatibility, Etcd backup, Worker/MCP plan
• airgap: add Mirror artifacts, Signatures, Catalogs, IDMS/ITMS, OSUS/update graph
• security: add Subject, SA/SCC/RBAC path, Least-privilege alternative, Audit concern
• review: use Verdict, Blockers, Risks, Evidence, Suggested fixes, Smallest next step

Guardrails

• Do not recommend disabling SCC, granting privileged, granting SCCs to broad groups, or binding anyuid to a namespace default ServiceAccount as casual fixes.
• Do not recommend editing rendered-* MachineConfigs, operator-owned Deployments, static-pod manifests, or node files when a source CR exists.
• Do not recommend oc adm upgrade --force, --to-image, --allow-not-recommended, etcd restore, quorum restore, or manual static-pod surgery without explicit break-glass framing and approval.
• Do not treat OADP/Velero as a substitute for etcd snapshots.
• Do not treat snapshots, image registry data, Prometheus data, Loki data, and etcd as the same recovery class.
• Do not assume an image registry on emptyDir, automatic InstallPlan approval, or an unbounded catalog mirror is production-safe.
• Do not use podman system prune guidance on OpenShift nodes; prefer documented CRI-O cleanup workflows and support docs.
• Do not bulk-approve CSRs without node identity validation in bare-metal/UPI environments.
• Do not delete kubeadmin without a verified alternate cluster-admin and break-glass path.
• Do not expose local paths, customer names, internal domains, credentials, tokens, pull-secret contents, or private topology in public examples.

Success Criteria

Pass when all are true:

• guidance is OpenShift-specific and version-aware
• owner/operator/source CR is identified where relevant
• read-only evidence precedes mutation
• risky operations include preflight, approval, rollback/restore, post-checks, and stop conditions
• OCP/OKD/managed/MicroShift support boundaries are explicit when relevant
• SCC, MCO, CVO, OLM, Routes, OVN, storage, and etcd are not conflated
• public outputs use generic placeholders and pass anti-leak checks

Fail when any are true:

• answer is generic Kubernetes advice that ignores OpenShift operators and CRs
• command-first response mutates a cluster before diagnosis
• unsupported or break-glass commands are normalized
• security guidance broadens SCC/RBAC without least-privilege reasoning
• upgrade advice lacks etcd backup and operator/MCP preflight
• examples leak private customer, industry, local path, internal domain, or credential details

Failure Modes

| Scenario | Detection | Fallback | |---|---|---| | Version unclear | No OCP/OKD minor/z-stream provided | Ask for oc version, oc get clusterversion, and relevant operator CSV versions | | Vague incident | Only symptom provided | Start with oc get co, events sorted by timestamp, oc get mcp, and scoped oc adm inspect | | Owner unclear | Multiple operators could own the surface | Find API group, namespace, ownerReferences, ClusterOperator status, and source CR before changing anything | | Risky mutation requested | Upgrade, network, SCC, storage, node OS, pull-secret, cert, etcd, or identity change | Require preflight, approval, rollback/restore, and post-checks | | Public artifact requested | Skill/doc/runbook content could expose context | Use generic examples and run anti-leak checks from day2-runbooks | | External docs needed | Local source/version evidence is insufficient for high-impact behavior | Prefer versioned Red Hat docs, local oc explain, installed CLI help, or upstream source; label uncertainty |

When in doubt

• Ask for the smallest missing fact that changes the decision.
• Diagnose at the platform layer before the workload layer.
• Prefer supported CR-driven changes over node shell edits.
• Prefer a lab rehearsal over a clever production shortcut.
• Prefer public-safe placeholders over realistic internal names.