curiositech/kubernetes-deployment-automation

Kubernetes Deployment Automation

Operational expertise for the deployment lifecycle — from kubectl apply to production-verified rollout. Owns Day 2 operations: deploying manifests, monitoring rollouts, troubleshooting failures, managing secrets/namespaces, and GitOps workflows.

When to Use

Use for:

• Deploying manifests to Kubernetes clusters (kubectl, Helm, Kustomize)
• Setting up GitOps with ArgoCD or Flux
• Monitoring and verifying rollout health
• Troubleshooting stuck or failed deployments
• Multi-environment promotion (dev → staging → prod)
• Namespace, RBAC, and secret lifecycle for deployments
• Rollback decisions and execution

NOT for:

• Generating K8s YAML/Helm charts → kubernetes-manifest-generator
• Blue-green/canary traffic shifting → blue-green-deployment-orchestrator
• Cluster provisioning with Terraform → terraform-module-builder
• CI/CD pipeline YAML → github-actions-pipeline-builder
• Application-level debugging → language-specific skills

Core Process

flowchart TD
  START[Deployment Request] --> ASSESS{What stage?}
  ASSESS -->|New deployment| PREFLIGHT[Pre-flight Checks]
  ASSESS -->|Stuck rollout| TROUBLESHOOT[Troubleshoot]
  ASSESS -->|Setup GitOps| GITOPS[GitOps Setup]
  ASSESS -->|Promote env| PROMOTE[Promotion Pipeline]

  PREFLIGHT --> NS[Namespace exists?]
  NS -->|No| CREATE_NS[Create namespace + RBAC]
  NS -->|Yes| SECRETS[Secrets provisioned?]
  CREATE_NS --> SECRETS
  SECRETS -->|No| PROVISION_SECRETS[Provision secrets]
  SECRETS -->|Yes| DEPS[Dependencies ready?]
  PROVISION_SECRETS --> DEPS
  DEPS -->|No| WAIT_DEPS[Wait / deploy deps first]
  DEPS -->|Yes| DEPLOY[Deploy]

  DEPLOY --> METHOD{Deployment method?}
  METHOD -->|kubectl| KUBECTL[kubectl apply -f / rollout]
  METHOD -->|Helm| HELM[helm upgrade --install]
  METHOD -->|ArgoCD| ARGOCD[argocd app sync]
  METHOD -->|Flux| FLUX[flux reconcile]

  KUBECTL --> VERIFY[Verify Rollout]
  HELM --> VERIFY
  ARGOCD --> VERIFY
  FLUX --> VERIFY

  VERIFY --> HEALTHY{Healthy?}
  HEALTHY -->|Yes| DONE[Deployment Complete]
  HEALTHY -->|No| TROUBLESHOOT

  TROUBLESHOOT --> TRIAGE{Failure type?}
  TRIAGE -->|ImagePullBackOff| IMG[Check image tag, registry auth, pull policy]
  TRIAGE -->|CrashLoopBackOff| CRASH[Check logs, resource limits, probes]
  TRIAGE -->|Pending| PENDING[Check node resources, PVC, scheduling]
  TRIAGE -->|OOMKilled| OOM[Increase memory limits, check for leaks]
  TRIAGE -->|Timeout| TIMEOUT[Check readiness probe, startup time]
  IMG --> DECIDE_ROLLBACK{Fixable quickly?}
  CRASH --> DECIDE_ROLLBACK
  PENDING --> DECIDE_ROLLBACK
  OOM --> DECIDE_ROLLBACK
  TIMEOUT --> DECIDE_ROLLBACK
  DECIDE_ROLLBACK -->|Yes| FIX[Fix and redeploy]
  DECIDE_ROLLBACK -->|No| ROLLBACK[kubectl rollout undo]
  FIX --> DEPLOY
  ROLLBACK --> DONE

Deployment Methods

kubectl Direct Apply

# Dry-run first — always
kubectl apply -f manifests/ --dry-run=server

# Apply with record for rollback
kubectl apply -f manifests/

# Watch rollout
kubectl rollout status deployment/<name> -n <ns> --timeout=300s

When to use: Simple deployments, scripted pipelines, single-cluster.

Helm

# Diff before upgrade (requires helm-diff plugin)
helm diff upgrade <release> <chart> -f values-prod.yaml -n <ns>

# Deploy with atomic (auto-rollback on failure)
helm upgrade --install <release> <chart> \
  -f values-prod.yaml \
  -n <ns> \
  --atomic \
  --timeout 5m \
  --wait

When to use: Parameterized deployments, multiple environments from one chart.

ArgoCD GitOps

# Create application
argocd app create <name> \
  --repo <git-url> \
  --path <manifest-path> \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace <ns> \
  --sync-policy automated \
  --auto-prune \
  --self-heal

# Sync with prune
argocd app sync <name> --prune

# Check health
argocd app get <name>

When to use: Multi-cluster, audit trail required, team deployments.

Flux GitOps

# Bootstrap Flux
flux bootstrap github \
  --owner=<org> \
  --repository=<repo> \
  --path=clusters/<cluster> \
  --personal

# Create Kustomization
flux create kustomization <name> \
  --source=GitRepository/<repo> \
  --path=<manifest-path> \
  --prune=true \
  --interval=5m

# Force reconcile
flux reconcile kustomization <name>

When to use: Lightweight GitOps, multi-tenant clusters, prefer pull-based.

Rollout Verification

Always verify after deploy. Minimum checks:

# 1. Rollout status
kubectl rollout status deployment/<name> -n <ns> --timeout=300s

# 2. Pod health
kubectl get pods -n <ns> -l app=<name> -o wide

# 3. Recent events (failures surface here)
kubectl events -n <ns> --for deployment/<name> --types=Warning

# 4. Endpoint readiness
kubectl get endpoints <service-name> -n <ns>

# 5. Logs from new pods (spot crashes early)
kubectl logs -n <ns> -l app=<name> --since=2m --tail=50

Anti-Patterns

kubectl apply Without Dry-Run

Novice: "Just apply it, YOLO" Expert: Always --dry-run=server first. Server-side dry-run catches quota violations, admission webhook rejections, and schema errors that client-side misses. Timeline: 2022: client-side dry-run was common → 2024+: server-side dry-run is standard (catches real admission issues).

Helm Without --atomic

Novice: helm upgrade then manually check and rollback on failure Expert: --atomic auto-rolls back on any failure. Without it, you get half-deployed states where some resources updated and others didn't, requiring manual cleanup.

Ignoring Resource Requests/Limits at Deploy Time

Novice: "The manifest generator already set them" Expert: Verify limits match the target environment. Dev limits in prod cause OOMKills. Run kubectl describe node to check available capacity before deploying resource-heavy workloads.

Secrets in Git

Novice: Base64-encoded secrets committed to the repo Expert: Base64 is encoding, not encryption. Use Sealed Secrets, SOPS, External Secrets Operator, or Vault. GitOps repos should contain SealedSecret or ExternalSecret resources, never plain Secret. Detection: grep -r "kind: Secret" manifests/ | grep -v SealedSecret | grep -v ExternalSecret

Multi-Environment Promotion

flowchart LR
  DEV[Dev Cluster] -->|Tests pass| STAGE[Staging Cluster]
  STAGE -->|Smoke tests + soak| PROD[Production Cluster]

  subgraph "Promotion Gate"
    TESTS[Automated tests]
    APPROVAL[Manual approval]
    METRICS[Metric check]
  end

  DEV --> TESTS
  TESTS --> STAGE
  STAGE --> APPROVAL
  APPROVAL --> METRICS
  METRICS --> PROD

Pattern: Same manifests, different values per environment.

| Approach | How | Best For | |----------|-----|----------| | Kustomize overlays | base/ + overlays/{dev,staging,prod} | Simple, native K8s | | Helm values | values-dev.yaml, values-prod.yaml | Parameterized charts | | ArgoCD ApplicationSets | Generator matrix across clusters | Multi-cluster GitOps | | Flux Kustomization | Per-cluster paths in fleet repo | Pull-based multi-cluster |

References

• references/troubleshooting-guide.md — Consult when diagnosing specific K8s failure states (ImagePullBackOff, CrashLoopBackOff, eviction, node pressure)
• references/gitops-patterns.md — Consult when setting up ArgoCD or Flux from scratch, or designing repo structure for GitOps
• references/rollback-strategies.md — Consult when deciding between revision rollback, Helm rollback, and GitOps revert