ahmedasmar/gitops-workflows

GitOps Workflows

Overview

This skill provides comprehensive GitOps workflows for continuous deployment to Kubernetes using ArgoCD 3.x and Flux 2.7+.

When to use this skill:

• Setting up GitOps from scratch (ArgoCD or Flux)
• Designing Git repository structures
• Multi-cluster deployments
• Troubleshooting sync/reconciliation issues
• Implementing secrets management
• Progressive delivery (canary, blue-green)
• Migrating between GitOps tools

---

Core Workflow: GitOps Implementation

Use this decision tree to determine your starting point:

Do you have GitOps installed?
├─ NO → Need to choose a tool
│   └─ Want UI + easy onboarding? → ArgoCD (Workflow 1)
│   └─ Want modularity + platform engineering? → Flux (Workflow 2)
└─ YES → What's your goal?
    ├─ Sync issues / troubleshooting → Workflow 7
    ├─ Multi-cluster deployment → Workflow 4
    ├─ Secrets management → Workflow 5
    ├─ Progressive delivery → Workflow 6
    ├─ Repository structure → Workflow 3
    └─ Tool comparison → Read references/argocd_vs_flux.md

---

1. Initial Setup: ArgoCD 3.x

Latest Version: v3.1.9 (stable), v3.2.0-rc4 (October 2025)

Quick Install

# Create namespace
kubectl create namespace argocd

# Install ArgoCD 3.x
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.1.9/manifests/install.yaml

# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Port forward to access UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Access: https://localhost:8080

→ Template: assets/argocd/install-argocd-3.x.yaml

ArgoCD 3.x New Features

Breaking Changes:

• ✅ Annotation-based tracking (default, was labels)
• ✅ RBAC logs enforcement enabled
• ✅ Legacy metrics removed

New Features:

• ✅ Fine-grained RBAC (per-resource permissions)
• ✅ Better defaults (resource exclusions for performance)
• ✅ Secrets operators endorsement

Deploy Your First Application

# CLI method
argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

# Sync application
argocd app sync guestbook

Health Check

# Check application health
python3 scripts/check_argocd_health.py \
  --server https://argocd.example.com \
  --token $ARGOCD_TOKEN

→ Script: scripts/check_argocd_health.py

---

2. Initial Setup: Flux 2.7

Latest Version: v2.7.1 (October 2025)

Quick Install

# Install Flux CLI
brew install fluxcd/tap/flux  # macOS
# or: curl -s https://fluxcd.io/install.sh | sudo bash

# Check prerequisites
flux check --pre

# Bootstrap Flux (GitHub)
export GITHUB_TOKEN=<your-token>
flux bootstrap github \
  --owner=<org> \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --personal

# Enable source-watcher (Flux 2.7+)
flux install --components-extra=source-watcher

→ Template: assets/flux/flux-bootstrap-github.sh

Flux 2.7 New Features

• ✅ Image automation GA
• ✅ ExternalArtifact and ArtifactGenerator APIs
• ✅ Source-watcher component for better performance
• ✅ OpenTelemetry tracing support
• ✅ CEL expressions for readiness evaluation

Deploy Your First Application

# gitrepository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master
---
# kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  path: "./kustomize"
  prune: true
  sourceRef:
    kind: GitRepository
    name: podinfo

Health Check

# Check Flux health
python3 scripts/check_flux_health.py --namespace flux-system

→ Script: scripts/check_flux_health.py

---

3. Repository Structure Design

Decision: Monorepo or Polyrepo?

Monorepo Pattern

Best for: Startups, small teams (< 20 apps), single team

gitops-repo/
├── apps/
│   ├── frontend/
│   ├── backend/
│   └── database/
├── infrastructure/
│   ├── ingress/
│   ├── monitoring/
│   └── secrets/
└── clusters/
    ├── dev/
    ├── staging/
    └── production/

Polyrepo Pattern

Best for: Large orgs, multiple teams, clear boundaries

infrastructure-repo/     (Platform team)
app-team-1-repo/        (Team 1)
app-team-2-repo/        (Team 2)

Environment Structure (Kustomize)

app/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── replica-patch.yaml
    ├── staging/
    └── production/

→ Reference: references/repo_patterns.md

Validate Repository Structure

python3 scripts/validate_gitops_repo.py /path/to/repo

→ Script: scripts/validate_gitops_repo.py

---

4. Multi-Cluster Deployments

ArgoCD ApplicationSets

Cluster Generator (deploy to all clusters):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
spec:
  generators:
  - cluster:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: '{{name}}-myapp'
    spec:
      source:
        repoURL: https://github.com/org/apps
        path: myapp
      destination:
        server: '{{server}}'

→ Template: assets/applicationsets/cluster-generator.yaml

Performance Benefit: 83% faster deployments (30min → 5min)

Generate ApplicationSets

# Cluster generator
python3 scripts/applicationset_generator.py cluster \
  --name my-apps \
  --repo-url https://github.com/org/repo \
  --output appset.yaml

# Matrix generator (cluster x apps)
python3 scripts/applicationset_generator.py matrix \
  --name my-apps \
  --cluster-label production \
  --directories app1,app2,app3 \
  --output appset.yaml

→ Script: scripts/applicationset_generator.py

Flux Multi-Cluster

Hub-and-Spoke: Management cluster manages all clusters

# Bootstrap each cluster
flux bootstrap github --context prod-cluster --path clusters/production
flux bootstrap github --context staging-cluster --path clusters/staging

→ Reference: references/multi_cluster.md

---

5. Secrets Management

Never commit plain secrets to Git. Choose a solution:

Decision Matrix

| Solution | Complexity | Best For | 2025 Trend | |----------|-----------|----------|------------| | SOPS + age | Medium | Git-centric, flexible | ↗️ Preferred | | External Secrets Operator | Medium | Cloud-native, dynamic | ↗️ Growing | | Sealed Secrets | Low | Simple, GitOps-first | → Stable |

Option 1: SOPS + age (Recommended 2025)

Setup:

# Generate age key
age-keygen -o key.txt
# Public key: age1...

# Create .sops.yaml
cat <<EOF > .sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
EOF

# Encrypt secret
kubectl create secret generic my-secret --dry-run=client -o yaml \
  --from-literal=password=supersecret > secret.yaml
sops -e secret.yaml > secret.enc.yaml

# Commit encrypted version
git add secret.enc.yaml .sops.yaml

→ Template: assets/secrets/sops-age-config.yaml

Option 2: External Secrets Operator (v0.20+)

Best for: Cloud-native apps, dynamic secrets, automatic rotation

Option 3: Sealed Secrets

Best for: Simple setup, static secrets, no external dependencies

→ Reference: references/secret_management.md

Audit Secrets

python3 scripts/secret_audit.py /path/to/repo

→ Script: scripts/secret_audit.py

---

6. Progressive Delivery

Argo Rollouts (with ArgoCD)

Canary Deployment:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 2m}
      - setWeight: 50
      - pause: {duration: 2m}
      - setWeight: 100

→ Template: assets/progressive-delivery/argo-rollouts-canary.yaml

Flagger (with Flux)

Canary with Metrics Analysis:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
spec:
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99

→ Reference: references/progressive_delivery.md

---

7. Troubleshooting

Common Issues

ArgoCD OutOfSync:

# Check differences
argocd app diff my-app

# Sync application
argocd app sync my-app

# Check health
python3 scripts/check_argocd_health.py --server https://argocd.example.com --token $TOKEN

Flux Not Reconciling:

# Check resources
flux get all

# Check specific kustomization
flux get kustomizations
kubectl describe kustomization my-app -n flux-system

# Force reconcile
flux reconcile kustomization my-app

Detect Drift:

# ArgoCD drift detection
python3 scripts/sync_drift_detector.py --argocd --app my-app

# Flux drift detection
python3 scripts/sync_drift_detector.py --flux

→ Script: scripts/sync_drift_detector.py

→ Reference: references/troubleshooting.md

---

8. OCI Artifacts (Flux 2.6+)

GA Status: Flux v2.6 (June 2025)

Use OCIRepository for Helm Charts

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo-oci
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/charts/podinfo
  ref:
    semver: ">=6.0.0"
  verify:
    provider: cosign

→ Template: assets/flux/oci-helmrelease.yaml

Verify OCI Artifacts

python3 scripts/oci_artifact_checker.py \
  --verify ghcr.io/org/app:v1.0.0 \
  --provider cosign

→ Script: scripts/oci_artifact_checker.py

→ Reference: references/oci_artifacts.md

---

Quick Reference Commands

ArgoCD

# List applications
argocd app list

# Get application details
argocd app get <app-name>

# Sync application
argocd app sync <app-name>

# View diff
argocd app diff <app-name>

# Delete application
argocd app delete <app-name>

Flux

# Check Flux status
flux check

# Get all resources
flux get all

# Reconcile immediately
flux reconcile source git <name>
flux reconcile kustomization <name>

# Suspend/Resume
flux suspend kustomization <name>
flux resume kustomization <name>

# Export resources
flux export source git --all > sources.yaml

---

Resources Summary

Scripts (automation and diagnostics)

• check_argocd_health.py - Diagnose ArgoCD sync issues (3.x compatible)
• check_flux_health.py - Diagnose Flux reconciliation issues (2.7+ compatible)
• validate_gitops_repo.py - Validate repository structure and manifests
• sync_drift_detector.py - Detect drift between Git and cluster
• secret_audit.py - Audit secrets management (SOPS, Sealed Secrets, ESO)
• applicationset_generator.py - Generate ApplicationSet manifests
• promotion_validator.py - Validate environment promotion workflows
• oci_artifact_checker.py - Validate Flux OCI artifacts and verify signatures

References (deep-dive documentation)

• argocd_vs_flux.md - Comprehensive comparison (2024-2025), decision matrix
• repo_patterns.md - Monorepo vs polyrepo, app-of-apps, environment structures
• secret_management.md - SOPS+age, Sealed Secrets, ESO (2025 best practices)
• progressive_delivery.md - Argo Rollouts, Flagger, canary/blue-green patterns
• multi_cluster.md - ApplicationSets, Flux multi-tenancy, hub-spoke patterns
• troubleshooting.md - Common sync issues, debugging commands
• best_practices.md - CNCF GitOps principles, security, 2025 recommendations
• oci_artifacts.md - Flux OCI artifacts (GA v2.6), signature verification

Templates (production-ready configurations)

• argocd/install-argocd-3.x.yaml - ArgoCD 3.x installation with best practices
• applicationsets/cluster-generator.yaml - Multi-cluster ApplicationSet example
• flux/flux-bootstrap-github.sh - Flux 2.7 bootstrap script
• flux/oci-helmrelease.yaml - OCI artifact + HelmRelease example
• secrets/sops-age-config.yaml - SOPS + age configuration
• progressive-delivery/argo-rollouts-canary.yaml - Canary deployment with analysis