harsh040506/ci-cd-pipelines

CI/CD Pipelines

Design fast, reliable, and secure automated delivery pipelines.

---

Pipeline Design Principles

1. FAST OR FIX IT

Engineer feedback loops as short as possible. A pipeline that takes 30 minutes trains developers to stop running it frequently.

| Stage | Target Duration | |---|---| | Lint + type-check | < 2 minutes | | Unit tests | < 5 minutes | | Build | < 5 minutes | | Integration tests | < 10 minutes | | End-to-end tests | < 15 minutes | | Total (parallel) | < 15 minutes |

Optimize ruthlessly: cache dependencies, run stages in parallel, skip unaffected paths.

2. FAIL FAST

Run the fastest, cheapest checks first. Don't spend 10 minutes building a Docker image if a lint error would have caught the problem in 30 seconds.

Order: lint → type-check → unit tests → build → integration tests → e2e → deploy

3. EVERY MERGE TO MAIN MUST BE DEPLOYABLE

The main branch is always production-ready. Enforce this by blocking merges on CI failure. Deploy to staging automatically on every merge. Require human approval to promote to production.

4. IDEMPOTENT DEPLOYS

Running the same pipeline twice with the same inputs produces the same result. No side effects, no partial states.

---

GitHub Actions: Production Template

# .github/workflows/ci-cd.yml
name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true   # Cancel stale PR runs on new push

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # ──────────────────────────────────────────
  # Stage 1: Lint & Type Check (fast feedback)
  # ──────────────────────────────────────────
  lint:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    
    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'           # Cache node_modules by package-lock hash
    
    - run: npm ci
    - run: npm run lint
    - run: npm run type-check

  # ──────────────────────────────────────────
  # Stage 2: Unit Tests (parallel with lint)
  # ──────────────────────────────────────────
  test-unit:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'
    - run: npm ci
    - run: npm run test:unit -- --coverage
    
    - name: Upload coverage
      uses: codecov/codecov-action@v4
      with:
        token: ${{ secrets.CODECOV_TOKEN }}

  # ──────────────────────────────────────────
  # Stage 3: Build Docker Image
  # ──────────────────────────────────────────
  build:
    name: Build & Push Image
    runs-on: ubuntu-latest
    needs: [lint, test-unit]   # Only build if lint and tests pass
    permissions:
      contents: read
      packages: write
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
      image-digest: ${{ steps.push.outputs.digest }}
    steps:
    - uses: actions/checkout@v4
    
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    
    - name: Log in to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    
    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=sha,prefix=sha-
          type=ref,event=branch
          type=semver,pattern={{version}}
    
    - name: Build and push
      id: push
      uses: docker/build-push-action@v5
      with:
        context: .
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha       # Use GitHub Actions cache for layers
        cache-to: type=gha,mode=max
    
    - name: Scan for vulnerabilities
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
        format: 'sarif'
        severity: 'HIGH,CRITICAL'
        exit-code: '1'             # Fail pipeline on HIGH/CRITICAL CVEs

  # ──────────────────────────────────────────
  # Stage 4: Integration Tests
  # ──────────────────────────────────────────
  test-integration:
    name: Integration Tests
    runs-on: ubuntu-latest
    needs: [build]
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_PASSWORD: testpass
          POSTGRES_DB: testdb
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'
    - run: npm ci
    - run: npm run test:integration
      env:
        DATABASE_URL: postgresql://postgres:testpass@localhost:5432/testdb

  # ──────────────────────────────────────────
  # Stage 5: Deploy to Staging (auto on main)
  # ──────────────────────────────────────────
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: [test-integration]
    if: github.ref == 'refs/heads/main'
    environment:
      name: staging
      url: https://staging.example.com
    steps:
    - uses: actions/checkout@v4
    
    - name: Configure kubectl
      uses: azure/setup-kubectl@v3
    
    - name: Set kubeconfig
      run: |
        echo "${{ secrets.STAGING_KUBECONFIG }}" | base64 -d > kubeconfig
        echo "KUBECONFIG=$PWD/kubeconfig" >> $GITHUB_ENV
    
    - name: Deploy to staging
      run: |
        kubectl set image deployment/api-service \
          api-service=${{ needs.build.outputs.image-tag }} \
          -n staging
        kubectl rollout status deployment/api-service -n staging --timeout=5m

  # ──────────────────────────────────────────
  # Stage 6: Deploy to Production (manual gate)
  # ──────────────────────────────────────────
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: [deploy-staging]
    if: github.ref == 'refs/heads/main'
    environment:
      name: production        # Requires manual approval in GitHub Environments
      url: https://example.com
    steps:
    - uses: actions/checkout@v4
    
    - name: Configure kubectl
      uses: azure/setup-kubectl@v3
    
    - name: Set kubeconfig
      run: |
        echo "${{ secrets.PROD_KUBECONFIG }}" | base64 -d > kubeconfig
        echo "KUBECONFIG=$PWD/kubeconfig" >> $GITHUB_ENV
    
    - name: Deploy to production (canary)
      run: |
        # Deploy canary with 10% traffic
        kubectl set image deployment/api-service-canary \
          api-service=${{ needs.build.outputs.image-tag }} \
          -n production
        kubectl rollout status deployment/api-service-canary -n production --timeout=5m
        
        # Wait 5 minutes, then promote
        sleep 300
        
        kubectl set image deployment/api-service \
          api-service=${{ needs.build.outputs.image-tag }} \
          -n production
        kubectl rollout status deployment/api-service -n production --timeout=5m

---

Caching Strategies

Dependency Caching (by lockfile hash)

# Node.js
- uses: actions/setup-node@v4
  with:
    cache: 'npm'

# Python
- uses: actions/setup-python@v5
  with:
    cache: 'pip'

# Go
- uses: actions/setup-go@v5
  with:
    cache: true

Docker Layer Caching

# GitHub Actions Cache (free, fast)
cache-from: type=gha
cache-to: type=gha,mode=max

# Registry cache (better for large teams)
cache-from: type=registry,ref=ghcr.io/myorg/myapp:cache
cache-to: type=registry,ref=ghcr.io/myorg/myapp:cache,mode=max

Build Parallelism

Run independent jobs in parallel. needs: creates the dependency graph — only use it when there's a real dependency:

jobs:
  lint:        # No dependencies — runs immediately
  test-unit:   # No dependencies — runs immediately
  build:
    needs: [lint, test-unit]  # Waits for both
  deploy:
    needs: [build]

---

Branch & Release Strategies

Trunk-Based Development (Recommended)

• All developers merge to main frequently (daily or more)
• Feature flags control feature visibility, not branches
• No long-lived feature branches
• Main is always deployable

main ──●──●──●──●──●──●──●
       │     │           │
       PR    PR           Release tag

GitFlow (for products with fixed release cycles)

main       ────────────────────────────●──
               ↑                      ↑
release/1.2 ──●──────────────────────●
                ↑                  ↑
develop    ──●──●──●──●──●──●──●──●──

Use trunk-based for web services. Use GitFlow for mobile apps or packaged software.

---

Deployment Gate Patterns

Quality Gates (block bad deploys)

# Block if test coverage drops
- name: Coverage check
  run: |
    COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
    if (( $(echo "$COVERAGE < 80" | bc -l) )); then
      echo "Coverage $COVERAGE% is below 80% threshold"
      exit 1
    fi

# Block if bundle size increases >10%
- name: Bundle size check
  run: |
    npx bundlewatch --config bundlewatch.config.js

Approval Gates

Use GitHub Environments with required reviewers for production deploys. Configure in GitHub Settings → Environments → Required reviewers.

Automated Rollback Gate

- name: Deploy with automatic rollback
  run: |
    kubectl set image deployment/api-service api-service=$IMAGE -n production
    if ! kubectl rollout status deployment/api-service -n production --timeout=3m; then
      echo "Rollout failed — initiating rollback"
      kubectl rollout undo deployment/api-service -n production
      exit 1
    fi

---

Security Best Practices

1. Never print secrets in logs. Use ::add-mask:: in GitHub Actions:

   - run: echo "::add-mask::${{ secrets.API_KEY }}"
   

2. Pin action versions to SHA, not tag:

   uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
   

   Tags can be moved by the action author (supply chain attack vector).

3. Use minimal permissions:

   permissions:
     contents: read
     packages: write   # Only what's needed
   

4. Scan dependencies and images on every build.

5. Use OIDC for cloud authentication instead of long-lived credentials:

   - uses: aws-actions/configure-aws-credentials@v4
     with:
       role-to-assume: arn:aws:iam::123456789:role/github-actions-role
       aws-region: us-east-1
   

Deeper Reference

For ready-to-use pipeline templates and advanced workflow patterns, see:

• references/pipeline-patterns.md — complete GitHub Actions and GitLab CI pipeline configurations for common deployment strategies
• references/workflow-templates.md — reusable workflow templates for Docker builds, semantic releases, and environment promotions