jaykim88/cicd-pipeline

CI/CD Pipeline (Backend)

Purpose

Make every backend deploy automated, gated, and reversible — containerized build, quality gates, migrations applied safely as an explicit step, and a deploy strategy that can roll back without downtime.

Universal — the gated pipeline (build → test → migrate → deploy), SHA-tagged images, and blue-green/canary with rollback are CI/CD principles; the CI platform and registry differ.

Procedure

1. Quality gates first (parallel)
   • type-check, lint, test (unit + integration + contract) run on every PR — parallel jobs
   • Failure blocks merge (branch protection)
   • Concurrency cancellation: cancel the stale in-progress run when a new push to the same PR arrives (concurrency: { group, cancel-in-progress: true } on GHA) — saves minutes + CI cost
   • Filter triggers (paths-ignore for docs-only) so a README edit doesn't run the full pipeline

1b. Harden the pipeline itself (it runs with your secrets)

• Least-privilege permissions: at the top (contents: read); widen per-job only as needed — the default token is over-permissioned
• Pin third-party actions to a full commit SHA, not a moving tag; let Dependabot bump them
• Never expose secrets to untrusted fork PRs — pull_request (default) has no secret access; pull_request_target runs with secrets against PR code, use only with extreme care
• Use OIDC for cloud deploys (no long-lived static keys in secrets)

2. Containerized build, SHA-tagged

   • Multi-stage Dockerfile (build stage → slim runtime, e.g., distroless/Alpine)
   • Tag the image with the commit SHA (immutable, traceable) — not just latest
   • Cache dependency + build layers

3. DB migration as an EXPLICIT, gated step — before the deploy

   • Run migrate deploy as its own pipeline step, before routing traffic to new code
   • Migrations must be expand-contract safe (see migration-strategy) so old code still works during rollout
   • A failed migration halts the deploy (don't deploy code expecting a schema that didn't apply)

4. Deploy strategy with rollback

   • Blue-green: deploy to idle environment, switch traffic, keep old for instant rollback
   • Canary: route a small % to new version, watch metrics, ramp or roll back
   • Gate production behind environment protection rules + health checks

5. Health checks + auto-rollback

   • Readiness/liveness probes; deploy waits for healthy before shifting traffic
   • Auto-rollback (or fast manual) on health-check failure or error-rate spike (tie to observability-setup)

6. Security in the pipeline

   • Dependency audit (npm audit), secret scan (gitleaks), image scan (trivy)
   • Fail or warn on Critical/High vulns (see backend-security-audit)

7. Validate (validation loop)

   • Run a real deploy to staging; verify: migration gate runs before code, health check gates traffic switch, rollback actually restores the previous version
   • Force a failing health check → verify traffic does NOT switch / auto-rolls-back

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | Migrations run inside app startup | Explicit gated migration step before deploy | | latest image tag | SHA-tagged immutable images | | Deploy with no rollback path | Blue-green / canary with instant rollback | | Traffic switched before health check | Health-gated traffic shift | | Sequential type-check→lint→test | Parallel quality-gate jobs | | Stale runs piling up on every PR push | concurrency: cancel-in-progress | | Workflow with default (over-permissioned) GITHUB_TOKEN | Least-privilege permissions: contents: read | | uses: action@v4 (moving tag — hijack risk) | Pin to a full commit SHA | | Secrets exposed via pull_request_target against fork code | Use pull_request; pull_request_target only with extreme care | | Long-lived cloud access keys in CI secrets | OIDC federation to the cloud |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | No rollback path; migrations run unsafely on startup with no gate; secrets not scanned (leak risk); secrets exposed to untrusted fork PRs via pull_request_target | Fix immediately | | Major | latest tags (non-reproducible deploys); no health-gated traffic switch; default over-permissioned GITHUB_TOKEN; third-party actions pinned to a moving tag | Fix this sprint | | Minor | Sequential CI jobs (slow); image scan not yet wired; no concurrency: cancel-in-progress (stale runs pile up) | Schedule within 2 sprints |

Completion Criteria

• [ ] Quality gates (type/lint/test) parallel + blocking
• [ ] SHA-tagged container images
• [ ] Migration runs as an explicit gate before deploy (expand-contract safe)
• [ ] Blue-green/canary deploy with rollback verified
• [ ] Health-gated traffic switch
• [ ] Dependency + secret + image scanning in pipeline

Output

• Pipeline config: .github/workflows/*.yml (build/test/migrate/deploy stages)
• Dockerfile: multi-stage, slim runtime
• Deploy config: blue-green/canary + health checks + rollback
• Commit format: chore(ci): containerized build + migration gate / chore(deploy): blue-green with rollback

Implementation

TypeScript + Node + Docker (default)

• GitHub Actions: docker/build-push-action + setup-buildx; multi-stage Dockerfile (deps → build → distroless runtime)
• Layer cache: npm cache + Docker layer cache keyed on lockfile
• Migration gate: a job running prisma migrate deploy before the deploy job; deploy needs: it
• Deploy: Vercel (serverless) / Fly.io / Railway / k8s; SHA-tagged image; environment protection rules

Other stacks

• Python / FastAPI: same GHA + Docker; alembic upgrade head as the migration gate
• Go: multi-stage build to a scratch/distroless image (tiny); golang-migrate gate
• Universal: SHA-tagging, gated migrations, blue-green/canary, and health-gated rollout are platform-agnostic; GitLab CI / CircleCI use the same job-DAG structure

Related skills

• migration-strategy — migrations run as a gated pipeline step
• test-strategy — tests + contract verification gate the build
• backend-security-audit — dependency + secret scanning in CI

Reference

• Key insight encoded: Tag images with commit SHA, gate prod behind environment protection, run DB migrations as an explicit pipeline gate before the deploy step, and deploy staging→prod with auto-rollback on health-check failure.