rileyhilliard/managing-pipelines

Pipeline Management

Decision guidance for GitHub Actions CI/CD pipelines, deployment strategies, and infrastructure automation.

Contents

• When to use which pattern
• Security quick reference
• Performance quick reference
• Workflow architecture quick reference
• Deployment quick reference
• Infrastructure as code quick reference
• Observability quick reference
• Cross-pipeline conventions
• Pipeline debugging checklist

When to use which pattern

| Scenario | Reference | Why | | --- | --- | --- | | Hardening against supply chain attacks | Security | SHA pinning, permissions, OIDC | | Speeding up slow CI builds | Performance | Caching, matrix builds, concurrency | | DRY-ing up duplicated workflow YAML | Workflow architecture | Reusable workflows vs composite actions | | Setting up staging/production deploys | Deployment | Environment promotion, protection rules | | Adding Terraform/OpenTofu to CI | Infrastructure | Plan-on-PR, apply-on-merge, drift detection | | Tracking pipeline reliability | Observability | OTel, DORA metrics, SLOs | | Reviewing a PR that modifies workflows | Security + Workflow | Permissions audit, secret exposure review | | Debugging flaky pipelines | Observability + Performance | Metrics, cache hit rates, concurrency | | Migrating from Jenkins/CircleCI | Workflow architecture | Action patterns, reusable workflow design | | Setting up monorepo CI | Performance | Path filtering, selective job execution |

Security quick reference

Use for: Preventing supply chain attacks, minimizing credential exposure, hardening runner environments.

Key decisions:

• Pin all third-party actions to full commit SHAs, not tags
• Set org-level default token permissions to read-only
• Use OIDC for cloud auth instead of stored credentials
• Never use pull_request_target without understanding the security model

See references/security-hardening.md for attack patterns and mitigations.

Performance quick reference

Use for: Reducing CI times, optimizing runner costs, parallelizing builds.

Key decisions:

• Cache dependency installs AND build artifacts (not just node_modules)
• Use fail-fast: false for CI matrices, true for deployment
• Set concurrency groups with cancel-in-progress: true for CI, false for deploys
• Use path filtering in monorepos to skip irrelevant jobs

See references/performance-optimization.md for caching strategies and runner selection.

Workflow architecture quick reference

Use for: Structuring reusable CI/CD components, managing action dependencies.

Key decisions:

• Reusable workflows for entire pipeline templates; composite actions for shared steps
• Pass secrets explicitly, not with secrets: inherit
• Automate SHA pin updates with Dependabot or Renovate
• Restrict allowed actions at the org level

See references/workflow-architecture.md for patterns and versioning.

Deployment quick reference

Use for: Environment promotion, deployment gates, progressive delivery.

Key decisions:

• Use GitHub Environments with branch restrictions for production
• Release-based promotion gives the cleanest audit trail
• Progressive delivery (canary/blue-green) via Argo Rollouts or Flagger
• Custom deployment protection rules for SLO-gated deployments

See references/deployment-strategies.md for promotion patterns and rollback strategies.

Infrastructure as code quick reference

Use for: Terraform/OpenTofu pipelines, drift detection, policy enforcement.

Key decisions:

• Always save plan output and apply the saved plan (never plan-then-apply without -out)
• Post plan output as PR comments for review
• Segment state by functional boundary, not geography
• Run scheduled drift detection separately from code-triggered deploys

See references/infrastructure-pipelines.md for IaC workflow patterns.

Observability quick reference

Use for: Pipeline reliability tracking, incident response, capacity planning.

Key decisions:

• Instrument pipelines with OpenTelemetry (runs as traces, jobs as spans)
• Track DORA metrics: deployment frequency, lead time, change failure rate, MTTR
• Set SLOs for pipeline reliability (e.g., 99% main branch build success)
• Monitor cache hit rates and queue times as leading indicators

See references/pipeline-observability.md for instrumentation and metrics.

Pipeline debugging checklist

Slow CI builds

1. Check cache hit rates (low = cold start overhead)
2. Look for sequential jobs that could run in parallel
3. Verify concurrency groups aren't queuing unnecessarily
4. Check runner specs (CPU-bound work on small runners)
5. Look for full-repo checkouts when sparse checkout would work

Failed deployments

1. Check environment protection rule approvals
2. Verify OIDC token audience and subject claims
3. Check if concurrency group blocked/cancelled the run
4. Review Terraform plan output for unexpected changes
5. Check if deployment protection rules (Datadog, etc.) rejected

Security incidents

1. Audit recent changes to workflow files and action versions
2. Check for new pull_request_target usage
3. Review GITHUB_TOKEN permissions in affected workflows
4. Scan for secrets in workflow logs (step outputs, artifacts)
5. Check if any action SHAs were recently changed

Flaky pipelines

1. Check if tests have timing dependencies (see condition-based-waiting skill)
2. Look for shared state between matrix jobs
3. Verify caches aren't corrupted (clear and rebuild)
4. Check for rate limiting from external services
5. Review runner availability (self-hosted runner capacity)