lv416e/devops-infrastructure

DevOps Infrastructure

Overview

Manual infrastructure changes are incidents waiting to happen. Unversioned configs are undocumented debt.

Core principle: EVERY piece of infrastructure is defined in code, stored in version control, and deployed through automation.

Violating the letter of this process is violating the spirit of infrastructure engineering.

The Iron Law

NO MANUAL INFRASTRUCTURE CHANGES - EVERYTHING IS CODE, VERSIONED, AND REVIEWABLE

If you clicked through a console UI to create it, it doesn't exist yet. Write the code.

When to Use

Use for ANY infrastructure work:

• Provisioning cloud resources
• Building container images
• Configuring CI/CD pipelines
• Setting up deployment strategies
• Managing secrets and credentials
• Configuring monitoring and alerting
• Setting up networking and DNS

Use this ESPECIALLY when:

• Under deadline pressure ("just create it in the console for now")
• A quick manual fix seems faster
• Someone says "we'll codify it later"
• You're setting up a "temporary" environment
• Debugging a production issue by hand

Don't skip when:

• It's "just one resource" (one resource becomes twenty)
• It's a dev environment (dev environments become templates)
• It's a one-time setup (nothing is one-time)

The Five Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Define Infrastructure as Code

BEFORE creating ANY resource:

1. Choose Your IaC Tool

   • Terraform: Multi-cloud, mature ecosystem, declarative
   • Pulumi: General-purpose languages, strong typing
   • CloudFormation: AWS-native, deep integration
   • Pick one per project and stick with it

2. Structure Your Code

   infrastructure/
   ├── modules/           # Reusable components
   │   ├── networking/
   │   ├── compute/
   │   └── database/
   ├── environments/      # Per-environment configs
   │   ├── dev/
   │   ├── staging/
   │   └── production/
   ├── variables.tf       # Input definitions
   └── outputs.tf         # Exported values
   

3. State Management

   • Remote state (S3 + DynamoDB, GCS, Terraform Cloud)
   • State locking enabled - always
   • Never commit state files to git
   • One state per environment minimum

4. Plan Before Apply

   • Always run plan/preview first
   • Review every change in the plan
   • Automate plan output in PRs
   • Never apply without reviewing the plan

Phase 2: Container Best Practices

Every container image follows these rules:

1. Multi-Stage Builds

   <Good> ```dockerfile FROM node:20-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm ci --production=false COPY . . RUN npm run build

   FROM node:20-alpine AS runtime WORKDIR /app RUN addgroup -g 1001 appgroup && adduser -u 1001 -G appgroup -s /bin/sh -D appuser COPY --from=builder /app/dist ./dist COPY --from=builder /app/node_modules ./node_modules USER appuser EXPOSE 3000 CMD ["node", "dist/index.js"]

   Small image, non-root user, only production artifacts
   </Good>
   
   <Bad>
   ```dockerfile
   FROM node:20
   WORKDIR /app
   COPY . .
   RUN npm install
   CMD ["npm", "start"]
   

   Bloated image, root user, dev dependencies included, source code exposed </Bad>

2. Minimal Base Images

   • Alpine or distroless
   • Pin exact versions (not latest)
   • Rebuild regularly for security patches

3. Security Scanning

   • Scan images in CI (Trivy, Snyk, Grype)
   • Block deployment on critical/high CVEs
   • No secrets in images - ever
   • No secrets in build args - they leak in layer history

4. Image Hygiene

   • .dockerignore for every project
   • One process per container
   • Health checks defined
   • Graceful shutdown handling (SIGTERM)

Phase 3: Kubernetes Deployment Patterns

Match the workload to the right abstraction:

1. Deployments - Stateless services

   • Rolling updates by default
   • Resource limits set (CPU and memory)
   • Readiness and liveness probes configured
   • Pod Disruption Budgets defined

2. StatefulSets - Databases, caches, queues

   • Stable network identities
   • Persistent volume claims
   • Ordered startup/shutdown
   • Don't use for stateless workloads

3. CronJobs - Scheduled tasks

   • Concurrency policy set
   • Failure history limits
   • Deadline seconds configured
   • Idempotent by design

4. Resource Definitions

   <Good> ```yaml resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "512Mi" ``` Explicit requests and limits </Good> <Bad> ```yaml # No resource limits defined - hope the node has enough ``` Unbounded resource usage, noisy neighbor problems </Bad>

Phase 4: CI/CD Pipeline Design

Every pipeline follows this structure:

1. Pipeline Stages

   Lint → Test → Build → Scan → Deploy(staging) → Verify → Deploy(production)
   

   • No stage can be skipped
   • Failure at any stage stops the pipeline
   • Production deploy requires explicit approval

2. Deployment Strategies

   | Strategy | When | Risk | Rollback | |----------|------|------|----------| | Rolling | Default for most services | Medium | Automatic | | Blue-Green | Zero-downtime critical services | Low | Instant switch | | Canary | High-traffic, risk-sensitive | Lowest | Route back to stable |

   • Rolling: Good default, gradual replacement
   • Blue-Green: Full parallel environment, instant cutover
   • Canary: Percentage-based traffic shifting, metrics-driven promotion

3. Rollback Plan

   • Every deployment MUST have a rollback plan
   • Automated rollback on health check failure
   • Database migrations must be backward-compatible
   • "We'll figure it out" is not a rollback plan

4. Pipeline Security

   • Secrets injected at runtime, never in code
   • Least-privilege service accounts
   • Signed artifacts and images
   • Audit trail for every deployment

Phase 5: Secrets Management and Monitoring

Secrets:

1. Never in Code

   • Not in source files
   • Not in Dockerfiles
   • Not in CI config files
   • Not in environment variable definitions committed to git
   • Use vault (HashiCorp Vault, AWS Secrets Manager, 1Password)
   • Inject at runtime via environment or mounted files

2. Rotation

   • Secrets have expiration dates
   • Automated rotation where possible
   • Revoke immediately when compromised

Monitoring and Alerting:

3. The Four Golden Signals

   • Latency: How long requests take
   • Traffic: How many requests per second
   • Errors: Rate of failed requests
   • Saturation: How full your resources are

4. Alert Design

   • Alert on symptoms, not causes
   • Every alert has a runbook
   • No alert fatigue - if you ignore it, delete it
   • Page for user-facing impact only

Red Flags - STOP and Follow Process

If you catch yourself thinking:

• "Just create it in the console, we'll codify later"
• "This is a one-time setup"
• "I'll hardcode the secret for now"
• "We don't need monitoring yet"
• "Skip the staging deploy, push straight to prod"
• "The rollback plan is to fix forward"
• "It's just a small config change, no PR needed"
• "We'll add resource limits later"
• "Root user is fine for now"

ALL of these mean: STOP. Follow the process.

Common Rationalizations

| Excuse | Reality | |--------|---------| | "Console is faster for one resource" | One resource becomes twenty. Codify from the start. | | "We'll codify it later" | You won't. "Later" means "never" in infrastructure. | | "It's just a dev environment" | Dev environments are production templates. Treat them the same. | | "Hardcode the secret for now" | Secrets in code get committed, pushed, leaked. Use a vault. | | "We don't need monitoring yet" | You need monitoring BEFORE the first incident, not after. | | "Skip staging, it works on my machine" | Your machine is not production. Deploy to staging first. | | "Fix forward is our rollback" | Fix forward under pressure creates new incidents. Have a real rollback. | | "Resource limits slow us down" | Unbounded containers slow everyone down when they consume the node. | | "Manual change, just this once" | Snowflake servers start with "just this once." | | "One-time setup doesn't need code" | Nothing is one-time. You'll rebuild, migrate, or recover. |

Anti-Patterns

| Anti-Pattern | Consequence | Correct Approach | |-------------|-------------|-----------------| | Snowflake servers | Unreproducible, undocumented, irreplaceable | Everything in IaC, immutable infrastructure | | Secrets in code | Credential leaks, security incidents | Vault/env injection, runtime secrets | | No rollback plan | Extended outages, panic-driven fixes | Automated rollback, backward-compatible migrations | | Deploy without approval | Unreviewed changes in production | PR-based deployments, required approvals | | No resource limits | Noisy neighbors, node exhaustion, cascading failures | Explicit requests and limits on every workload | | latest tag | Unreproducible builds, surprise breaking changes | Pin exact versions, rebuild intentionally |

Quick Reference

| Phase | Key Activities | Success Criteria | |-------|---------------|------------------| | 1. IaC | Define resources in code, remote state, plan before apply | All infrastructure in version control | | 2. Containers | Multi-stage builds, minimal images, security scanning | Small, secure, non-root images | | 3. Kubernetes | Right abstraction, resource limits, probes | Workloads are resilient and bounded | | 4. CI/CD | Pipeline stages, deployment strategy, rollback plan | Automated, gated, reversible deployments | | 5. Secrets/Monitoring | Vault injection, four golden signals, alert runbooks | No secrets in code, actionable alerts |

Verification Checklist

Before marking infrastructure work complete:

• [ ] All resources defined in IaC (no console-created resources)
• [ ] State stored remotely with locking
• [ ] Container images are multi-stage, minimal, non-root
• [ ] Images scanned for vulnerabilities in CI
• [ ] No secrets in code, Dockerfiles, or CI configs
• [ ] Secrets injected from vault/secrets manager at runtime
• [ ] Resource limits set on all workloads
• [ ] Health checks and probes configured
• [ ] Deployment strategy chosen with rollback plan documented
• [ ] Production deploy requires approval
• [ ] Monitoring covers four golden signals
• [ ] Every alert has a runbook

Can't check all boxes? You're not done.

Integration with Other Skills

This skill requires using:

• test-driven-development - REQUIRED for testing IaC changes (Terratest, infrastructure integration tests)

Complementary skills:

• systematic-debugging - Use when deployments fail or infrastructure behaves unexpectedly
• documentation-generation - Generate runbooks, architecture diagrams, and operational docs

Final Rule

If it's not in code, it doesn't exist.
If it's not in version control, it's not real.
If it can't be reviewed, it can't be trusted.

No exceptions without your human partner's explicit approval.