aj-geddes/domain-devops

DevOps Domain Skill

Docker

Quick Commands

docker build -t myapp:v1.0.0 .                          # Build image
docker build --target production -t myapp:prod .         # Multi-stage build
docker run --cpus=0.5 --memory=512m myapp:v1.0.0         # Run with limits
docker history myapp:v1.0.0                              # Inspect layers
docker image prune -f                                    # Remove dangling
docker logs -f --tail=100 container_id                   # Tail logs

Dockerfile Best Practices

• Use specific base image tags (never :latest)
• Multi-stage builds for minimal runtime images
• Copy dependency files first for layer caching
• Run as non-root user
• Use .dockerignore to exclude unnecessary files
• Minimize layers (combine RUN commands with &&)
• Use distroless or alpine for production
• Set health checks
• Label images with metadata

See: references/docker-complete.md for optimization techniques and Compose configurations.

CI/CD Pipelines

Pipeline Stages

1. Lint - Code quality checks (parallel with tests)
2. Test - Unit/integration tests with coverage
3. Build - Container image build and push
4. Deploy - Environment-specific deployments
5. Verify - Smoke tests and health checks

Best Practices

• Cache dependencies between runs
• Matrix builds for multi-version testing
• Separate fast checks (lint) from slow (integration)
• Fail fast on quality gates
• Tag images with commit SHA and semantic versions
• Store secrets in CI secret store, never in code
• Use environments for staging/production approvals

See: references/github-actions.md for complete workflows, matrix builds, caching, and deployment automation.

Kubernetes

Essential Resources

| Resource | Purpose | |----------|---------| | Deployment | Manages replica sets and rolling updates | | Service | Stable networking endpoint for pods | | Ingress | HTTP(S) routing to services | | ConfigMap | Non-sensitive configuration | | Secret | Sensitive data (credentials, tokens) | | HPA | Horizontal Pod Autoscaler |

Key kubectl Commands

kubectl apply -f deployment.yaml                         # Apply manifests
kubectl get pods,svc,ing -n production                   # Resource status
kubectl logs -f deployment/myapp -n production           # View logs
kubectl exec -it pod/myapp-xxx -- /bin/sh                # Shell into pod
kubectl port-forward svc/myapp 8080:80                   # Port forward
kubectl rollout status deployment/myapp                  # Rollout status
kubectl rollout undo deployment/myapp                    # Rollback
kubectl scale deployment/myapp --replicas=5              # Manual scale
kubectl top pods -n production                           # Resource usage

Deployment Checklist

• [ ] Resource requests and limits defined
• [ ] Liveness and readiness probes configured
• [ ] Running as non-root user
• [ ] Secrets externalized (not in manifests)
• [ ] Labels for monitoring and service discovery
• [ ] Multiple replicas for high availability
• [ ] Rolling update strategy configured
• [ ] HPA configured for auto-scaling

See: references/kubernetes-manifests.md for manifest examples, Helm charts, and security configurations.

Observability

1. Logging (What happened?)

• Structured JSON logs with context (request_id, user_id, service_name)
• Levels: DEBUG < INFO < WARNING < ERROR < CRITICAL
• Centralize with Loki, ElasticSearch, or CloudWatch
• Never log sensitive data

2. Metrics (How much/how many?)

• Counter: Monotonically increasing (requests_total)
• Gauge: Current value (active_connections)
• Histogram: Distribution (request_duration_seconds)
• Summary: Quantiles (p95, p99 latency)
• Track: request rate, error rate (4xx/5xx), latency percentiles, saturation (CPU/memory/disk)

3. Tracing (Where did time go?)

• Distributed tracing across services with OpenTelemetry
• Track request path, identify bottlenecks and slow queries

See: references/observability-stack.md for Prometheus, Grafana, Loki, Jaeger, and OpenTelemetry configurations.

Deployment Strategies

| Strategy | How It Works | When to Use | Trade-off | |----------|-------------|-------------|-----------| | Rolling | Gradually replace old pods | Standard deploys, backward-compatible changes | Slower rollout | | Blue-Green | Two environments, instant switch | DB migrations, major version updates | 2x infrastructure cost | | Canary | Route small % to new version, increase if healthy | High-risk changes, need real-traffic validation | Complexity, needs metrics |

All strategies: use readiness probes, have rollback plan, monitor error rate and latency during rollout.

See: references/deployment-strategies.md for rollback procedures and automated canary configurations.

Security Hardening

Container Security

• Scan images for vulnerabilities (Trivy, Snyk)
• Minimal base images (distroless, scratch)
• Non-root user, read-only root filesystem, drop all capabilities
• Regular image updates

Kubernetes Security

• Network policies (deny-all by default)
• Pod Security Standards (restricted mode)
• RBAC for least privilege access
• External secrets management (Vault, AWS Secrets Manager)
• Encrypt secrets at rest
• Admission controllers for policy enforcement

Secrets Management

• Never commit to Git
• Use external secret stores, rotate regularly
• Audit access, mount as files (not env vars when possible)
• Scope to namespaces

See: references/security-hardening.md for network policies, image scanning automation, and compliance configurations.

Infrastructure as Code

Principles

• Version control all manifests
• Declarative over imperative
• Separate environment configs (dev/staging/prod)
• Validate before apply (kubectl dry-run, helm lint)
• Use GitOps (ArgoCD, Flux) -- Git as single source of truth

Helm Best Practices

• Template for environment differences, override with env-specific values files
• Version charts semantically
• Include sane defaults, validate before release

Troubleshooting Checklist

| Symptom | Commands | Common Causes | |---------|----------|---------------| | Pod not starting | kubectl describe pod <name>, kubectl logs <name>, kubectl get events --sort-by=.metadata.creationTimestamp | Image pull errors, resource limits, health check failures | | Service unreachable | kubectl get svc,endpoints <name>, kubectl describe svc <name> | Label mismatch, port misconfiguration, network policies | | High resource usage | kubectl top pods, kubectl describe node <name> | No resource limits, memory leaks, inefficient code | | Deployment stuck | kubectl rollout status deployment/<name>, kubectl get events \| grep <name> | Failing health checks, insufficient resources, image issues |

Quick Reference Links

• references/docker-complete.md - Comprehensive Docker guide
• references/kubernetes-manifests.md - K8s manifests and Helm charts
• references/github-actions.md - Complete CI/CD workflows
• references/observability-stack.md - Monitoring and logging setup
• references/deployment-strategies.md - Deployment patterns and rollbacks
• references/security-hardening.md - Security best practices

Key Principles

1. Automate Everything - Manual processes are error-prone
2. Measure Everything - Can't improve what you don't measure
3. Fail Fast - Catch issues early in pipeline
4. Immutable Infrastructure - Replace, don't modify
5. Infrastructure as Code - Version control all config
6. Monitor Proactively - Alert before users notice
7. Practice Chaos - Test failure scenarios regularly
8. Document Runbooks - Incident response should be scripted

<!-- Last reviewed: 2026-03 -->