dwoolworth/deploy

OPS Deployment Skill

Overview

The deployment skill covers the full lifecycle of deploying code to production: pre-checks, execution, verification, rollback, and documentation. Every deployment follows this process. No shortcuts.

Pre-Deployment Checklist

Before any deployment, verify ALL of the following:

• [ ] CQ Review passed: Confirm a CQ approval comment exists on the ticket
• [ ] QA Testing passed: Confirm a QA PASS comment exists on the ticket
• [ ] Ticket is in rfp status: Only deploy tickets that have reached Ready for Production
• [ ] Infrastructure health is green: All existing services are healthy before introducing changes
• [ ] Dependencies are in place: Database migrations staged, environment variables set, external service dependencies confirmed
• [ ] Rollback plan documented: You know exactly how to revert this deployment
• [ ] Deployment notes reviewed: Any special instructions from DEV, CQ, or QA are understood
• [ ] No conflicting deployments in progress: One deployment at a time to isolate issues

If ANY item on this checklist is not satisfied, STOP. Do not deploy. Communicate the blocker in #standup and on the ticket.

Docker Deployment Commands

Build and Tag the Image

docker build -t ${SERVICE_NAME}:${VERSION} -f Dockerfile .
docker tag ${SERVICE_NAME}:${VERSION} ${REGISTRY}/${SERVICE_NAME}:${VERSION}
docker tag ${SERVICE_NAME}:${VERSION} ${REGISTRY}/${SERVICE_NAME}:latest

Push to Registry

docker push ${REGISTRY}/${SERVICE_NAME}:${VERSION}
docker push ${REGISTRY}/${SERVICE_NAME}:latest

Deploy with Docker Compose

# Pull latest images
docker compose -f docker-compose.prod.yml pull

# Deploy with zero-downtime (if configured)
docker compose -f docker-compose.prod.yml up -d --remove-orphans

# Verify containers are running
docker compose -f docker-compose.prod.yml ps

Direct Container Deployment

# Stop the old container
docker stop ${SERVICE_NAME} || true
docker rm ${SERVICE_NAME} || true

# Run the new container
docker run -d \
  --name ${SERVICE_NAME} \
  --restart unless-stopped \
  --network ${NETWORK_NAME} \
  -p ${HOST_PORT}:${CONTAINER_PORT} \
  --env-file .env.production \
  ${REGISTRY}/${SERVICE_NAME}:${VERSION}

# Verify the container is healthy
docker inspect --format='{{.State.Health.Status}}' ${SERVICE_NAME}

Kubernetes Deployment Commands

Apply Deployment Manifest

# Apply the deployment
kubectl apply -f k8s/deployment.yaml -n ${NAMESPACE}

# Watch the rollout
kubectl rollout status deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE} --timeout=300s

Verify the Deployment

# Check pod status
kubectl get pods -n ${NAMESPACE} -l app=${APP_LABEL}

# Check pod logs for errors
kubectl logs -n ${NAMESPACE} -l app=${APP_LABEL} --tail=50

# Describe deployment for events
kubectl describe deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE}

Scale a Deployment

# Scale up/down
kubectl scale deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE} --replicas=${REPLICA_COUNT}

# Verify scaling
kubectl get pods -n ${NAMESPACE} -l app=${APP_LABEL}

Rollback Procedures

Docker Rollback

# Stop the current (broken) container
docker stop ${SERVICE_NAME}
docker rm ${SERVICE_NAME}

# Run the previous version
docker run -d \
  --name ${SERVICE_NAME} \
  --restart unless-stopped \
  --network ${NETWORK_NAME} \
  -p ${HOST_PORT}:${CONTAINER_PORT} \
  --env-file .env.production \
  ${REGISTRY}/${SERVICE_NAME}:${PREVIOUS_VERSION}

# Verify rollback
docker inspect --format='{{.State.Health.Status}}' ${SERVICE_NAME}

Docker Compose Rollback

# Revert to previous image tags in compose file, then:
docker compose -f docker-compose.prod.yml up -d --remove-orphans
docker compose -f docker-compose.prod.yml ps

Kubernetes Rollback

# Rollback to the previous revision
kubectl rollout undo deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE}

# Watch the rollback
kubectl rollout status deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE} --timeout=300s

# Verify pods are healthy after rollback
kubectl get pods -n ${NAMESPACE} -l app=${APP_LABEL}

# Check rollout history
kubectl rollout history deployment/${DEPLOYMENT_NAME} -n ${NAMESPACE}

Rollback Decision Criteria

Execute a rollback immediately if any of the following occur:

• Health checks fail after deployment
• Error rates spike above baseline threshold
• Response times degrade significantly
• Critical functionality is broken
• Data integrity issues detected

Do NOT attempt to "fix forward" if the issue is unclear. Roll back first, then investigate.

Post-Deployment Verification

After every deployment, verify ALL of the following:

Health Checks

# HTTP health check
curl -sf ${SERVICE_URL}/health || echo "HEALTH CHECK FAILED"

# Detailed health with dependencies
curl -s ${SERVICE_URL}/health/detailed | jq .

Smoke Tests

# Verify critical endpoints are responding
curl -sf -o /dev/null -w "%{http_code}" ${SERVICE_URL}/api/status
curl -sf -o /dev/null -w "%{http_code}" ${SERVICE_URL}/api/ping

Monitoring Verification

# Check error rates (should not spike)
# Check response times (should not degrade)
# Check resource usage (should be within normal bounds)
# Check logs for new error patterns
docker logs ${SERVICE_NAME} --since 5m 2>&1 | grep -i error || echo "No errors found"

Monitoring Setup

Every deployed service must have:

• Health check endpoint: Returns 200 when healthy, non-200 when degraded
• Metrics exposure: CPU, memory, request count, error rate, response time
• Log aggregation: Structured logs shipped to central logging
• Alerting rules: Alerts on health check failure, error rate spike, resource exhaustion

Deployment Documentation Template

Every deployment gets a comment on the ticket with this structure:

## Deployment Record

**Ticket:** #[TICKET_ID]
**Deployed by:** ops
**Timestamp:** [ISO 8601 timestamp]
**Environment:** [production/staging]
**Version:** [version tag or commit hash]
**Previous Version:** [what was running before]

### Changes Deployed
[Brief description of what this deployment includes]

### Deployment Method
[Docker Compose / Kubernetes / Direct Container]

### Pre-Deployment Checks
- [x] CQ review confirmed
- [x] QA pass confirmed
- [x] Infrastructure health verified
- [x] Rollback plan prepared
- [x] Dependencies in place

### Post-Deployment Verification
- [x] Health checks passing
- [x] Smoke tests passing
- [x] Error rates nominal
- [x] Response times nominal
- [x] Monitoring dashboards green

### Rollback Plan
To rollback this deployment:
1. [Exact rollback command or steps]
2. [Verification after rollback]
3. [Notify team in #standup]

### Notes
[Any observations, warnings, or follow-up items]