michaelsvanbeek/cicd

CI/CD Standards (Harness Drone)

When to Use

• Creating or modifying .drone.yml pipelines
• Designing build, test, and deploy stages
• Adding code coverage reporting
• Publishing Docker images or other artifacts
• Configuring secrets, triggers, or cron schedules
• Reviewing pipeline design for correctness and efficiency
• Auditing an existing pipeline for missing stages (lint, test, coverage, security scan)
• Improving pipeline speed (parallelism, caching, fail-fast ordering)

Platform

• Harness Drone (OSS / Community Edition) running Docker pipelines.
• Pipeline config lives in .drone.yml at the repo root.
• Every pipeline is kind: pipeline, type: docker.

---

Core Principles

1. Pipeline as code — .drone.yml is version-controlled, reviewed, and tested like application code.
2. Fail fast — cheapest checks (lint, format) run first. Expensive steps (integration tests, builds) run only if cheap checks pass.
3. Hermetic builds — every step starts from a known Docker image. No reliance on host state.
4. Secrets never in YAML — use Drone secrets (from_secret). No env var values in pipeline definitions.
5. Idempotent deploys — running the same pipeline twice produces the same result.

---

Pipeline Architecture

Separate concerns into distinct pipelines (Drone --- document separators):

| Pipeline | Trigger | Purpose | |----------|---------|---------| | lint-and-test | push, pull_request on main | Quality gate — must pass before merge | | build-and-publish | push to main (post-merge) | Build artifacts, push images | | deploy | push to main OR manual promote | Ship to staging/production | | scheduled | cron | Recurring tasks (backups, scans, syncs) |

Pipeline Ordering

# Pipeline 1: Gate — blocks merge if it fails
kind: pipeline
name: lint-and-test
trigger:
  branch: [main]
  event: [push, pull_request]

# Pipeline 2: Build — only on merged code
kind: pipeline
name: build-and-publish
trigger:
  branch: [main]
  event: [push]
depends_on: [lint-and-test]

# Pipeline 3: Deploy — after build succeeds
kind: pipeline
name: deploy
trigger:
  branch: [main]
  event: [push]
depends_on: [build-and-publish]

---

Stage 1: Lint and Format

Linting is the fastest, cheapest check. It runs first and blocks everything else.

steps:
  - name: lint
    image: python:3.12-slim
    commands:
      - pip install ruff
      - ruff check .
      - ruff format --check .

Rules:

• Lint checks are non-negotiable — no || true, no allow_failure.
• Use the exact same tool versions in CI and local dev.
• For multi-language repos, run language-specific linters in parallel steps.

| Language | Lint step | |----------|-----------| | Python | ruff check . && ruff format --check . | | TypeScript | npx prettier --check . && npx eslint . | | YAML / Markdown | npx prettier --check "**/*.{yml,yaml,md}" | | Dockerfile | hadolint docker/*.Dockerfile |

---

Stage 2: Test

Tests run after lint passes. Use parallel steps per test target to maximize throughput.

  - name: test-shared
    image: python:3.12-slim
    commands:
      - pip install pytest pytest-cov python-dotenv
      - pytest shared/ -v --tb=short

  - name: test-my-script
    image: python:3.12-slim
    commands:
      - pip install pytest pytest-cov -r scripts/my_script/requirements.txt
      - >-
        pytest scripts/my_script/ -v --tb=short
        --cov=scripts/my_script
        --cov-report=term-missing
        --cov-fail-under=80

Rules:

• Every test step installs only the dependencies it needs — no global requirements.txt.
• Use --tb=short for concise failure output in CI.
• Use --cov-fail-under=80 to enforce minimum coverage (see coverage section).
• If a test step needs system packages, use a custom Docker image (see build stage).
• Never skip tests in CI with @pytest.mark.skip unless there's a tracked issue.

Test Parallelism

Drone runs steps within a pipeline in parallel by default. Steps that depend on each other need explicit depends_on:

steps:
  - name: lint
    ...

  - name: test-unit
    depends_on: [lint]
    ...

  - name: test-integration
    depends_on: [lint]
    ...

Lint → then unit + integration in parallel → then build (depends on both).

---

Stage 3: Code Coverage

Configuration

  - name: test-with-coverage
    image: python:3.12-slim
    commands:
      - pip install pytest pytest-cov
      - >-
        pytest
        --cov=src
        --cov-report=term-missing
        --cov-report=xml:coverage.xml
        --cov-fail-under=80

Coverage Rules

| Rule | Value | Rationale | |------|-------|-----------| | Minimum threshold | 80% | Floor — not a target. New code should aim higher. | | Fail the build | Yes | --cov-fail-under=80 makes the step exit non-zero | | Report format | term-missing + xml | Terminal for quick review, XML for tooling integration | | What counts | Tested lines / total lines | Branch coverage is ideal but line coverage is the practical minimum |

Coverage Anti-Patterns

• Don't chase 100% — diminishing returns above 90%. Focus coverage on business logic and error paths.
• Don't exclude files just to raise the number — if code exists, it should be tested.
• Don't count generated code — exclude auto-generated files in .coveragerc or pyproject.toml.

[tool.coverage.run]
source = ["src", "scripts"]
omit = ["*/test_*", "*/__pycache__/*"]

[tool.coverage.report]
fail_under = 80
show_missing = true

---

Stage 4: Build Artifacts

Docker Images

Use the Drone Docker plugin for building and pushing images:

  - name: build-and-push
    image: plugins/docker
    settings:
      dockerfile: docker/my-service.Dockerfile
      repo: ${DRONE_REPO_OWNER}/my-service
      tags:
        - latest
        - ${DRONE_COMMIT_SHA:0:8}
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
    when:
      branch: [main]
      event: [push]

Rules:

• Always tag with both latest and a commit-specific tag (short SHA or semver).
• Build images only on main branch, never on PRs (PRs run lint + test only).
• Use multi-stage Dockerfiles to keep published images small (see docker skill).
• Store registry credentials as Drone secrets, never in pipeline YAML.
• For release builds, prefer semver tags over SHA — see the release pipeline section below.

Python Packages

For internal Python packages:

  - name: build-package
    image: python:3.12-slim
    commands:
      - pip install build
      - python -m build
    when:
      branch: [main]
      event: [push]

---

Stage 5: Deploy

Deployment Strategies

| Strategy | When to use | Risk | |----------|-------------|------| | Direct deploy | Homelab scripts, internal tools | Low — rollback is a git revert + re-deploy | | Blue-green | Services with zero-downtime requirement | Medium — need two environments | | Canary | User-facing APIs with high traffic | Low — gradual rollout, auto-rollback | | Manual promote | Production deployments requiring approval | Lowest — human gate |

Homelab Deploy Pattern

For homelab scripts running on a schedule, deployment means the cron pipeline pulls the latest image:

---
kind: pipeline
name: deploy-to-homelab
trigger:
  branch: [main]
  event: [push]
depends_on: [build-and-publish]

steps:
  - name: deploy
    image: appleboy/drone-ssh
    settings:
      host:
        from_secret: deploy_host
      username:
        from_secret: deploy_user
      key:
        from_secret: deploy_ssh_key
      script:
        - cd /opt/homelab && docker compose pull && docker compose up -d

Serverless Deploy Pattern

For AWS Lambda services:

  - name: deploy-staging
    image: node:20-slim
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    commands:
      - npm ci
      - npx serverless deploy --stage dev
    when:
      branch: [main]
      event: [push]

  - name: deploy-production
    image: node:20-slim
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    commands:
      - npm ci
      - npx serverless deploy --stage prod
    when:
      event: [promote]
      target: [production]

Promote to production manually: drone build promote <repo> <build> production

---

Secrets Management

• All credentials go through Drone secrets — from_secret: key_name.
• Store secrets via drone secret add CLI or Drone UI, never in .drone.yml.
• Secrets are not exposed in logs, pull request builds, or fork builds by default.
• For multi-repo secrets (Docker Hub creds), use organization-level secrets.

environment:
  API_KEY:
    from_secret: api_key        # correct
  # API_KEY: "sk-1234..."       # NEVER — exposed in version control

---

Cron / Scheduled Pipelines

For recurring tasks (backups, health checks, cleanup):

---
kind: pipeline
name: backup-daily
trigger:
  event: [cron]
  cron: [backup-daily]

steps:
  - name: run-backup
    image: python:3.12-slim
    environment:
      BACKUP_SOURCE:
        from_secret: backup_source
    commands:
      - pip install -r scripts/backup/requirements.txt
      - python -m scripts.backup.backup

Rules:

• Cron pipeline names match the Drone cron job name for clarity.
• Cron pipelines should not share triggers with CI pipelines — use separate --- documents.
• Always include error notifications (webhook, email) for scheduled jobs.

---

Step Design Best Practices

Base Images

| Language | Image | Notes | |----------|-------|-------| | Python | python:3.12-slim | Minimal, fast pull | | Node.js | node:20-slim | For Serverless, frontend builds | | Docker builds | plugins/docker | Drone's Docker build plugin | | SSH deploy | appleboy/drone-ssh | Remote execution via SSH | | General CLI | alpine:3.19 | When you just need curl, jq, etc. |

Caching

Drone Docker pipelines don't have built-in caching. Options:

• Pre-built images with dependencies baked in — fastest, most reliable.
• Drone volume (/tmp/cache mounted) — works for single-node setups.
• Layer caching for Docker builds — use plugins/docker with cache_from.

Notifications

  - name: notify-failure
    image: plugins/slack
    settings:
      webhook:
        from_secret: slack_webhook
      channel: ci-alerts
      template: >
        {{repo.name}} build {{build.number}} failed on {{build.branch}}.
        {{build.link}}
    when:
      status: [failure]

---

Performance Pipeline

Integrate performance testing into CI with a tiered approach (see performance-testing and ui-performance skills for full details):

| Trigger | Test type | Purpose | |---------|-----------|----------| | Every PR | Smoke (k6, 1 min, 5 VUs) | Catch gross regressions cheaply | | Release tag | Load + Stress (k6, 15 min) | Validate capacity before shipping | | Release tag | Lighthouse audit | Enforce frontend performance score ≥ 80 | | Weekly cron | Soak (k6, 1-4 hours) | Detect memory leaks and connection exhaustion |

  - name: perf-smoke
    image: grafana/k6:latest
    commands:
      - k6 run --duration=1m --vus=5 tests/perf/smoke.js
    depends_on: [test]

---

Release Pipeline

Tag-Triggered Releases

Add a dedicated release pipeline that triggers only on semver tags. This separates release builds from regular CI (see release-management skill for full versioning conventions):

---
kind: pipeline
name: release
trigger:
  event: [tag]
  ref: ["refs/tags/v*"]

steps:
  - name: validate-tag
    image: alpine:3.19
    commands:
      - |
        TAG="${DRONE_TAG}"
        if ! echo "$TAG" | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$'; then
          echo "ERROR: Tag '$TAG' does not match semver format"
          exit 1
        fi
        echo "Valid semver tag: $TAG"

  - name: lint
    image: python:3.12-slim
    commands:
      - pip install ruff
      - ruff check .
      - ruff format --check .
    depends_on: [validate-tag]

  - name: test
    image: python:3.12-slim
    commands:
      - pip install pytest pytest-cov
      - pytest -v --tb=short --cov=src --cov-fail-under=80
    depends_on: [validate-tag]

  - name: build-and-publish
    image: plugins/docker
    settings:
      repo: ${DRONE_REPO_OWNER}/my-service
      tags:
        - latest
        - ${DRONE_TAG}
        - ${DRONE_TAG##v}
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
    depends_on: [lint, test]

Release Pipeline Rules

| Rule | Rationale | |------|-----------| | Validate tag format | Reject accidental or malformed tags before building | | Re-run lint + test | Tag may point to a commit that wasn't the latest CI run | | Tag with semver | v1.5.0 and 1.5.0 alongside latest for flexibility | | Skip latest for pre-releases | v2.0.0-beta.1 should not be tagged latest | | Version-file validation | Ensure pyproject.toml / package.json version matches the tag |

Version-File Validation Step

If the project stores a version in a file, add a validation step to prevent drift:

  - name: validate-version
    image: python:3.12-slim
    commands:
      - |
        FILE_VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
        TAG_VERSION="${DRONE_TAG#v}"
        if [ "$FILE_VERSION" != "$TAG_VERSION" ]; then
          echo "ERROR: pyproject.toml version ($FILE_VERSION) != tag ($TAG_VERSION)"
          exit 1
        fi
    depends_on: [validate-tag]

---

Pipeline Checklist

Use this checklist when reviewing any .drone.yml:

| Check | What to verify | |-------|---------------| | Fail fast | Lint/format steps run before test steps | | No secrets in YAML | All credentials use from_secret | | Pinned images | Step images use specific tags, not :latest | | Test coverage | --cov-fail-under is set and enforced | | PR safety | Build/deploy steps have when: event: [push] (not on PRs) | | Cron isolation | Scheduled pipelines use separate --- documents | | Dependencies | depends_on ensures correct ordering between pipelines | | Notifications | Failure notifications are configured | | Minimal installs | Each step installs only what it needs | | Short SHA tags | Docker images tagged with commit SHA, not just latest | | Release pipeline | Tag-triggered pipeline exists for semver releases | | Tag validation | Release pipeline validates semver format before building |