1mangesh1/ci-cd-pipelines

CI/CD Pipelines

Comprehensive guide to building, configuring, and optimizing CI/CD pipelines across multiple platforms including GitLab CI, Jenkins, CircleCI, and more.

Cross-reference: For GitHub Actions specifically, see the dedicated github-actions skill which covers workflows, matrix builds, caching, secrets, and reusable actions in depth.

CI/CD Pipeline Fundamentals

Core Concepts

• Continuous Integration (CI): Automatically build and test code on every commit
• Continuous Delivery (CD): Automatically prepare releases for deployment (manual trigger)
• Continuous Deployment (CD): Automatically deploy every change that passes the pipeline

Pipeline Anatomy

Commit -> Build -> Test -> Scan -> Stage -> Deploy -> Monitor

• Stages: Logical groupings of jobs (build, test, deploy)
• Jobs: Individual units of work within a stage
• Steps: Commands or actions within a job
• Artifacts: Files produced by jobs and passed between stages
• Triggers: Events that start a pipeline (push, merge, schedule, manual)

Pipeline Design Principles

1. Fail fast - Run quick checks (lint, syntax) before slow tests
2. Parallelize - Run independent jobs concurrently
3. Cache aggressively - Avoid rebuilding unchanged dependencies
4. Minimize artifacts - Pass only what downstream jobs need
5. Idempotent deployments - Running deploy twice yields the same result

---

GitLab CI/CD

Basic .gitlab-ci.yml

# .gitlab-ci.yml
stages:
  - build
  - test
  - scan
  - deploy

variables:
  NODE_VERSION: "20"
  DOCKER_DRIVER: overlay2

# Global defaults
default:
  image: node:${NODE_VERSION}
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
      - .npm/

# Build stage
build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

# Test stage with parallel jobs
unit-tests:
  stage: test
  needs: [build]
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run test:unit -- --coverage
  coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

integration-tests:
  stage: test
  needs: [build]
  services:
    - postgres:16
    - redis:7
  variables:
    POSTGRES_DB: test_db
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: secret
    DATABASE_URL: "postgresql://runner:secret@postgres:5432/test_db"
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run test:integration

# Security scanning
sast:
  stage: scan
  needs: []
  image: semgrep/semgrep:latest
  script:
    - semgrep --config auto --json --output semgrep-results.json .
  artifacts:
    reports:
      sast: semgrep-results.json
  allow_failure: true

dependency-scan:
  stage: scan
  needs: []
  script:
    - npm audit --audit-level=high
  allow_failure: true

# Deploy to staging
deploy-staging:
  stage: deploy
  needs: [unit-tests, integration-tests]
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - ./deploy.sh staging
  only:
    - main

# Deploy to production with manual approval
deploy-production:
  stage: deploy
  needs: [deploy-staging]
  environment:
    name: production
    url: https://example.com
  script:
    - ./deploy.sh production
  when: manual
  only:
    - main

GitLab CI Rules and Conditions

# Advanced rules
deploy-preview:
  stage: deploy
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: always
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: never
    - when: never
  script:
    - ./deploy-preview.sh

# Trigger downstream pipeline
trigger-downstream:
  stage: deploy
  trigger:
    project: mygroup/downstream-project
    branch: main
    strategy: depend
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

GitLab CI Matrix Builds

test-matrix:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ["18", "20", "22"]
        DB: ["postgres", "mysql"]
  image: node:${NODE_VERSION}
  services:
    - name: ${DB}:latest
  script:
    - npm ci
    - npm test

---

Jenkins

Declarative Jenkinsfile

// Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any

    environment {
        DOCKER_REGISTRY = 'registry.example.com'
        APP_NAME = 'my-app'
        NODE_HOME = tool('NodeJS-20')
        PATH = "${NODE_HOME}/bin:${env.PATH}"
    }

    options {
        timeout(time: 30, unit: 'MINUTES')
        disableConcurrentBuilds()
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
    }

    parameters {
        choice(name: 'DEPLOY_ENV', choices: ['staging', 'production'], description: 'Target environment')
        booleanParam(name: 'SKIP_TESTS', defaultValue: false, description: 'Skip test stage')
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
                script {
                    env.GIT_COMMIT_SHORT = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()
                    env.VERSION = "${env.BUILD_NUMBER}-${env.GIT_COMMIT_SHORT}"
                }
            }
        }

        stage('Install') {
            steps {
                sh 'npm ci'
            }
        }

        stage('Lint & Format') {
            steps {
                sh 'npm run lint'
                sh 'npm run format:check'
            }
        }

        stage('Test') {
            when {
                not { params.SKIP_TESTS }
            }
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit -- --ci --reporters=jest-junit'
                    }
                    post {
                        always {
                            junit 'junit.xml'
                        }
                    }
                }
                stage('Integration Tests') {
                    steps {
                        sh 'docker compose -f docker-compose.test.yml up -d'
                        sh 'npm run test:integration'
                    }
                    post {
                        always {
                            sh 'docker compose -f docker-compose.test.yml down -v'
                        }
                    }
                }
            }
        }

        stage('Build Docker Image') {
            steps {
                script {
                    docker.build("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}")
                }
            }
        }

        stage('Push Image') {
            steps {
                script {
                    docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-creds') {
                        docker.image("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}").push()
                        docker.image("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}").push('latest')
                    }
                }
            }
        }

        stage('Deploy') {
            input {
                message "Deploy to ${params.DEPLOY_ENV}?"
                ok 'Deploy'
                submitter 'admin,deployers'
            }
            steps {
                withCredentials([
                    string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN'),
                    file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')
                ]) {
                    sh """
                        kubectl set image deployment/${APP_NAME} \
                            ${APP_NAME}=${DOCKER_REGISTRY}/${APP_NAME}:${VERSION} \
                            -n ${params.DEPLOY_ENV}
                    """
                }
            }
        }
    }

    post {
        success {
            slackSend(
                channel: '#deployments',
                color: 'good',
                message: "SUCCESS: ${APP_NAME} ${VERSION} deployed to ${params.DEPLOY_ENV}"
            )
        }
        failure {
            slackSend(
                channel: '#deployments',
                color: 'danger',
                message: "FAILED: ${APP_NAME} pipeline - ${env.BUILD_URL}"
            )
        }
        always {
            cleanWs()
        }
    }
}

Scripted Jenkinsfile

// Jenkinsfile (Scripted Pipeline)
node('docker') {
    def app
    def version

    try {
        stage('Checkout') {
            checkout scm
            version = sh(script: 'git describe --tags --always', returnStdout: true).trim()
        }

        stage('Build') {
            sh 'npm ci'
            sh 'npm run build'
        }

        stage('Test') {
            parallel(
                'unit': { sh 'npm run test:unit' },
                'lint': { sh 'npm run lint' }
            )
        }

        stage('Docker') {
            app = docker.build("my-app:${version}")
            docker.withRegistry('https://registry.example.com', 'docker-creds') {
                app.push()
                app.push('latest')
            }
        }

        stage('Deploy') {
            if (env.BRANCH_NAME == 'main') {
                timeout(time: 15, unit: 'MINUTES') {
                    input message: 'Deploy to production?', ok: 'Deploy'
                }
                sh "./deploy.sh production ${version}"
            }
        }

        currentBuild.result = 'SUCCESS'
    } catch (e) {
        currentBuild.result = 'FAILURE'
        throw e
    } finally {
        cleanWs()
        emailext(
            subject: "${currentBuild.result}: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
            body: "Build URL: ${env.BUILD_URL}",
            recipientProviders: [culprits(), requestor()]
        )
    }
}

Jenkins Shared Library

// vars/standardPipeline.groovy (Shared Library)
def call(Map config = [:]) {
    pipeline {
        agent any
        stages {
            stage('Build') {
                steps {
                    sh config.buildCommand ?: 'npm ci && npm run build'
                }
            }
            stage('Test') {
                steps {
                    sh config.testCommand ?: 'npm test'
                }
            }
            stage('Deploy') {
                when { branch 'main' }
                steps {
                    sh config.deployCommand ?: './deploy.sh'
                }
            }
        }
    }
}

// Usage in Jenkinsfile:
// @Library('my-shared-lib') _
// standardPipeline(buildCommand: 'make build', testCommand: 'make test')

---

CircleCI

.circleci/config.yml

# .circleci/config.yml
version: 2.1

orbs:
  node: circleci/[email protected]
  docker: circleci/[email protected]
  slack: circleci/[email protected]

executors:
  node-executor:
    docker:
      - image: cimg/node:20.11
    working_directory: ~/project
    resource_class: medium

  node-with-db:
    docker:
      - image: cimg/node:20.11
      - image: cimg/postgres:16.2
        environment:
          POSTGRES_USER: test
          POSTGRES_DB: test_db
          POSTGRES_PASSWORD: test
      - image: cimg/redis:7.2

commands:
  install-deps:
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
          cache-path: node_modules
          cache-version: v1

jobs:
  lint:
    executor: node-executor
    steps:
      - install-deps
      - run:
          name: Lint
          command: npm run lint
      - run:
          name: Type Check
          command: npm run typecheck

  unit-test:
    executor: node-executor
    parallelism: 4
    steps:
      - install-deps
      - run:
          name: Run Unit Tests
          command: |
            TESTFILES=$(circleci tests glob "src/**/*.test.ts" | circleci tests split --split-by=timings)
            npx jest $TESTFILES --ci --reporters=jest-junit
          environment:
            JEST_JUNIT_OUTPUT_DIR: ./reports
      - store_test_results:
          path: ./reports
      - store_artifacts:
          path: ./coverage

  integration-test:
    executor: node-with-db
    steps:
      - install-deps
      - run:
          name: Wait for DB
          command: dockerize -wait tcp://localhost:5432 -timeout 30s
      - run:
          name: Run Migrations
          command: npm run db:migrate
      - run:
          name: Integration Tests
          command: npm run test:integration

  build-and-push:
    executor: docker/docker
    steps:
      - setup_remote_docker:
          version: "20.10.24"
      - checkout
      - docker/build:
          image: myorg/myapp
          tag: "${CIRCLE_SHA1:0:7},latest"
      - docker/push:
          image: myorg/myapp
          tag: "${CIRCLE_SHA1:0:7},latest"

  deploy:
    executor: node-executor
    parameters:
      environment:
        type: enum
        enum: ["staging", "production"]
    steps:
      - checkout
      - run:
          name: Deploy to << parameters.environment >>
          command: ./deploy.sh << parameters.environment >>
      - slack/notify:
          event: pass
          template: basic_success_1

workflows:
  build-test-deploy:
    jobs:
      - lint
      - unit-test:
          requires:
            - lint
      - integration-test:
          requires:
            - lint
      - build-and-push:
          requires:
            - unit-test
            - integration-test
          filters:
            branches:
              only: main
      - deploy:
          name: deploy-staging
          environment: staging
          requires:
            - build-and-push
      - hold-production:
          type: approval
          requires:
            - deploy-staging
      - deploy:
          name: deploy-production
          environment: production
          requires:
            - hold-production

---

Pipeline Design Patterns

Fan-Out / Fan-In

Run multiple jobs in parallel, then converge on a single downstream job.

# GitLab CI example
stages:
  - build
  - test
  - merge-results
  - deploy

build:
  stage: build
  script: npm run build
  artifacts:
    paths: [dist/]

test-unit:
  stage: test
  needs: [build]
  script: npm run test:unit
  artifacts:
    reports:
      junit: reports/unit.xml

test-e2e:
  stage: test
  needs: [build]
  script: npm run test:e2e
  artifacts:
    reports:
      junit: reports/e2e.xml

test-performance:
  stage: test
  needs: [build]
  script: npm run test:perf
  artifacts:
    paths: [perf-results/]

merge-and-report:
  stage: merge-results
  needs: [test-unit, test-e2e, test-performance]
  script: ./generate-report.sh

deploy:
  stage: deploy
  needs: [merge-and-report]
  script: ./deploy.sh

Matrix Builds

Test across multiple versions, platforms, or configurations simultaneously.

# CircleCI matrix
workflows:
  multi-platform:
    jobs:
      - test:
          matrix:
            parameters:
              node-version: ["18", "20", "22"]
              os: ["linux", "macos"]

Conditional Pipelines (Monorepo Strategy)

# GitLab CI - trigger jobs only when relevant files change
frontend:
  stage: build
  rules:
    - changes:
        - "frontend/**"
        - "shared/**"
  script:
    - cd frontend && npm ci && npm run build

backend:
  stage: build
  rules:
    - changes:
        - "backend/**"
        - "shared/**"
  script:
    - cd backend && pip install -r requirements.txt && pytest

infrastructure:
  stage: deploy
  rules:
    - changes:
        - "terraform/**"
  script:
    - cd terraform && terraform plan

---

Caching Strategies

Layer Caching for Speed

# GitLab CI - granular caching
variables:
  NPM_CACHE: "${CI_PROJECT_DIR}/.npm"

default:
  cache:
    - key:
        files:
          - package-lock.json
      paths:
        - node_modules/
      policy: pull-push
    - key: "${CI_JOB_NAME}-build-cache"
      paths:
        - .next/cache/
        - dist/
      policy: pull-push

# Jobs that only read cache
test:
  cache:
    - key:
        files:
          - package-lock.json
      paths:
        - node_modules/
      policy: pull   # Read-only, don't waste time uploading
  script:
    - npm test

Docker Layer Caching

# CircleCI
jobs:
  build-docker:
    machine:
      image: ubuntu-2204:current
      docker_layer_caching: true
    steps:
      - checkout
      - run: docker build -t myapp:latest .

# Jenkins
pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                script {
                    docker.build("myapp:${env.BUILD_NUMBER}",
                        "--cache-from myapp:latest .")
                }
            }
        }
    }
}

---

Secret Management in Pipelines

Best Practices

1. Never hardcode secrets in pipeline files
2. Use platform-native secret stores (GitLab CI Variables, Jenkins Credentials, CircleCI Contexts)
3. Mask secrets in logs
4. Rotate secrets regularly
5. Limit scope - use environment-specific secrets

GitLab CI Variables

# Protected and masked variables set in GitLab UI
# Settings > CI/CD > Variables

deploy:
  script:
    - echo "Deploying with token"
    - curl -H "Authorization: Bearer ${DEPLOY_TOKEN}" https://api.example.com/deploy
  variables:
    DEPLOY_TOKEN: ${PRODUCTION_DEPLOY_TOKEN}  # Pulled from CI/CD settings

Jenkins Credentials

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                withCredentials([
                    usernamePassword(
                        credentialsId: 'docker-hub',
                        usernameVariable: 'DOCKER_USER',
                        passwordVariable: 'DOCKER_PASS'
                    ),
                    string(credentialsId: 'api-key', variable: 'API_KEY'),
                    file(credentialsId: 'service-account', variable: 'SA_KEY_FILE')
                ]) {
                    sh 'docker login -u $DOCKER_USER -p $DOCKER_PASS'
                    sh 'gcloud auth activate-service-account --key-file=$SA_KEY_FILE'
                }
            }
        }
    }
}

HashiCorp Vault Integration

# GitLab CI with Vault
deploy:
  variables:
    VAULT_AUTH_ROLE: "gitlab-ci"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  secrets:
    DATABASE_PASSWORD:
      vault: production/db/password@secrets
      file: false
  script:
    - ./deploy.sh

---

Docker-Based Pipelines

Multi-Stage Docker Build in CI

# GitLab CI with Docker-in-Docker
build-image:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
    IMAGE_TAG: "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build
        --build-arg BUILDKIT_INLINE_CACHE=1
        --cache-from ${CI_REGISTRY_IMAGE}:latest
        --tag ${IMAGE_TAG}
        --tag ${CI_REGISTRY_IMAGE}:latest
        .
    - docker push ${IMAGE_TAG}
    - docker push ${CI_REGISTRY_IMAGE}:latest

Kaniko (Rootless Docker Builds)

# GitLab CI with Kaniko (no Docker daemon needed)
build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.21.0-debug
    entrypoint: [""]
  script:
    - /kaniko/executor
        --context "${CI_PROJECT_DIR}"
        --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
        --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
        --cache=true
        --cache-repo="${CI_REGISTRY_IMAGE}/cache"

---

Deploy Strategies

Blue-Green Deployment

# GitLab CI blue-green deploy
deploy-blue-green:
  stage: deploy
  script:
    - |
      # Determine current active (blue or green)
      CURRENT=$(kubectl get svc myapp -o jsonpath='{.spec.selector.slot}')
      if [ "$CURRENT" = "blue" ]; then TARGET="green"; else TARGET="blue"; fi

      # Deploy to inactive slot
      kubectl set image deployment/myapp-${TARGET} app=${IMAGE_TAG}
      kubectl rollout status deployment/myapp-${TARGET} --timeout=300s

      # Run smoke tests against inactive slot
      ./smoke-test.sh https://${TARGET}.internal.example.com

      # Switch traffic
      kubectl patch svc myapp -p "{\"spec\":{\"selector\":{\"slot\":\"${TARGET}\"}}}"

      echo "Switched traffic from ${CURRENT} to ${TARGET}"

Canary Deployment

# GitLab CI canary deploy
deploy-canary:
  stage: deploy
  script:
    - |
      # Deploy canary (10% traffic)
      kubectl apply -f k8s/canary-deployment.yaml
      kubectl set image deployment/myapp-canary app=${IMAGE_TAG}

      # Wait and monitor error rates
      sleep 300
      ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')

      if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
        echo "Error rate too high (${ERROR_RATE}), rolling back canary"
        kubectl delete deployment myapp-canary
        exit 1
      fi

      # Promote canary to full deployment
      kubectl set image deployment/myapp app=${IMAGE_TAG}
      kubectl rollout status deployment/myapp --timeout=300s
      kubectl delete deployment myapp-canary

deploy-rolling:
  stage: deploy
  script:
    - |
      kubectl set image deployment/myapp app=${IMAGE_TAG}
      kubectl rollout status deployment/myapp --timeout=600s

      # If rollout fails, auto-rollback
      if [ $? -ne 0 ]; then
        kubectl rollout undo deployment/myapp
        exit 1
      fi

---

Environment Promotion (Dev -> Staging -> Production)

# GitLab CI promotion pipeline
stages:
  - build
  - test
  - deploy-dev
  - deploy-staging
  - deploy-production

build:
  stage: build
  script:
    - docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA} .
    - docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

test:
  stage: test
  script:
    - npm ci && npm test

deploy-dev:
  stage: deploy-dev
  environment:
    name: development
    url: https://dev.example.com
    on_stop: stop-dev
  script:
    - helm upgrade --install myapp ./chart
        --namespace dev
        --set image.tag=${CI_COMMIT_SHORT_SHA}
        --set ingress.host=dev.example.com
  rules:
    - if: '$CI_COMMIT_BRANCH != "main"'

stop-dev:
  stage: deploy-dev
  environment:
    name: development
    action: stop
  script:
    - helm uninstall myapp --namespace dev
  when: manual

deploy-staging:
  stage: deploy-staging
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - helm upgrade --install myapp ./chart
        --namespace staging
        --set image.tag=${CI_COMMIT_SHORT_SHA}
        --set ingress.host=staging.example.com
    - ./run-smoke-tests.sh https://staging.example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

deploy-production:
  stage: deploy-production
  environment:
    name: production
    url: https://example.com
  script:
    - helm upgrade --install myapp ./chart
        --namespace production
        --set image.tag=${CI_COMMIT_SHORT_SHA}
        --set ingress.host=example.com
        --set replicas=3
  when: manual
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

---

Pipeline Security (SAST, DAST, Dependency Scanning)

# GitLab CI security scanning stages
stages:
  - build
  - test
  - security
  - deploy

# Static Application Security Testing
sast-semgrep:
  stage: security
  image: semgrep/semgrep:latest
  script:
    - semgrep --config=auto --json --output=semgrep.json .
  artifacts:
    reports:
      sast: semgrep.json
  allow_failure: true

# Dependency vulnerability scanning
dependency-check:
  stage: security
  script:
    - npm audit --json > npm-audit.json || true
    - npx audit-ci --config audit-ci.json
  artifacts:
    paths:
      - npm-audit.json

# Container image scanning
container-scan:
  stage: security
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image
        --exit-code 1
        --severity HIGH,CRITICAL
        --format json
        --output trivy-results.json
        ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}
  artifacts:
    paths:
      - trivy-results.json
  allow_failure: true

# Dynamic Application Security Testing
dast:
  stage: security
  needs: [deploy-staging]
  image: owasp/zap2docker-stable
  script:
    - zap-baseline.py
        -t https://staging.example.com
        -r zap-report.html
        -l WARN
  artifacts:
    paths:
      - zap-report.html
  allow_failure: true

# Secret detection
secret-detection:
  stage: security
  image: trufflesecurity/trufflehog:latest
  script:
    - trufflehog git --json --fail file://. > secrets-report.json
  artifacts:
    paths:
      - secrets-report.json

---

Notification Integration

Slack Notifications

# GitLab CI with Slack
.notify_slack: &notify_slack
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = "success" ]; then
        COLOR="#36a64f"
        STATUS="succeeded"
      else
        COLOR="#dc3545"
        STATUS="failed"
      fi
      curl -X POST "$SLACK_WEBHOOK_URL" \
        -H 'Content-type: application/json' \
        -d "{
          \"attachments\": [{
            \"color\": \"${COLOR}\",
            \"title\": \"Pipeline ${STATUS}: ${CI_PROJECT_NAME}\",
            \"text\": \"Branch: ${CI_COMMIT_REF_NAME}\nCommit: ${CI_COMMIT_SHORT_SHA}\nJob: ${CI_JOB_NAME}\",
            \"footer\": \"<${CI_PIPELINE_URL}|View Pipeline>\"
          }]
        }"

deploy-production:
  stage: deploy
  <<: *notify_slack
  script:
    - ./deploy.sh production

Email Notifications (Jenkins)

// Jenkinsfile post section
post {
    failure {
        emailext(
            subject: "FAILED: ${env.JOB_NAME} [${env.BUILD_NUMBER}]",
            body: """
                <h2>Build Failed</h2>
                <p>Job: ${env.JOB_NAME}</p>
                <p>Build: <a href="${env.BUILD_URL}">#${env.BUILD_NUMBER}</a></p>
                <p>Changes: ${currentBuild.changeSets.collect { it.items.collect { it.msg } }.flatten().join(', ')}</p>
            """,
            mimeType: 'text/html',
            recipientProviders: [culprits(), developers(), requestor()]
        )
    }
}

---

Artifact Management

# GitLab CI artifact strategies
build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
      - build/
    exclude:
      - dist/**/*.map
    expire_in: 1 week
    when: on_success

test:
  stage: test
  script:
    - npm test
  artifacts:
    # Upload test results for GitLab UI integration
    reports:
      junit: reports/junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    # Also keep raw coverage for downstream
    paths:
      - coverage/
    when: always    # Keep artifacts even on failure
    expire_in: 30 days

---

Pipeline as Code Best Practices

1. Version control your pipeline - Pipeline config lives with the code it builds
2. Use templates and includes to reduce duplication

# GitLab CI includes
include:
  - project: 'devops/pipeline-templates'
    ref: main
    file:
      - '/templates/node-ci.yml'
      - '/templates/docker-build.yml'
      - '/templates/deploy.yml'

  - local: '.gitlab/ci/security.yml'

  - template: Security/SAST.gitlab-ci.yml

3. Pin versions of tools and images
4. Use linting to validate pipeline configs before commit

# Validate GitLab CI locally
gitlab-ci-lint .gitlab-ci.yml

# Validate CircleCI config
circleci config validate

# Validate Jenkins pipeline
# Use Jenkins CLI or replay feature

5. Keep pipelines fast - target under 10 minutes for CI, under 30 for full CD
6. Use YAML anchors to reduce repetition

.deploy_template: &deploy_defaults
  image: alpine/k8s:1.28.4
  before_script:
    - kubectl config use-context ${KUBE_CONTEXT}
  script:
    - helm upgrade --install ${APP_NAME} ./chart --namespace ${NAMESPACE}

deploy-staging:
  <<: *deploy_defaults
  variables:
    KUBE_CONTEXT: staging-cluster
    NAMESPACE: staging
  environment:
    name: staging

deploy-production:
  <<: *deploy_defaults
  variables:
    KUBE_CONTEXT: production-cluster
    NAMESPACE: production
  environment:
    name: production
  when: manual

---

Common Pipeline Failures and Fixes

| Problem | Cause | Fix | |---------|-------|-----| | npm ci fails | Lock file mismatch | Run npm install locally and commit package-lock.json | | Docker build OOM | Insufficient memory | Increase runner resources or optimize Dockerfile | | Tests pass locally, fail in CI | Environment differences | Use same Docker image locally; check env vars | | Cache not restoring | Key mismatch | Verify cache key references correct files | | Pipeline hangs | Waiting for approval or stuck service | Add timeouts; check when: manual stages | | Permission denied | File not executable | Run chmod +x script.sh and commit | | Secret not available | Variable scope mismatch | Check protected/masked settings; branch protection | | Flaky tests | Race conditions or external deps | Retry mechanism; mock external services | | Slow pipelines | No parallelism or caching | Add parallel, needs (DAG), and caching | | Deploy rollback fails | No rollback strategy | Implement kubectl rollout undo or Helm rollback |

Retry Flaky Jobs

# GitLab CI
flaky-integration-test:
  stage: test
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - script_failure
  script:
    - npm run test:integration

Timeout Configuration

# GitLab CI
job-with-timeout:
  timeout: 15 minutes
  script:
    - npm run long-test

# CircleCI
jobs:
  slow-test:
    steps:
      - run:
          name: Long running test
          command: npm run test:e2e
          no_output_timeout: 20m

---

References

• GitLab CI/CD Documentation
• Jenkins Pipeline Syntax
• CircleCI Configuration Reference
• 12 Factor App - Build/Release/Run
• DORA Metrics for CI/CD Performance
• See also: github-actions skill for GitHub-specific workflows