pantheon-org/azure-pipelines-validator

Azure Pipelines Validator

Validates, lints, and security-scans Azure DevOps Pipeline configurations (azure-pipelines.yml, azure-pipelines.yaml). Runs four validation layers via a single orchestrator script.

Basic Usage

# Full validation (all layers)
bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml

Layers executed in order: 0. YAML lint (yamllint) — formatting, indentation, trailing spaces

1. Syntax validation — schema, required fields, stages/jobs/steps hierarchy, task format, dependencies
2. Best practices — display names, task version pinning, pool image specificity, caching, timeouts
3. Security scan — hardcoded secrets/API keys/AWS/Azure credentials, dangerous script patterns, SSL bypasses, container :latest tags

Common Options

# Targeted runs
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --syntax-only
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only

# Skip layers
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --skip-yaml-lint
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-best-practices
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-security

# Strict mode (fail on warnings)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict

Individual Scripts

python3 scripts/validate_syntax.py azure-pipelines.yml
python3 scripts/check_best_practices.py azure-pipelines.yml
python3 scripts/check_security.py azure-pipelines.yml

Output and Error Recovery

════════════════════════════════════════════════════════════════════════════════
  Azure Pipelines Validator
════════════════════════════════════════════════════════════════════════════════

[1/3] Running syntax validation...
✓ Syntax validation passed

[2/3] Running best practices check...
SUGGESTIONS (2):
  INFO: Line 15: Job 'BuildJob' should have displayName [missing-displayname]
  💡 Add 'displayName: "Your Job Description"' to job 'BuildJob'

  WARNING: Line 25: Task 'Npm@1' could benefit from caching [missing-cache]
  💡 Add Cache@2 task to cache dependencies and speed up builds

[3/3] Running security scan...
MEDIUM SEVERITY (1):
  MEDIUM: Line 8: Container 'linux' uses ':latest' tag [container-latest-tag]
  🔒 Pin container images to specific versions or SHA digests

When validation fails:

1. Note the rule code in brackets (e.g., [missing-displayname]) — see references/ for rule details.
2. Fix the flagged line and re-run the same layer (--syntax-only, --security-only, etc.) to iterate quickly.
3. Run full validation once all targeted fixes are applied to confirm no regressions.
4. For MEDIUM/HIGH security findings, do not merge until resolved; INFO findings are advisory.

Common Scenarios

New pipeline validation

bash scripts/validate_azure_pipelines.sh new-pipeline.yml

Security audit before merge

bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only --strict

Pipeline optimisation

bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices

CI/CD self-validation in Azure Pipelines

steps:
- script: |
    bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict
  displayName: 'Validate Pipeline Configuration'

Auto-detection

Run without arguments to auto-detect azure-pipelines*.yml files in the current directory (up to 3 levels deep).

Fetching Live Documentation

The validator performs static analysis only. For dynamic lookups (task versions, input parameters, feature docs), use:

# Context7 MCP
mcp__context7__resolve-library-id("azure-pipelines")
mcp__context7__get-library-docs(context7CompatibleLibraryID, topic="deployment")

# Or WebSearch / WebFetch
WebSearch("Azure Pipelines Docker@2 task documentation 2025")
WebFetch("https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/docker-v2")

Requirements

• Python 3.7+, Bash
• PyYAML and yamllint: auto-installed in a persistent .venv if not available system-wide — no manual setup required.

# Optional manual install
pip3 install PyYAML yamllint

Troubleshooting

| Problem | Fix | |---|---| | ModuleNotFoundError: PyYAML | pip3 install PyYAML | | Permission denied | chmod +x scripts/*.sh scripts/*.py | | Unexpected validation errors | Check references/azure-pipelines-reference.md or Microsoft Learn |

Anti-Patterns

NEVER skip validation on template files

• WHY: Templates called from the main pipeline can contain schema violations, missing parameters, or invalid task references that only surface at runtime. Validating only the entry-point file leaves template errors undetected until the pipeline actually runs.
• BAD: Validate only azure-pipelines.yml and skip all files under templates/*.yml.
• GOOD: Run the validator on every .yml file in the pipeline directory, including all templates.

NEVER treat YAML lint warnings as informational

• WHY: Azure Pipelines YAML is whitespace-significant. Indentation errors and trailing spaces can silently restructure the pipeline — turning a step into a job property, or merging two separate blocks — without any obvious parse error.
• BAD: Ignore yamllint warnings such as "trailing spaces" or "wrong indentation" because the pipeline appears to run.
• GOOD: Fix all YAML formatting issues before submitting; a clean yamllint pass is a prerequisite for a trustworthy pipeline.

NEVER use the strict mode flag on an unreviewed pipeline

• WHY: --strict fails on all warnings, which is the correct setting for a CI gate. Applied to a brand-new pipeline with dozens of warnings, it produces so much noise that engineers discard the output entirely and disable validation rather than fix the root causes.
• BAD: Run --strict on a new pipeline, see 30 warnings, and remove validation from the workflow because "it's too noisy."
• GOOD: Run without --strict first, fix critical errors, then warnings, then graduate to strict mode as a CI gate.

NEVER skip security scanning before merging pipeline changes

• WHY: A passing syntax validation does not imply a secure pipeline. Security scanning catches hardcoded credentials, injection vectors, and insecure patterns that syntax checks are not designed to detect.
• BAD: Merge a pipeline change after syntax validation passes with no security check.
• GOOD: Always run --security as part of the validation workflow, treating MEDIUM and HIGH findings as merge blockers.

References

• references/azure-pipelines-reference.md — full YAML syntax reference and rule definitions
• assets/examples/basic-pipeline.yml — simple CI pipeline
• assets/examples/docker-build.yml — Docker build and push
• assets/examples/deployment-pipeline.yml — multi-environment deployment with approval gates
• assets/examples/multi-platform.yml — multi-platform build matrix
• assets/examples/template-example.yml — reusable templates

# Test with a bundled example
bash scripts/validate_azure_pipelines.sh assets/examples/basic-pipeline.yml

Extending the Skill

Add custom rules to the appropriate script:

• Syntax rules → scripts/validate_syntax.py
• Best practice rules → scripts/check_best_practices.py
• Security rules → scripts/check_security.py

# Example custom best-practice rule in check_best_practices.py
def _check_custom_rule(self):
    for job in self._get_all_jobs():
        job_name = job.get('job') or job.get('deployment')
        if 'tags' not in pool:
            self.issues.append(BestPracticeIssue(
                'warning',
                self._get_line(job_name),
                f"Job '{job_name}' should specify agent tags",
                'custom-missing-tags',
                "Add 'tags' to pool to select appropriate agents"
            ))

---

Note: This skill validates pipeline configurations but does not execute pipelines. Use Azure DevOps Pipeline validation or Azure CLI to test actual pipeline execution.