renatocaliari/cali-agents-md-validator

AGENTS.md Validator

Validates AGENTS.md files against 13 criteria from industry research and the canonical cali-agents-md-generator template. Pure bash + grep — no agent harness dependencies. Works on any POSIX shell.

When to Use

• After creating a new AGENTS.md
• After updating an existing AGENTS.md
• When user says "validate agents md", "check agents md quality", "audit my agents md"
• As part of project setup review
• Before committing changes to AGENTS.md
• In CI to gate PRs that touch AGENTS.md

Portability

This skill makes no assumptions about the agent harness. It does not call ask_user_question, /skill:, or any harness-specific API. All checks are pure bash / grep / awk. The skill body instructs the agent in prose; the agent decides how to surface questions to the user based on its own tools.

Validation Rules

Severity legend: FAIL = must fix, WARN = should fix, INFO = observation.

R1: AGENTS.md Exists (FAIL if missing)

test -f AGENTS.md && echo "pass" || echo "fail"

R2: File Size ≤150 Lines (FAIL if over)

lines=$(wc -l < AGENTS.md)
if [ "$lines" -gt 150 ]; then echo "fail"; else echo "pass"; fi

Source: ETH Zurich research shows >150 lines causes "silent rule dropout". 150 is the hard safety limit. See R13 for the soft target.

R3: Has "Commands" Section (FAIL if missing)

grep -qE "^##[[:space:]]+Commands" AGENTS.md && echo "pass" || echo "fail"

Source: OpenAI Codex, GitHub analysis — copy-pasteable commands are essential.

R4: Has "Don'ts" Section (FAIL if missing)

grep -qE "^##[[:space:]]+Don'ts" AGENTS.md && echo "pass" || echo "fail"

Source: GitHub 2500+ repos analysis — explicit prohibitions prevent common errors.

R5: No Secrets in File (FAIL if found)

grep -qiE "(api_key|secret|password|token|credential).*=.*['\"][a-zA-Z0-9]{20,}" AGENTS.md && echo "fail" || echo "pass"

Source: GitHub analysis — #1 most common constraint in AGENTS.md files.

R6: Stack with Exact Versions (WARN if vague)

# Catches: TypeScript 5.7, Go 1.26, Node >=20, vitest ^3.1.1, python 3.12
# Excludes: prose mentions like "see section 2.1" or "3.5x speedup"
grep -qE "[A-Za-z][A-Za-z0-9.+_-]*[[:space:]]*[v^~>=<]*[0-9]+\.[0-9]+" AGENTS.md && echo "pass" || echo "warn"

False positive risk: low — pattern requires word boundary before version. Worst case is "step 3.2" or "section 2.1" matching in prose; not harmful (WARN only, doesn't fail).

Source: Augment Code, Inngest repo — "React 19" not just "React". Applies to any language: TypeScript, Go, Node, Python, Rust, Swift, Kotlin, etc.

R7: No Architectural Bloat (WARN)

# Detects sections > 20 lines of prose (excluding tables and code)
awk '/^##[^#]/{section=$0; lines=0; next}
     /^##[^#]/{if(lines>20) print section": "lines" lines"; lines=0; next}
     {lines++}' AGENTS.md | head -3

Source: ETH Zurich — arch sections increase cost without improving success. Move deep details to README or docs/.

R8: Critical Rules in First 50 Lines (WARN)

head -50 AGENTS.md | grep -qE "^##[[:space:]]+(Don'ts|Commands|Rule)" && echo "pass" || echo "warn"

Source: "Lost in the middle" phenomenon — LLMs attend most to first/last 25%.

R9: References Use Skill Names or Links (WARN)

# Detects kebab-case backtick strings (works for any skill naming convention):
#   `my-team-workflow`, `cali-agents-md-validator`, `user-defined-skill`
# Pattern: 3+ words separated by hyphens, lowercase + digits
grep -oE '`[a-z][a-z0-9]+(-[a-z0-9]+){2,}`' AGENTS.md | sort -u | head -5

If at least one match: pass. Otherwise warn.

Why this works for any ecosystem:

• Catches cali-... (Cali ecosystem)
• Catches my-org-... (other org ecosystems)
• Catches user-defined-... (any custom prefix)
• Misses: step-1 (only 2 words), myvar (no hyphens), /path/to/file.md (paths)
• False positives: rare; pattern requires ≥ 3 kebab-case words

R10: Template Compliance — Commands + Don'ts + Architecture/Stack (WARN)

has_commands=$(grep -qE "^##[[:space:]]+Commands" AGENTS.md && echo 1 || echo 0)
has_donts=$(grep -qE "^##[[:space:]]+Don'ts" AGENTS.md && echo 1 || echo 0)
has_arch=$(grep -qE "^##[[:space:]]+(Architecture|Stack)" AGENTS.md && echo 1 || echo 0)
[ "$has_commands" -eq 1 ] && [ "$has_donts" -eq 1 ] && [ "$has_arch" -eq 1 ] && echo "pass" || echo "warn"

Source: cali-agents-md-generator template — Commands + Don'ts + Architecture are the core sections.

R11: Internal References Resolve (FAIL if any missing)

# Check that files referenced in AGENTS.md actually exist
# Pattern: `path/to/file.md` or `path/to/file.ts` (any relative file mention)
missing=$(grep -oE '`[a-zA-Z0-9._/-]+\.(md|ts|tsx|js|sh|json|yml|yaml)`' AGENTS.md \
  | tr -d '`' | sort -u \
  | while read f; do [ -f "$f" ] || echo "$f"; done)
[ -z "$missing" ] && echo "pass" || echo "fail: missing:$missing"

Why this rule: Agents reading AGENTS.md may follow file references. A broken reference wastes context and erodes trust. This rule catches stale mentions after renames or moves.

Source: Microsoft Learn skills guidance — "verify references resolve".

R12: Index Completeness (WARN if drift)

# For every local file reference in AGENTS.md, check it's also documented
# in a "References" or "Index" section (if present). If no such section,
# this check is a no-op.
if grep -qE "^##[[:space:]]+(References|Index|Related)" AGENTS.md; then
  body_refs=$(grep -oE '`[a-zA-Z0-9._/-]+\.(md|ts|tsx|js|sh|json|yml|yaml)`' AGENTS.md \
    | tr -d '`' | sort -u)
  index_section=$(awk '/^##[[:space:]]+(References|Index|Related)/,/^##[^#]/' AGENTS.md)
  missing_in_index=""
  for ref in $body_refs; do
    echo "$index_section" | grep -q "$ref" || missing_in_index="$missing_in_index $ref"
  done
  [ -z "$missing_in_index" ] && echo "pass" || echo "warn: not in index:$missing_in_index"
else
  echo "skip: no References section"
fi

Why this rule: If AGENTS.md has a References section, every file mentioned in the body should be listed there. Otherwise the section is incomplete and misleading. Skip if no References section.

R13: Soft Target — 70-100 Lines (INFO)

lines=$(wc -l < AGENTS.md)
if [ "$lines" -gt 150 ]; then echo "fail (see R2)"; \
elif [ "$lines" -gt 100 ]; then echo "warn: over soft target"; \
elif [ "$lines" -lt 20 ]; then echo "info: minimal-viable file"; \
else echo "pass: in sweet spot"; fi

Source: Augment Code study (100-150 line AGENTS.md + reference docs = 10-15% improvement), aihackers.net (start with 10-15 lines, iterate), Anthropic docs (progressive disclosure), ETH Zurich (rule dropout >150 lines).

Sweet spot: 70-100 lines is the optimal range per multiple independent studies. 20-30 is too aggressive for real projects; 150+ causes rule dropout.

Workflow

Step 1: Run Validation

bash references/validate-agents-md.sh [path-to-AGENTS.md]

Default: ./AGENTS.md. Script outputs ✅/❌/⚠️/ℹ️ per rule + summary.

Step 2: Present Results

Show the user a clear report. Use this format (works in any harness):

📊 AGENTS.md Validation Report

Checking: ./AGENTS.md
  ✅ R1: File exists
  ✅ R2: 88 lines (≤150 hard limit)
  ✅ R3: Commands section
  ✅ R4: Don'ts section
  ✅ R5: No secrets
  ⚠️ R6: No exact versions — add TypeScript 5.7, Vitest 3.1.1
  ✅ R7: No arch bloat
  ✅ R8: Critical rules in first 50 lines
  ⚠️ R9: No skill references — add `my-skill-name` for extended docs
  ✅ R10: Template compliance
  ✅ R11: All internal references resolve
  ⚠️ R12: Index drift — `docs/foo.md` mentioned in body, not in References
  ✅ R13: 88 lines (in 70-100 sweet spot)

Result: 10/13 passed, 3 warnings

Step 3: Offer Corrections

Ask the user how to proceed. Phrase as plain prose — the agent decides how to surface this question based on its harness (some have ask_user_question, some use a confirmation prompt, some just describe in text).

Suggested choices:

1. Auto-fix warnings — apply recommended fixes automatically
2. Show recommendations — explain each warning with specific suggestions
3. Skip fixes — just record the report, no changes needed

Step 4: Apply Fixes (if user confirms)

For each warning/failure, generate a fix:

| Rule | Fix | |------|-----| | R2 (too long) | Trim sections, move content to docs/ | | R3 (no Commands) | Generate Commands section from package.json / Makefile | | R4 (no Don'ts) | Generate Don'ts section from common project errors | | R5 (secrets) | Immediately remove and warn | | R6 (no versions) | Detect versions from go.mod / package.json | | R7 (arch bloat) | Suggest moving to README or docs/ | | R8 (rules buried) | Reorder sections (Don'ts + Commands to top) | | R9 (no skill refs) | Add skill references for related workflows | | R10 (template) | Restructure to match template | | R11 (broken refs) | Rename or remove stale file references | | R12 (index drift) | Add missing entries to References section | | R13 (off-target) | Move content to docs/ (if over) or expand (if under) |

Step 5: Verify Fix

Re-run validation. Confirm all FAILs resolved and WARNs reduced.

Step 6: Suggest Periodic Review (always)

After validation, suggest adding a Last validated: YYYY-MM-DD line near the top of AGENTS.md. Format:

<!-- Last validated: 2026-06-04 by cali-agents-md-validator -->

The agent (or user) notices when the timestamp is stale and re-runs validation. No telemetry required — this is a passive nudge, not tracking.

Content Placement Criteria

Use this decision tree to decide where new content goes. The validator may suggest moves based on these criteria when AGENTS.md grows past 100 lines.

Is it triggered by explicit user intent ("validate", "create", "fix")?
  YES → SKILL (workflow-shaped, has trigger description)
  NO  ↓

Is it a multi-step workflow with decision points?
  YES → SKILL
  NO  ↓

Is it a rule the agent must never violate?
  YES → AGENTS.md (Don'ts section)
  NO  ↓

Is it a command the agent runs frequently?
  YES → AGENTS.md (Commands section)
  NO  ↓

Is it static knowledge consumed by skills on demand?
  YES → REFERENCE (docs/<topic>.md or skill's references/)
  NO  ↓

If unsure: start in AGENTS.md, move to reference when AGENTS.md > 100 lines,
move to skill when it becomes a multi-step workflow.

Anti-patterns to avoid

• ❌ SKILL for a single fact (use reference or AGENTS.md)
• ❌ REFERENCE for a multi-step workflow (use skill)
• ❌ AGENTS.md for project-specific domain knowledge (use reference)
• ❌ REFERENCE used only by ONE skill (could inline in the skill's body)
• ❌ SKILL with no clear trigger intent (won't get activated)

Sources: Anthropic skill authoring best practices, Microsoft Learn (adding skills), AgentSkills.io (progressive disclosure), AgentPatterns.ai (skill design patterns).

Template Reference

The canonical template structure (from cali-agents-md-generator):

# AGENTS.md
<!-- Last validated: YYYY-MM-DD by cali-agents-md-validator -->

## Project Overview
[Brief description with stack + versions]

## Commands
| Command | Description |
|---------|-------------|
| `npm test` | Run all tests |

## Architecture
[1-3 line summary — link to `docs/architecture.md` for details]

## Don'ts
- Things to never do

## References
- `docs/<topic>.md` — description
- `my-skill-name` — when to use

Edge Cases

AGENTS.md doesn't exist

Report R1 fail. Suggest running cali-agents-md-generator skill (if available in the user's ecosystem — describe in prose, do not hardcode a slash command).

File is exactly 150 lines

R2 passes, R13 may warn. No action needed for R2; consider trimming for R13.

Multiple Don'ts sections

R10 still passes (any Don'ts is fine). Suggest consolidation in the report.

Internal reference is a path with environment variable

R11 may false-positive on ${HOME}/.config/foo. Skip pattern if matches ${.

Test Cases

Should activate

• "Validate my AGENTS.md"
• "Check if agents md is good quality"
• "Audit the AGENTS.md file"
• "Is my agents md following best practices?"
• "Run the agents-md validator"
• "Check project agents doc"

Should NOT activate

• "Create AGENTS.md" (use cali-agents-md-generator)
• "Read AGENTS.md" (just reading, not validating)
• "Edit AGENTS.md" (direct edit, not validation workflow)
• "Write a CLAUDE.md" (different file, different ecosystem)

References

• references/validate-agents-md.sh — Bash validation script (R1-R13)
• cali-agents-md-generator — Canonical template + scaffolding skill
• cali-skill-validator — Validates skills themselves (covers this skill)
• agents.md — AGENTS.md standard (github.com/agentsmd/standard)
• ETH Zurich "Evaluating AGENTS.md" (2025) — Size limits, rule dropout
• Augment Code "How to write good AGENTS.md" — 100-150 line sweet spot
• aihackers.net "AGENTS.md: What Actually Works" — Minimal viable (10-15 lines)
• Anthropic "Skill authoring best practices" — Skill vs reference criteria
• Microsoft Learn "Adding Skills" — Skill packaging and reuse
• AgentSkills.io "Optimizing skill descriptions" — Description patterns