axiomantic/spellbook-auditing

Audit Spellbook

You are auditing the spellbook project itself. This skill orchestrates parallel subagents to comprehensively analyze skills, commands, docs, and prompts for optimization opportunities.

Invariant Principles

1. Parallelism maximizes audit coverage - All audit agents launch simultaneously; sequential execution wastes context
2. Token efficiency compounds - Small savings multiply across always-loaded descriptions, skill bodies, and runtime
3. CSO prevents workflow leak - Descriptions trigger only; workflow in description = Claude follows description not skill
4. Evidence over claims - Every finding requires file/line/example proof; no unsubstantiated optimization recommendations
5. Actionable over diagnostic - Report must produce implementable items with clear priority

Trigger Conditions

Use this skill when:

• User asks to "audit spellbook", "optimize skills", "review spellbook"
• Before major releases to ensure quality
• When concerned about token usage or instruction bloat
• Periodically for maintenance

Execution Flow

Phase 1: Launch Parallel Audit Subagents

Launch ALL of these subagents in a SINGLE message (parallel execution):

1. Factcheck Agent

Audit all documentation in spellbook for factual accuracy.

Files to check:
- README.md
- docs/**/*.md
- CHANGELOG.md
- Any claims in skill/command descriptions

For each claim found:
1. Identify the assertion
2. Verify against: code, external sources, logical consistency
3. Flag unverifiable or incorrect claims

Output: JSON array of {file, line, claim, status: "verified"|"unverified"|"incorrect", evidence}

2. Instruction Engineering Compliance Agent

Audit all instruction files against instruction-engineering principles.

Files: skills/*/SKILL.md, commands/*.md, AGENTS.spellbook.md

Check for:
- Clear role definition
- Explicit trigger conditions
- Structured output formats
- Edge case handling
- Appropriate use of examples (not excessive)
- Action-oriented language
- Avoidance of ambiguity

Output: JSON array of {file, issues: [{principle, violation, suggestion}], score: 0-100}

3. Description Quality Agent (CSO Compliance)

Audit skill/command descriptions for Claude Search Optimization (CSO) compliance.

IMPORTANT: Reference the writing-skills skill for authoritative CSO guidance.

For each skill and command, analyze against these principles:

1. **Trigger-Only Rule**: Description should ONLY describe when to use, NEVER summarize workflow
   - BAD: "dispatches subagent per task with code review between tasks" (workflow summary)
   - GOOD: "Use when executing implementation plans with independent tasks" (trigger only)

2. **Start with "Use when..."**: Focus on triggering conditions, symptoms, situations

3. **Include natural keywords**: What would a user say when they need this skill?
   - Include problem symptoms (race conditions, flaky tests, merge conflicts)
   - Include specific contexts (before commit, after test failure, during PR review)

4. **Avoid workflow leakage**: If description mentions steps/phases/process, Claude may
   follow the description instead of reading the full skill (documented bug!)

5. **Third person**: Descriptions are injected into system prompt

6. **Technology-agnostic unless skill is technology-specific**

7. **Under 500 characters** (max 1024 total frontmatter)

8. **Clear either/or delineation**: When multiple trigger conditions exist, use explicit
   enumeration to make each condition clearly independent:
   - BAD: "Use when writing subagent prompts or invoking Task tool or improving skills"
   - GOOD: "Use when: (1) constructing prompts for subagents, (2) invoking the Task tool, or (3) writing/improving skill instructions"

   Ambiguous "or" chains make it unclear which conditions are independent triggers vs. related concepts.

For each description, classify as:
- CSO_COMPLIANT: Follows all principles
- WORKFLOW_LEAK: Contains process/workflow that Claude might follow instead of skill
- MISSING_TRIGGERS: Too vague, missing "Use when..." or specific symptoms
- TOO_BROAD: Would trigger for unrelated tasks
- TOO_NARROW: Missing keywords users would naturally say
- AMBIGUOUS_TRIGGERS: Multiple conditions without clear enumeration (fix with numbered list)

Output: JSON array of {
  file,
  current_desc,
  cso_status: "CSO_COMPLIANT"|"WORKFLOW_LEAK"|"MISSING_TRIGGERS"|"TOO_BROAD"|"TOO_NARROW"|"AMBIGUOUS_TRIGGERS",
  issues: [list of specific violations],
  proposed_desc,
  rationale
}

4. Instruction Optimizer Agent

Deep audit of instruction content for token optimization.

For each skill/command, identify:
- Semantic overlap between sections
- Extraneous examples that could be removed
- Verbose phrasing that could be tightened
- Sections that could be collapsed/merged
- Overcomplicated workflows that could be simplified
- Repeated patterns that could be extracted

CRITICAL: Optimizations must NOT reduce intelligence or capability.
The goal is SMARTER and SMALLER, not dumber.

Output: JSON array of {file, optimizations: [{section, issue, before_tokens, after_tokens, proposed_change}], total_savings}

5. MCP Candidate Agent

Analyze tool call patterns across skills/commands for MCP extraction candidates.

Look for:
- Repeated sequences of tool calls (e.g., "read file, grep pattern, edit file")
- Common workflows that multiple skills perform
- Bash commands that could be MCP tools
- File operations that are repeated verbatim

A good MCP candidate:
- Is used 3+ times across different skills
- Has clear input/output contract
- Would save tokens by reducing instruction repetition
- Provides atomic, reusable functionality

Output: JSON array of {pattern, occurrences: [{file, context}], proposed_mcp_name, proposed_signature, token_savings_estimate}

6. YAGNI Analysis Agent

Audit spellbook for unnecessary complexity and unused features.

Check for:
- Skills that duplicate functionality
- Features that seem unused or untested
- Overly complex workflows that could be simplified
- Configuration options nobody uses
- Dead code paths in instructions
- Skills that are too narrow (could be merged)
- Skills that are too broad (should be split)

Apply the principle: "You Aren't Gonna Need It"

Output: JSON array of {item, type: "skill"|"command"|"feature", concern, recommendation, confidence: "high"|"medium"|"low"}

7. Persona Quality Agent (if fun-mode exists)

Audit persona/context/undertow lists for quality and variety.

Files: skills/fun-mode/personas.txt, contexts.txt, undertows.txt

Check for:
- Duplicates or near-duplicates
- Entries that are too similar in vibe
- Missing variety in weirdness tiers
- Entries that are too long (token waste)
- Entries that don't synthesize well together
- Quality of creative writing

Output: JSON with {personas: {count, duplicates, quality_issues}, contexts: {...}, undertows: {...}, cross_synthesis_issues}

8. Consistency Audit Agent

Audit for consistency across all skills and commands.

Check for:
- Inconsistent formatting (some use tables, some don't)
- Inconsistent terminology (same concept, different words)
- Inconsistent section structure
- Inconsistent trigger condition formats
- Inconsistent output format specifications
- Style drift between older and newer skills

Output: JSON array of {inconsistency_type, examples: [{file1, file2, difference}], suggested_standard}

9. Dependency Analysis Agent

Map dependencies between skills, commands, and MCP tools.

Build a dependency graph:
- Which skills invoke other skills?
- Which skills depend on specific MCP tools?
- Which skills have circular dependencies?
- Which skills are orphaned (nothing invokes them)?
- Which skills are over-invoked (too central, single point of failure)?

Output: JSON with {graph: {nodes, edges}, orphans, circular_deps, hotspots}

10. Test Coverage Agent

Analyze test coverage for spellbook components.

Check:
- Which MCP tools have tests?
- Which don't?
- Are there integration tests for skill workflows?
- Test quality (do tests actually verify behavior?)

Output: JSON array of {component, type, has_tests, test_quality: "good"|"weak"|"none", gaps}

11. Token Counting Agent

Measure actual token costs across all spellbook content.

For each file, calculate:
- Total tokens (words * 1.3 as estimate, or use tiktoken if available)
- Tokens by section
- Comparison to similar skills (is this one bloated?)

Produce rankings:
- Largest skills by token count
- Largest commands by token count
- Total tokens in AGENTS.spellbook.md
- Total tokens in all skill descriptions (always-loaded cost)

Output: JSON with {
  total_tokens: N,
  always_loaded_tokens: N,  // descriptions only
  deferred_tokens: N,       // skill bodies
  by_file: [{file, total, sections: [{name, tokens}]}],
  rankings: {largest_skills: [], largest_commands: []}
}

12. Conditional Extraction Agent

Find large conditional blocks that should become skills.

Scan for patterns in:
- AGENTS.spellbook.md
- commands/*.md
- Any non-skill instruction file

Look for:
- "If X, then [20+ lines of instructions]"
- "When Y happens: [large block]"
- "For Z situations: [detailed workflow]"
- Platform-specific sections (macOS/Linux/Windows)
- Language-specific sections (Python/TypeScript/etc.)

A block should become a skill if:
- It's 15+ lines
- It's conditionally triggered
- It could stand alone as a coherent workflow

Output: JSON array of {
  file,
  line_start,
  line_end,
  trigger_condition,
  block_tokens,
  proposed_skill_name,
  extraction_difficulty: "easy"|"medium"|"hard"
}

13. Tables-Over-Prose Agent

Identify prose sections that would be more token-efficient as tables.

Look for:
- Lists of "X does Y" statements
- Repeated structural patterns in prose
- Option/flag documentation
- Comparison content
- Any enumeration that follows a pattern

Calculate savings:
- Current prose token count
- Estimated table token count
- Percentage savings

Output: JSON array of {
  file,
  section,
  current_format: "prose"|"list",
  current_tokens,
  proposed_tokens,
  savings_pct,
  example_conversion
}

14. Glossary Opportunity Agent

Find repeated term definitions that could use a shared glossary.

Look for:
- Same concept explained multiple times across files
- Inline definitions ("X, which means Y")
- Repeated explanations of spellbook-specific terms
- Acronym expansions repeated

Good glossary candidates:
- Terms used in 3+ files
- Definitions that are 10+ words
- Spellbook-specific jargon

Output: JSON array of {
  term,
  occurrences: [{file, line, definition_text}],
  proposed_canonical_definition,
  token_savings_estimate
}

15. Naming Consistency Agent

Audit all skill, command, and agent names for semantic consistency.

NAMING CONVENTIONS:
| Type | Pattern | Examples |
|------|---------|----------|
| Commands | Imperative verb(-noun) | execute-plan, verify, write-plan |
| Skills | Gerund (-ing) OR Noun-phrase | debugging, test-driven-development, brainstorming |
| Agents | Noun-agent (role) | code-reviewer, fact-checker |

RATIONALE:
- Commands tell the system to DO something (imperative mood)
- Skills describe WHAT you're doing/learning (descriptive)
- Agents ARE something (role/identity)

For each skill:
- Flag if name is imperative verb pattern (should be gerund/noun)
- Examples: skills should use gerunds like "debugging", "fixing-tests", "developing", or short descriptives like "develop"

For each command:
- Flag if name is noun-phrase without action verb (should be imperative)
- Examples: commands should use imperatives like "handoff", "audit-green-mirage"

For each agent:
- Flag if name is not noun-agent pattern

Output: JSON array of {
  name,
  type: "skill"|"command"|"agent",
  current_pattern: "imperative"|"gerund"|"noun-phrase"|"noun-agent"|"ambiguous",
  expected_pattern,
  is_compliant: boolean,
  suggested_rename,
  severity: "high"|"medium"|"low"
}

16. Reference Validation Agent

Validate that all skill/command references in documentation actually exist.

Scan all files for references to skills and commands:
- Backtick references: `skill-name`, `command-name`
- Prose references: "use the X skill", "invoke X command"
- Table references: skill/command names in Helper tables

For each reference found:
1. Check if it's a skill reference - verify skills/{name}/SKILL.md exists
2. Check if it's a command reference - verify commands/{name}.md exists
3. Check for type mismatches (referencing command as skill or vice versa)

KNOWN PATTERNS TO CHECK:
- Helper Skills tables (audit-spellbook has one)
- Cross-references in skill bodies
- AGENTS.spellbook.md skill listings
- README.md feature lists

Output: JSON array of {
  file,
  line,
  reference,
  reference_type: "skill"|"command"|"ambiguous",
  exists: boolean,
  actual_type: "skill"|"command"|"none",
  type_mismatch: boolean,
  suggestion
}

17. Orphaned Docs Agent

Find documentation files without corresponding source files.

Check for orphaned docs:
- docs/skills/*.md without matching skills/*/SKILL.md
- docs/commands/*.md without matching commands/*.md

Check for missing docs:
- skills/*/SKILL.md without matching docs/skills/*.md
- commands/*.md without matching docs/commands/*.md

Note: skills/commands/agents docs are generated by pre-commit hooks.
Focus on:
1. Orphans: docs that reference deleted/renamed items
2. Missing docs: items that should have docs/ entries

Output: JSON array of {
  file,
  issue: "orphaned"|"missing_docs",
  expected_source,
  recommendation: "delete"|"create"|"rename"
}

Phase 2: Compile Report

After all agents complete, compile results into a unified report:

# Spellbook Audit Report
Generated: [timestamp]

## Executive Summary
- Total token savings opportunity: X tokens (~Y%)
- Critical issues: N
- Optimization opportunities: M
- MCP candidates: K

## Factcheck Results
[summary + critical issues]

## Instruction Engineering Compliance
[summary + worst offenders]

## Description Optimization
[table of proposed changes with savings]

## Instruction Optimization
[grouped by file, sorted by savings potential]

## MCP Candidates
[prioritized list with implementation notes]

## YAGNI Analysis
[recommendations sorted by confidence]

## Persona Quality
[if applicable]

## Consistency Issues
[grouped by type]

## Dependency Analysis
[graph summary, orphans, hotspots]

## Test Coverage
[gaps and recommendations]

## Token Analysis
[total costs, rankings, always-loaded vs deferred breakdown]

## Conditional Extraction Candidates
[blocks that should become skills, sorted by token savings]

## Tables-Over-Prose Opportunities
[sections to convert, with example conversions]

## Glossary Candidates
[terms to define once, with occurrence counts]

## Naming Consistency
[skills/commands/agents with non-compliant names]

## Reference Validation
[broken or mistyped skill/command references]

## Orphaned Documentation
[docs without corresponding source files]

## Actionable Items
1. [High priority items]
2. [Medium priority items]
3. [Low priority items]

Save report to: ~/.local/spellbook/docs/<project-encoded>/audits/spellbook-audit-[timestamp].md

Phase 3: Implementation Prompt

After presenting the report summary, ask the user:

The audit identified [N] actionable items with potential savings of ~[X] tokens.

How would you like to proceed?
1. Implement high-priority items now
2. Implement all items
3. Review report first, decide later
4. Skip implementation

Use AskUserQuestion tool with these options.

If user chooses implementation:

1. Use writing-plans skill to create implementation plan
2. Ask any clarifying questions upfront using AskUserQuestion
3. Execute plan using appropriate skills/subagents

Helper Skills and Commands

When implementing fixes, these can be invoked:

| Name | Type | Use For | |------|------|---------| | writing-skills | skill | AUTHORITATIVE guide for skill structure, CSO, and description writing | | instruction-engineering | skill | Restructuring poorly-organized instructions | | optimizing-instructions | skill | Compressing verbose instructions | | writing-plans | skill | Creating implementation plans | | fact-checking | skill | Deep-diving on specific claims | | finding-dead-code | skill | Identifying unused code in MCP tools | | auditing-green-mirage | skill | Auditing test quality | | /simplify | command | Simplifying overcomplicated workflows |

Naming Convention Reference

| Type | Pattern | Examples | |------|---------|----------| | Commands | Imperative verb(-noun) | execute-plan, verify, handoff | | Skills | Gerund/Noun-phrase | debugging, test-driven-development | | Agents | Noun-agent (role) | code-reviewer, fact-checker |

Notes

• All subagents run in PARALLEL for speed
• Each agent should be thorough but focused on its specific concern
• Token estimates can be approximate (count words * 1.3)
• When in doubt, flag for human review rather than making assumptions
• The report should be actionable, not just diagnostic
• Run this audit before major releases
• Consider running monthly for maintenance

Critical: Claude Search Optimization (CSO)

When auditing or fixing descriptions, follow CSO principles from the writing-skills skill:

The Workflow Leak Bug: If a description summarizes workflow (steps, phases, process), Claude may follow the description instead of reading the full skill content. This is a documented bug that caused real failures (e.g., "code review between tasks" in description caused ONE review instead of the TWO specified in the actual skill).

Description Formula:

"Use when [triggering conditions/symptoms/situations]"

NOT:

"Use when X - does Y then Z then W"  # Workflow leak!

Verification: After fixing descriptions, test that Claude actually reads the full skill content rather than just following the description.