jmagly/research-quality-audit

Research Quality Audit

Audit the research corpus for shallow stubs, incomplete documentation, and missing source files. Detects analysis docs written from abstracts alone (the root cause of the 88-stub incident) and reports doc depth metrics across the corpus.

Triggers

• "audit research quality"
• "check for stubs"
• "find shallow docs"
• "research quality audit"
• "how deep are the analysis docs?"
• /research-quality-audit

Parameters

--range REF-XXX:YYY (optional)

Audit a specific range of REF identifiers. Default: entire corpus.

--fix (optional)

Auto-dispatch expansion agents to deepen stubs. Each stub gets a focused agent that reads the full PDF/source and rewrites the analysis doc.

--threshold N (optional)

Minimum line count for a doc to be considered non-stub. Default: 80.

--format (optional)

Output format: full (default), summary, or json.

--pdf-check (optional)

Also verify that each REF has an actual PDF or source file, not just metadata.

Execution Flow

Phase 1: Corpus Scan

1. Glob all finding docs: .aiwg/research/findings/REF-*.md (and/or documentation/references/REF-*.md depending on corpus layout)
2. For each doc, collect:
   • Line count (total lines)
   • Content lines (non-empty, non-frontmatter, non-heading lines)
   • Section count (number of ## headings)
   • Key quote count (blockquotes or inline quotes)
   • Source availability — does the PDF exist at the referenced pdf_location?
   • Full text available — does sources/text/REF-XXX.txt exist?
   • Frontmatter completeness — required fields present?

Phase 2: Classification

Classify each doc into quality tiers:

| Tier | Content Lines | Sections | Quotes | Verdict | |------|-------------|----------|--------|---------| | Full | >= 150 | >= 8 | >= 3 | Comprehensive analysis | | Adequate | 80-149 | >= 5 | >= 1 | Meets minimum depth | | Stub | 40-79 | >= 3 | 0 | Written from abstract — needs expansion | | Skeleton | < 40 | any | 0 | Placeholder only — needs full rewrite |

Additional flags:

• No PDF: analysis exists but source PDF is missing
• No full text: PDF exists but text extraction was not run
• Abstract-only indicators: doc mentions "abstract" but no methodology/results sections

Phase 3: Report

Research Quality Audit
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Corpus: 372 documents
Threshold: 80 content lines

Quality Distribution:
  Full (150+):      124 (33%)  ████████████████░░░░░░░░░░
  Adequate (80-149): 89 (24%)  ████████████░░░░░░░░░░░░░░
  Stub (40-79):      98 (26%)  █████████████░░░░░░░░░░░░░
  Skeleton (<40):    61 (16%)  ████████░░░░░░░░░░░░░░░░░░

Statistics:
  Mean content lines:  112
  Median:              94
  Min:                 12 (REF-299)
  Max:                 591 (REF-018)

Source Availability:
  PDF present:         348 / 372 (94%)
  Full text extracted:  201 / 372 (54%)
  Missing PDF:          24 papers
  Missing text:        171 papers

Stubs Requiring Expansion (159):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  REF-253  22 lines  skeleton  No PDF    "Agentic Design Patterns"
  REF-254  35 lines  skeleton  Has PDF   "Multi-Agent Debate"
  REF-255  45 lines  stub      Has PDF   "Language Agent Tree Search"
  REF-256  48 lines  stub      No text   "ReAct: Synergizing Reasoning"
  ...

Top 10 Shallowest (candidates for immediate expansion):
  1. REF-299  12 lines  skeleton  "Toolformer: Language Models Can..."
  2. REF-312  15 lines  skeleton  "WebArena: A Realistic Web..."
  3. REF-253  22 lines  skeleton  "Agentic Design Patterns..."
  ...

Phase 4: Auto-Fix (if --fix)

When --fix is specified:

1. Filter fixable stubs — only expand docs that have a PDF or full text available
2. Batch by priority — shallowest docs first, batch into groups of 10
3. Dispatch expansion agents — each agent:
   • Reads the full PDF/extracted text for the source
   • Rewrites the analysis doc with comprehensive content
   • Target: 150+ content lines with methodology, findings, limitations, key quotes
4. Re-audit after expansion — run Phase 1-3 again to verify improvements
5. Report — docs expanded, mean line improvement, remaining stubs

Auto-Fix Results:
  Dispatched: 10 expansion agents (batch 1 of 16)
  Expanded: 10 / 10
  Mean improvement: 77 → 161 lines (+109%)
  Remaining stubs: 149

  Run again with --fix to process next batch.

Integration Points

| Component | Relationship | |-----------|-------------| | induct-research | Quality audit should auto-run after batch induction | | corpus-snapshot | Gates on stub rate > 10% (#814) | | research-lint | ref-frontmatter rule catches incomplete metadata; quality-audit catches shallow content | | research-status | Doc depth is a component of corpus health scoring | | research-acquire | For stubs with missing PDFs, triggers acquisition before expansion |

Distinction from Other Tools

| Tool | What it checks | |------|---------------| | research-lint | Structural — frontmatter fields, naming, references resolve | | research-quality-audit | Depth — is the content substantive? Was the source actually read? | | research-quality | Evidence — GRADE assessment of the source's research quality | | corpus-health | Aggregate — overall corpus metrics including depth, structure, coverage |

Examples

# Full corpus audit
/research-quality-audit

# Audit specific range
/research-quality-audit --range REF-253:372

# Auto-expand stubs (batch of 10)
/research-quality-audit --fix

# Strict threshold (120 lines minimum)
/research-quality-audit --threshold 120

# Check source file availability
/research-quality-audit --pdf-check

# JSON for programmatic use
/research-quality-audit --format json

References

• @$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/induct-research/SKILL.md — Source of stubs when acquisition is skipped
• @$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-acquire/SKILL.md — Acquires PDFs for stub expansion
• @$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-lint/SKILL.md — Structural validation (complementary)
• @$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-quality/SKILL.md — GRADE evidence assessment (complementary)
• @$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-status/SKILL.md — Health scoring includes depth metrics