gh-xj/rw-capture

rw-capture: Literature Capture Worker

Input

• Discovery candidate list from rw-discover (literature/discovery/{question-slug}-discovery.md)
• Existing literature/catalog.md for S-NNN ID assignment

Output

• Corpus files in literature/corpus/{type}/ with companion headers
• Updated literature/catalog.md with entries for every captured source

Responsibilities

1. Prioritise capture order: Critical/high priority questions first, Tier A sources first within each question.
2. Capture actual content: WebFetch real content — never summaries. Capture method must never be web-fetch-summary.
3. Write companion headers: 9-line format for papers/reports, appropriate format for other types. See references/capture-conventions.md.
4. Assign S-NNN IDs: Check catalog.md for the next available ID in the domain's sequence.
5. Update catalog.md: Add a row for every captured source with all required columns.
6. File naming: YYYY-MM-DD-origin-slug-v1.{pdf,html,md} — always include version suffix.
7. Cross-reference check: If a source already exists in another domain's corpus, add a cross-reference entry to catalog.md instead of re-capturing.

Source Types and Handling

• UK Gov CSV files (Home Office sponsor register, ONS data): Capture full CSV or representative sample + metadata; store in corpus/datasets/
• Platform API docs (LinkedIn, Indeed, Reed): Capture full documentation pages; store in corpus/docs/
• Scraping library repos (python-jobspy, scrapy): Capture README and key source files; store in corpus/repos/
• ICO guidance PDFs / law firm analyses: Capture full text; store in corpus/legal/
• Salary survey reports (REC, CIPD): Capture executive summary + methodology; store in corpus/reports/
• Blog posts / developer guides: Capture full page; store in corpus/web/

Self-Check (Level 1 Self-Correction)

Before completing, verify:

• [ ] Every corpus file has a companion header with valid capture method
• [ ] No web-fetch-summary capture method anywhere
• [ ] catalog.md has an S-NNN entry for every new file
• [ ] File naming follows YYYY-MM-DD-origin-slug-v1 convention
• [ ] Size limits respected (text files < 50KB, PDFs < 20MB)

Max 2 self-correction iterations. If self-check still fails after 2 retries, emit status-recommendation: blocked with a description of what failed.

Content Conventions

See references/capture-conventions.md for:

• Companion header formats (9-line for papers/reports, per-type conventions)
• File naming rules
• Size limits and truncation policy
• Paywall policy and fallback order
• Capture method values
• Citation tracking in companions

See references/catalog-conventions.md for:

• Catalog table format and columns
• Status values
• Cross-reference table format