pantheon-org/triage-tool

Triage Tool

Add a new tool or library to the research repo as a structured reference summary.

When to Use

• User provides a GitHub URL, npm/PyPI package name, or tool name
• User says "triage this tool", "add this tool", "analyse this tool/library"
• Evaluating whether a tool belongs in the repo

When Not to Use

• The tool has already been triaged (check REVIEWED.md first)
• The tool is clearly out of scope (not relevant to the research domain)
• User wants a full deep-dive analysis — use triage-tool first, then promote to ANALYSIS-<name>.md

Mindset

Triage is a quality gate, not a catalogue entry. The goal is a reproducible, honest assessment.

1. Scope before summary: confirm the tool is relevant to the research domain before investing in a full write-up. A well-reasoned out-of-scope ruling is as valuable as a full summary.
2. Evidence first: quote README and docs; never infer capabilities not described by the author.
3. Triage-then-promote: every tool enters via REVIEWED.md; promotion to ANALYSIS-*.md and vendoring both require explicit user confirmation — never automatic.

Workflow

1. Resolve the source

• Fetch the repo README and any linked docs/architecture pages.
• Record: tool name, author/org, license, language, latest version/commit, primary URL.
• Derive a stable slug: always <author-or-org>-<tool-name> — no exceptions (e.g. versatly-clawvault, press-longchat, rtk-ai-rtk, safishamsi-graphify).

2. Check for duplicates

• Search REVIEWED.md and references/REFERENCE_INDEX.md for the slug or repo URL.
• If already present, report it and stop.

3. Classify the tool

Assign one or more tags from the controlled list. See classification guide for the full taxonomy and scope assessment criteria.

4. Assess scope fit

Confirm the tool is relevant to the research domain. See classification guide for in-scope / borderline / out-of-scope criteria. Note any borderline ruling in REVIEWED.md.

5. Fill in the reference summary

Read assets/templates/REFERENCE-tool.yaml to get the required frontmatter fields and section structure. Create references/<slug>.md with a YAML frontmatter block (all required_fields from the template) followed by the required sections.

• TL;DR: 3–8 bullets capturing what the tool does and why it matters.
• What's novel: one paragraph — what does this do that adjacent tools do not?
• Architecture overview: describe the core design and how the tool works. Skip sections that are not applicable (mark N/A).
• Deployment model: runtime, language, dependencies, storage.
• Self-reported metrics: quote numbers with source — always mark (as reported).
• Open questions: gaps, risks, unverified claims, missing benchmarks.

Keep language precise. Do not pad. Mark all unverified claims.

6. Decide on vendoring

Ask whether to vendor the repo. Clone as a git submodule into tools/<repo-name>/ for tools warranting code-level inspection, or link-only for large or peripheral tools. Do not vendor without confirmation. When vendored, set local_clone in both the reference and analysis frontmatter to the submodule path relative to the document (e.g. relative to references/ or analysis/, it will be one level up then into tools/).

7. Update REVIEWED.md

Add a row to the summary table at the top (reverse-chronological):

| <today's date> | <slug> | tool | pending | <one-line description> |

8. Update REFERENCE_INDEX.md

Add a row under the most relevant category table.

9. Report and offer next step

Summarise what was created. Ask the user whether to promote to ANALYSIS-<slug>.md, vendor into tools/, keep as pending, or skip with reasoning.

10. If user confirms promotion

Read assets/templates/ANALYSIS-tool.yaml to get the required frontmatter fields and section structure. Create analysis/ANALYSIS-<slug>.md with a YAML frontmatter block (all required_fields from the template) followed by the required stage sections. Update the disposition in REVIEWED.md from pending to analysis.

10b. Create repro file

After creating ANALYSIS-<slug>.md, also create benchmarks/sources/<slug>-repro.md using the REPRO-tool template (assets/templates/REPRO-tool.yaml). Locate the benchmark harness in the vendored source by checking common directories in order: benchmarks/, eval/, evals/, scripts/. Set harness_present: true and harness_path to the relative path within the vendored repo if found, otherwise set harness_present: false and harness_path: null. Set outcome to one of: verified, partially verified, attempted — inconclusive, repro guide (not run), or stub — no harness found.

Quick Commands

# Check for duplicate before starting
grep -i "<slug-or-repo-url>" REVIEWED.md references/REFERENCE_INDEX.md

# Fetch tool README
curl -s "https://raw.githubusercontent.com/<owner>/<repo>/main/README.md"

# Create reference file from template
cp templates/REFERENCE-tool.md references/<slug>.md

# Validate the completed file
./scripts/validate-reference-tool.sh references/<slug>.md
./scripts/validate-analysis-tool.sh ANALYSIS-<slug>.md

# Vendor as submodule (only after explicit user confirmation)
git submodule add <repo-url> tools/<repo-name>

# | YYYY-MM-DD | <slug> | tool | pending | <one-line description> |

Anti-Patterns

NEVER omit YAML frontmatter

WHY: Files without frontmatter fail schema validation and break indexing tools that rely on structured metadata.

BAD Start the file with # ANALYSIS: <slug> followed by bold-text fields. → GOOD Open with --- YAML frontmatter block containing all required fields before any prose.

NEVER treat self-reported benchmarks as verified

WHY: Tool benchmarks are often run on curated inputs under favourable conditions.

BAD "Achieves 40% latency reduction." → GOOD "Reports 40% latency reduction (as reported, README)."

NEVER skip the duplicate check

WHY: Re-triaging creates conflicting entries and wastes effort.

BAD Create new file without checking REVIEWED.md. → GOOD Run grep -i "<slug>" REVIEWED.md references/REFERENCE_INDEX.md first.

NEVER vendor without user confirmation

WHY: Vendoring changes git state and submodule config — a deliberate action, not automatic.

BAD Run git submodule add during triage. → GOOD Offer vendoring as a follow-up after the summary is written.

NEVER promote without user confirmation

WHY: Promotion to ANALYSIS-*.md is a quality gate, not automatic.

BAD Create ANALYSIS-*.md as part of triage. → GOOD Triage to REVIEWED.md, then ask the user.

References

• classification guide — tag taxonomy and scope assessment criteria (in/borderline/out); customise for your research domain
• Reference artifacts: YAML template · schema · validator
• Analysis artifacts: YAML template · schema · validator
• Repro artifacts: YAML template · schema