a-green-hand-jack/research-results-auditor

Research Results Auditor

Produce a structured validity verdict for completed experiment results before they become paper claims. This skill asks: is the evidence scientifically defensible as stated?

Use this skill when:

• experiments are complete and results are about to be written into the paper as claims
• a result looks convincing but the connection between the measurement and the stated conclusion feels stretched
• a reviewer is likely to ask "what else could explain this?"
• an ablation result is used to attribute a performance gain to a specific component
• class imbalance, calibration, or threshold choice may be driving the metric
• the gap between what was measured and what is claimed in the paper needs to be checked before submission

Do not use this skill when experiments are still running or results are not yet final — use result-diagnosis for in-progress diagnosis and next-action decisions. Do not use this skill for test selection or variance reporting — use statistical-analysis-planner.

Pair this skill with:

• result-diagnosis upstream: resolve engineering and scientific issues before auditing validity
• statistical-analysis-planner upstream: ensure variance is characterized before the audit
• paper-evidence-board downstream: update evidence slots with the validity verdict
• paper-writing-contract-planner downstream: narrow or forbid claims that fail the audit
• experiment-design-planner when the audit identifies a controlled experiment needed to rule out a confound

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
└── references/
    └── audit-criteria.md

Progressive Loading

• Always read references/audit-criteria.md.
• Read memory/claim-board.md and memory/evidence-board.md before auditing.
• Read paper/.agent/paper-evidence-board.md when claims have paper-facing evidence slots.

Core Principles

• Separate data observations from author interpretation before auditing.
• A claim is only as strong as its weakest alternative explanation.
• Ablation results prove component contribution only when all other factors are controlled.
• A benchmark metric is not the same as the real-world property being claimed.
• Confidence should decrease as the distance between measurement and claim increases.
• A failed audit should result in a narrowed claim, not silence.

Step 1 — Reconstruct the Claim

For each result to audit, write in one sentence:

Claim: [method/component X] causes [effect Y] on [task/benchmark Z], as shown by [metric M].

Then identify:

• the artifact(s) that support it: experiment reports, tables, figures, logs
• what the metric M actually measures vs what Y asserts
• any gap between M and Y (e.g. M = accuracy on a benchmark, Y = "robustness to distribution shift")

If the claim cannot be written in one sentence without hedges, that is an early signal of drift.

Step 2 — Protocol Integrity Check

Read references/audit-criteria.md.

For each claim, verify:

• Metric–task alignment: does the metric directly measure the claimed property, or is it a proxy?
• Comparable conditions: same data split, same compute budget, same preprocessing for all compared methods?
• Ablation isolation: when attributing a gain to component X, is exactly one variable changed?
• Evaluation coverage: is the test set representative of the claim's stated scope?
• Reproducibility: is the result averaged over enough seeds to be stable? (use statistical-analysis-planner if not)

Flag each violation as:

• blocker: the claim cannot be made as stated without fixing this
• risk: the claim is defensible but a reviewer will likely raise it
• note: disclose in limitations or experiment settings

Step 3 — Confound Audit

For each claim, list at least two alternative explanations and classify:

Alternative: [alternative explanation]
Likelihood: high / medium / low
Ruling-out evidence: [experiment, analysis, or prior work that rules it out]
Status: ruled-out / possible / unaddressed

Common confounds in ML:

• Scale confound: larger model / more parameters / more compute explains the gain
• Data confound: the method was tuned on data that overlaps with test distribution
• Hyperparameter confound: baseline was not fairly tuned; proposed method was
• Metric confound: metric rewards a property (e.g. output length, confidence calibration) that correlates with but is not the claimed property
• Selection confound: result is reported only for seeds or checkpoints where it worked

Any unaddressed confound with high or medium likelihood is an audit blocker.

Step 4 — Claim-Drift Check

Compare the measured quantity to the stated conclusion. Flag drift if:

• the paper uses language that implies a broader scope than the benchmark covers (e.g. "general reasoning" from a single benchmark)
• a controlled lab result is described as if it transfers to deployment conditions
• a relative improvement % is used to support an absolute capability claim
• performance on a specific demographic or domain is generalized to all users
• a correlation is described as causation without a controlled intervention

For each drift finding, record:

• the measured quantity
• the stated conclusion
• the drift type: scope / generalization / causation / magnitude
• the maximum defensible claim given only the evidence

Step 5 — Inferential Quality Check

Check that comparative claims meet minimum statistical support:

• Is there a reported uncertainty interval (std, CI, or range) for comparative claims?
• If multiple comparisons are made, has multiple-comparison correction been applied?
• Does class imbalance in the test set affect the metric? (accuracy on imbalanced data overstates performance on the minority class)
• Is calibration relevant? (for claims about probability outputs or confidence-weighted metrics)
• Does threshold choice affect the reported metric? (precision/recall/F1 are threshold-sensitive)

Any comparative claim without reported uncertainty is classified as a risk.

Step 6 — Write the Validity Verdict

For each audited claim, produce:

Claim: [one-sentence claim]
Protocol verdict: pass | conditional | fail
Confound verdict: clean | possible-[confound-name] | unaddressed-[confound-name]
Drift verdict: none | scoped-[drift-type]
Inferential verdict: supported | conditional | unsupported
Overall verdict: valid | narrowed | blocked | insufficient-evidence

Recommended wording:
  Allowed: [maximum defensible phrasing]
  Forbidden: [specific language that overclaims]

Required actions:
  - [action to resolve each blocker or risk]

Save the audit report to paper/.agent/results-audit-<date>.md.

Step 7 — Memory Writeback

• Update memory/claim-board.md: narrow or strengthen claims based on verdicts
• Update memory/risk-board.md: add confound and drift risks with severity
• Update memory/action-board.md: add required actions for each blocker
• Update paper/.agent/paper-evidence-board.md: annotate evidence slots with validity verdict

Final Sanity Check

Before finishing:

• every main claim has a one-sentence reconstruction
• alternative explanations are listed and classified for each claim
• every blocker has an action
• the audit report is saved with date
• allowed and forbidden claim wordings are explicit for each result