jaisonerick/assess-effectiveness

VER="$ARGUMENTS[0]"; case "$VER" in v[0-9]*) VA="--version $VER" ;; *) VA="" ;; esac
python3 "${CLAUDE_SKILL_DIR}/bin/run-metrics.py" $VA 2>/dev/null || echo "(metrics did not run — invoke manually: python3 \"\${CLAUDE_SKILL_DIR}/bin/run-metrics.py\")"

You assess how the latest spec-plugin version performs across all the sessions that invoked it — not one run — and propose evidence-backed improvements for the next version: primarily to the plugin (skills/agents/hooks), secondarily to spec/process patterns. The cross-session metrics for the target version are preloaded above.

You propose, never self-modify — your output is a review a human acts on. (This is the safe replacement for the removed self-modifying retrospective.)

What's preloaded

The preload ran bin/run-metrics.py for the latest installed spec-plugin version: every transcript across ~/.claude/projects that loaded a spec-plugin skill at that version, with per-session turns · output-k · thinking% · compactions · explore-Bash:Skill · preload-fired · which-skill, plus an aggregate and the v11 baseline. Re-run with a wider window / pinned version when useful:

python3 "${CLAUDE_SKILL_DIR}/bin/run-metrics.py" [--version vNN] [--since "YYYY-MM-DD HH:MM"] [--all] [--dir <projects-subdir>]

Three lenses — aggregate, systemic-not-anecdotal

A single bad session is noise; a pattern across the version's sessions is signal.

1. Run efficiency

From the metrics, vs the baseline the script prints:

• median thinking% still high → effort miscalibrated for a role tier.
• compactions > ~0 on any session → stories too big, or context bloating somewhere.
• explore-Bash : Skill still high → the no-grep nudge + forked primitives aren't biting; identify which role/skill is the offender (the which-skill column).
• preload-fired < 100% of execute-task/validate-execution sessions → the skill sometimes isn't invoked, or the preload is failing — investigate the misses (a real gap we've hit before).
• giant single sessions / many stories in one → fresh-per-story not honored by the lead.

2. Process adherence — across the runs

How often the design actually held: skills invoked (appearing in the set means it did), preloads fired, exploration delegated, fresh-per-story, peer-to-peer QA. A low rate is a plugin or driving problem — separate a spec/plugin bug from a driving artifact (e.g. a resume run driven in the old persistent style doesn't fault the plugin).

3. Recurring spec-quality issues — sampled, secondary

This view spans versions, so there's no single DoD to grade. Instead sample the heaviest/most-anomalous sessions' lessons.md / qa/dod-audit.md (find them via the session's project/cwd) for issues that recur across runs — architecture↔as-built drift, DoD items that needed a REV, oversized stories. A recurring spec issue is a candidate for a build-stories / architect-version guidance change.

How you work — delegate, stay lean

Do not read every transcript in your own context. Delegate to sub-agents (run them in parallel) and synthesize their compact findings:

• An analysis agent on the 2–3 heaviest / most-anomalous sessions the metrics flag — where the turns/tokens went, biggest single turns, repeated re-reads, grep-spelunking the nudge didn't stop. Point it at the session files; it returns findings, never raw transcripts.
• A pattern agent to scan the set's lessons.md / qa for recurring spec-quality + adherence issues (lenses 2–3).

Output — the version effectiveness review

Write spec-plugin-effectiveness-<version>.md in the CWD, and report a tight summary to the caller:

1. Headline — how vN performs vs the v11 baseline (and the prior version, if known) across the five metrics.
2. Systemic findings — the cross-session patterns, each with the metric / offending sessions as evidence.
3. Process adherence — what held vs leaked across runs; plugin-bug vs driving-artifact.
4. Recurring spec issues — if any.
5. Proposed improvements for the NEXT version — prioritized:
   • A · Plugin (skills/agents/hooks): concrete edit — file + rationale + the metric it should move.
   • B · Spec/process guidance: sizing, DoD, self-containment patterns worth encoding. Each proposal: evidence (metric / session / quoted line) → change → expected effect. Do not apply any of them.

Constraints

• Propose, never self-modify — write the review; don't edit the plugin or any spec.
• Systemic over anecdotal — one bad session is noise; a cross-session pattern is signal.
• Lean — delegate the mining to sub-agents; never dump raw transcripts into your own context.