oduffy-delphi/plan-delivery-audit

Plan-Delivery Audit

Spec backlink: archive/specs/2026-05/2026-05-28-archive-aware-review-oracle-and-audit-skill.md § Chunk C5. Origin: Distilled from the 2026-05-27 example-game-repo audit (X:\example-game-workbench-repo\tasks\recovery\2026-05-27-plan-delivery-audit.md) where a live-only review-trail read produced a false "most work unreviewed" alarm — 22 archived records had been moved by /workweek-complete Step 13 and were invisible to the live glob.

When to invoke

• After any session crash or partition that puts shipping state in doubt.
• Mid-quarter "what shipped this month?" reconciliation.
• Post-merge audit when a chain of handoffs lost a clean stopping point.
• PM question: "Did we actually finish what we think we finished?"
• Any time a plan's status: implemented self-assertion needs independent verification.

Out-of-scope sidecars

Sidecar files generated by the coordinator review pipeline inherit their parent plan's status: frontmatter and must be excluded from the plan set being audited. Exclude any file matching:

• *.prior-art-check.md
• *.coverage-check.md
• *.docs-check.md
• *.review-the Staff Engineer.md
• *.review-sid.md
• *.review-the Data Science Reviewer.md
• Any file whose basename contains .review- or .check.

Apply the exclusion at glob time. Default glob: docs/plans/*.md minus the patterns above. The example-game-repo audit (2026-05-27) confirmed this exclusion is load-bearing — sidecars were the source of misleading status: reviewed entries in the candidate set before filtering.

The three oracles (read each independently)

Read each oracle independently before comparing. Oracle 1 is the thing under test; Oracles 2 and 3 are the falsifiers. Never let Oracle 1's self-assertion influence your Oracle 2 or 3 read.

Oracle 1 — Plan-claim

The plan's own frontmatter and AC status column. Read:

• status: field (e.g. implemented, in-progress, draft, superseded, abandoned)
• Any per-AC Status column entries in the plan body
• Execution notes, reviewed: fields

Treat these as the plan's claim, not as ground truth. Oracle 1 is the hypothesis; Oracles 2 and 3 either confirm or falsify it. A plan confidently self-reporting status: implemented with all ACs green is a starting point, not a verdict.

Oracle 2 — Code-reality

Does the code at HEAD actually deliver what the plan claims? Two sub-checks, both required:

(a) Cited-artifact existence. For each file, path, symbol, or structural element the plan claims it created or changed (read from the plan's scope section, chunk list, and AC criteria prose), confirm it exists at HEAD in the claimed shape:

• Named files / paths → ls <path> or test -f <path> on disk
• Named symbols or function signatures → grep -n '<symbol>' <file> on disk
• Named schema fields, frontmatter keys, config entries → grep the owning file

Any claimed artifact absent or wrong-shaped at HEAD = PARTIAL, regardless of what Oracle 1 asserts. If the plan names no concrete artifacts to verify (purely narrative ACs with no file/path/symbol claims), note that Oracle 2 is unverifiable and treat the plan conservatively.

(b) Project test suite. Run the project's configured fast-test/validate command at HEAD (the coordinator:validate skill / fast_test_cmd in the project config). A failing test suite is independent falsifying evidence even when artifact checks pass.

Dispatch shape: For audits covering ≥4 plans, dispatch parallel read-only Sonnet scouts (one per plan). Each scout reads the plan's scope/chunks/AC prose, extracts the concrete artifact claims, runs existence/grep checks for each, and reports PRESENT / ABSENT / UNVERIFIABLE per artifact. EM runs the project test suite once (not per-plan) and applies the result to all implemented plans in the batch. Do NOT modify files, commit, or push during Oracle 2 runs.

Oracle 3 — Review (archive-aware)

Does an independent review-trail record cover the plan's delivery commits?

A commit C is covered by trail record [A..B] if and only if:

  git merge-base --is-ancestor C B   # must succeed (exit 0): C is within the reviewed window (at or before B)
! git merge-base --is-ancestor C A   # must succeed (exit 0): C is NOT before the window start (after A, exclusive)

Both clauses are required: is-ancestor C B AND ! is-ancestor C A. The negative clause prevents a review record from absorbing commits that predate the reviewed window — without it, any record with a sufficiently old start SHA would appear to "cover" any commit in the repo. The ! negation in front of the second git merge-base is load-bearing — omitting it inverts the test from "after the window start" to "before the window start" and silently breaks the audit.

Trail records live at BOTH state/review-trail/**/*.json AND archive/review-trail/**/*.json. Read via:

list-review-trail-records.sh

DO NOT glob state/review-trail/ alone. The /workweek-complete Step 13 archival moves all current-week records into archive/review-trail/<week-starting>/ on every weekly reset. A live-only read systematically under-counts coverage for anything older than the current week. This is the exact failure mode that prompted this skill — the 2026-05-27 example-game-repo audit found 22 archived records invisible in the live dir, producing a false "most work unreviewed" alarm. See docs/wiki/workstream-complete-review.md § Archive-Aware Glob.

For each trail record returned by list-review-trail-records.sh:

1. Read the record's sha_range field (format: "<start>..<end>")
2. For each of the plan's delivery commits C: run the two-clause ancestry test above
3. If any record covers all delivery commits, Oracle 3 = COVERED; else UNCOVERED

If the plan's delivery commits cannot be identified from frontmatter or execution notes, search git log --oneline for commits that touch files the plan's ACs describe.

Bucket decision tree

Apply this decision tree to each plan after running all three oracles. Every plan resolves into exactly one bucket — no plan should fall into two.

| Oracle 1 (claim) | Oracle 2 (code) | Oracle 3 (review) | Bucket | |------------------|-----------------|-------------------|--------| | status: implemented (or shipped) | Cited artifacts exist at HEAD AND project tests green | ≥1 trail record covers delivery commits | DELIVERED+REVIEWED | | status: implemented (or shipped) | Cited artifacts exist at HEAD AND project tests green | No trail record covers delivery commits | DELIVERED-UNREVIEWED | | status: implemented (or shipped) | ≥1 cited artifact absent/wrong at HEAD, OR project tests red, OR no cited artifacts to verify (claim unverifiable) | (any) | PARTIAL | | status: in-progress / draft / reviewed (not yet implemented) | (any) | (any) | IN-FLIGHT | | status: superseded / abandoned / cancelled | (any) | (any) | ABANDONED |

Tie-breaking rules:

• A plan with status: implemented but unverifiable Oracle 2 (no concrete artifact claims to check, and no project test suite configured) goes into PARTIAL, not DELIVERED. Self-assertion without any independently checkable evidence is not delivery.
• A plan with status: draft AND commit evidence of substantial shipped work still goes into IN-FLIGHT — the correct response is to flip the frontmatter, not to reclassify here.
• DELIVERED-UNREVIEWED is a real state, not an error. The correct follow-up is to dispatch code-reviewer against the delivery diff and record the trail entry. Do not skip the review or backdoor a "reviewed" claim without running the actual reviewer.
• ABANDONED plans found with Oracle 2 evidence of shipped code should surface as a concern in the audit output — the plan may need a frontmatter flip to superseded with a superseded_by: pointer.

Output format

Emit a markdown table — one row per audited plan:

| Plan | Oracle 1 (claim) | Oracle 2 (code) | Oracle 3 (review) | Bucket |
|------|-----------------|-----------------|-------------------|--------|
| `docs/plans/YYYY-MM-DD-name.md` | `status: X`; artifact count | N/M artifacts present; missing if any; tests green/red | trail record `A..B` covers commits / UNCOVERED | **BUCKET** |

Follow the table with:

1. Summary — count per bucket.
2. Catch-up review queue — list every DELIVERED-UNREVIEWED plan with its delivery commits.
3. Frontmatter flip recommendations — list any ABANDONED plans with stale status: draft.
4. Method notes — flag any plan with unverifiable Oracle 2 (no concrete artifact claims to check) or missing delivery commits.

Save to state/audits/YYYY-MM-DD-<SID_SHORT>-plan-delivery-audit.md, where SID_SHORT is the first 8 characters of the session id resolved via cs_resolve_session_id (4-tier: COORDINATOR_SESSION_ID → CLAUDE_SESSION_ID → CLAUDE_CODE_SESSION_ID → sentinel), e.g.:

SID="$(cs_resolve_session_id)"; SID="${SID:-unknown}"
SID_SHORT="${SID:0:8}"

If cs_resolve_session_id is unavailable in the current shell, fall back to a short nonce (e.g. $(date +%s%N 2>/dev/null || date +%s) truncated to 8 characters) rather than omitting the key entirely. The session/nonce key exists so two plan-delivery audits run the same day do not silently clobber each other's output. Create the state/audits/ directory if absent.

Worked example — 2026-05-27 example-game-repo audit

the Staff Engineer F3 falsifiability hook. The rows below are drawn directly from the example-game-repo audit at X:\example-game-workbench-repo\tasks\recovery\2026-05-27-plan-delivery-audit.md. A reviewer can walk the decision tree against each row independently and verify the bucket assignment. Plans are in the example-game-repo repo (X:\example-game-workbench-repo\docs\plans\).

Plans audited (4 real plans; sidecar exclusion applied first):

| Plan | Oracle 1 (claim) | Oracle 2 (code) | Oracle 3 (review) | Bucket | |------|-----------------|-----------------|-------------------|--------| | 2026-05-24-patch-and-send-back-contribution-invite.md | status: implemented; cites commits d67f7371b, d368b6269, a92447c17, b5b824d47; 10 ACs | 9/10 ACs verified at disk; AC3 (URL liveness) inherently manual — remaining 9 pass grep/cited checks | COVERED (archive-aware). archive/review-trail/2026-05-21/2026-05-24-115430145738-08b5c444.json — range 977a40b29..cc4c936d6; all 4 delivery commits satisfy is-ancestor C cc4c936d6 && ! is-ancestor C 977a40b29. A live-only Oracle 3 (state/review-trail/ glob without archive/review-trail/**) would mis-classify this plan as DELIVERED-UNREVIEWED — see Key finding below. | DELIVERED+REVIEWED (under archive-aware Oracle 3) — would have been DELIVERED-UNREVIEWED under live-only Oracle 3 | | 2026-05-26-game-dev-ownership-and-bidirectional-install-drift.md | status: implemented; 9 delivery commits; 13 ACs; reviewed: the Staff Engineer 2026-05-26; code-reviewer 2026-05-26 | 12/13 ACs fully in-repo; AC7 realized as external coordinator dependency (landed) | COVERED. Live record state/review-trail/2026-05-26-131032300171-1afa35ae.json — range 609399fcc..HEAD; all 9 delivery commits fall inside range | DELIVERED+REVIEWED | | 2026-05-26-headless-extractor-seam-buildout.md | status: draft; active workstream with recovery handoff 2026-05-27_084305_e40956a9.md ("review-complete, ZERO C++ authored; resume by dispatching Phase 1 H-2 resolver") | Not run — no terminal close; Oracle 2 does not apply to in-flight work | Not applicable — delivery not claimed | IN-FLIGHT | | 2026-05-19-headless-extraction-buildout.md | status: draft; kind: roadmap-lite; explicitly superseded by the 05-26 seam plan | Not run — no delivery expected | Not applicable — abandoned by supersession | ABANDONED |

On DELIVERED-UNREVIEWED and PARTIAL buckets: the worked example's 4 plans landed cleanly in DELIVERED+REVIEWED / IN-FLIGHT / ABANDONED in their resolved state — but the 05-24 row is annotated above with what it would have been under a live-only Oracle 3 (DELIVERED-UNREVIEWED). That's the falsifying demonstration of why archive-aware reading is load-bearing: the same plan, audited the same day, resolves to a different bucket depending on whether Oracle 3 globs the archive. PARTIAL is exercised when Oracle 1 claims implemented but Oracle 2 finds a claimed artifact absent/wrong at HEAD or the project test suite red — none of the 4 audited plans hit that path; the bucket is reached via the decision tree's Oracle 2 branch when artifact checks or project tests fail.

Decision tree walk for each row:

1. patch-and-send-back (05-24): Oracle 1 = implemented → proceed to Oracle 2. Oracle 2 = cited artifacts verified at HEAD: 9/10 claims pass (grep/file checks); AC3 is a URL liveness check — inherently unverifiable by grep/ls → "all verifiable artifact claims confirmed". Proceed to Oracle 3. Oracle 3 = archived trail record covers all delivery commits via range-membership → COVERED. Bucket: DELIVERED+REVIEWED. ✓

2. game-dev-ownership (05-26): Oracle 1 = implemented → proceed to Oracle 2. Oracle 2 = cited artifacts verified at HEAD: 12/13 claims pass (grep/file checks); AC7 is an external coordinator dependency (landed externally) → "all verifiable artifact claims confirmed". Proceed to Oracle 3. Oracle 3 = live trail record covers all 9 delivery commits → COVERED. Bucket: DELIVERED+REVIEWED. ✓

3. headless-extractor-seam (05-26): Oracle 1 = draft → bucket resolves immediately at Oracle 1. No delivery claimed; active recovery handoff confirms in-flight status. Bucket: IN-FLIGHT. ✓ (Oracle 2 and 3 not needed.)

4. headless-extraction-buildout (05-19): Oracle 1 = draft with explicit supersession pointer → bucket resolves at Oracle 1. No delivery expected or claimed. Bucket: ABANDONED. ✓ (Audit recommends a frontmatter flip to status: superseded + superseded_by: for hygiene — the current draft falsely signals "resumable" to pickup candidates.)

Key finding from this audit: The handoff's alarm — "only 4 review-trail records, most shipped work unreviewed" — was an archival artifact, not a coverage gap. The /workweek-complete run on 2026-05-24 (commit db151655e) moved 22 review-trail records into archive/review-trail/2026-05-21/. A live-only read of state/review-trail/ saw only the current week's 4 records and missed them. Oracle 3 reading ONLY the live dir would have mis-classified the 05-24 plan as DELIVERED-UNREVIEWED — a false verdict. The archive-aware read via list-review-trail-records.sh is load-bearing, not optional.

Why a skill, not a one-shot

This shape recurs after every crash and every "did we actually ship what we think we shipped?" moment — it has fired at least twice in the coordinator workstream and once explicitly in example-game-repo (2026-05-27). Re-deriving the three-oracle protocol, the git range-membership formula, the archive-aware glob, and the sidecar exclusion rule by hand each time is the cost this skill exists to amortise.

Closest analogues in skills/: bug-sweep (sweep-and-classify), architecture-audit (multi-oracle assessment), validate (project test suite runner). All three support skill-shape for recurring structured procedures. A wiki-only home would re-derive the dispatch shape every invocation and would not be greppable from coordinator:plan or /workstream-complete as a routine option.

Dispatch sequencing

The skill orchestrates its own work in three phases:

Phase 1 — Gather and filter (~2 min, EM does this directly)

1. Glob the plan set: docs/plans/*.md (or the caller-supplied glob).
2. Filter out sidecars (any filename containing .prior-art-check, .coverage-check, .docs-check, .review-the Staff Engineer, .review-, or .check.).
3. For each remaining plan, read frontmatter status: to rough-sort into buckets:
   • status: superseded / abandoned / cancelled → ABANDONED (no further oracle work needed)
   • status: draft / in-progress / reviewed → IN-FLIGHT (no further oracle work needed)
   • status: implemented / shipped → needs Oracle 2 + 3 (add to work queue)

Phase 2 — Oracle 2 + 3 on implemented plans (~10 min, parallel scouts)

For each implemented plan, dispatch a read-only Sonnet scout with:

"Read <plan-path> and extract every concrete artifact claim (files, paths, symbols, schema fields, config entries) from its scope section, chunk list, and AC criteria prose. Verify each claim exists at HEAD using grep/ls/symbol lookup. Report each claim as PRESENT / ABSENT / UNVERIFIABLE. Do NOT modify files, commit, or push. Return results inline."

Concurrently (EM-side), run Oracle 3 for each plan:

list-review-trail-records.sh   # get all live + archived records
# for each record, read sha_range and test delivery commits for range membership:
  git merge-base --is-ancestor <commit> <end_sha>    # must succeed (exit 0)
! git merge-base --is-ancestor <commit> <start_sha>  # must succeed (exit 0) — note the leading !

Phase 3 — Synthesise and write output (~3 min)

Apply the bucket decision tree to each plan. Write the output table + summary to state/audits/YYYY-MM-DD-<SID_SHORT>-plan-delivery-audit.md (your session's audit — see "Output format" above for how SID_SHORT is resolved). Surface any DELIVERED-UNREVIEWED plans to the PM with a code-reviewer dispatch recommendation — do not dispatch autonomously, as the PM may choose to defer.