phrazzld/harness-engineering

/harness-engineering

Engineer the harness. Keep it thin.

Route

| Need | Load | |---|---| | create skill or prompt | references/mode-create.md | | eval skill | references/mode-eval.md | | lint skill | references/mode-lint.md | | apply skill-design lessons | references/skill-design-principles.md | | clean Codex skill catalog | external steipete-skill-cleaner | | convert agent/skill | references/mode-convert.md | | sync externals | references/mode-sync.md | | engineer doctrine/gates/hooks | references/mode-engineer.md | | measure skill usage/health/staleness | references/mode-audit.md | | current model/provider/harness facts | the roster skill's references/model-provider-harness-index.md | | open-model defaults | references/open-model-roster.md |

Repo-local skills for consumer repos (bespoke QA drivers, persona probes) are written directly into that repo's .agents/skills/ with its real routes and commands; this skill owns the craft either way. For a repo's verification skill, interview the operator first: the manual checks they run after the agent responds and before merge are the spec — encode each check that has a tool. Turning a proven session pattern into a first-party primitive starts at the primitive test below — most patterns are prompts, not skills.

Contract

• Fix root cause in the highest-leverage layer: type/test/hook/gate/skill/ AGENTS, in that order.
• Prefer deletion. Harness prose is context tax.
• Cross-harness first: Claude, Codex, Pi. Filesystem + SKILL.md is primary.
• Skills stay self-contained: scripts/references under the skill; state roots from invoking repo.
• Treat a skill as a folder, not a markdown file. Use scripts, references, examples, templates, assets, evals, or append-only data when prose would make the agent reconstruct repeatable work.
• Model/provider/harness selection facts live in the roster skill (skills/roster/references/model-provider-harness-index.md). Keep that file factual: model ids, context, price, latency/smoke evidence, tool support, benchmark sources, deprecations, and freshness. Do not encode role-fit policy there; the lead agent composes task-specific teams from current evidence.
• Harness Kit source skills live in skills/; repo-local .agents/skills/ and harness-specific skill bridge dirs are /seed output for consumer repos.
• Generated/root AGENTS.md is a router, not a manual. Keep non-obvious facts only.
• System bootstrap exposes the full first-party skill catalog; repo-local vendoring is exceptional and must earn its complexity.
• Provider CLIs are tools. Do not wrap them in semantic orchestration unless a shaped ticket explicitly asks.
• Harness-specific accelerators (e.g. orchestration-workflow templates) may ride inside a skill folder as assets the prose names as optional — a harness without the feature must lose nothing by ignoring them. Build one only after telemetry shows the pattern recurring; never pre-author.

Delegation Judgment

delegate on judgment per the shared Roster contract: native subagents by default; add cross-model critics, roster providers, or sprite lanes (/sprites) only when they answer a distinct question. See harnesses/shared/AGENTS.md (Roster).

Local lane guidance: Use lanes for doctrine critique, runtime compatibility, gate design, and regression risk. Do not treat a missing repo-local roster as a waiver; use the resolver-backed probe.

Primitive Test

Before creating or growing anything, classify it (2026-06 audit, backlog 103):

• Prompt — "is this just what the operator would retype to a strong model?" One file, no references/, no receipts, exempt from skill machinery.
• Skill — "does this change what a frontier model does, for the better, repeatedly?" Judgment + context the model can't derive.
• Doctrine line — "worth paying for in every session?" Goes in AGENTS.md, not a folder.
• Mode B — event-triggered (on PR-ready, on production error, on schedule)? It belongs in the event plane (bitterblossom), not this harness. This repo is the ad-hoc operator layer plus the shared disk contracts.

History: slash commands were collapsed into skills when skills arrived, so saved prompts masqueraded as skills and the catalog tripled. Do not recreate that.

Quality Bar

• SKILL.md encodes judgment, not a procedure the model already knows.
• Frontmatter descriptions are model trigger classifiers, not human summaries: include explicit Use when: phrases and Trigger: aliases.
• Instruction prose is compression, not literature. Sacrifice grammar before clarity; keep terse imperatives, named failure modes, and concrete oracles.
• Put long mode detail in references/; keep the entry file short.
• Build gotchas from repeated agent failures. If a gotcha can be asserted by a script, hook, or eval, codify it there and point the skill at the artifact.
• Ad-hoc roster lanes beat static project subagents unless tool permissions must be isolated.
• New mechanisms include a gate, eval, or smoke path.
• Every run ends clean: no untracked or modified files.

Post-Sync Acceptance

After changing skills, shared doctrine, generated docs, bootstrap, roster, or harness projections, prove the output is repo-fit, not merely structurally valid.

## Acceptance Evidence
- Live repo evidence read: source skill, shared doctrine, generated docs, bootstrap output, roster, or harness projection inspected.
- Acceptance source: backlog oracle, skill contract, generated index/docs contract, bootstrap contract, or explicit absence.
- Evidence that proves it: command output, diff, generated artifact, bootstrap transcript, or gate output.
- Exact command/path/route exercised: check, generator, bootstrap, smoke path, projection path, or route run.
- Oracle / acceptance artifact hash: sha256 digest for any fixture, generated artifact, transcript, or contract used as the oracle, or state that no artifact-backed oracle exists.
- Contract-change acknowledgment: reason when the change alters an acceptance contract, generated source, or assertion surface, or state that no contract changed.
- Repo-fit check: source/generator/projection agree; no stale generated docs, wrong skill root, stale command, or copied bridge remains.
- Structural gate: `check --repo .` result, or the specific sub-gate exercised.
- Residual risk: skipped harness, external dependency, or none with reason.

Gotchas

• Phase prose is not judgment. Frontier models know the SDLC; a skill that restates implement/refactor/review steps is railroading (Anthropic's own top skill lesson: don't state the obvious, avoid railroading). Encode the bespoke part — oracles, repo facts, taste — or nothing.
• Process bureaucracy trains checkbox compliance, not quality. Multi-field completion gates, oracle hashes, and learning-packet ceremonies get filled in plausibly by strong models. Verification is tests, CI, and driving the live surface.
• Deterministic scaffold is the historical failure mode here: agents unsure of harness engineering fall back to Rust/scripts that enforce prose. Every gate must answer "what real failure did you catch in the last 90 days?" — no answer, delete it.
• Check telemetry before adding or keeping a skill. The 2026-06 audit found ~15 of 36 skills unused; usage is a power law. Low usage with high value-when-used is fine (say so); low usage with no story is deletion.
• A new frontier model release silently converts some skill prose from judgment into railroading: instructions tuned for the last model anchor the new one to stale patterns. After a major model ships, re-audit skill and doctrine prose; prefer deleting an instruction over updating it.
• Name harness features by capability, not vendor. "Use the harness's large-scale orchestration feature when it has one" degrades gracefully across harnesses; "use dynamic workflows" confuses every harness that lacks them.
• Meta-work ratio: if this repo's commit rate rivals the product repos', the flywheel is feeding itself.
• Stale AGENTS prose is worse than missing prose.
• Duplicated repo-local skill copies are usually stale context unless a repo needs checked-in vendored harness state.
• Generated catalog/docs drift means the source skill changed but the harness projection did not.
• Unsupported invocation hooks mean usage telemetry is structurally shaped but not empirically proven for that harness.
• Structural eval trees are not semantic proof; objective graders must assert behavior or carry an explicit waiver.
• Helper scripts that are not wired into a gate become optional folklore.
• Regexes over agent prose are usually the wrong boundary.
• If a rule matters, enforce it outside prose.

Verification

Run cargo run --locked -p harness-kit-checks -- check --repo . after changing harness primitives, gates, roster, bootstrap, or sync logic. For bootstrap changes, also re-run the bootstrap and confirm the installed symlinks (skills, prompts, configs) match the source tree.