arisng/harness-engineering

Harness Engineering

Design environments, feedback loops, and control systems that enable AI coding agents to produce reliable software at scale. A harness is everything around the code that keeps agents effective: structured knowledge, mechanical constraints, application legibility, and entropy management.

Core insight: When agents struggle, the fix is almost never "try harder." Ask: what capability is missing, and how do we make it both legible and enforceable for the agent?

Variant: harness-engineering-copilot — strategies and patterns specialized for GitHub Copilot agent customization.

Sources: OpenAI — Harness Engineering | Martin Fowler — Harness Engineering

Three Harness Components

| Component | What it does | Deterministic? | |---|---|---| | Context Engineering | Curated, structured knowledge base in the repo + dynamic context (observability, browser) | Mixed | | Architectural Constraints | Layered architecture enforced by custom linters and structural tests | Deterministic | | Garbage Collection | Periodic agents that detect drift, stale docs, and constraint violations | LLM-based |

Assess Current Harness

Before building, audit what exists. Run through this checklist with the user:

1. Repository knowledge — Is there an AGENTS.md or equivalent entry point? Is it a concise map (~100 lines) or a bloated monolith?
2. Documentation structure — Is there a docs/ directory with architecture, design docs, product specs, and plans? Are they versioned and cross-linked?
3. Architectural constraints — Are module boundaries defined? Are dependency directions enforced mechanically (linters, structural tests)?
4. Application legibility — Can an agent boot, drive, and observe the application (logs, metrics, traces, browser)?
5. Feedback loops — When the agent produces drift, is there a process to detect and correct it?
6. CI/CD gates — Do linters and tests catch constraint violations before merge?

Rate each 0–2 (absent / partial / enforced). Total ≤ 5 = early stage; 6–8 = developing; 9–12 = mature.

1. Context Engineering

The repository is the agent's only world. Anything not accessible in-context effectively does not exist.

Repository Knowledge as System of Record

Anti-pattern: One giant AGENTS.md with everything. It crowds out task context, rots instantly, and is unverifiable.

Pattern: AGENTS.md as table of contents (~100 lines), pointing to structured docs/:

AGENTS.md              ← map, not manual
ARCHITECTURE.md        ← top-level domain + layer map
docs/
├── design-docs/       ← indexed, with verification status
│   ├── index.md
│   └── core-beliefs.md
├── exec-plans/        ← active, completed, tech-debt
│   ├── active/
│   └── completed/
├── product-specs/     ← requirements + acceptance criteria
├── references/        ← external docs, llms.txt files
├── DESIGN.md
├── FRONTEND.md
├── QUALITY_SCORE.md
├── RELIABILITY.md
└── SECURITY.md

Key principles:

• Progressive disclosure: agents start with a small stable entry point and are taught where to look next
• Plans as first-class artifacts: execution plans with progress logs checked into the repo
• Mechanical freshness: CI jobs validate the knowledge base is up-to-date, cross-linked, and correctly structured
• Doc-gardening agent: periodic background task scans for stale docs and opens fix-up PRs

Dynamic Context Providers

Extend what agents can observe beyond static files:

• Browser automation: Wire Chrome DevTools Protocol (CDP) into agent runtime. Agents take DOM snapshots, screenshots, and navigate UI to reproduce bugs and validate fixes.
• Observability stack: Ephemeral per-worktree observability (logs via LogQL, metrics via PromQL, traces via TraceQL). Agents query: "ensure startup < 800ms" or "no span in critical journeys exceeds 2s."
• App bootability: Make the application bootable per git worktree so agents can launch isolated instances per task.

2. Architectural Constraints

Agents replicate patterns that exist in the repository — including bad ones. Mechanical enforcement prevents drift.

Layered Domain Architecture

Divide each business domain into fixed layers with strictly validated dependency directions:

Types → Config → Repo → Service → Runtime → UI
                         ↑
                    Providers (auth, connectors, telemetry, feature flags)

• Code may only depend "forward" through the layers
• Cross-cutting concerns enter through a single explicit interface: Providers
• Enforced by custom linters and structural tests (not just conventions)

Mechanical Enforcement

| Mechanism | Purpose | Examples | |---|---|---| | Custom linters | Catch violations with remediation instructions in error messages | Dependency direction, structured logging, naming conventions, file size limits | | Structural tests | Validate architecture invariants | ArchUnit-style tests, import graph validation | | Pre-commit hooks | Block violations before they enter the repo | Schema validation, boundary checks | | CI validation | Catch what slips through locally | Full lint + test suite, doc freshness checks |

Critical detail: Write custom lint error messages that include remediation instructions. These messages become agent context when violations occur, teaching the agent how to fix issues.

Enforce Invariants, Not Implementations

Be prescriptive about boundaries (e.g., "parse data shapes at the boundary"), but not about specific tools. Specify what, let agents decide how within boundaries.

3. Garbage Collection

Full agent autonomy introduces entropy. Without active management, drift compounds.

Golden Principles

Define opinionated, mechanical rules that keep the codebase legible:

• Prefer shared utility packages over hand-rolled helpers (centralized invariants)
• Validate data at boundaries — don't probe data "YOLO-style"
• Enforce consistency in naming, logging, and error handling

Periodic Cleanup Agents

Schedule recurring background tasks that:

1. Scan for deviations from golden principles
2. Update quality grades per domain and layer
3. Open targeted refactoring PRs (most reviewable in under a minute)

Cadence: Daily or per-sprint. Treat like garbage collection — continuous small increments, not painful bursts.

Quality Scoring

Maintain a quality document that grades each product domain and architectural layer, tracking gaps over time. This gives both humans and agents a map of where debt lives.

Agent-First Workflow

When adopting harness engineering, the development loop changes:

Human: Define task → Write acceptance criteria
  ↓
Agent: Validate codebase state → Implement → Self-review → Request agent reviews
  ↓
Agent: Respond to feedback → Iterate until reviewers satisfied
  ↓
Agent: Detect/remediate build failures → Escalate only when judgment required
  ↓
Merge (minimal blocking gates — corrections are cheap, waiting is expensive)

Humans work at a different abstraction layer: prioritize work, translate user feedback into acceptance criteria, validate outcomes. When agents struggle, identify what's missing and feed it back into the repo.

Applying to Existing Codebases

Not every technique applies to brownfield code. Assess feasibility:

| Technique | Retrofit difficulty | Start here | |---|---|---| | AGENTS.md + docs/ structure | Low | Create entry point, incrementally document | | Custom linters | Medium | Start with 2–3 high-value rules | | Structural tests | Medium | Enforce module boundaries first | | App legibility (CDP, observability) | High | Requires infra investment | | Full garbage collection agents | High | Start with doc-gardening only |

Pragmatic approach: Start with context engineering (cheapest, highest ROI), add architectural constraints incrementally, garbage collection last.

Harness as Service Template

For organizations with 2–3 main tech stacks, consider packaging harnesses as reusable templates:

• Pre-configured linters and structural tests for the stack
• Knowledge base skeleton (AGENTS.md, docs/ structure)
• CI pipeline with doc-freshness and constraint-validation jobs
• Starter golden principles for the technology

This parallels "golden path" service templates, but optimized for agent-driven development.