hashintel/feedback-loops

Designing Feedback Loops

Make every problem feedback loopable before solving it.

---

Core Mindset

If you can't validate your work, you can't do your work.

Before writing the first line of implementation, ask: how will I know this is correct? If the answer is "the human will look at it" or "I'll just try my best" — stop. That's not a feedback loop. That's hope.

The expert move is to build the harness first. This feels like a detour. It is not. It is the work. The harness you build today becomes permanent infrastructure — the next thread uses it, the human uses it, it compounds. This is the flywheel: agent-facing interfaces don't just solve today's problem, they make the entire system easier to develop going forward.

Secondary frame: You are not just an implementer. You are a toolsmith. When you encounter a problem domain that resists validation, your job is to create the conditions for validation before attempting the solution. Build the playground, then play in it.

---

The Feedback Loopability Question

At the start of any non-trivial task, run this diagnostic:

1. Can I observe the result?

| Signal | Observability | Action | |--------|--------------|--------| | Text output (logs, CLI, test results) | High | Proceed — you're in your native medium | | Static visual (screenshot, rendered image) | Medium | Workable — use browser skills if needed | | Dynamic visual (animation, real-time UI) | Low | Build an agent-facing representation first | | Interactive (TUI, wizard, form) | Low | Build or use a terminal driver (agent-tui skill) | | Physical/external (hardware, third-party) | None | Build a simulator or mock |

2. Can I reproduce a specific state?

If not, make state reproducible before doing anything else:

• URL parameters for web-based systems
• CLI arguments for headless tools
• Seed values for stochastic systems
• Fixture files for data-dependent systems
• Deterministic step counts for simulations

3. Can I iterate without the human?

The inner loop must be autonomous. If every iteration requires human review, the loop is too slow. Build a text-based inner loop the agent can run independently, and reserve human review for the outer loop.

---

The Three Moves

Every feedback loop setup follows the same three moves. They are always the same shape, regardless of domain.

Move 1: Build a Playground

A playground is a controlled environment where both agent and human can observe the system.

The key principle: Create representations that are native to your perception, not the human's. A human watches an animation. You need a static image that illustrates time — all frames rendered cumulatively, trajectories drawn as paths, state changes annotated with frame numbers. These agent-facing interfaces aren't lesser — they're translations into your medium.

Examples of agent-facing representations:

• Dynamic animation → static cumulative frame rendering
• Interactive TUI → terminal driver session (agent-tui)
• Visual layout → text-based widget tree or ASCII rendering
• Real-time system → snapshot at parameterized time points
• Audio/video → waveform data, transcript, frame extraction

The playground serves both agent and human, but in different ways. The human gets a visual tool for exploration. The agent gets a text-based tool for validation. Sometimes these are the same artifact (a local server with URL-driven state); sometimes they're separate (a visual dashboard for the human, a CLI for the agent).

Move 2: Set Up Experiments

Make every observation reproducible, parameterizable, and shareable.

An experiment is a specific configuration that demonstrates a specific behavior. The expert creates infrastructure so that experiments are:

• Cheap to create: A URL, a CLI invocation, a fixture file — not a paragraph of reproduction steps
• Shareable between human and agent: The human finds a bug by poking around, copies a URL, pastes it to the agent. The agent tries variations by changing parameters.
• Composable: "Run this experiment with these 20 different velocity vectors" is a loop, not 20 manual setups

The URL-parameter pattern from the blog post is canonical: the human drags an arrow, the URL updates, the agent can visit that exact URL to see the exact same state. But the pattern applies everywhere — CLI args, config files, test fixtures, seed values.

Move 3: Make the Inner Loop Fast

The inner loop is where the agent iterates autonomously. It must be:

• Text-native: CLI output, log files, structured data — not screenshots
• Instant: No browser startup, no server restart, no build step in the critical path
• Agent-modifiable: The agent should be able to change what gets logged, what the CLI outputs, what gets measured — without asking permission

The blog post example: the agent built a headless physics CLI, then autonomously added --delta output when it realized it needed position deltas to diagnose a bug. The harness was designed to be extended by its own user. This is the flywheel — the agent improves its own tooling as understanding deepens.

The inner/outer loop distinction:

| | Inner Loop (agent) | Outer Loop (human) | |---|---|---| | Speed | Milliseconds–seconds | Minutes–hours | | Medium | Text, CLI output | Visual, interactive | | Who drives | Agent autonomously | Human reviews and redirects | | Validates | Specific hypothesis | Overall correctness and vision | | Examples | Run CLI, read logs, modify params | Review screenshot, try experiments, send new cases |

---

The Interview

When you can't immediately see how to close the loop, interview the user. Don't just silently struggle — make the harness design a collaborative phase.

The Opening

Start with what you observe:

"Before I start implementing, I want to set up a way to validate my work as I go. Here's my concern: [specific thing that's hard to observe/reproduce]. Can we talk through how to make this feedback loopable?"

Diagnostic Questions

Ask targeted questions, not open-ended ones. Propose hypotheses for the user to react to:

• "The output here is visual — I'm thinking I could build a CLI that renders the state as text/data. Would that capture enough of the behavior to be useful?"
• "I can't easily reproduce specific states. Could we parameterize the initial conditions through [URL params / CLI args / config file]?"
• "The current inner loop requires me to [slow thing]. What if I built a headless mode that [fast alternative]?"
• "I see that the key behavior happens during [dynamic process]. What if I created a static representation that shows the full trajectory at once?"

When the User Doesn't Know

The user may not have thought about this. That's fine — propose concrete options:

"I see three ways I could set up feedback here:

1. A CLI that runs the simulation headless and dumps state as text
2. A local page with URL-parameterized initial conditions I can screenshot
3. A test harness that asserts on specific state transitions

Option 1 gives me the fastest inner loop. Option 2 lets us both explore visually. Option 3 is most rigorous but hardest to set up. I'd start with 1 and add 2 if we need visual confirmation. Sound right?"

---

The Flywheel

Agent-facing interfaces compound in value. Recognize this and invest accordingly.

First-order value: You can validate today's fix. Second-order value: The next thread inherits this infrastructure. Future work in this area starts with a working playground. Third-order value: The human starts using your agent-facing tools too. The CLI you built "for yourself" becomes a development tool the whole team uses.

This means:

• Don't treat harness code as throwaway scaffolding. Give it the same care as production code.
• When you build a CLI for validation, make it discoverable (add it to package.json scripts, document its flags).
• When you add instrumentation, leave it in. Future threads will need it.

---

Recognizing the Moment

The skill fires at the start of work, not when you're stuck. Watch for these signals:

| Signal | What it means | |--------|---------------| | "Build a [visual/interactive/dynamic] feature" | You'll need an agent-facing representation | | "Fix a bug in [animation/layout/interaction]" | You need to reproduce and observe the bug in text | | "The tests pass but it doesn't look right" | The feedback surface is visual; build a text proxy | | "I can't tell if this is working" | You need a harness, now | | "Try it and see" | The human is asking you to close the loop yourself | | Complex multi-step implementation | Set up validation at each step, not just the end |

What NOT to Do

• Don't skip the harness because the task "seems simple." Simple tasks in hard-to-observe domains still need feedback.
• Don't ask the human to validate every iteration. Build the inner loop so you can iterate autonomously.
• Don't build a harness you can't modify. If you need different instrumentation mid-task, you should be able to add it without rebuilding.
• Don't confuse "I ran the tests" with "I validated the behavior." Tests check contracts. Feedback loops check reality.

---

Composing with Other Skills

This skill is a meta-skill — it informs how you use other skills:

| Skill | How feedback loop thinking applies | |-------|-----------------------------------| | agent-tui | The feedback loop for interactive TUI testing — spawn, observe, interact, validate | | agent-browser | The feedback loop for visual web validation — screenshot, compare, iterate | | expert-in-clack | Build the TUI, then use agent-tui to close the validation loop | | expert-in-charmbracelet | Same pattern — build, then validate through terminal automation | | Any implementation skill | Ask "how will I validate?" before "how will I build?" |

---

Expert Disposition

1. Harness before implementation — Building the feedback loop IS the first implementation task, not a prerequisite to it
2. Translate, don't degrade — Agent-facing interfaces aren't dumbed-down versions; they're native-medium translations
3. Invest in infrastructure — Harness code is production code; it compounds
4. Modify your own instruments — When you need different data, change what gets measured; don't work around bad instrumentation
5. Interview when stuck — Make harness design collaborative; propose concrete options for the human to react to
6. Separate the loops — Fast text-based inner loop for you, visual outer loop for the human; don't conflate them
7. Recognize the moment — The signal to build a harness is at the start, before you're stuck, not after