codyswanngt/lisa-reproduce-bug

Reproduce Bug

A bug that cannot be reproduced cannot be verified as fixed. Root cause analysis does not begin until a reliable reproduction exists.

The reproduction must exercise the real path

Left alone, an agent asked to demonstrate a defect will build the environment in which the defect appears — a stub, a fake, a fresh harness — and present that as the reproduction. It looks like proof and it is not: a fix that satisfies it may never touch the path the user is on.

Two kinds of setup get confused here, and only one is a problem:

• Prerequisites — state the real path genuinely needs: a seeded record, a logged-in user, a feature flag, a fixture recreating production-equivalent data. These are part of the reproduction. Removing them changes the precondition rather than testing anything, so do not remove them.
• Replacement scaffolding — anything standing in for behaviour the real path would perform: a mocked service, a stubbed function, a fake clock, a bespoke harness that bypasses the normal entry point. This is what makes a reproduction suspect.

So the reproduction states, explicitly:

• The entry point the user actually hits — the route, endpoint, command, or interaction.
• Prerequisites, listed as setup.
• Replacement scaffolding, each item with the real behaviour it substitutes.
• Whether the failure survives with prerequisites in place and replacement scaffolding removed. If it does not, this is a lead rather than a reproduction. Say so and keep going.

If the real path is unreachable — no credentials, no environment, no data — that is a blocked reproduction. Report the missing access instead of substituting scaffolding and calling the bug reproduced.

Choose the method from the symptom

| Symptom | Reach for | | --- | --- | | Wrong value, bad output, thrown error | Failing test at the narrowest layer that still crosses the real path | | Broken interface or journey | Browser or device driver against a running build (Playwright, Maestro) | | Service or API behaviour | Direct request to the running service — client script or curl | | Depends on particular data | Seeded fixture recreating that state, recorded as a prerequisite | | Intermittent or timing-shaped | Loop the trigger; capture timestamps around async boundaries | | Works locally, fails deployed | Do not chase it locally — reproduce against the environment that fails |

A failing test is the preferred form wherever it can cross the real path: it runs in CI, it becomes the regression guard, and codify-verification expects it. A script is the fallback. Manual steps are the last resort and must carry their prerequisites.

Report the failure rate, do not round it

Run the reproduction enough times to state a rate. A reproduction that fails half the time cannot prove a fix — one passing run afterwards means nothing at that rate. If the rate is too low to distinguish a fix from luck, say what would raise it: more iterations, a forced schedule, a narrowed trigger, a seeded clock.

When it will not reproduce

The difference is nearly always one of: runtime version, configuration or feature flags, data state, credentials and permissions, network posture, or platform. Diff the two environments along those axes rather than guessing between them, and report which axes you compared and what you found. That comparison is the finding when the bug stays hidden.

Output

## Reproduction

**Entry point:** the route, command, or action the user hits
**Command or steps:** exactly what to run
**Actual:** what happens · **Expected:** what should happen
**Prerequisites:** seeded data, auth state, flags the real path needs — or "none"
**Replacement scaffolding:** each mock, stub, fake, or bespoke harness and the
  real behaviour it substitutes — or "none"
**Survives without replacement scaffolding:** yes / no — if no, this is a lead,
  not a reproduction
**Observed failure rate:** n failures in m runs
**Form:** failing test `path` | script `path` | manual steps above
**Environment:** runtime, platform, relevant dependency versions

Capture output whole — a truncated stack trace loses the line that mattered. But evidence carries whatever the system was holding, so redact secrets, tokens, credentials, and personal data before a reproduction is handed on or attached to a work item, and keep any unredacted capture only where the data class it contains is already permitted to live.