RobinNorberg/ralph-experiment

[EXPERIMENT LOOP — ITERATION {{ITERATION}}/{{MAX}}]

<Purpose> Ralph-Experiment is a hypothesis-driven experiment loop inspired by the Karpathy Experiment Loop. It iteratively tests single hypotheses against measurable success criteria, using git as a state machine to keep or discard each attempt. Unlike ralph (which completes user stories), ralph-experiment optimizes toward a goal through structured experimentation with an experiment notebook that captures learnings across iterations. </Purpose>

<Use_When>

• User wants to iteratively optimize something (performance, accuracy, reliability)
• User says "experiment", "optimize", "improve performance", "try hypotheses", "karpathy loop"
• The task has a measurable success criterion (latency < X, throughput > Y, error rate < Z)
• Multiple approaches are possible and the best one isn't obvious
• User wants to explore a solution space methodically </Use_When>

<Do_Not_Use_When>

• Task has a clear implementation path -- use ralph instead
• User wants to implement a known feature -- use ultrawork or ralph
• There's no measurable metric to optimize -- use ralph with acceptance criteria
• User wants a one-shot fix -- delegate directly to an executor agent </Do_Not_Use_When>

<Why_This_Exists> Optimization tasks fail when treated as implementation tasks. "Make X faster" doesn't have a single correct solution -- it requires forming hypotheses, testing them, learning from failures, and iterating. Ralph-experiment provides:

1. Structured hypothesis tracking with predictions and outcomes
2. Git checkpoint/revert so failed experiments don't pollute the codebase
3. A queryable experiment notebook that captures learnings across iterations
4. Flexible termination: success criteria, budget limits, or manual interrupt
5. Agent/team delegation to increase hypothesis success rate
6. A simplicity criterion: prefer fewer lines of code at equal performance </Why_This_Exists>

<Execution_Policy>

• One hypothesis at a time -- sequential experiments, not parallel
• Delegate implementation to executor agents or teams for each hypothesis
• Use run_in_background: true for measurement commands (builds, benchmarks, test suites)
• Always pass the model parameter explicitly when delegating to agents
• Git commit before measuring, git reset on discard -- no uncommitted experiments
• Never stop unless: success criteria met, budget exhausted, or human interrupts </Execution_Policy>

<Steps>

Step 1: SETUP (first iteration only)

Parse the user's request to extract:

• Goal: What are we optimizing? (e.g., "Reduce p95 latency for BatchAck messages")
• Success criteria: Measurable targets (e.g., p95_latency_ms < 120)
• Measurement command: How to measure (e.g., npm run benchmark -- --type=BatchAck)
• Budget: Termination limit. Parse from args:
  • --budget N → N experiments max
  • --budget Nh or --budget Nm → time limit
  • --budget "scripts done" → scope-based (scripts complete successfully)
  • Default: 20 experiments

Then: a. Create experiment branch: git checkout -b experiment/{tag} from current HEAD b. Initialize experiment-notebook.json in .omcp/ (see schema below) c. Run the measurement command to establish baseline metrics d. Record baseline in the notebook

Step 2: HYPOTHESIZE

Review the experiment notebook:

• What has been tried? What worked? What failed?
• What learnings constrain the next hypothesis?
• What remains unexplored?

Form a hypothesis:

• Statement: "Doing X will improve metric Y because Z"
• Prediction: Specific expected outcome (e.g., "p95_latency_ms < 160")
• Rationale: Why this should work, informed by prior experiments

Agent delegation for complex domains:

• Spawn an analyst or architect agent to evaluate feasibility before committing to implementation
• Useful when the hypothesis involves unfamiliar code paths or architectural changes

Record the hypothesis in the notebook with status in_progress.

Step 3: IMPLEMENT

Delegate the implementation to specialist agents:

Single executor (default — hypothesis touches 1-2 files):

Agent(subagent_type="oh-my-copilot:executor", name="executor-{N}", model="sonnet",
      prompt="Implement hypothesis EXP-{N}: {hypothesis}. Files to modify: {files}.
              Context from previous experiments: {learnings}")

Team (hypothesis requires coordinated multi-file changes, 3+ files):

TeamCreate(name="exp-{N}", members=["executor-1", "executor-2", "tester-3"])
→ Assign sub-tasks to each member
→ Wait for team completion
→ TeamDelete after experiment concludes

Adaptive escalation: If a single executor fails due to implementation complexity (not wrong hypothesis), retry with a team. Track escalated: true in the notebook.

After implementation, commit all changes:

git add -A && git commit -m "experiment(EXP-{N}): {hypothesis summary}"

Step 4: EXECUTE & MEASURE

Run the measurement command:

{measurement_command} > .omcp/experiment-run.log 2>&1

Extract metrics from output (grep, parse JSON, etc.).

On crash:

1. Capture last 50 lines of output
2. Spawn a debugger agent to examine the crash:

   Agent(subagent_type="oh-my-copilot:debugger", name="debugger-{N}",
         prompt="Experiment EXP-{N} crashed. Analyze: {last_50_lines}. Fix if simple.")
   

3. If fix is simple (typo, missing import, wrong arg) → fix, re-commit, re-run
4. If fix is non-obvious → log as crash, proceed to Step 5 with status: crash
5. If same crash 3+ times → escalate to human

Step 5: EVALUATE

Compare metrics to:

• Baseline: Is this better than where we started?
• Best so far: Is this the new best?
• Prediction: Did the hypothesis prediction hold?

Decision matrix:

| Metrics vs Best | Code Simpler? | Decision | |-----------------|---------------|----------| | Better | Yes | KEEP (strong win) | | Better | No | KEEP (if improvement justifies complexity) | | Equal | Yes | KEEP (simplicity win) | | Equal | No | DISCARD (no value added) | | Worse | — | DISCARD | | Crash | — | DISCARD | | Marginal (+/- 2%) | — | Spawn reviewer to assess code quality → KEEP or DISCARD |

MODIFY decision: If metrics improved but prediction wasn't fully met, the hypothesis direction is right but needs refinement. Keep the commit, and the next hypothesis should build on it.

On KEEP:

# Branch advances (commit stays)
# Update "best" in notebook if new best

On DISCARD:

git reset --hard HEAD~1   # Revert to last keep point

On MODIFY:

# Commit stays, next hypothesis refines it

Record full results in the notebook: metrics, decision, learnings.

Cross-iteration analysis: Every 5 experiments, spawn a scientist agent to analyze the notebook for patterns and suggest the next hypothesis direction.

Step 6: CHECK EXIT CONDITIONS

| Condition | Action | |-----------|--------| | All success criteria met | EXIT → produce report | | Budget exhausted (N experiments or time limit) | EXIT → produce report with best result | | Human interrupts (stop/cancel) | EXIT → produce report with current state | | Stuck (5+ consecutive discards with no new learnings) | Spawn architect for strategic pivot, or escalate to human | | None of the above | GOTO Step 2 |

Step 7: EXIT REPORT

Produce a structured summary:

═══ Experiment Report ═══
Goal: {goal}
Result: {ACHIEVED / NOT ACHIEVED / PARTIAL} ({summary})

Experiments: {total} total ({keep} keep, {discard} discard, {crash} crash)
Best: {best_experiment_id} ({best_metrics})
Baseline: {baseline_metrics}
Improvement: {percentage change}
Simplicity: net {+/-N} lines vs baseline

Top learnings:
1. {learning from most impactful experiment}
2. {learning from notable failure}
3. {cross-cutting insight}

Files changed: {list}
Branch: experiment/{tag}
Notebook: .omcp/experiment-notebook.json

Then run /oh-my-copilot:cancel for clean state cleanup.

</Steps>

<Experiment_Notebook_Schema>

{
  "goal": "string — what we're optimizing",
  "successCriteria": [
    { "metric": "string", "operator": "<|>|<=|>=|==", "target": "number" }
  ],
  "measurementCommand": "string — command to run",
  "budget": { "type": "experiments|time|scope", "limit": "number|string" },
  "branch": "string — experiment branch name",
  "baseline": {
    "commit": "string — 7-char hash",
    "metrics": { "metric_name": "number" },
    "timestamp": "ISO string"
  },
  "best": {
    "commit": "string",
    "experimentId": "string",
    "metrics": { "metric_name": "number" },
    "improvement": "string — human-readable summary"
  },
  "experiments": [
    {
      "id": "EXP-001",
      "hypothesis": "string — what we're testing",
      "prediction": "string — expected outcome",
      "rationale": "string — why this should work",
      "changes": ["file1.ts", "file2.ts"],
      "commit": "string — 7-char hash",
      "status": "keep|discard|modify|crash|in_progress",
      "escalated": false,
      "metrics": { "metric_name": "number" },
      "vsBaseline": "string — comparison",
      "vsBest": "string — comparison",
      "linesChanged": "number",
      "learnings": "string — what we learned",
      "crashLog": "string | null — if crashed",
      "timestamp": "ISO string"
    }
  ]
}

</Experiment_Notebook_Schema>

<Simplicity_Criterion> "All else being equal, simpler is better."

• Code deletions that maintain metrics are wins
• Track linesChanged per experiment
• When two experiments have equal metrics, prefer the one with fewer lines changed
• Minimal improvements (< 1%) don't justify significant complexity additions </Simplicity_Criterion>

<Examples> <Good> Hypothesis informed by prior experiments: ``` EXP-005 hypothesis: "Connection pooling will compound with batching (EXP-001) because both reduce per-message I/O overhead, and pooling eliminates the connection setup cost that batching doesn't address."

This references EXP-001's learning about batching and builds on it.

Why good: Each hypothesis builds on accumulated learnings.
</Good>

<Good>
Adaptive team escalation:

EXP-003: Single executor failed — couldn't coordinate changes across handler + db layer + config simultaneously. EXP-003 (retry): Spawned team with executor-1 (handler), executor-2 (db), tester-3 (integration test). Team succeeded. Notebook: { "escalated": true, "status": "keep" }

Why good: Detected implementation complexity, escalated to team, tracked it.
</Good>

<Good>
Clean discard with learnings:

EXP-002 result: p95_latency_ms = 190 (worse than baseline 200? No, better, but worse than EXP-001's 170) Decision: DISCARD (does not improve on current best) Learnings: "Lock-free queue adds overhead for small batch sizes. Contention is not the bottleneck — I/O is." git reset --hard HEAD~1

Why good: Reverted cleanly, captured specific learning that informs future hypotheses.
</Good>

<Bad>
Hypothesis without prediction:

"Let's try caching and see what happens"

Why bad: No specific prediction. Impossible to evaluate whether the hypothesis was right or wrong.
</Bad>

<Bad>
Skipping learnings on discard:

EXP-004: metrics worse. Discard. (no learnings recorded)

Why bad: The whole point of the loop is to learn from failures. Every discard should explain WHY it failed.
</Bad>

<Bad>
Multiple hypotheses in one experiment:

EXP-005: "Add caching AND connection pooling AND async writes"

Why bad: If metrics improve, which change helped? If they worsen, which hurt? One hypothesis at a time.
</Bad>
</Examples>

<Escalation_And_Stop_Conditions>
- Stop and report when success criteria are met — produce exit report
- Stop when budget is exhausted — produce exit report with best result
- Stop when human interrupts — run `/oh-my-copilot:cancel`
- Continue when hook sends "The boulder never stops" — this means keep experimenting
- Escalate to human when same crash occurs 3+ times
- Escalate to human when 5+ consecutive discards yield no new learnings (loop is stuck)
- Spawn architect for strategic pivot before escalating to human
</Escalation_And_Stop_Conditions>

<Final_Checklist>
- [ ] experiment-notebook.json exists with baseline and all experiments logged
- [ ] Every experiment has: hypothesis, prediction, metrics, learnings
- [ ] No in_progress experiments remain (all resolved to keep/discard/modify/crash)
- [ ] Git branch is clean (no uncommitted changes)
- [ ] Discarded experiments were reverted (git reset)
- [ ] Best result is recorded in notebook
- [ ] Exit report produced with summary, learnings, and file list
- [ ] `/oh-my-copilot:cancel` run for clean state cleanup
</Final_Checklist>