lebsral/ai-watching-optimization

Watch What Happens During Optimization

Guide the user through monitoring their optimizer.compile() runs so they can see progress, catch problems early, and know when to stop.

The problem

You run optimizer.compile(program, trainset=trainset) and wait. Minutes pass. Sometimes hours. You have no idea if scores are improving, if the optimizer is stuck, or if you should stop and try something different.

This skill helps you pick the right monitoring approach and interpret what you see.

What you can observe

| Observable | Why it matters | Tools that show it | |-----------|---------------|-------------------| | Scores over time | Are they improving? Plateauing? Dropping? | All tools below | | Instructions evolving | What is the optimizer changing in your prompts? | GEPA logger (prompt diffs), LangWatch (predictor states) | | Cost accumulating | How much are you spending? Worth continuing? | LangWatch, MLflow | | Convergence pattern | Should you stop early or keep going? | All tools below (scores over time) | | LM calls | Is the optimizer calling the right model? | inspect_history, LangWatch, MLflow | | Acceptance decisions | Why did the optimizer accept/reject a candidate? | GEPA logger (event log), BaseCallback |

Pick your tool

Decision tree

1. Using GEPA?
   • Want a web dashboard? -> dspy-gepa-logger (Option 2)
   • Just want basic stats? -> track_stats=True (Option 1)
2. Using MIPROv2, BFS, BRS, or COPRO?
   • Want a cloud dashboard with live scores? -> LangWatch (Option 3)
   • Want a local dashboard? -> MLflow (Option 4)
   • Just want console output? -> BaseCallback (Option 1)
3. Any optimizer, post-hoc?
   • inspect_history() always works (Option 1)

Comparison

| Tool | Optimizers | Setup | Dashboard | Local/Cloud | |------|-----------|-------|-----------|-------------| | Built-in (track_stats) | GEPA only | One flag | No (dict) | Local | | Built-in (BaseCallback) | All | ~20 lines | No (console) | Local | | inspect_history(n) | All (post-hoc) | Zero setup | No (console) | Local | | dspy-gepa-logger | GEPA only | pip install | Yes (web) | Local | | LangWatch | BFS/BRS/COPRO/MIPROv2 | pip install + API key | Yes (cloud) | Cloud | | MLflow | All (via autolog) | pip install | Yes (local) | Local |

Option 1 - Built-in (zero dependencies)

Always do this - baseline evaluation

Run dspy.Evaluate before AND after optimization. Without a baseline, you cannot tell if optimization helped.

from dspy.evaluate import Evaluate

evaluator = Evaluate(devset=devset, metric=your_metric, num_threads=8)

# Before optimization
baseline_score = evaluator(program)
print(f"Baseline: {baseline_score}")

# Run optimizer
optimized = optimizer.compile(program, trainset=trainset)

# After optimization
optimized_score = evaluator(optimized)
print(f"Optimized: {optimized_score}")
print(f"Improvement: {optimized_score - baseline_score:+.1f}")

GEPA - track_stats=True

GEPA supports a built-in stats flag that records detailed results per iteration.

optimizer = dspy.GEPA(
    metric=your_metric,
    task_lm=task_lm,
    reflection_lm=reflection_lm,
    track_stats=True,  # Enable tracking
)

optimized = optimizer.compile(program, trainset=trainset)

# Inspect results after compilation
stats = optimizer.detailed_results
for iteration, result in enumerate(stats):
    print(f"Iteration {iteration}: score={result['score']:.3f}")

Any optimizer - custom BaseCallback

Write a callback that prints progress as the optimizer evaluates candidates.

import dspy

class OptimizationProgressCallback(dspy.BaseCallback):
    def __init__(self):
        super().__init__()
        self.eval_count = 0

    def on_evaluate_end(self, instance, inputs, outputs, exception):
        self.eval_count += 1
        score = outputs.get("score", None)
        if score is not None:
            print(f"[Eval {self.eval_count}] Score: {score:.3f}")

# Register the callback
progress = OptimizationProgressCallback()
dspy.configure(callbacks=[progress])

# Now run your optimizer -- progress prints automatically
optimized = optimizer.compile(program, trainset=trainset)

Post-hoc - inspect_history

After optimization completes (or if you interrupt it), inspect recent LM calls.

# Show the last 5 LM calls
dspy.inspect_history(n=5)

This shows the full prompt and completion for each call, useful for verifying the optimizer is calling the correct model and seeing what instructions it tried.

Option 2 - GEPA Logger (web dashboard)

A drop-in replacement for GEPA's internal logger that adds a web dashboard with real-time stats, eval tables, and prompt diffs.

pip install dspy-gepa-logger

import dspy
from dspy_gepa_logger import GEPALogger

# Create the logger (starts web dashboard on port 3000)
logger = GEPALogger()

optimizer = dspy.GEPA(
    metric=your_metric,
    task_lm=task_lm,
    reflection_lm=reflection_lm,
)

# Register the logger as an observer
optimizer.register_observer(logger)

optimized = optimizer.compile(program, trainset=trainset)

The dashboard shows:

• Real-time scores as iterations progress
• Eval tables with per-example pass/fail
• Prompt diffs showing how instructions changed between iterations
• LM call logs with full prompts and completions
• Event timeline with 8 event types (SeedValidation, Reflection, ValsetEval, etc.)

The web dashboard requires Node.js and uses SQLite for storage.

For detailed GEPA usage, see /dspy-gepa.

Option 3 - LangWatch (cloud dashboard)

LangWatch patches DSPy optimizers to stream live progress to a cloud dashboard. This is the only tool that shows real-time optimizer progress for non-GEPA optimizers.

pip install langwatch

import langwatch
import dspy

langwatch.dspy.init(experiment="my-optimization-run")

optimizer = dspy.MIPROv2(metric=your_metric, auto="light")
optimized = optimizer.compile(program, trainset=trainset)

The LangWatch dashboard shows:

• Live scores updating as each candidate is evaluated
• Cost tracking so you know how much the optimization is spending
• Predictor states showing the current instructions and demos for each module
• Run comparison across multiple optimization experiments

Supported optimizers: BootstrapFewShot, BootstrapFewShotWithRandomSearch, COPRO, MIPROv2.

Not supported: GEPA (use dspy-gepa-logger instead).

Requires a LangWatch API key. Free tier available at app.langwatch.ai.

For detailed setup, see /dspy-langwatch.

Option 4 - MLflow (local dashboard)

MLflow's DSPy autolog captures optimization as parent/child runs with traces.

pip install mlflow

import mlflow
import dspy

mlflow.dspy.autolog(log_compiles=True)

# Set up experiment
mlflow.set_experiment("my-optimization")

with mlflow.start_run():
    optimizer = dspy.MIPROv2(metric=your_metric, auto="light")
    optimized = optimizer.compile(program, trainset=trainset)

# View results
mlflow ui
# Open http://localhost:5000

The MLflow dashboard shows:

• Parent run for the optimization, child runs for each candidate
• Metrics logged per candidate (score, parameters tried)
• Traces tab showing individual LM calls
• Artifacts including the optimized program state

Works with all optimizers via autolog.

For detailed setup, see /dspy-mlflow.

Interpreting what you see

This is the most important section. Knowing which tool to use is step one. Knowing what the data means is what saves you hours.

Converging?

• Scores climbing then plateauing -- good. The optimizer found improvements and saturated. You can stop.
• Scores climbing steadily -- keep going. The optimizer is still finding improvements.
• Flat from the start -- the optimizer is not finding better candidates. Possible causes:
  • Your metric is saturated (already near 100%)
  • Your metric is broken (always returns the same score)
  • The search space is too constrained
  • The task is too easy for optimization to help

Overfitting?

If you have a separate validation set:

• Train score up, val score also up -- good. Real improvement.
• Train score up, val score flat or dropping -- overfitting. The optimizer is memorizing training examples.
  • Fix: use a larger, more diverse trainset
  • Fix: reduce the optimization budget (fewer iterations)
  • Fix: keep train and val sets strictly separate

# Always evaluate on held-out data
eval_train = Evaluate(devset=trainset, metric=your_metric)
eval_val = Evaluate(devset=valset, metric=your_metric)

train_score = eval_train(optimized)
val_score = eval_val(optimized)
print(f"Train: {train_score}, Val: {val_score}")
if train_score - val_score > 10:
    print("Warning: possible overfitting")

Stuck?

• Same score for many iterations -- the optimizer cannot find better candidates.
  • GEPA: check if the reflection LM is providing useful feedback. Try a stronger reflection model.
  • MIPROv2: try a heavier preset (auto="medium" or auto="heavy").
  • BootstrapFewShot: increase max_bootstrapped_demos or max_labeled_demos.
  • Any optimizer: check if your metric has enough granularity (binary pass/fail gives less signal than a 0-1 float score).

When to stop early?

• Plateaued for 5+ iterations -- diminishing returns. Stop and use the best so far.
• Most improvement happens in the first 30-50% of budget -- if you have seen minimal gains in the second half, further iterations are unlikely to help.
• Cost per improvement point increasing -- track cost alongside score. When each additional point costs 10x more than the previous, you have hit practical limits.

Wrong model being called?

A common bug: the optimizer or your program is calling a different LM than you intended.

• inspect_history: look at the model name in the prompt headers
• LangWatch dashboard: shows which model each call uses
• Fix: make sure you called dspy.configure(lm=...) and that per-module LM assignments are correct

# Check what model was actually used
dspy.inspect_history(n=1)
# Look for the model identifier in the output

Cost signals

• Track total cost as optimization runs. Compare cost-per-point-of-improvement across iterations.
• If early iterations gave +5 points for $2, and recent iterations gave +0.5 points for $2, you are in diminishing returns territory.
• LangWatch shows cost natively. For other tools, estimate from token counts and model pricing.

Gotchas

1. track_stats=True is GEPA-specific. It does not work with MIPROv2 or other optimizers. Use BaseCallback or an external tool for those.
2. LangWatch does not support GEPA. LangWatch patches BFS/BRS/COPRO/MIPROv2 internals. For GEPA monitoring, use dspy-gepa-logger.
3. Always run baseline evaluation. Without a before score, you cannot tell if optimization helped. This is the most common mistake.
4. Flat score does not mean the optimizer is broken. It could mean your metric is saturated, your metric is broken (always returns the same value), or the task does not benefit from optimization.
5. inspect_history() is post-hoc, not live. It shows recent LM calls from memory. It does not stream progress during optimization.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• GEPA optimizer details -- see /dspy-gepa
• MIPROv2 optimizer details -- see /dspy-miprov2
• LangWatch setup and features -- see /dspy-langwatch
• MLflow setup and features -- see /dspy-mlflow
• Arize Phoenix tracing -- see /dspy-phoenix
• Experiment tracking for comparing completed runs -- see /ai-tracking-experiments
• Improving accuracy for the full measure-improve-verify loop -- see /ai-improving-accuracy
• Cutting costs when optimization is too expensive -- see /ai-cutting-costs
• For worked examples, see examples.md
• Install /ai-do if you do not have it -- it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do