curiositech/automatic-stateful-prompt-improver

Automatic Stateful Prompt Improver

DECISION POINTS

PROMPT ASSESSMENT:
├── Simple question/command (what, when, how)
│   └── Skip optimization → Answer directly
├── Complex task (multi-step, reasoning, technical)
│   ├── Token budget < 1000
│   │   └── APE: 3-5 iterations
│   ├── Token budget 1000-5000
│   │   └── OPRO: 5-10 iterations
│   └── Token budget > 5000
│       └── DSPy compilation: 10-20 iterations
└── Reusable template/system prompt
    └── Full optimization with historical retrieval

OPTIMIZATION TECHNIQUE SELECTION:
├── Instruction rewriting needed
│   └── Use APE (Automatic Prompt Engineer)
├── Parameter tuning with constraints  
│   └── Use OPRO (Optimization by PROmpting)
├── Complex pipeline with multiple modules
│   └── Use DSPy compilation patterns
└── Unknown/exploratory domain
    └── Hybrid APE→OPRO→DSPy cascade

ITERATION CONTROL:
├── Improvement < 1% for 3 rounds → STOP
├── Quality score > 0.95 → STOP  
├── Max iterations reached → STOP
├── User satisfaction confirmed → STOP
└── Continue → Next iteration

FEEDBACK INTEGRATION:
├── Task successful (user confirms/metrics good)
│   └── Record positive feedback + embed for retrieval
├── Task failed/poor quality
│   └── Record negative feedback + analyze failure mode
└── Unclear outcome
    └── Ask user for explicit feedback before recording

FAILURE MODES

Over-Optimization Spiral

• SYMPTOM: Prompt grows to 500+ tokens with many nested constraints
• DIAGNOSIS: Chasing diminishing returns instead of stopping at "good enough"
• FIX: Apply 80/20 rule - if improvement drops below 5% per iteration, stop

Template Obsession

• SYMPTOM: Spending iterations on formatting/structure vs. task clarity
• DIAGNOSIS: Confusing presentation with performance
• FIX: Measure actual task success, not template conformity

Historical Overfitting

• SYMPTOM: Optimized prompt works for past examples but fails on new inputs
• DIAGNOSIS: Training on too narrow a dataset from retrieval
• FIX: Include diverse examples in optimization, test on held-out cases

Capability Misjudgment

• SYMPTOM: Adding extensive scaffolding for tasks model handles natively
• DIAGNOSIS: Assuming model limitations without testing
• FIX: Test baseline capability before heavy prompting

Measurement Blindness

• SYMPTOM: Multiple iterations without clear success metrics
• DIAGNOSIS: Optimizing without knowing what "better" means
• FIX: Define measurable success criteria in first step

WORKED EXAMPLES

Example 1: Code Optimization Request

Original: "Make this code better"

def process_data(data):
    results = []
    for item in data:
        if item > 0:
            results.append(item * 2)
    return results

Decision Point Navigation:

1. Assessment: Complex task (requires code analysis) → Trigger optimization
2. Technique Selection: Code review + improvement → APE with 5 iterations
3. Retrieved Context: Similar code optimization prompts from history
4. Optimized Prompt: "Analyze this Python function for performance, readability, and Pythonic patterns. Identify specific improvements: algorithmic complexity, memory usage, edge cases, and style. Provide refactored code with explanations."

What Novice Misses: Vague "make better" doesn't specify criteria What Expert Catches: Need explicit dimensions (performance, style, edge cases)

Result: Clear analysis of list comprehension opportunity, edge case handling, type hints

Example 2: Reasoning Task Template

Original: "Help me think through this decision"

Decision Point Navigation:

1. Assessment: Reusable template + reasoning task → Full optimization
2. Historical Retrieval: Found decision framework prompts (0.87 similarity)
3. Technique Selection: DSPy compilation (structured reasoning pipeline)
4. Iteration Strategy: 10 rounds, measuring decision quality

Optimized Template:

Decision Analysis Framework:
1. SITUATION: State the decision clearly with constraints
2. STAKEHOLDERS: List affected parties and their interests  
3. OPTIONS: Generate 3-5 distinct alternatives
4. CRITERIA: Define success metrics and weighting
5. TRADE-OFFS: Analyze each option against criteria
6. RECOMMENDATION: Select best option with confidence level

Quality Gates Applied: Template completeness, reusability score, user satisfaction

Example 3: Ambiguous Technical Request

Original: "Set up monitoring"

Decision Point Navigation:

1. Assessment: Underspecified + technical → Trigger optimization
2. Clarification Strategy: OPRO with constraint elicitation
3. Domain Context: Retrieved monitoring setup patterns

Optimized Prompt: "Design monitoring setup by specifying: (1) Infrastructure scope (servers, containers, applications), (2) Key metrics (performance, availability, business), (3) Alert thresholds and escalation, (4) Technology stack constraints, (5) Budget/complexity limits. Provide implementation roadmap with priorities."

Before/After Trade-offs:

• Before: Endless back-and-forth clarification
• After: Structured requirements gathering in single exchange
• Cost: Longer initial prompt
• Benefit: Complete specification in one round

QUALITY GATES

Pre-execution checklist before calling optimize_prompt:

• [ ] Task complexity score > 3 (multi-step/reasoning required)
• [ ] Clear success criteria defined or derivable
• [ ] Token budget estimated and technique selected
• [ ] Domain context identified for retrieval
• [ ] Iteration limit set based on complexity

Post-optimization validation:

• [ ] Optimized prompt is specific and actionable
• [ ] Success metrics are measurable
• [ ] Constraint coherence verified (no contradictions)
• [ ] Token efficiency: improvement justifies added length
• [ ] Historical context integrated appropriately
• [ ] User confirmation obtained for major changes

Quality scoring rubric (0-100):

• Clarity: Can naive user understand requirements? (25 pts)
• Specificity: Concrete vs. abstract instructions? (25 pts)
• Completeness: Covers edge cases and constraints? (25 pts)
• Efficiency: Achieves goals without bloat? (25 pts)

NOT-FOR BOUNDARIES

Do NOT use this skill for:

• Simple factual questions → Answer directly with knowledge
• File operations without reasoning → Use file-management skill
• Direct command execution → Execute immediately
• Conversational responses → Respond naturally
• Already optimized prompts → Check history first to avoid re-optimization
• User explicitly says "don't optimize" → Respect user preference

Delegate instead:

• For mathematical problems → Use calculation-focused skills
• For creative writing → Use creative-writing skill (unless template creation)
• For data analysis → Use data-analysis skill (unless complex reasoning required)
• For debugging → Use debugging skill (unless systematic improvement needed)

Gray areas requiring judgment:

• Medium complexity tasks (score 2-4) → Test baseline performance first
• Domain expertise requests → Optimize only if reusable template potential
• Follow-up questions → Optimize if expanding scope significantly