stevengonsalvez/cost-aware-pipeline

Cost-Aware LLM Pipeline

Model Routing Strategy

Route tasks to the cheapest model that can handle them reliably.

Pricing Reference (per 1M tokens, USD)

| Model | Input | Output | Relative Cost | |-------|-------|--------|---------------| | Haiku | $0.80 | $4.00 | 1x (baseline) | | Sonnet | $3.00 | $15.00 | ~4x | | Opus | $15.00 | $75.00 | ~19x |

Routing Rules

| Task Complexity | Route To | Examples | |-----------------|----------|----------| | Simple (< 100 lines, clear pattern) | Haiku | File renaming, simple search, format conversion, status checks | | Moderate (100-500 lines, some judgment) | Sonnet | Code review, test writing, refactoring, documentation | | Complex (500+ lines, deep reasoning) | Opus | Architecture design, debugging subtle issues, multi-file refactoring | | Creative (open-ended, high quality bar) | Opus | System design, novel algorithms, critical security review |

Implementation with Claude Code Agent Tool

# In agent orchestration, specify model based on task complexity
def route_to_model(task_description: str, estimated_complexity: str) -> str:
    """Return model parameter for Agent tool."""
    routing = {
        "simple": "haiku",
        "moderate": "sonnet",
        "complex": "opus",
        "creative": "opus",
    }
    return routing.get(estimated_complexity, "sonnet")

When spawning sub-agents:

• Use model: "haiku" for Explore agents doing simple file searches
• Use model: "sonnet" (default) for most implementation work
• Use model: "opus" only for architecture reviews, complex debugging, distinguished-engineer tasks

Cost Savings Examples

| Scenario | Before (all Opus) | After (routed) | Savings | |----------|-------------------|----------------|---------| | 10 file searches | $0.15 | $0.008 (haiku) | 95% | | 5 code reviews | $0.75 | $0.20 (sonnet) | 73% | | 1 architecture review | $0.15 | $0.15 (opus) | 0% | | Typical session | $1.05 | $0.36 | 66% |

Narrow Retry Strategy

When an LLM call fails, retry intelligently — not blindly.

Principles

1. Retry the minimum scope — Don't re-run the entire pipeline, retry only the failed step
2. Escalate model on retry — If haiku fails, retry with sonnet, not haiku again
3. Add context on retry — Include the error in the retry prompt
4. Max 2 retries — If it fails 3 times, escalate to human

Retry Escalation Ladder

Attempt 1: haiku (cheapest)
    | fail
    v
Attempt 2: sonnet + error context
    | fail
    v
Attempt 3: opus + error context + examples
    | fail
    v
Escalate to human

Implementation

RETRY_LADDER = [
    {"model": "haiku", "extra_context": None},
    {"model": "sonnet", "extra_context": "Previous attempt failed: {error}"},
    {"model": "opus", "extra_context": "Two attempts failed. Errors: {errors}. Please reason carefully."},
]

async def call_with_retry(prompt: str, task: str) -> str:
    errors = []

    for attempt in RETRY_LADDER:
        try:
            result = await call_llm(
                model=attempt["model"],
                prompt=prompt,
                extra_context=attempt["extra_context"].format(
                    error=errors[-1] if errors else "",
                    errors="; ".join(errors)
                ) if attempt["extra_context"] and errors else prompt,
            )
            return result
        except Exception as e:
            errors.append(str(e))

    raise EscalationNeeded(f"Failed after {len(RETRY_LADDER)} attempts: {errors}")

Prompt Caching Strategies

Structure prompts to maximize cache hits and reduce costs.

Cache-Friendly Prompt Structure

+-------------------------------------+
| STATIC SYSTEM PROMPT (cached)       |  <- Rarely changes, high cache hit rate
| - Agent instructions                |
| - Tool definitions                  |
| - Project context (CLAUDE.md)       |
+-------------------------------------+
| SEMI-STATIC CONTEXT (cached)        |  <- Changes per-task, not per-turn
| - File contents being worked on     |
| - Relevant documentation            |
| - Test results                      |
+-------------------------------------+
| DYNAMIC CONTENT (not cached)        |  <- Changes every turn
| - User message                      |
| - Latest tool results               |
| - Conversation tail                 |
+-------------------------------------+

Key Principles

1. Front-load static content — Put rarely-changing content at the start of the prompt
2. Batch file reads — Read all needed files at once, not one at a time
3. Minimize prompt churn — Avoid reformatting the same content differently each turn
4. Use system prompts for stable context — CLAUDE.md, agent definitions, skill bodies

Cost Impact of Caching

With Anthropic's prompt caching:

• Cached tokens cost 90% less on input
• Cache write costs 25% more (one-time)
• Cache TTL: 5 minutes (refreshed on use)

Strategy: Keep system prompt + project context stable across turns. Only the user message and tool results should change.

Budget Planning

Estimating Session Costs

| Session Type | Typical Tokens | Estimated Cost | |-------------|----------------|----------------| | Quick fix (5 min) | 50K in / 10K out | $0.15 - $0.90 | | Feature implementation (30 min) | 500K in / 100K out | $1.50 - $9.00 | | Deep debugging (1 hr) | 1M in / 200K out | $3.00 - $18.00 | | Multi-agent swarm (2 hr) | 5M in / 1M out | $15.00 - $90.00 |

Setting Budgets

# Per-session cost cap (used by agent-ops skill)
export AGENT_COST_CAP=5.00

# Per-agent cost cap for sub-agents
export SUBAGENT_COST_CAP=1.00

# Warning threshold (fraction of cap)
export AGENT_COST_CAP_WARN=0.80

Monitoring

# View today's costs
grep "$(date +%Y-%m-%d)" "$HOME/{{TOOL_DIR}}/metrics/costs.jsonl" | \
  python3 -c "import sys,json; rows=[json.loads(l) for l in sys.stdin]; \
  print(f'Sessions: {len(set(r[\"session_id\"] for r in rows))}'); \
  print(f'Total: \${sum(r[\"estimated_cost_usd\"] for r in rows):.4f}')"