mohitagw15856/model-migration-plan

Model Migration Plan Skill

A model swap changes every output of your feature at once. This skill plans the migration like the risky deploy it is: eval first, shadow second, canary third — with numbers, not vibes, deciding each promotion.

What This Skill Produces

• A phased migration plan (eval → shadow → canary → full) with promotion criteria per phase
• Prompt adaptation notes — what typically shifts between models and what to re-tune
• Rollback triggers and the mechanics of rolling back fast
• A cost/latency delta forecast for the new model

Required Inputs

Ask for (if not already provided):

• Current and target model (and why: deprecation, quality, cost, latency)
• The feature's traffic and blast radius — requests/day, who sees the output, what a bad output costs
• Existing evals — a regression suite (see prompt-regression-suite) or at minimum golden examples; if none exist, phase 0 is building one
• The deadline, if the migration is forced by a deprecation date

Migration Phases

Phase 0 — Baseline. Freeze a regression suite against the current model. Without a baseline, "the new model is fine" is unfalsifiable. Record current cost, latency (p50/p95), and quality scores.

Phase 1 — Offline eval. Run the suite against the target model with the prompt as-is, then with adapted prompts. Promotion criteria: pass rate ≥ baseline, no canary failures, cost/latency within budget. Expect to iterate here — most "model regressions" are prompt-fit issues.

Phase 2 — Shadow. Mirror a sample of real traffic to the new model; log, never serve. Compare distributions: refusal rate, output length, format-violation rate, judge scores on a sample. Duration: long enough to cover weekly traffic patterns.

Phase 3 — Canary. Serve the new model to [1-5]% of traffic behind a flag, tagged in analytics. Watch the same metrics plus user-visible signals (regenerate rate, thumbs-down, support tickets). Widen in steps; each step has the same promotion criteria.

Phase 4 — Full cutover + cleanup. 100% traffic, old model kept warm behind the flag for [period], then removed. Update model pins everywhere (including the eval judge if it referenced the old model), and re-baseline the regression suite on the new model.

Prompt Adaptation Notes

Between model generations, re-check: instruction-following strictness (newer models often follow the letter, exposing sloppy prompts), format compliance (JSON/markdown habits differ), verbosity defaults, refusal boundaries, tool-calling style, and system-prompt sensitivity. Adapt the prompt per model rather than writing to the lowest common denominator — keep per-model prompt versions if both run simultaneously.

Rollback

• Triggers (numbers, set in advance): canary quality below baseline by [X], refusal/format-violation rate above [Y], p95 latency above [Z], or any safety incident.
• Mechanics: the model is a config flag, not a code deploy — rollback is a flag flip taking effect in [minutes]. State who can flip it and how it's tested before the canary starts.

Output Format

Model Migration Plan: [feature] — [current model] → [target model]

Why now: [driver + deadline]. Blast radius: [traffic, audience, cost of a bad output].

| Phase | Gate to pass | Duration | Owner | |---|---|---|---| | 0 Baseline | suite frozen; cost/latency recorded | | | | 1 Offline eval | [criteria] | | | | 2 Shadow | [criteria] | | | | 3 Canary [x]% → [y]% | [criteria] | | | | 4 Cutover + cleanup | [criteria] | | |

Prompt adaptations found/expected: [list]

Rollback: triggers [numbers]; mechanism [flag]; owner [who].

Cost/latency forecast: [current] → [projected], at [traffic].

Quality Checks

• [ ] Every phase promotion criterion is a number against the recorded baseline
• [ ] Shadow phase compares distributions, not anecdotes ("outputs look good" is not a gate)
• [ ] Rollback is a config flip with a named owner, tested before canary
• [ ] The plan re-baselines the regression suite after cutover — the new model becomes the new normal
• [ ] Deprecation deadlines leave slack for at least one failed phase-1 iteration

Anti-Patterns

• [ ] Do not skip shadow because offline evals passed — real traffic finds what golden sets miss
• [ ] Do not migrate the feature and the prompt redesign in one change — you won't know which moved the metrics
• [ ] Do not compare models with an unpinned judge, or a judge that is the target model grading itself
• [ ] Do not leave the old model path in code indefinitely "just in case" — set the removal date in the plan
• [ ] Do not treat a cheaper model as free savings without re-checking quality at the tails, not just the mean