sakanaai/shinka-run

Shinka Run CLI Skill

Run a batch of program mutations using ShinkaEvolve's CLI interface.

When to Use

Use this skill when:

• evaluate.py and initial.<ext> already exist
• The user wants to run code evolution using the ShinkaEvolve/Shinka library
• You want configurable program evolution runs using explicit CLI args

Do not use this skill when:

• You need to scaffold a new task from scratch (use shinka-setup)

What is ShinkaEvolve?

A framework developed by SakanaAI that combines LLMs with evolutionary algorithms to propose program mutations, that are then evaluated and archived. The goal is to optimize for performance and discover novel scientific insights.

Repo and documentation: https://github.com/SakanaAI/ShinkaEvolve Paper: https://arxiv.org/abs/2212.04180

Workflow

1. Inspect task directory

ls -la <task_dir>

Confirm evaluate.py and initial.<ext> exist.

2. Inspect CLI reference quickly

shinka_run --help

3. Check model availability before proposing a run

shinka_models
shinka_models --verbose

Validate the exact run config against shinka_models:

• Mutation models: every entry in evo.llm_models must appear in the llm list.
• Meta recommendation models: if evo.meta_rec_interval is set and evo.meta_llm_models is set, every meta model must appear in the llm list.
• Prompt evolution models: if evo.evolve_prompts=true, use evo.prompt_llm_models when provided, otherwise evo.llm_models; every selected model must appear in the llm list.
• Embedding model: if evo.embedding_model is set, it must appear in the embedding list.
• Local OpenAI-compatible models are allowed for LLMs and embeddings via local/<model>@http(s)://host[:port]/v1, and these local models are not expected to appear in shinka_models.

Important runtime rules:

• Do not assume meta recommendations fall back to evo.llm_models. In the current runner, meta recommendations are only enabled when evo.meta_llm_models is explicitly set.
• Prompt evolution does fall back to evo.llm_models when evo.prompt_llm_models is unset.
• Treat local/<model>@http(s)://host[:port]/v1 values as an explicit exception to the shinka_models membership check. Instead, confirm the local endpoint URL and serving status separately before running.
• If any required model is missing from shinka_models, stop and ask the user to either change the config or set the missing credentials first.

4. Confirm first-batch configuration with the user

• Minimum: budget scope, generation count, critical overrides.
• Explicitly confirm the mutation LLMs, meta recommendation LLMs, prompt evolution LLMs, and embedding model after checking them against shinka_models.
• If unclear, ask before running.
• Do not override any non-confirmed arguments.

5. Launch main run with explicit knobs

shinka_run \
  --task-dir <task_dir> \
  --results_dir <results_dir> \
  --num_generations 40 \
  --set db.num_islands=3 \
  --set job.time=00:10:00 \
  --set evo.task_sys_msg='<task-specific system message guiding search>'\
  --set evo.llm_models='["gpt-5-mini","gpt-5-nano"]' \
  --set evo.meta_llm_models='["gpt-5-mini"]' \
  --set evo.prompt_llm_models='["gpt-5-mini"]' \
  --set evo.embedding_model='text-embedding-3-small' \
  # Concurrency settings for parallel sampling and evaluation
  --max-evaluation-jobs 2 \
  --max-proposal-jobs 2 \
  --max-db-workers 2

6. Verify outputs before handoff

ls -la <results_dir>

Expect artifacts like run log, generation folders, and SQLite DBs.

7. Between-batch handoff (unless explicitly autonomous)

• Summarize outcomes from the finished batch.
• Ask user for the next batch config before running again.
• Explicitly ask: "What new directions should we push next batch? Please include algorithm ideas, constraints, and failure modes to avoid."
• Turn user feedback into a revised system prompt and pass it via --set evo.task_sys_msg=... in the next shinka_run call.
• If the prompt is long/multiline, put it in a config file and use --config-fname instead of shell-escaping.
• Unless the user explicitly wants a fresh run/fork, keep the same --results_dir for follow-up batches.

Example next-batch command with feedback-driven prompt:

shinka_run \
  --task-dir <task_dir> \
  --results_dir <results_dir> \
  --num_generations 20 \
  --set evo.task_sys_msg='<new system prompt derived from user feedback>' \
  --set db.num_islands=3

Batch Control Policy (Required)

Treat one shinka_run invocation as one batch of program evaluations/generations.

• Default mode: human-in-the-loop between batches.
• After each batch and before the first, always ask the user what configuration to run next (budget, --num_generations, model/settings overrides, concurrency, islands, output path).
• Do not start the next batch until the user confirms the next config.
• Keep --results_dir fixed across continuation batches so Shinka can reload prior results.
• Exception: if the user explicitly asks for fully autonomous execution, you may continue across batches without re-asking between runs.