mims-harvard/devtu-self-evolve

ToolUniverse Self-Evolution Orchestrator

Coordinates the full development lifecycle by dispatching to specialized devtu skills.

The Cycle

Discover → Create → Test → Fix → Optimize → Ship → Repeat

Each phase maps to a dedicated skill:

| Phase | Skill | What it does | |-------|-------|-------------| | Discover | devtu-auto-discover-apis | Gap analysis, web search for APIs, batch discovery | | Create | devtu-create-tool | Build tool class + JSON config + test examples | | Test | (this skill) | Launch researcher persona agents to find issues | | Fix | devtu-fix-tool | Diagnose failures, implement fixes, validate | | Optimize | devtu-optimize-skills | Improve skill reports, evidence handling, UX | | Optimize | devtu-optimize-descriptions | Improve tool JSON descriptions for clarity | | Docs | devtu-docs-quality | Validate documentation accuracy | | Ship | devtu-github | Branch, commit, push, create PR |

Quick Start

Pick an entry point based on what's needed:

• "Run a test round" → jump to Testing Phase
• "Expand coverage" → invoke Skill(skill="devtu-auto-discover-apis")
• "Create a new tool" → invoke Skill(skill="devtu-create-tool")
• "Fix a broken tool" → invoke Skill(skill="devtu-fix-tool")
• "Improve skills" → invoke Skill(skill="devtu-optimize-skills")
• "Full cycle" → follow all phases below in order

---

Phase 1: Discovery (optional)

Invoke Skill(skill="devtu-auto-discover-apis") to:

1. Run gap analysis on current tool categories
2. Search for life science APIs in underrepresented domains
3. Score and prioritize APIs by coverage, reliability, documentation

Phase 2: Tool Creation (optional)

Invoke Skill(skill="devtu-create-tool") for each new API:

1. Create Python tool class implementing the API
2. Create JSON config with parameters, descriptions, test examples
3. Register in _lazy_registry_static.py and default_config.py
4. Validate: python -m tooluniverse.cli test <ToolName>

Phase 3: Testing Phase

This is the core testing loop, run directly by this skill.

Setup

1. Check for open PRs: gh pr list --state open
2. If unmerged PR → use that branch; if merged → new branch from origin/main
3. Rebase: git fetch origin && git rebase origin/main

Researcher Persona Agents

Launch 2 agents per round (A + B) using the Agent tool with these parameters:

Each agent gets:

• Domain specialty (oncology, genomics, pharmacology, etc.)
• Research question (specific biological question)
• 5-7 test scenarios exercising different tools
• Instructions to report issues with severity (HIGH/MEDIUM/LOW)
• Issue IDs: Feature-{round}{letter}-{num} (e.g., Feature-59A-001)

Agent prompt template — see references/persona-template.md

Verification (CRITICAL)

Before implementing ANY agent-reported issue, verify via CLI:

python3 -m tooluniverse.cli run <ToolName> '<json_args>'

50%+ of agent reports are false positives from MCP interface confusion. Only fix verified issues.

Fix Principles

1. Prevent, don't recover — fix root cause, not symptoms
2. Validate at input — reject bad params early with clear guidance
3. Distinguish "no data" from "bad query" — different messages for each
4. Fix the abstraction — don't add alias lists that grow forever

Anti-patterns: hint text instead of validation, parameter aliases instead of fixing naming, post-hoc probing instead of pre-validation.

Skill Usefulness Testing (NEW — beyond tool testing)

Standard testing verifies tools work. Usefulness testing verifies skills actually solve scientist problems. Run this after standard testing:

1. Pick a real research question that the skill claims to answer (not a tool-level test)
2. Launch an agent following the skill workflow on the real question
3. Assess honestly: Does the skill produce an actionable answer, or just a data dump?

Score 1-10 rubric:

• 1-3: Tool catalog — lists tools without interpretation
• 4-6: Data collector — gathers data but doesn't help combine/interpret
• 7-8: Reasoning framework — guides interpretation with tables/scoring/synthesis
• 9-10: Decision engine — produces concrete, defensible recommendations

Common failure patterns found in usefulness tests:

| Pattern | Score Impact | Fix | |---------|-------------|-----| | "Call A, then B, then C" without explaining what to DO with results | -3 | Add interpretation tables | | Tool params wrong (tool works but skill documents wrong names) | -2 | Verify ALL tool params via get_tool_info() | | Promises data the API can't deliver (e.g., DepMap CRISPR scores) | -2 | Be honest about limitations; add computational procedure workaround | | No synthesis phase at the end | -2 | Add "so what?" phase that combines all evidence | | No evidence grading | -1 | Add T1-T4 or similar confidence tiers | | No computational procedures for things tools can't do | -1 | Add Python code blocks using scipy/pandas/numpy |

When tools can't help, add computational procedures: Some analyses need Python code, not API calls. Skills should include working code blocks for:

• Statistical testing (scipy.stats, FDR correction)
• Data analysis from downloaded files (pandas + CSV from DepMap, TCGA, etc.)
• Scoring algorithms (ACMG classification, viability scores)
• Sequence analysis (Biopython)

See devtu-optimize-skills Patterns 14-15 for full guidance.

Phase 3.5: Benchmark Evaluation

Quantify plugin performance after testing. Uses Skill(skill="devtu-benchmark-harness").

# Run lab-bench (20 MCQ)
python skills/devtu-benchmark-harness/scripts/run_eval.py --benchmark lab-bench --mode plugin-only --n 20

# Run BixBench (computational, use first 20)
python skills/devtu-benchmark-harness/scripts/run_eval.py --benchmark bixbench --mode plugin-only --n 20

# Analyze results
python skills/devtu-benchmark-harness/scripts/analyze_results.py --results <results-file>

# Generate report
python skills/devtu-benchmark-harness/scripts/generate_report.py --results <results-file> --output BENCHMARK_REPORT.md

Compare with previous round. If any category regresses, prioritize fixing that skill/tool in Phase 4.

Phase 4: Fix & Commit

1. Implement verified fixes (see references/bug-patterns.md for code-level patterns)
2. Run code-simplifier: Skill(skill="simplify") — always after writing or modifying code
3. Lint: ruff check src/tooluniverse/<file>.py
4. Verify syntax: python -c "from tooluniverse.<module> import <Class>"
5. Test: python -m tooluniverse.cli run <Tool> '<json>'
6. Pre-commit hook pattern: stage → commit (fails, reformats) → re-stage → commit
7. Push: git push origin <branch>

Also see Skill(skill="devtu-code-optimization") for reusable fix patterns and anti-patterns.

Phase 5: Optimize (optional)

After fixes are stable:

• Skill(skill="devtu-optimize-descriptions") — improve tool descriptions
• Skill(skill="devtu-optimize-skills") — improve research skill quality
• Skill(skill="devtu-docs-quality") — validate docs accuracy

Phase 6: Ship

Invoke Skill(skill="devtu-github") or manually:

1. Rebase: git fetch origin && git stash && git rebase origin/main && git stash pop
2. git push --force-with-lease origin <branch>
3. Create or update PR: gh pr create / verify with gh pr view <N> --json mergeable
4. Verify "mergeable": "MERGEABLE" before reporting done

GitHub repo: mims-harvard/ToolUniverse — always verify with git remote -v before pushing.

---

Git Rules (CRITICAL)

• NEVER push to main — all work on feature branches
• NEVER have multiple open fix PRs — keep adding to current branch
• Always rebase before push: git fetch origin && git rebase origin/main
• Commit message format: no "BUG" terminology, use "Feature" or "Fix"
• No AI attribution in commits

Common Issue Categories

| Category | Signal | |----------|--------| | Silent parameter miss | Wrong-field check; param ignored | | Always-fires conditional | .get("field") on wrong type | | Silent normalization | Auto-transform not disclosed | | Wrong notation/case | Gene fusions, Title Case names | | Substring match | Short symbol returns multiple targets | | try/except indent | Mismatched → SyntaxError |

Full patterns → references/bug-patterns.md

Round Tracking

After each round: advance counter, update patterns file, keep this SKILL.md under 150 lines.

Current round: 127 (rounds completed: 52-126)