lamm-mit/autoresearch

autoresearch

Autonomous AI agent that modifies and iteratively improves a GPT language model training setup, running experiments within a 5-minute time budget to optimize validation bits-per-byte.

Code repository

https://github.com/karpathy/autoresearch

Use this as the implementation source: clone the repo and follow its README for install, dependencies, and how to run code or experiments. The generated client prints JSON with a suggested git clone command.

Primary resource (landing page)

https://github.com/karpathy/nanochat

This is the paper or artifact home from DOI/registry metadata — not a JSON API. If this URL is arXiv, the generated client can still fetch live Atom metadata (title, abstract, authors) without a BASE_URL. For other hosts, the client uses stub mode until you set a real BASE_URL for a REST service.

What “running” this client does

The *_client.py script prints JSON that combines a GitHub repository (clone URL + suggested git clone) with optional paper context from arXiv (live Atom metadata when reference_url is arXiv). Run the real code by cloning the repo and following its README — the skill is your agent-facing entrypoint, not a substitute for the repo’s install steps.

To call a REST API instead, set BASE_URL in scripts/autoresearch_client.py or wrap the upstream CLI with subprocess after clone.

How to run the method (from the source)

Extracted for operators and agents. Confirm against the upstream repository or paper before relying on it in production.

Prerequisites

• Single NVIDIA GPU (tested on H100)
• Python 3.10+
• uv project manager

Installation

# 1. Install uv project manager (if you don't already have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install dependencies
uv sync

# 3. Download data and train tokenizer (one-time, ~2 min)
uv run prepare.py

How to run

Manual single training experiment (~5 min):

uv run train.py

Autonomous agent mode:

Point your AI agent (Claude, Codex, etc.) to the program.md file and prompt:

Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.

The agent will autonomously:

1. Read program.md for instructions
2. Modify train.py (hyperparameters, architecture, optimizer, batch size, etc.)
3. Run training for exactly 5 minutes
4. Evaluate using val_bpb (validation bits per byte)
5. Keep or discard changes based on improvement
6. Repeat autonomously

Configuration

Key files to understand:

• prepare.py — Fixed constants, one-time data prep (downloads training data, trains BPE tokenizer), runtime utilities. Do not modify.
• train.py — Single file edited by the agent. Contains GPT model, optimizer (Muon + AdamW), training loop. Fair game: architecture, hyperparameters, batch size, optimizer settings.
• program.md — Baseline instructions for agents. Edit this to customize agent behavior and research setup.

Training constraints:

• Fixed 5-minute time budget (wall clock, excluding startup/compilation) regardless of compute platform
• Metric: val_bpb (validation bits per byte, lower is better, vocab-size-independent)
• Expected frequency: ~12 experiments/hour, ~100 experiments overnight

For smaller compute platforms (MacBook, etc.), tune in prepare.py and train.py:

• Use smaller dataset (e.g., TinyStories)
• Decrease vocab_size (from 8192 to 4096, 2048, or 256 bytes)
• Lower MAX_SEQ_LEN (down to 256)
• Reduce EVAL_TOKENS for faster validation
• Lower DEPTH (default 8, try 4)
• Use WINDOW_PATTERN: "L" instead of "SSSL"
• Reduce TOTAL_BATCH_SIZE to 2**14 (~16K) or lower

Refer to notable forks for CPU/MacOS/Windows/AMD variants.

The same text lives in scripts/USAGE.md for tools that prefer reading files under scripts/.

Parameters

--time-budget (int) [optional, default=5] Fixed wall-clock training duration in minutes (default: 5) --metric (str) [optional, default=val_bpb] Optimization metric: val_bpb (validation bits per byte, lower is better)

Usage

python3 scripts/autoresearch_client.py uv run train.py

Example Output

{"val_bpb": 1.234, "epoch": 1, "loss": 2.567}