A metric-gated autonomous experiment loop lets an AI agent run many bounded experiments without human supervision. The human defines the arena: the editable files, the fixed evaluation harness, the metric to improve, the time budget, and the rules for keeping or discarding changes. The agent proposes code changes, runs the experiment, reads the metric, records the result, and either advances the branch or reverts.
Karpathy’s autoresearch repository demonstrates the pattern with small language-model training. The setup deliberately narrows the environment to three important files: fixed data preparation and evaluation code, one training script the agent may edit, and a Markdown program that tells the agent how to operate. Each run receives a fixed five-minute training budget and is judged by validation bits per byte, so changes to architecture, optimizer, batch size, and schedules can be compared on one machine.
The transferable idea is not specific to language-model training. Any domain with a reliable scalar metric, a repeatable harness, a bounded edit surface, and reversible state can use the same loop. The pattern sits between coding-agent orchestration and black-box optimization: the agent searches by editing source code, but the keep/discard decision is grounded in measured behavior rather than persuasion.
The loop is only meaningful if the agent cannot move the goalposts. Data loading, evaluation, metric calculation, run timeout, and constants that define comparability should live outside the editable region. The agent can change implementation details, but not the judge.
The editable surface should be small enough for the agent to understand and for humans to review afterward. Autoresearch uses a single training file. Other systems might allow one prompt file, one ranking function, one SQL query, or one policy module. A smaller surface reduces accidental coupling and makes rollback simple.
Every experiment produces a decision: keep, discard, or crash. A kept change lowers the target metric or improves it enough to justify complexity. A discarded change is logged but reverted. A crash is either fixed if trivial or recorded as a failed idea. The branch history becomes the surviving lineage of measured improvements.
The agent can be creative about proposals, but the environment supplies the judgment function. That separation is what makes the loop safe enough to run while the human is absent.
Before the loop starts, the system verifies dependencies, prepares data, creates a fresh branch, initializes a result log, and records the baseline. The first run matters because it anchors every later keep/discard decision. If the baseline is noisy or unreproducible, the loop will chase randomness.
# results.tsv
commit metric memory_gb status description
3f21a9b 0.997900 44.0 keep baselineThe agent edits the allowed file with one hypothesis, commits the attempt, runs the fixed command, extracts the metric, and records the outcome. If the metric improves, the commit remains as the new base. If it regresses, the branch resets to the previous best while the failed idea remains in the external log.
while true; do
base=$(git rev-parse --short HEAD)
agent_edit_train_py_with_one_idea
git add train.py && git commit -m "experiment: describe idea"
timeout 10m uv run train.py > run.log 2>&1
metric=$(grep "^val_bpb:" run.log | awk '{print $2}')
if improved "$metric" best_metric.txt; then
status=keep
echo "$metric" > best_metric.txt
else
status=discard
git reset --hard "$base"
fi
log_result "$base" "$metric" "$status" "short description"
doneThe human-facing program file is the strategy layer. It can tell the agent which kinds of ideas to try, how to trade complexity against metric gains, how to handle crashes, and how to avoid repeated dead ends. Keeping strategy outside the editable experiment file lets teams compare different agent instructions against the same arena.
Small model pretraining is a natural fit because it has repeatable scripts, measurable validation loss, and many possible code-level interventions. Fixed wall-clock budgets make the loop optimize for useful progress on the available hardware rather than abstract benchmark purity.
Search teams can let an agent mutate feature weights, fusion formulas, query rewrites, or reranking prompts while a frozen evaluation suite computes NDCG, recall, latency, and regression checks. The keep gate should include both quality and operational constraints so the agent does not win by making queries too slow.
When a benchmark is deterministic and cheap enough, an agent can explore source-level optimizations overnight. The harness should pin inputs, check correctness, collect latency distributions, and reject changes that improve the mean while breaking tail latency or memory budgets.
Git gives the loop a simple memory model: current best commit, attempted commit, reset on failure, and inspectable history for kept changes. The result log should include discarded ideas too, because otherwise the agent will rediscover the same bad changes after the branch forgets them.
A tiny improvement that adds brittle code may not be worth keeping. Include a simplicity rule in the program: prefer deletions, small diffs, and explainable changes when metric differences are close. For production systems, the gate should include tests, style checks, safety constraints, and reviewability.
def should_keep(new_metric, best_metric, changed_lines, crashed):
if crashed:
return False
delta = best_metric - new_metric
if delta >= 0.005:
return True
if delta >= 0.001 and changed_lines <= 20:
return True
if delta >= 0 and changed_lines < 0:
return True
return FalseDiscarded experiments are valuable training data for the next proposal. A compact result table should preserve the idea, metric, memory, crash reason, and why it was rejected. The agent can periodically read this table to avoid loops and combine near-misses into better hypotheses.