Andrej Karpathy has a talent for explaining complicated things in a way that makes you wonder why anyone found them complicated in the first place. His latest project, autoresearch, is almost offensively simple: hand an AI agent a training script, tell it to make the script better, and go to bed.
That’s it. That’s the whole thing.
You wake up to a hundred experiments, neatly logged. The winners survived. The losers got into oblivion. The commit history reads like the lab notebook of a tireless postdoc with no ego and excellent taste — which, if you squint, is exactly what it is.
Three files. One prompt. No framework. The absence of machinery is not laziness. It’s the entire point.
The shape of the loop
Every system worth studying has a loop at its center. This one’s is almost suspiciously clean.
The agent reads the code. It has an idea — maybe a different activation function, maybe a tweak to the learning rate schedule, maybe something genuinely unhinged. It commits the change. It trains for exactly five minutes. It checks one number: validation bits per byte. Number went down? Keep the commit. Number didn’t? Erase it. Pretend it never happened.
Then it does it again. And again. And again. While you dream about whatever it is engineers dream about. Probably uptime.
The five-minute time budget is doing more work than it looks. Because every experiment gets the same wall-clock allocation, the agent can change anything — model size, optimizer, architecture — and the results stay comparable. No apples-to-oranges. Just one question: what’s the best model you can cook in five minutes on this GPU?
Git is the state machine. Branch tip equals current best. Failed experiments aren’t filed, debated, or sent to committee — they’re deleted. There’s a separate TSV log for the full history, deliberately untracked. What worked and what was tried are two different stories, stored in two different places. This is not an accident.
And the evaluation harness? Locked. Immutable. The agent cannot touch the file that defines “better.” It cannot move the goalposts. It can only, you know, actually improve.
The constraints are the design
It would be easy — and wrong — to look at autoresearch and catalogue what’s missing. No planner. No chain-of-thought. No retrieval-augmented anything. No multi-agent symposium where four LLMs argue about hyperparameters while a fifth takes minutes.
But the absence of all that is not a gap. It’s the architecture.
The system works because the problem is well-posed. One file to edit. One number to optimize. One fixed budget. Get those three things right, and even greedy hill-climbing — the dumbest search strategy that still deserves a name — produces genuinely good results.
There’s a lesson here that the AI agent community keeps paying tuition on and refusing to learn: a well-posed problem with a simple agent will embarrass a poorly-posed problem with a brilliant one. Every single time. The bottleneck is almost never the agent’s intelligence. It’s the clarity of what you’ve asked it to do.
The pattern beneath the pattern
Forget the neural networks for a moment. Strip the domain away and what’s left is a pattern:
An autonomous agent iterates on a mutable artifact against a fixed evaluation protocol, keeping improvements and discarding regressions, indefinitely.
This has nothing to do with machine learning. It’s a search strategy over the space of text artifacts — code, config, prompts, anything that diffs cleanly and rolls back cheaply.
Compiler optimization? The artifact is a set of flags. The eval is a benchmark suite. The agent tries combinations overnight. By morning, your build is faster and you did nothing.
Infrastructure tuning? The artifact is a database config. The eval is a load test. The agent hill-climbs toward better throughput while you catch up on sleep you’ve owed yourself since 2019.
Prompt engineering? The artifact is a system prompt. The eval is a test suite. The agent refines it, version by version, keeping what scores higher. No vibes-based iteration. No “hmm, that feels better.” Just numbers.
The recipe is the same every time: scope the artifact tight enough to review, make the evaluation honest enough that gaming it is harder than just being good, fix the resource budget, and let the loop run.
The human’s actual job
Here’s the quiet part. In autoresearch, the human doesn’t write the training code. The human writes the program that tells the agent how to write the training code. The skill file — program.md — is the real product. The training script is just the output.
This is management, distilled. Write a clear spec. Encode your taste. Get out of the way. The quality of the outcome depends almost entirely on the quality of the constraints. Vague spec? Wandering experiments. Precise spec? Focused progress. Same as it ever was.
The best line in the entire repository is buried in the prompt:
“A 0.001 improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 improvement from deleting code? Definitely keep.”
One sentence. One heuristic. Applied a hundred times overnight, it shapes the entire trajectory of the search. That’s not programming in any traditional sense. That’s writing taste into a system and watching it compound.
The real insight
Here’s the thing nobody says plainly enough: agents do their best work when you give them a reward function that lives outside their training distribution.
The LLM has seen a lot of code. It has intuitions about what good training scripts look like. But it has never optimized this specific script against this specific metric on this specific hardware under this specific time budget. The evaluation protocol is novel. The search space is familiar enough to navigate but alien enough that memorization won’t save you.
That’s the sweet spot. The agent brings general competence — pattern recognition, code fluency, architectural taste absorbed from millions of repositories. The evaluation brings specificity — a concrete, unseen objective that can only be solved by actually trying things and measuring what happens.
This is why autoresearch works and why “write me a good training script” doesn’t. The open-ended request invites the agent to regurgitate. The constrained loop forces it to discover. One is retrieval. The other is research.
And it generalizes. Every time you set up an agent with a clear metric it hasn’t been trained to game, you’re creating the conditions for genuine exploration. The agent stops pattern-matching from its training data and starts hill-climbing through territory it has never seen. That’s when the interesting things happen.
So what
The AI agent ecosystem is accumulating complexity at a rate that would make a Rube Goldberg machine blush. New frameworks weekly. Multi-agent architectures with communication patterns that need their own architecture diagrams. Planning systems that plan about planning. It’s frameworks all the way down, and the turtles are getting expensive.
Autoresearch is the counterargument you didn’t know you needed. It says: maybe the thing that matters is not the agent. Maybe what matters is the loop, the metric, and the constraint. Get those right, and the agent barely needs to be clever.
The hard part was never building a smarter agent. The hard part — the part that actually requires human judgment — was knowing what to measure.
It always is.
autoresearch is open source. The pattern it demonstrates — autonomous hill-climbing over a constrained artifact — is worth stealing.