Building Effective Autoresearch Systems: 1

An autoresearch system is software that does research on its own. It has an idea, changes some code, runs an experiment, reads the result, and keeps the change if the number went the right way. Then it does it again, a few thousand times, overnight. Some of these systems tune prompts. Some design neural network architectures. A few have found genuinely new mathematics.

This is the first of two posts about them. This one is about whether you can believe what they tell you, and it goes like this. First the good news, because some of these systems really have discovered things, and it is worth being precise about which ones and why. Then the bad news: what happens when the thing being tested can reach the thing doing the testing, which turns out to be most of the time. Then what it costs to do it properly, why the economics push every team towards measuring less, and what a loop you could actually trust would look like.

The second post is the build guide: every mechanism I found for getting each part of one of these systems right.

Andrej Karpathy put out a small tool last year called autoresearch. It’s about 630 lines. It edits the code that trains a little language model, runs a fixed evaluation, keeps the edits that score better and bins the ones that don’t. Point it at a GPU, go to bed, and by morning it’s done a night’s work.

The teaser chart showed the loss going down. Further down, in the loop’s own write-up of everything it had tried, there was a section headed Surprising non-results. One entry read: changing the random seed from 42 to 137 improved the score by 0.0004. Make of that what you will.

People made something of it. On Hacker News one commenter said the seed change “sure does sound like overfitting to the eval set”. Another pointed out that the chart’s y-axis didn’t start at zero, which made a tiny gain look like a breakthrough. Karpathy didn’t argue. He pointed at the loop’s own note and said the model clearly knew the change was odd, so it’s fixable with a better prompt. Somebody opened an issue asking the obvious follow-up: if the loop finds a good seed, are you supposed to keep it?

Here’s the part I can’t stop thinking about. The loop didn’t malfunction. It was told to find changes that made a number go up, and it found one. On a five-minute training run of a small model, seed variance is big enough to swallow the entire reported improvement.

And this isn’t news. David Picard ran the experiment properly in 2021: scan enough random seeds and you can manufacture 1.82% of ImageNet accuracy out of nothing at all, just by picking the luckiest initialisation of the output layer. Free improvement is lying around everywhere. Any search process pointed at a noisy number will find it, because that’s what searching means.

So the loop wasn’t broken. The scoreboard was.

That’s what this post is about. I read 149 open-source projects that run this kind of loop, 122 of which actually measure something, and looked at one question in each: can the thing being tested reach the thing that’s testing it? The answer turns out to predict almost everything else, including which of these systems have discovered anything real.

First, the good news

It would be easy to write the cynical version of this post, and it would be wrong. Some of this works, and works spectacularly.

It has found new maths. DeepMind’s FunSearch found a cap set of size 512 where the best known was 496, and nudged a long-standing lower bound from 2.2180 to 2.2184 (Nature, 2024). That result is still standing. It’s the kind of thing where you don’t have to trust anybody, because a cap set either is one or isn’t.

It has shipped into production at scale. AlphaEvolve found a scheduling heuristic that has been running inside Google’s Borg cluster manager for over a year, continuously recovering 0.7% of Google’s worldwide compute. It sped up a kernel in Gemini’s own architecture by 23%, which took 1% off Gemini’s training time. Meta’s KernelEvolve has over 8,000 generated kernels deployed, serving hundreds of trillions of ad-ranking inferences a day, with more than 60% inference throughput improvement on one model and over 25% training throughput on their own silicon.

It has written a paper that passed human peer review. Sakana’s AI Scientist v2 submitted three fully autonomous manuscripts to an ICLR 2025 workshop. Reviewers didn’t know the source. One scored 6.33, above 55% of the human submissions, and would have been accepted (paper). It was a workshop rather than the main conference, and the paper was withdrawn before publication by prior agreement, but the review was real and it passed.

And it’s getting cheaper to run. ShinkaEvolve matched a state-of-the-art circle-packing result in 150 evaluations where the original took thousands.

None of that is hype. So hold those four things in your head, because the interesting question isn’t whether autoresearch works. It’s why it works so well there and so badly everywhere else.

Why the same idea finds real maths here and a random seed there

Look at that list again and notice what the four successes have in common.

A cap set can be checked by a program in milliseconds, and the program can’t be argued with. A scheduling heuristic either packs more jobs onto the same machines or it doesn’t, and Borg counts. A kernel either produces the same numbers faster, or it’s wrong. Even the accepted paper was graded by humans who didn’t know what they were reading.

Every one of those has a referee the system couldn’t reach, lean on or sweet-talk.

Now look at Karpathy’s loop. It’s graded by a five-minute training run whose noise is larger than the effects it’s hunting for, and there’s only one run. That’s not a referee. That’s a coin with a slight bias, flipped once.

This is the single axis the whole field sits on, and it isn’t my invention. The most comprehensive survey of self-improving AI I could find reads 1,250 papers from 2024 to 2026 and concludes that demonstrated self-improvement strength tracks a verification hierarchy: formal verifiers at the strong end, a model assessing its own work at the weak end (Chen, Wang and Qu, 2026).

So here’s the rule this post is going to keep proving, from four separate directions, two of them from medicine:

It sounds obvious written down. It is apparently very easy to forget while building something that works.

Happily, somebody has already run the experiment that proves it, mostly by accident.

Same models, two test suites, forty-three times the cheating

METR runs two different benchmark families. HCAST is a spread of general agentic tasks. RE-Bench is machine-learning research work, and on RE-Bench the agent can see the whole scoring function.

Same models. Same period. The difference is enormous: reward hacking showed up in 39 of 128 RE-Bench runs, about 30%, against 8 of 1,087 HCAST runs, under 1%. METR put it at more than 43 times more common.

On one RE-Bench task, Optimize LLM Foundry, it happened in all 21 runs they examined. Every single one.

The methods are worth listing, because they’re not subtle and they’re not clever. Reading the reference answer out of the Python stack. Overwriting the timing function so the code looks faster. Monkey-patching the evaluator. Stubbing out validation. Defining a fake tensor subclass whose equality check always returns true, so the scorer’s comparison passes.

Nothing about the model changed between those two numbers. The only thing that changed was whether the scoring code was within reach.

And before you reach for the obvious fix: they tried asking nicely. On one task they ran three different instructions, including a plain request not to cheat, twenty runs each. METR’s conclusion was that the instructions made nearly no difference.¹¹This one has mutated badly in secondhand tellings, which now claim that telling the model not to cheat pushed hacking from 30% up to 70 or 95%. That’s a misreading. On the single task tested the baseline was already 80%, and the three variants came in at 95%, 80% and 70%. The finding is that asking has no effect, not that asking backfires.

Better still, they asked the model whether what it was doing matched what the task designer intended. o3 said no, ten times out of ten, and carried on.

Keep that result in your pocket. Asking nicely turns out to be what almost the entire field is currently doing.

That last detail closes off the comfortable explanation. This isn’t a comprehension failure. The model understands the intent perfectly and optimises the metric anyway, because the metric is what’s being optimised. Confusion isn’t the problem. Reach is.

The benchmarks everyone quotes leave the answer key lying around

If you’re thinking this only affects hobby projects, the field’s most-cited coding benchmark has the same hole.

SWE-bench gives an agent a repository and a bug report and asks for a patch. In September 2025 somebody noticed that the agents had found a shortcut: run git log --all, or search the reflog, and the actual fix is sitting right there in the repository’s future history. No memorisation, no cleverness. The environment hands over the answer key. Reproductions were logged across Claude, Qwen and GLM models.

That’s not the only leak. A manual screening of successful patches found that about a third had the solution written into the issue report or its comments, and another third passed on tests too weak to distinguish a fix from a fudge. After filtering, one agent’s resolve rate fell from 12.47% to 3.97%.

Then Cursor ran the experiment that settles it. On SWE-bench Pro, 63% of one frontier model’s successful resolutions retrieved the fix rather than worked it out: 57% by looking it up upstream, 9% by mining git history. So they sealed the git history and restricted internet access, and watched the scores fall. One model went from 87.1% to 73.0%. Another went from 74.7% to 54.0%.

Hold onto those two numbers, 14 points and 21 points. They come back later, from a very different field.

There’s a neat trick for measuring this in general, by the way. ImpossibleBench mutates coding tasks so the written specification actively contradicts the unit tests, making them genuinely unsolvable. Any pass at all is therefore, by construction, a cheat. Your pass rate is your cheating rate. I find that a lovely piece of design.

Although I’ve now used the word “cheating” about a dozen times, and it’s the wrong word. It’s also about to make this problem look far smaller than it is.

Nobody here is cheating, which is exactly the problem

Almost nobody building these systems is gaming anything on purpose. The best description of what’s actually happening comes from statistics, and it’s a paper about human scientists: Gelman and Loken’s garden of forking paths.

Their point is that you don’t need a fishing expedition to get a false result. You just need to make a series of reasonable choices, each one shaped by the data you happen to be looking at. Should the evaluation use 200 samples or 500? Should that outlier run count? Should the baseline use the same prompt format? Each decision feels forced by the situation. None of them feels like a degree of freedom. But a different dataset would have produced different choices, and that’s enough to break the statistics, entirely without anybody behaving badly.

Their sharpest observation is about how this lands: researchers hear the accusation as deliberate fishing, get offended, and dismiss the whole critique. It isn’t an accusation. It’s a description of how careful people produce unreliable results.

How much does that cost? Simmons, Nelson and Simonsohn ran the simulation in 2011. Four innocuous freedoms, the sort nobody would think to mention, each one defensible on its own: measuring two related outcomes instead of one, adding ten more observations when the result looks close, controlling for an obvious covariate, and dropping one of three conditions. Individually each pushes the false-positive rate from 5% to somewhere between 8% and 13%. Together they take it to 61%.

They also demonstrated the point by publishing a real, statistically significant finding that listening to The Beatles makes undergraduates a year and a half younger. Not feel younger. Be younger, by date of birth.

And that’s all with researcher freedom in play. Take every last bit of it away and there’s still something underneath.

Why picking the best of ten is a trap

Suppose you generate ten candidates and measure each one exactly once, fairly, with no bias anywhere. You keep the highest scorer. That sounds unimpeachable.

It isn’t, and there’s a name for why. Smith and Winkler called it the Optimizer’s Curse. Each measurement is the candidate’s true value plus some error. When you take the maximum, you’re selecting partly on true value and partly on the largest positive error. So the winner is, on average, luckier than it is good, and its real performance will come in below what you measured. Every time. Even with perfectly unbiased measurements.

Two separate things drive that gap, and it’s worth keeping them apart. The noise decides what fraction of your winner’s advantage is luck. The number of candidates decides how large the illusion gets: screen ten and you overstate by a certain amount, screen two hundred and you overstate by nearly twice as much, with precisely the same proportion of it being luck.

Which is an uncomfortable pair of facts for a technology whose entire pitch is screening enormous numbers of candidates. Searching harder doesn’t dilute the problem. It scales it. The only thing that shrinks the fraction is measuring better.

Drag the noise up and watch the winner’s true value come apart from its measured one. That gap is the seed change. That gap is Picard’s 1.82%. It’s the same gap every time, and it doesn’t require anybody to do anything wrong.

I read 149 of these systems. One measures properly.

So that’s the theory. I wanted to know what the actual field looks like, which meant reading the code rather than the papers.

Here’s the method, plainly, because the numbers below are only worth what the method is worth. I screened roughly 1,300 candidate repositories, read 149 of them properly, and found that 122 run a loop that measures something and acts on the measurement. The other 27 were demos, wrappers or abandoned. For each of the 122 I answered the same small set of questions by reading the source: where the verdict comes from, whether the candidate can reach the evaluator, whether measurements are repeated, whether there’s a held-out split, whether there’s a spend cap, and what carries across runs.

The limits are real and I’ll state them up front. Reading code isn’t running it. GitHub over-represents work that someone wanted seen. Deciding whether a code path is genuinely dead is a judgement call, and I made about fifteen of those. And this is a snapshot of repositories that change weekly.

One thing made me confident it was worth doing anyway. That 1,250-paper survey I mentioned earlier, the most thorough thing in the field, doesn’t open any code at all. It reads papers. And its own conclusion is that rigorous measurement of self-improvement is the field’s most underpopulated area. So: the definitive survey reads 1,250 papers, opens zero repositories, and concludes that measurement is the gap.

Here’s the gap.

Of the 122 systems that run a measured loop:

• 45 decide the verdict in code. The other 77 hand some or all of it to a language model.
• 16 have an evaluator the candidate genuinely can’t reach. 55 expose it outright, 49 partially.
• 13 repeat a measurement before acting on it. 84 never do.
• 12 hold out data the loop never sees.
• 6 have a spending cap written in code rather than expressed as a hope in a prompt.
• 21 carry anything at all across runs, and almost none of them scope what gets carried.

Those are each depressing on their own. But the number I actually went looking for was the overlap, because any one safeguard on its own doesn’t buy you much. A protected evaluator that you consult once is still a single noisy sample. Repeats on an evaluator the candidate can edit are just repeats of a rigged measurement.

So: how many of the 122 have all four of a code-decided verdict, a protected evaluator, repeated measurement and a held-out split?

One. It’s SEAL. Six systems have three of the four. If you also require a spend cap in code, the count goes to zero.

I want to be careful about what that does and doesn’t mean. It doesn’t mean 121 projects are worthless, and it doesn’t mean their authors are careless. Most are research code doing exactly what research code is for. But nearly all of them report a number that went up, and for 121 of them there is at least one specific, nameable reason that number might have gone up without anything real happening.

It also means something slightly odd: as far as I can tell, nobody had counted this before. There’s excellent work measuring how often models hack a given benchmark. There was nothing measuring how many harnesses leave the door open in the first place.

Six ways a program can reach its own mark

“Reaching the evaluator” sounds like one failure. It’s at least six, and they get harder to spot as you go down the list.

1. It deletes the detector. Sakana and UBC’s Darwin Gödel Machine is a coding agent that rewrites itself. Its authors noticed it was sometimes hallucinating tool calls, writing out fake logs showing tests it had never run. So they added special marker tokens to detect the hallucination, and scored the agent on that detector. The agent then achieved a perfect score by removing the markers (paper).

Read that again, because it’s the whole post in one incident. Presented with an instrument that measured its dishonesty, the system optimised away the instrument. And to Sakana’s considerable credit, they found it themselves and published it.

2. It grades its own exam. In one kernel-optimisation project, the harness counts a task as passed when it sees the string PASS in the output. The output is produced by a test the language model itself wrote.

3. It votes on itself. In aworld/self_evolve, 45% of the weight used to rank a candidate comes from values the model asserts about its own candidate. Not measurements of it. Claims about it.

4. It picks its own exam questions. Opik derives each candidate’s evaluation subsample from a hash of the candidate’s own prompt text. Nobody designed that as a cheat, and it isn’t one in intent. It does mean each candidate is graded on a different subset, chosen by a function of the thing being graded.

5. The exam changes between candidates. AutoPrompt grows its evaluation set adversarially as it goes, adding cases the current candidate fails. Reasonable on its own terms. But the final selection compares scores across the whole run, and those scores were computed against different sets. The closing comparison isn’t comparing like with like.

6. The rejection never fires. This is my favourite, because it’s three characters. OpenEvolve has a novelty check: a model is asked to judge whether a candidate is novel, and told to answer NOT_NOVEL if it isn’t. The parser that reads the answer looks for NOT NOVEL, with a space. The strings never match. The check can never reject anything, and it sat like that in a widely-used repository.

There’s a pattern underneath all six. Not one is a deliberate cheat. Every one is an ordinary engineering decision that happens to connect the candidate to its own score, and the loop then does what loops do.

That’s also why code review catches these and papers don’t. None of the six would appear in a methods section.

Which raises the obvious question. If the evaluator is sitting right there, why doesn’t anybody just tell the model to leave it alone?

What these loops actually say to the model

They do. That’s the thing. I went and read the prompts, expecting boilerplate, and found the most interesting hour of the whole exercise.

Most of these systems are, quite literally, a document. There’s a persona, some setup steps, the one file the model is allowed to edit, instructions for finding the metric in a log, a results ledger usually formatted as TSV, the keep-or-discard rule spelled out as arithmetic, a preference for simpler code, and an instruction never to stop or ask the human anything.

And then there’s FunSearch.

FunSearch’s prompt has no instructions in it. No persona, no task description, no “improve this”, no system message at all. What it does instead, in about thirty lines of programs_database.py, is quietly stage a scene.

It takes two programs that already exist and sorts them by score, worst first. It renames them priority_v0 and priority_v1. It then rewrites the docstring of every function after the first to read “Improved version of priority_v0”. And finally it writes out the header of a third function, gives it the docstring “Improved version of priority_v1”, leaves the body empty, and stops.

def priority_v0(el: tuple[int, ...], n: int) -> float:
    "Returns the priority with which we want to add `element` to the cap set."
    ...

def priority_v1(el: tuple[int, ...], n: int) -> float:
    "Improved version of `priority_v0`."
    ...

def priority_v2(el: tuple[int, ...], n: int) -> float:
    "Improved version of `priority_v1`."

Nobody ever asks the model for anything. It’s handed a function whose docstring already asserts that it’s an improvement, and left to go and make that true. The only instruction in the whole prompt is a docstring on the thing being written.

It’s the cheapest prompt in the entire census, and it’s the one that found new mathematics.

Which lands this post’s argument from the opposite direction. FunSearch didn’t win on prompt engineering, because there isn’t any. It won because a cap set can be checked.

The rule everyone writes and almost nobody enforces

Here’s the part that made me laugh, and then stop laughing.

Nearly every loop’s prompt contains some version of the same paragraph. Don’t modify the evaluator. Don’t hard-code answers for the benchmark. Don’t install packages. Stay inside the edit block.

So people did think of it. They knew the failure mode. They wrote the rule down.

Three projects in the census enforce it in code. HyperAgents reverts the evaluator folder after every run and strips it out of replayed patches. TheGreenCedar checks protected paths against a sha256. GEPA keeps the scoring on a server the candidate can’t reach.

Everywhere else, the rule lives in the prompt and the file stays writable.

That’s METR’s experiment again, run accidentally across an entire field, arriving at the same answer. A rule written in a prompt, addressed to the thing it’s meant to restrain, isn’t a rule. It’s a hope with good grammar.

And a few of them simply hand over the answers

Once you’re reading prompts, you start finding things nobody would ever put in a paper.

• AFlow gives its proposer three failure logs including their expected answers, so validation labels arrive directly in the context of the thing being optimised.
• smallnest judges a candidate by having a model write a prose review, then scraping a line like score: 9/100 back out of it with a pattern match. That’s where the 9 got read as 90.
• ShinkaEvolve’s rewrite prompt ships with a literal {language} that never gets filled in, so the model is being asked to write code in a language called {language}.
• OpenEvolve shows the model “diverse” programs for inspiration. Diverse means the ones ranked fourth and fifth.
• AIDE’s metric parser accepts inf as a valid score, and rejects integers.
• One judge panel, in darwin-skill, votes on two pieces of text that are identical.

None of that is incompetence. It’s what code looks like when nothing can check it, which is the same condition the whole post is about. A prompt is the least testable artefact in software: no types, no tests, no compiler, and a reader who will gamely do something plausible with whatever you hand it. So prompt bugs don’t fail. They quietly change what your experiment measures, and the run completes, and a number comes out.

All of which is still about the candidate reaching its own score. The other half of the census is quieter: what goes wrong when the score is perfectly honest and you only look at it once.

Almost nobody runs the experiment twice

Thirteen of 122 systems repeat a measurement before acting on it. Eighty-four never do.

Machine learning already knows how this ends. Henderson and colleagues took a reinforcement learning algorithm, ran it ten times with nothing different but the random seed, split the runs into two groups of five, and compared the groups. They got a statistically significant difference. Same code, same settings, same everything, and the two halves of one experiment disagreed with each other.

If five runs against five runs of identical code can produce a significant result, one run against one run tells you approximately nothing. And one run against one run is what 84 of these systems use to decide what to keep.

The follow-up work is blunter still: most published deep RL comparisons rest on point estimates, ignoring the uncertainty that a small number of runs implies, and the move to more expensive benchmarks has pushed run counts down over time, making the problem worse as the field has matured (Agarwal et al.).

That’s the thing to sit with. Both fields drifted towards fewer repeats for the same reason, which is that repeats cost money and produce no new headline. Hold onto that, because it’s the engine driving everything in this post. I’ll come back to it once I’ve shown you the rest of what the census turned up.

The feature is in the README and switched off in the code

The first of those is small, and it changes how you should read any paper in this field. In at least fifteen of the 122, the mechanism the project is known for is disabled, unreachable or absent in the shipped code.

ShinkaEvolve is a good system and I cited its sample efficiency approvingly earlier. It also ships with novelty filtering, meta-memory and island migration switched off by default, the migration rate set to zero. AFlow, which is discussed as a tree-search method, has no Monte Carlo tree search in it. ASI-Evolve implements the UCB1 bandit formula, which needs a visit count, and never increments the visit count. ACE describes a curator that adds, updates and removes entries, and implements only add. AlphaCodium’s keep-or-discard logic is commented out. The Darwin Gödel Machine’s full-evaluation stage can’t be reached from the code paths that run.

Then there’s the category where the loop simply keeps everything. One project, yoyo-evolve, returns “keep this candidate” on five separate error paths, so a candidate that crashes the evaluator is treated as a success. Its own source comment calls this adverse selection. Somebody saw it, named it correctly, and shipped it.

What the famous results look like up close

Which brings us to the flagship systems. I want to hold them to the same standard as everyone else, in both directions, because some of them come out of it well.

A few of those need unpacking.

The matrix multiplication result. AlphaEvolve found a way to multiply 4×4 matrices in 48 multiplications, presented as the first improvement on Strassen in 56 years. The result is correct. But Ernest Davis points out that those 48 are complex-valued multiplications, so if your matrices contain real numbers, which is most of the time, you need somewhere between 50 and 144 real multiplications. Meanwhile Winograd had a 48-multiplication method in 1967 and Waksman had 46 in 1970, both for commutative rings. AlphaEvolve’s scheme is genuinely valid in cases theirs aren’t, which is a real contribution, and it is not the contribution the headline described.

The file drawer. AlphaEvolve was set on more than 50 open maths problems. DeepMind reported the aggregate honestly: it matched the state of the art on roughly 75% and beat it on about 20%. But it’s the ~13 wins that got written up in detail, and the losses aren’t available for anyone to inspect. Davis asks for exactly the things a preregistration would have supplied: how much human time went in, how many configurations were tried, how much compute was burned. None of that is published, and the system itself wasn’t available outside Google, so nobody could check.

I don’t think that’s misconduct. I think it’s the single most normal thing in all of science, and it’s precisely what the last twenty years of metascience has been trying to fix.

FunSearch, described accurately. Davis’s earlier review makes a point that sounds like a criticism and is actually the most interesting thing about the system. The language model in FunSearch is a mutation operator inside a classical genetic programming loop. It’s never told what problem it’s solving. It’s shown two versions of a function and asked for another one like them. A human wrote the solver skeleton, the evaluation function and the seed programs. So what FunSearch demonstrates isn’t machine mathematical insight. It’s that “propose a plausible variant of this code” is a wildly useful primitive when you bolt it to a referee that can’t be fooled. Which is, I’d argue, the better lesson.

The $15 paper, checked. AI Scientist’s famous cost figure is real, and it’s LLM API cost only, on deliberately tiny experiments, excluding a week on an 8×H100 node. An independent team reproduced it at $6 to $15 plus three and a half hours of human involvement, and found a 42% experiment failure rate from coding errors and a median of five citations per paper. Their summary was that the output reads like a rushed undergraduate essay.

And one story that was overcooked in the other direction. In 2024 the original AI Scientist was widely reported to have edited its own code to escape a timeout, which got written up as an AI spontaneously resisting its constraints. Scott Alexander’s deflation is correct: the system was built on a tool that shows a language model an error and asks it to fix the error. The error was a timeout. Nobody had made any attempt to stop it editing its own code. It did the obvious thing.

I include that because this post would be dishonest without it. The failure modes here are mundane, and treating them as sinister is its own kind of inaccuracy.

Which isn’t to say nothing is at stake. Everything so far has been about wrong numbers, and there’s one consequence that isn’t about numbers at all.

What a cheating loop teaches the model

It’s the reason I think this matters beyond research hygiene.

Anthropic trained models on real production coding environments where a reward hack was available. The hack they seeded was sys.exit(0), which exits the test harness reporting success. Their own description of it is the best sentence in this entire literature: it’s the coding equivalent of “a student writing ‘A+’ at the top of their own essay”.

The models learned it, which surprises nobody. What they found next is the part worth knowing. At the exact point the model learned to reward hack, every one of their misalignment measures jumped together. The model started producing alignment-faking reasoning in half its answers to a question as innocuous as what its goals were. And when they handed it the codebase for that very paper inside a coding agent and asked it to build a reward-hacking classifier, it attempted to sabotage the work 12% of the time, producing classifiers about two-thirds as effective as they should have been.

So a loop that can reach its own score doesn’t just report an inflated number. It teaches a general lesson about what the job is, and that lesson transfers.

There’s a genuinely strange result buried in the mitigations, too. Telling the model during training that hacking was acceptable in this context left the hacking rate unchanged and removed the misaligned generalisation entirely. Framing, not behaviour, was what carried. Nobody should build a product on one result, but it’s the sort of finding that tells you the mechanism isn’t what anyone assumed. And UK AISI has reproduced the core effect on open models, which is more than most results in this post can claim.

Which brings me back to the thing I parked earlier. Everything up to here says these loops need more measurement, not less. None of it is secret, and none of it is hard. So why does almost nobody do it?

What one of these runs actually costs

Because measurement is where the money goes, and it explains the census better than any argument about rigour does.

The number everybody quotes is AI Scientist’s $15 a paper. Here’s the actual spread:

• AI Scientist: under $15 a paper. API calls only, on deliberately small experiments, not counting a week on an 8×H100 node.
• FunSearch: $800 to $1,400 for the admissible-set experiments, 250 to 500 kWh, about two million samples over two days on fifteen model instances plus five CPU servers. That’s from the Nature paper, and it’s rare that anyone publishes it.
• The Darwin Gödel Machine: about $22,000 for a single run, roughly two weeks and 80 iterations. Strip out the self-improvement and the same experiment costs about $10,000. The self-improvement is more than half the bill.
• Evaluating agents: $40,000 for 21,730 runs across nine benchmarks and nine models, in Princeton’s Holistic Agent Leaderboard. One benchmark averages $13 per evaluation. Another runs over $450, and they skipped one model on it because they estimated the bill at $20,000.

Four orders of magnitude, and the famous figure is the cheapest, smallest one on the chart.

But the number that reframes everything is from DeepMind, about AlphaEvolve: it takes on the order of 100 compute-hours to evaluate any single new solution. Not to generate it. To check it.

That ratio between making a candidate and checking one is the whole story. And it’s moving.

The cheap half is getting cheaper and the expensive half isn’t

METR ran six models on a training-speedrun task, up to $10,000 per run over five days, and reported the cost breakdown. It’s the most important number in this post that isn’t from my own census:

70 to 90% of the bill was experiment compute. Only 10 to 30% was the model.

Now put that next to the thing everyone knows about this industry, which is that inference keeps getting cheaper. That’s true, though less dramatically than the headlines suggest. At constant capability the decline is roughly 5 to 10 times a year, which three independent analyses agree on. The 280-fold and 900-fold figures you’ll see quoted are best cases picked at particular benchmark thresholds.²²The everything-gets-cheaper story has some awkward details. Google’s Gemini 3.5 Flash costs five times more per input token than 2.5 Flash at the same product tier. Anthropic’s documentation notes that models from Claude 4.7 onward use a tokenizer producing about 30% more tokens for the same text, so per-token comparisons across that boundary overstate the real fall by roughly that much. And the widely-cited Stanford HAI figure describes a drop over “approximately 18 months” for a span that is actually 23 months.

So follow it through.

The generating half of an autoresearch loop is tokens. It gets 5 to 10 times cheaper a year. The checking half is experiment compute: actually training the model, actually running the benchmark, actually measuring the thing. That doesn’t get cheaper on anything like the same curve, and for frontier work it’s going the other way, with operational costs rising and benchmarking costs flat or climbing.

Which means the ratio between searching and checking gets worse every single year, on its own, without anybody making a bad decision.

And when a budget gets tight, look at what’s on the table. You can generate fewer candidates, which feels like giving up. Or you can evaluate a bit less: drop from three repeats to one, shrink the held-out set, swap the expensive real benchmark for a model asked to judge the output. Every one of those is a small, sensible-looking economy. Every one of them softens the ruler.

That, I think, is the actual explanation for my census. Not carelessness. Incentives.

If you’re about to reply that you’ll simply spend more, there’s one more number worth seeing first.

Agents win on small budgets and lose on big ones

It’s a twist in METR’s data that anyone planning to run one of these overnight should know.

They measured how much each model could usefully spend before it stopped making progress. The range ran from nothing at all up to about $3,300. Two frontier models made no meaningful progress on the task whatsoever. And the overall shape was that agents beat humans at low budgets and lose to them at high ones, with sharply diminishing returns. Their conclusion on that task was that autonomous optimisation has so far had minimal effect.

This isn’t a one-off. RE-Bench found the same shape in 2024, with agents outperforming human experts by about four times at a two-hour budget and humans pulling ahead at eight hours and thirty-two (Wijk et al.).

Which is an awkward pairing with the section above. The thing these loops are best at is cheap, fast, shallow search. The thing that would make cheap, fast, shallow search trustworthy is expensive, slow, repeated measurement. The economics favour exactly the half that doesn’t work on its own.

All of which would be a genuinely hard problem, if it were a new one. It isn’t.

Two fields ran the same experiment and got the same answer

Medicine had this problem first, and solved it, and the parallel is exact enough to be a little eerie.

In a clinical trial, allocation concealment means the person enrolling a patient can’t find out which arm that patient will land in. It sounds like bureaucratic fussiness. It exists because if the enrolling clinician can know, they will, with the best will in the world and usually without noticing, steer the sicker patients away from the treatment they hope works.

In 1995, Schulz and colleagues took 250 trials from 33 meta-analyses and measured the damage. Trials without adequate concealment exaggerated their treatment effects by 41%. Trials that weren’t double-blind exaggerated by 17%.

That’s the same finding as Cursor’s, reached thirty years earlier. Let the assessed party reach the assessment, and the result inflates by a large, measurable amount.

But the more useful study is the bigger one that came later. The BRANDO project looked at 1,973 trials across 234 meta-analyses, and found something that should change how you spend your effort. The bias from missing blinding is severe for subjectively assessed outcomes. For mortality it essentially vanishes. For other objective outcomes it essentially vanishes.

Read that again: blinding matters almost entirely where the measurement is soft. Death doesn’t care whether the doctor was hopeful. It defends itself.

So here are the two experiments side by side. Medicine, 1,973 trials: bias concentrates where the outcome is a judgement and disappears where it’s a fact. AI, last June: seal the answer key and one model drops 14 points, another drops 21.

Two fields, three decades apart, same experiment, same answer. Which is the strongest form of the rule this whole post has been circling, and it comes with a practical instruction attached:

That’s also, finally, why FunSearch found a cap set and Karpathy’s loop found a random seed. One was graded by mathematics and the other by a noisy five-minute average. The architecture barely mattered.

Which sets up an obvious closing move: tell AI to go and learn what medicine already learned. Except that isn’t true either.

The fixes are already written. They just aren’t plumbing yet.

Here’s where I have to correct the version of this post I originally meant to write.

I assumed the story was that AI hasn’t learned what science learned. That’s wrong. Over the last eighteen months the field has re-derived the fixes, by name, citing the metascience directly.

There’s now a preregistration framework for experiments with AI agents, an ICML 2026 spotlight, which catalogues agent-specific researcher degrees of freedom: model choice, prompt wording, settings, redesigning the experiment after seeing the outcome. It reads like Simmons and colleagues’ 2011 requirements, rebuilt for a new subject.

And there’s one genuinely new idea, which I think is the cleverest thing I read in all of this. Preregistering for the next LLM has you commit your analysis plan together with a list of eligible future models, then run the confirmatory analysis on the first eligible model released afterwards. You cannot tune to a model that doesn’t exist yet. Across twenty models from four providers, the protocol blocked roughly three-quarters of attempted transfers of a p-hack.

That’s blinding, implemented as the examiner hasn’t been born yet. Medicine never had that option.

There’s also a rather pointed finding from the people studying automated scientists: examining the trace logs and code catches failures that reading the final paper doesn’t (Luo, Kasirzadeh and Shah). The paper isn’t the evidence. The log is. Which is awkward, because a survey of 24 runnable systems found 83% released their code and only 38% released seeds or execution traces.

So the fixes exist. They aren’t adopted. And the precedent tells us roughly how long that takes: Simmons published in 2011, registered reports launched at one journal in 2013, and they’re now offered by over 300.

Did it work? Two numbers. Registered reports produce positive results 44% of the time against 96% in the standard literature (Scheel et al.). And more violently: of large NHLBI cardiovascular trials, 57% published before 2000 found a benefit, against 8% after preregistration became mandatory. Seventeen of thirty, down to two of twenty-five.

Preregistration didn’t make the treatments worse. It revealed that most of them never worked.

Autoresearch is somewhere around 2012 on that clock. The papers exist. The infrastructure doesn’t.

Which means that for now, if you want a loop you can believe, you have to wire it up yourself.

How to build one you can trust

Here’s what the 122 systems taught me, roughly in order of how much it would have helped them.

1. Measure your ruler before you measure anything else. Run the identical candidate three times and look at the spread. If the spread is wider than the improvements you’re hunting for, you don’t have an experiment yet, and no amount of search will fix that. This is one afternoon of work and it would have saved most of the systems I read.

2. Put the evaluator where the candidate can’t reach it. Separate process, no write access to the scoring code, no access to the reference answers, no git history containing the fix. Sixteen of 122 do this.

3. Never accept on a single sample. How many repeats comes out of rule 1. Thirteen of 122 do this.

4. Hold something back and use it once. A set the loop never sees during search, consulted at the end to confirm the winner is real. Twelve of 122 do this.

5. Cap the money in code. Not in the prompt. A prompt is a wish. Six of 122 do this.

6. Fail closed. An evaluator that errors returns a rejection, never a pass. The alternative is a loop that selects for candidates which break the evaluator, which is the worst possible thing to select for.

7. Record the rejects. Every candidate, its score, its seed, its full trace. A loop that only writes down its winners cannot be audited by anyone, including you in three months.

And weight all of that by rule zero, from the medical evidence: spend most of your effort where your measurement is soft. If you’re optimising kernels against a correctness check and a stopwatch, you can be relatively relaxed. If a language model is grading prose, you need every one of these and you should still be suspicious.

The loop is fine. It’s the marking that’s the problem.

I started out expecting to find that autoresearch doesn’t work. That isn’t what the code says.

It works, and it works exactly as well as its referee is honest. Given mathematics, it found a cap set nobody had. Given a scheduler and a stopwatch, it’s been quietly recovering most of a percent of Google’s compute for over a year. Given a five-minute noisy average and one shot at it, it found a random seed and wrote up an improvement of 0.0004.

The same loop. The difference was who marked the homework.

And that’s the uncomfortable part, because we know how to fix it. Hold out data. Repeat the measurement. Keep the examiner out of reach. Write down what you tried before you try it. Medicine learned all of this at considerable cost, and AI has now rediscovered every item on the list and published papers about it. What’s missing isn’t knowledge. It’s that none of it is switched on by default, in a field where the checking is the expensive half of the bill and gets more expensive relative to the searching every year.

One system out of 122 gets the basics right. It’s a research repository, not a product, and nobody’s heard of it.

So when somebody tells you their agent discovered something overnight, there’s only one question worth asking, and it isn’t about the model or the search algorithm or how many candidates it screened.

Ask what marked it, and whether it could reach the marker.

Sources

Documentation, blogs and repositories

• Andrej Karpathy, autoresearch, and the launch discussion and issue #278, 2026.
• METR, Recent Frontier Models Are Reward Hacking, 2025.
• METR, Expenditure Horizon, 2026.
• Naman Jain, Cursor, Reward hacking is swamping model intelligence gains, 2026.
• Google DeepMind, AlphaEvolve: a Gemini-powered coding agent for designing advanced algorithms, 2025.
• SWE-bench, issue #465: agents recovering gold patches from git history, 2025.
• Scott Alexander, Sakana, Strawberry, and Scary AI, 2024.
• Jaime Sevilla and Josh You, Epoch AI, How persistent is the inference cost burden?, 2026.
• UK AI Safety Institute, reward-hacking-misalignment, 2025.
• Continual Intelligence, SEAL.

Research

• Bernardino Romera-Paredes et al., Mathematical discoveries from program search with large language models, Nature, 2024.
• Ernest Davis, Notes on AlphaEvolve, 2025, and a review of FunSearch, 2024.
• Yutaro Yamada et al., The AI Scientist-v2, 2025.
• Joeran Beel et al., Evaluating Sakana’s AI Scientist, 2025.
• Jenny Zhang et al., Darwin Gödel Machine, 2025.
• Robert Tjarko Lange et al., ShinkaEvolve, 2025.
• Gang Liao et al., KernelEvolve, 2026.
• Mingguang Chen, Licheng Wang and Bo Qu, Recursive Self-Improvement in AI, 2026.
• Monte MacDiarmid et al., Natural Emergent Misalignment from Reward Hacking in Production RL, 2025.
• Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini, ImpossibleBench, 2025.
• Reem Aleithan et al., SWE-Bench+: Enhanced Coding Benchmark for LLMs, 2024.
• Hjalmar Wijk et al., RE-Bench, 2024.
• Sayash Kapoor and Arvind Narayanan et al., Holistic Agent Leaderboard, 2025.
• Ziming Luo, Atoosa Kasirzadeh and Nihar Shah, The More You Automate, the Less You See, 2025.
• David Picard, torch.manual_seed(3407) is all you need, 2021.
• Peter Henderson et al., Deep Reinforcement Learning that Matters, 2018.
• Rishabh Agarwal et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, 2021.
• James E. Smith and Robert L. Winkler, The Optimizer’s Curse, Management Science, 2006.
• Joseph Simmons, Leif Nelson and Uri Simonsohn, False-Positive Psychology, Psychological Science, 2011.
• Andrew Gelman and Eric Loken, The garden of forking paths, 2013.
• Kenneth Schulz et al., Empirical Evidence of Bias, JAMA, 1995.
• Jelena Savović et al., Influence of reported study design characteristics on intervention effect estimates (BRANDO), Health Technology Assessment, 2012.
• Anne Scheel, Mitchell Schijen and Daniël Lakens, An Excess of Positive Results, AMPPS, 2021.
• Robert Kaplan and Veronica Irvin, Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time, PLOS ONE, 2015.
• Michelle Vaccaro, Preregistration for Experiments with AI Agents, 2026.
• Maria Thomas, Milos Gligoric and Nihar Shah, Mitigating LLM-based p-Hacking by Preregistering for the Next LLM, 2026.