Open-Endedness Bench

Open-Endedness Bench

AI agents are now given research jobs with no known answer: train a better model, speed up a chip design, beat a training-speed record. An agent works alone for hours or days, and every command it runs and every note it writes is saved in a log. Most benchmarks only look at the final score. Open-Endedness Bench reads the whole log and asks how the agent did the research: whether its claims rest on results it actually saw, whether its experiments gave answers, and what it did when an idea failed.

The reading is done in two parts. An AI model, which we call the judge, reads the log and answers fixed questions about it, quoting the log word for word; plain code then does all the counting. The final score of the task and any answer key are never used. Each run gets four scores, where higher is better, and six habits, where neither end is better.

What 119 runs show

We scored 119 runs on three benchmarks. A benchmark here is a fixed set of research tasks with its own way of scoring the result. PostTrainBench asks the agent to train a small language model for a test; Chip-Bench asks it to improve a hardware design; the nanoGPT speedrun asks it to make a model train in fewer steps. Six things stood out.

  1. Most improvements the agents announced were not real. On the speedrun, agents announced 64 improvements. We checked each one against the results the benchmark had logged, and only 10 (16%) actually beat the best result so far. On Chip-Bench, 6 of 21 announced improvements were real (29%). There, results the agent never commented on turned out to be real improvements just as often (31%).
  2. Those false wins then steered the work. Speedrun agents built later experiments on 91% of the results they had announced as wins. Of the 324 experiments that built on an earlier result, 194 (60%) built on a win that was not real.
  3. Many experiments ended without an answer. The share of experiments that ended in a clear answer the agent read was 37% on PostTrainBench, 24% on Chip-Bench and 85% on the speedrun.
  4. The best runs kept trying new ideas late. On 9 of the 10 tasks we could compare, the run with the best result tried more new ideas in its second half than the run with the worst result. On the speedrun the gap was 119 new ideas against 20. Simply taking more steps showed no link to a better result.
  5. Retrying the same idea again and again paid little. On Chip-Bench, the fourth and later attempts at an idea made up 36% of all experiments but brought only 10% of the real improvement. No run used all the time it was given; the typical run stopped after 19% of it.
  6. Each model has its own way of working. On PostTrainBench, 14 models each ran up to six tasks. For every habit, knowing which model ran tells you more than knowing which task it was given: the model accounts for a median of 43% of the differences between runs, the task for 7%. The chart near the bottom of this page shows each habit.

How each model does research

One card per model. The four petals are the four scores, with the numbers printed beside them; a longer petal is better. Each of the six lines runs between two opposite ways of working, and neither end is better. To make each model easier to remember, we compare it to a scientist who worked in a similar way, and give the reason in the model's own numbers. Each benchmark used its own judge, so the cards show one benchmark at a time.

Do habits come from the model or from the task?

Take one habit, such as how much an agent thinks before it acts. Runs differ on it. Part of that difference comes from which model ran, part from which task it was given, and the rest from neither. Each bar splits the difference that way, using the 91 PostTrainBench runs (14 models, 6 tasks). A long blue part means the model decides the habit, and giving it another task changes little.

How it is computed: we fit each habit on model and task together, and credit each with only what it explains beyond the other. From the paper's analysis.

Rank on each benchmark

Each column is ranked within its own benchmark by that benchmark's result. There is no overall ranking: the benchmarks share few models, measure different things, and use different judge models. Click a cell to open that benchmark's table.

A dot means the model was not run on that benchmark.