StateBench: Can AI Follow Every Change in a Long File?
StateBench: Can AI Follow Every Change in a Long File?
A model finds a name buried in a long file. Then it loses track of who holds the key after a few handovers. Both questions use the same document, but the second answer has to be computed from a changing state. That is the gap I wanted to understand in StateBench.
Ai2 introduced the benchmark on November 3, following a November 2 preprint. Its useful idea is to separate different kinds of updates instead of treating every long-context question as the same test. A bigger context window gives the model more text to work with; following all the changes remains a separate capability. Ai2’s announcement and the research paper make that distinction explicit.
I also ran the public generator and replayed its traces with exact code. No language model was called in that experiment. The model results below belong to the researchers; the missing-update experiment is our own small audit of how the generated problems behave.
What does the model have to keep track of?
Start with a setting that never changes. Finding its value is mostly a retrieval problem. Now overwrite that setting several times: the latest relevant write is enough. Replace those writes with increments, and the answer depends on the previous value. Finally, let one variable copy another variable’s current value. The chain can cross several registers before the question is asked.
The paper calls these r-trivial, aperiodic, soluble, and general finite state tracking. In the code, the last two appear as solvable and periodic. The arithmetic uses a finite set of values; a running bank balance is a useful analogy, but the actual increment task wraps around with modular arithmetic.
The paired construction is the detail I like. The generator starts from one source trace and rewrites its operations. The overwrite, increment, and copy versions reach the same state after every normal step. Their assignment positions and targets line up. The immutable version instead keeps each register’s initial value. That gives a much cleaner comparison than collecting unrelated “easy” and “hard” questions.
One programming detail matters: a <- b copies the value of b at that moment. It does not make a a live alias of b. If b changes later, an earlier copy stays as it was. The pinned README explains this explicitly.
What did the researchers actually measure?
There are different experiments in this paper, and merging them produces a misleading headline. Models trained specifically on these synthetic tasks answer a different question from chat models used off the shelf.
| Research setting | What the authors report | How I read it |
|---|---|---|
| Direct training from scratch | 275M-parameter architectures trained through 256 assignments; evaluation extended to 64,000. | Tests learning and length generalization under deliberately favorable training. |
| Open pretrained models | Strong immutable tracking; weaker overwrite tracking; failures on general finite tasks. | Having a recurrent component does not automatically teach the skill during pretraining. |
| Frontier model generation | Good easier-tier results; substantial decline on the harder tiers as inputs grow. | These models generated final states. A five-choice guessing baseline does not apply. |
In the announcement’s summary, the four tested frontier models—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna—fall to 13% or lower on the longest general finite problems. That is an authors’ report about one demanding part of this benchmark. It is not an overall accuracy score, and it does not mean that a model remembers only 13% of a document.
The direct-training result is also easy to overread. The maximum evaluation length was 250 times the maximum training length: 64,000 divided by 256 assignments. On the hardest tier, the flexible Gated DeltaNet variant stayed near perfect to roughly 10 times the training length before degrading. Those are different claims. Assignments here are operations, not tokens.
The theoretical results depend on the architecture, numerical precision, arithmetic, and memory-update constraints. They do not justify a blanket claim that every transformer can never track state. The framework also fixes the number of variables, while a real conversation can keep introducing new people, objects, and tasks.
A missing line changed the ending of our generated trace
For a closer look, I pinned the public code to commit 5b333b8d4266. I generated 32 seeded traces at each of four lengths: 8, 32, 128, and 512 post-initialization assignments. Each source had five ordinary registers, one scratch register, 32 possible values, and exactly one post-initialization constant write. There was no filler text.
The full audit produced 128 sources and 512 derived instances. It removed one post-initialization assignment at a time, replayed the rest, and checked the final state. An independent interpreter agreed with the official evaluator on all 87,040 deletion replays. A separate CLI build generated eight rendered examples, and the selected upstream semantic and soluble-family tests passed: 103 tests in total.
In that small source trace, both the copy version and its overwrite twin contain b <- 10. I omitted that same line. Later, the copy version executes c <- b, so the stale value 2 reaches c. Later changes repair b, but c keeps the old copy.
The overwrite twin instead contains c <- 10. It writes the needed value directly and recovers despite the earlier omission. Both complete, undamaged traces end with the same five ordinary values: [25, 16, 10, 31, 26]. Only the dependency structure differs. This is a concrete reason why finding the latest fact and following a chain of changes can behave differently.
How often did a skipped update survive to the end?
For the chart, I used a controlled subset: skip only assignments targeting the five ordinary registers, and score those same five registers at the end. The scratch register still participates in execution, but its own update positions are excluded from the omission set. The overwrite and copy twins use identical omission positions. A replay counts as changed if at least one scored final value differs.
At 512 updates, only 160 of 10,912 omissions changed the overwrite twin’s ending: 1.47%. In the copy twin, all 10,912 changed it. Many old explicit writes became irrelevant after later overwrites. In these particular copy traces, the missing changes continued to matter.
The 100% figure is a property of this sample and this deletion policy. It is not an estimated failure rate for any model, a prediction about real documents, or a proof that every copy task is maximally sensitive. A model can execute an update correctly; our deliberately damaged interpreter was forced to skip one. Different reset patterns, omission rules, or scored registers can change the result.
The reproduction pack includes the settings, script, generated traces, numerator and denominator counts, all-register and ordinary-target controls, licenses, and checks. That is the useful output of this exercise: a small, inspectable example of error recovery and error persistence.
What would I change in an agent workflow?
For an agent that maintains stock counts, task status, or ownership, I would keep an explicit event log and a current-state record. Let code apply the updates in order. Let the model explain the result and point to the events behind it. Before an important action, compare the proposed state with a fresh read from the owning system.
A checkpoint also needs its place in the timeline: which event was last applied, which values were stored, and where they came from. Otherwise a compact summary can preserve a stale answer while hiding the update that should have changed it. When an earlier event is corrected, recompute the affected state rather than appending another confident sentence to the conversation.
This connects with the earlier Mage-VL-Cowork analysis: choosing between clicks and code is only part of an agent’s job. It also needs a reliable account of what changed. The LFM3 output-contract check addressed another boundary—whether the output follows a usable shape. A valid JSON object can still contain an outdated value. Format validation and state validation deserve separate checks.
My take
StateBench gives me a better question to ask of a long-context system: can it reconstruct the current state after dependent updates? A needle-retrieval score does not answer that question by itself. The matched traces make the gap visible without requiring a sprawling real-world demonstration.
I would use this benchmark as a diagnostic alongside the application’s own tasks. Keep the source events, make important updates auditable, and test whether a missed step can be detected or repaired. I am much more interested in that recovery path than in another promise that the entire project fits inside one prompt.