StateBench: Can AI Follow Every Change in a Long File?
StateBench: Can AI Follow Every Change in a Long File? OPEN SOURCE FACTORY · NOVEMBER 6, 2026 Watch the English motion graphic Follow the Change — a 42-second English conceptual animation. Original 3D motion, newly composed music, and sound effects: Open Source Factory. The ledger scene is our design proposal; this is not footage of a model test. A model finds a name buried in a long file. Then it loses track of who holds the key after a few handovers. Both questions use the same document, but the second answer has to be computed from a changing state. That is the gap I wanted to understand in StateBench. Ai2 introduced the benchmark on November 3, following a November 2 preprint. Its useful idea is to separate different kinds of updates instead of treating every long-context question as the same test. A bigger context window gives the model more text to work with; following all the changes remains a separate capability. Ai2’s announcement and the research paper make that distincti...