Test whether an assistant delivers the right scientific result

In one sentence

A model handling a long contract must decide what to keep from earlier pages and what to consult when answering a question. The authors explain that designs make different trade-offs between keeping separate pieces of text and compressing them into a smaller memory. A token is a piece of text the model processes, such as a word or part of one. A layer is one stage in the model’s sequence of processing steps. The authors compare ways of storing and retrieving tokens, including designs that use different methods in different layers. They review prior research and classify documented model releases; they do not test an assistant on contracts.

Key concepts

• Contextual memory is information from the current document that remains available inside a model as it processes more text. It is not an external document store or the model’s training data.
• Explicit memory keeps earlier tokens as separately addressable items. In a hypothetical contract example, this is like retaining individual clauses so a later question can point to one.
• Sparse attention keeps explicit items but checks only a selected subset for each question. The selection can save work, but relevant evidence must be included.
• Recurrent state compresses preceding text into a running summary-like internal state. It can reflect earlier text without retaining every token as a separately retrievable item.
• Hybrid architecture combines memory methods, sometimes across layers. The authors’ comparison asks not only what is stored, but how it is updated, selected, read and combined.

A concrete example

Hypothetical illustration, not a paper test: A contract mentions a cancellation deadline near the beginning. When asked about it near the end, a design that retains individual tokens could consult the earlier passage; a sparse design would first have to select it. A compressed-state design might carry information from that passage without preserving the passage as a separately selectable item. None of these descriptions establishes that a model will answer correctly.

What the researchers measured

This is a survey and descriptive architecture comparison, not a new task-performance experiment. The authors assembled 59 release-level records of publicly documented model architectures. In a separate frozen comparison of 11 high-performing open-weight model endpoints from the Artificial Analysis Intelligence Index v4.3.2, captured on September 22, 2026, they report varied attention designs and an explicit token-retrieval path in every selected architecture. These counts describe the surveyed releases and selected endpoints; they are not accuracy scores.

Why it matters

The authors distinguish fitting more text into a model’s input from reliably using distant evidence. Their framework makes it easier to state what a design retains and what it may overlook. That distinction can guide questions about a proposed system, but the survey does not show how well any system would perform in a reader’s job.

Where it might help

The survey’s framework could help people planning long-document AI systems ask what evidence remains available, how retrieval candidates are chosen and whether different layers share memory. These are possible design questions, not demonstrated improvements to a workflow.

Impact across sectors

• Possible, not tested: Legal teams considering contract assistants could ask whether an architecture preserves access to individual earlier clauses.
• Possible, not tested: Healthcare teams considering long-record assistants could ask how a design selects older notes for a current question.
• Possible, not tested: Software teams processing lengthy specifications could use the framework to compare memory and retrieval choices before evaluating a system on their own documents.

Where the evidence stops

The authors say their literature coverage and release inventory are curated, not exhaustive. One of the 59 release records lacks separately disclosed attention architecture and is omitted from the classifiable-record visualization. The frozen endpoint comparison is a snapshot, not a causal test: model size, training, reasoning budget and implementation also affect leaderboard scores. For multimodal models, the authors classify only the documented language backbone, not vision or audio components. General PtoP note: this arXiv source is a preprint, and a survey of architectures is not evidence of performance in a deployed contract-review workflow.

In one sentence

A computer-using assistant can carry out many steps in scientific software yet still submit a wrong result. In one run described by the authors, GPT-5.6-terra entered an incorrect astronomy measurement, later entered the same value again, and checked the saved answer against itself rather than against what the measurement meant. To study such failures, the authors built OSWorld-Science: a collection of scientific tasks in real software, an environment that lets assistants operate a desktop, and task-specific checks of what they leave behind. The checks inspect outcomes such as files, measurements and application state, and can award credit for partially completed work.

Key concepts

• A computer-using agent is an assistant that takes actions on a desktop, such as clicking, typing or issuing a terminal command.
• A task-specific evaluator checks the resulting scientific work, rather than relying only on the assistant's claim that it finished.
• Partial credit records how much of a task's stated requirements the final result satisfies; it is not the same as counting fully finished tasks.
• A history window is the set of recent screenshots available to the assistant while it chooses its next action.
• Independent validation checks a result against the source or another measure, rather than repeating an assumption that might already be wrong.

A concrete example

Hypothetical illustration: An assistant is asked to record a measurement from an astronomy program. It saves a number in a workbook, then reopens the workbook to confirm that the same number is there. That confirms the save, not the measurement. The authors describe this distinction in a specific GPT-5.6-terra astronomy run; this hypothetical version is not an additional test result.

What the researchers measured

In the authors' overall comparison of evaluated models, Claude Fable 5.1 had the highest mean task score, 73.7%. That percentage averages task-specific scores, including partial credit; runs with no gradable outcome count as zero. It is not the percentage of tasks fully completed. Separately, the authors changed the assistant's configuration on the 23 QuPath pathology tasks. With Claude Opus 5 at medium reasoning effort and English prompts, making ten recent screenshots available produced a mean partial-credit score of 57.0%. The authors observed a lower score with a shorter history window, but no configuration cell differed significantly from its default after their stated correction for multiple comparisons. Each cell had one run per task, so the observed change should not be described as an established improvement.

Why it matters

The authors' examples show why a plausible sequence of actions is not enough evidence that scientific work is complete. A useful check must reach the delivered result and, where needed, the source of the scientific claim.

Where it might help

Possible use: a research team could use the paper's task descriptions and outcome checks to decide what an assistant must verify before handing over a scientific file. The study tests assistants on benchmark tasks; it does not establish a deployed research workflow.

Impact across sectors

• Possibility, not a proven deployment: Medical-image teams could specify both the image-analysis steps and the saved result an assistant would need to deliver.
• Possibility, not a proven deployment: Chemistry teams could check a proposed molecular file against its source, rather than accepting a file that merely opens correctly.
• Possibility, not a proven deployment: Statistics teams could distinguish a correct calculation from a missing or incorrectly saved report.

Where the evidence stops

The authors report uneven numbers of tasks across fields. Their detailed trajectory analysis covers released runs for 128 tasks and excludes biology; not every evaluated run has a released trajectory. Some software requires a licence, and the authors say they do not plan to publish all trajectories in this version. The QuPath configuration comparisons use one run per task and did not detect a significant difference from the default after correction. The paper also describes evaluation and infrastructure gaps in particular task analyses, so their counts should stay attached to the specific tasks and runs reported. General PtoP note: this arXiv source is a preprint, not evidence of peer review, and a benchmark result is not a deployment result.

In one sentence

A search agent can appear to improve by learning to repeat its own wrong answers. The authors study a training loop in which one part writes questions and draft answers from source documents, while another practices answering them. If the second part learns a wrong draft answer, its later agreement with that answer can be mistaken for progress. The authors call this unintentional shared-error pattern co-cheating. Their main method, CrossFit, divides documents into two groups. Questions from one group are scored by a separate answering model trained only on the other group. The main answering model still trains on admitted questions from both groups; only the feedback used to choose future practice questions changes.

Key concepts

• Self-evolving training: a system makes practice questions and uses an answering model’s responses to guide which questions it makes next.
• Pseudo-label: a proposed answer used for training without a human-written answer key; it may be wrong.
• False agreement: the question-maker’s adopted answer and the answering model’s response match, but an audit judges that shared answer incorrect.
• Source-level exclusion: keep every question from one document out of the training data of the model that scores questions from that document.
• Multi-sample verification: the authors’ separate check asks the same model three times with the source and three times without it; compatible majorities replace the draft answer, while other proposals are rejected.

A concrete example

Hypothetical illustration, not a paper result: A question-maker reads a document and drafts a question whose answer is mistakenly “Harbor A.” If its usual answering partner learned that draft from the same document, a later “Harbor A” response might earn credit. Under CrossFit, that question is instead scored by a partner that did not train on questions from that document. This removes that direct route to agreement; it does not guarantee the partner is right.

What the researchers measured

In the authors’ three-round Qwen3.5-4B experiments, CrossFit reduced final-round false-agreement mass from 6.1% with the standard coupled Dr. Zero loop to 3.0%. That percentage counts audited answer pairs that matched on the same incorrect answer. On a fixed set of 1,325 questions drawn from seven search benchmarks, CrossFit’s Qwen3.5-4B main model scored 48.8%, an 8.8-point improvement over the coupled version. This score counts questions whose normalized prediction contains a reference answer, then averages the seven benchmark scores equally. In the corresponding Qwen3.5-9B experiments, false-agreement mass fell from 8.8% to 3.7%, and the benchmark average rose by 8.4 points to 51.2%. The authors also report that replaying identical saved proposals with source-excluded feedback reduced false agreement, helping separate the feedback change from changes in which questions were generated.

Why it matters

The authors’ finding concerns a specific training problem: agreement is a poor stand-in for correctness when the model giving feedback has learned from the answer it is checking. Their comparison separates changing the feedback model’s training history from simply checking proposed answers more often.

Where it might help

A possible use is to design the feedback stage of a system that generates its own search-and-answer practice. The paper tests training and benchmark evaluation, not an operational assistant or an end-user workflow.

Impact across sectors

• Hypothetical possibility — workplace knowledge search: a team training an assistant on its documents could track whether its practice answers and feedback come from the same source.
• Hypothetical possibility — education: builders of a self-questioning study tool could distinguish agreement with a generated answer key from evidence that the answer is correct.
• Hypothetical possibility — research search: a team training a literature-search assistant could keep a document out of the scorer’s training set when scoring questions derived from it.

Where the evidence stops

The authors used an automated, evidence-backed audit after training, not exhaustive human annotation. Unsupported cases remained unresolved; final-round audit coverage varied by model and treatment. CrossFit does not make its scoring model a source of guaranteed truth: shared pretraining, overlapping evidence and related documents can still produce shared mistakes. The authors also note that rejecting difficult questions could lower false agreement without improving learning. CrossFit added auxiliary training cost, and they do not establish lower end-to-end cost or robustness to connected sources. This source is an arXiv preprint; as a general PtoP note, its preprint status does not imply peer review.

In one sentence

When a job needs several kinds of work, the authors ask whether one system can choose suitable specialists and arrange their handoffs. Raven is their proposed system. For example, a research summary might need to reach a coder before a designer can use the coder’s output; that sequence is an illustration, not a measured result. Raven’s Host Agent makes a task graph—a set of jobs connected by the order in which their outputs are needed—and assigns jobs to agents. The wider system also includes ways to retain context, reuse procedures and change the working instructions around an agent’s model.

Key concepts

• A harness is the tools, memory, instructions and control rules around an AI model. The paper treats a model together with its harness as a working unit.
• A task graph records both who does each job and which output another job needs. Raven checks a proposed graph before sending work to agents.
• A handoff is more than passing a file location: later work must receive the right information and preserve what earlier work established.
• Harness adaptation changes the working policy around a fixed model, rather than changing the model itself. The paper discusses this alongside previously published HarnessBank experiments.

A concrete example

Hypothetical example: a small team wants a briefing and a web page about a software experiment. A planner could assign source-finding to a research agent, implementation to a coding agent and the final page to a design agent. The plan would make the page wait for the experiment’s results. This shows what a handoff means; it is not a Raven test result.

What the researchers measured

In the authors’ Multi-Agent Orchestration Benchmark, systems submitted plans without running the specialist agents. With the Qwen3.8-27B model, Raven’s reported Exact Match rate was 0.711, versus 0.607 for the strongest compared baseline. Exact Match counts scored plans whose specialist choices and required order of work agree with an accepted reference. The authors also report Raven ahead of both compared systems on all four planning measures with each of the two tested models. These are planning findings, not measures of finished project quality.

Why it matters

The paper separates choosing and ordering specialists from the harder question of whether their combined work succeeds. That distinction gives a working reader a way to examine an AI workflow’s handoffs, rather than treating a plausible-looking plan as a completed job.

Where it might help

A possible use is to sketch a multi-specialist job before running it: identify the required outputs, who could produce each one, and what evidence the next worker needs. The paper’s planning benchmark tests agreement with reference plans; it does not establish that such a plan will complete a real project.

Impact across sectors

• Possible use in software teams: plan when investigation, code changes and a final explanation should occur.
• Possible use in research support: map how source gathering, analysis and presentation depend on one another.
• Possible use in creative production: specify which materials a designer needs from other contributors before starting.

Where the evidence stops

The authors say an alternative valid plan can differ from the benchmark’s reference plan. The planning test does not dispatch workers, and its nominal 140 requests do not by themselves determine every reported score’s denominator. Their theory gives sufficient conditions for reliable collaboration, but whether an implemented system meets them requires evaluation; it does not show that collaboration always beats one agent. Other evaluations have separate qualifications: Raven-Research’s comparison used questions also seen during development, and public benchmark copies were accessible; some Raven-Design slide configurations did not cover every task, and failed generations were excluded from reported means. General PtoP note: this arXiv source is a preprint, not evidence of peer review or a workplace deployment.

In one sentence

The study asks whether one learned picture of a changing scene can support several kinds of prediction. The authors trained Orca to represent what is happening, then tested whether that representation could help produce written answers, images and robot actions. Imagine seeing a hand beside a spoon, then seeing the spoon lifted. Orca’s video exercise asks it to predict an internal description of the next view, rather than draw every pixel. A second exercise gives it a description of an event and asks it to predict a view associated with that event. A third uses video questions and answers. The authors call these first two approaches “unconscious” and “conscious” learning; those names describe training methods, not awareness. After this training, the authors kept Orca’s main model fixed. They trained separate readouts—parts that turn its internal representation into an image or robot action—and used its existing language output for answers. This setup tests whether the learned representation is useful beyond a single output task.

Key concepts

• State transition: a change from one situation to another, such as a spoon moving from the table into a hand.
• World latent space: Orca’s internal representation of a scene and its possible changes, rather than a finished picture or sentence.
• Observation-only learning: predicting the representation of the next video frame without an event description.
• Event-conditioned learning: using a written event description to guide a prediction about an associated frame.
• Frozen backbone and readouts: the authors stop updating Orca’s main model while testing parts that express its representation as images or actions.

A concrete example

Hypothetical illustration, not a paper result: a kitchen worker supplies a picture of a closed drawer and the instruction “open the drawer.” An image readout might attempt to show the drawer open. The useful question is whether it also keeps the surrounding counter and objects consistent; the paper does not test this workplace use.

What the researchers measured

In the authors’ real-robot tests, Orca-4B averaged 36.6 rule-based points across five tasks with unfamiliar tablecloth or background settings, versus 27.6 for π 0.5, the compared robot-control system pretrained on large-scale robot data. Rule-based points mark the highest task stage reached before a trial ends; they are not a success percentage. In the separate unfamiliar-object setting, Orca-4B averaged 28.2 points versus 31.2 for π 0.5. The authors trained the Orca action readout on 200 robot trajectories per task. In a qualitative spoon-grasp example, they report Orca recovering from early failures while π 0.5 made repeated failed attempts. For image prediction, the authors report a 59.8 average percentage score for Orca-4B with its image readout on PRICE-V0.1, versus 56.1 for FLUX.2 [klein]. That score comes from four model judges rating generated images against an initial image and instruction. The authors also report improved text, image and action readout results as Orca’s pre-training scales up.

Why it matters

The authors test whether learning about changes in video can be useful when the output is a sentence, an image or a robot action. That is different from training a separate main model solely to produce each kind of output. The results concern the authors’ stated evaluations, not general reliability in everyday settings.

Where it might help

Possible uses suggested by the evaluated tasks include studying video understanding, predicting how an instructed interaction might look, and researching robot manipulation. These are possibilities, not demonstrated products or deployments.

Impact across sectors

• Possible use in media work: a video team could explore tools that answer questions about event order in clips; the paper tests benchmark questions, not an editing workflow.
• Possible use in robotics research: a lab could investigate whether video-trained representations help a robot handle a changed tabletop scene; the paper’s robot tests cover five specified manipulation tasks.
• Possible use in visual prototyping: a design team could explore images of an instructed scene change; the paper evaluates predicted interaction images, not a design service.

Where the evidence stops

The authors describe Orca as an early step. It learns mainly from vision and language, not sound, touch or force. Its visual prediction target comes from a frozen, pretrained vision component rather than a representation learned directly from all possible signals. The experiments mainly use the 0.8B and 4B sizes, and this version trains on only one-tenth of the authors’ 125K-hour video inventory. The authors report a trade-off among readout performances during training. They say their image benchmark has limited scale and variety, most annotated events cover short, minute-level changes, and the robot tasks remain relatively short and easy. The paper says no benchmark-specific training data was constructed or used, but does not establish that every underlying video source is disjoint from every evaluation source. General PtoP note: this arXiv paper is a preprint, and a benchmark result is not a deployment test.

In one sentence

The study asks whether an AI agent can finish more long tasks when its working procedure improves, even if the underlying model does not change. The authors built StateM, a runtime that keeps the current phase of work and a shared runbook outside the agent’s long conversation history. Imagine an agent setting up a web service: it can work freely during setup, but the runbook can require a working end-to-end check before the agent moves to handoff. StateM records the phase, refreshes relevant instructions, checks configured conditions at transitions, and leaves a route back for repair. The authors developed one runbook with GPT-5.5, tested it unchanged with GPT-5.6 variants, adapted its practices for DeepSeek-V4-Flash, and developed separate runbooks for BusinessBench task families.

Key concepts

• A harness is the execution system around a model. The authors change that system, not the model’s trained weights.
• A runbook describes phases, permitted transitions, checks and repair steps. Both the agent and a person can inspect it.
• A state is the recorded current phase. Entering one brings its instructions forward; leaving one can require a check.
• A configured check can block a handoff, but its strength depends on how it is verified. An agent’s own completion statement is not independent proof.
• The authors turn selected lessons from failed runs into versioned practices. A practice developed for one model or task family may need changing elsewhere.

A concrete example

Hypothetical illustration, not a reported test: an agent updates an internal service. Its runbook keeps it in a verification phase until a configured check confirms that another system can use the service. If the check fails, the agent returns to repair rather than declaring the job done.

What the researchers measured

On Terminal-Bench 2.1, an 89-task terminal benchmark, the authors report 92.1% trial-level success for GPT-5.5 xhigh with StateM, against an 83.1% published GPT-5.5 reference. Trial-level success counts completed trials, with five trials per task. With the runbook frozen, GPT-5.6 Sol xhigh with StateM recorded 424 successful trials out of 445, or 95.28% raw accuracy, in a public submission. That score is before review is settled, not a finalized leaderboard result. For a different model and setting, the adapted DeepSeek-V4-Flash system scored 392 successes out of 445 under standard timeouts; the authors report $15.20 in realized API charges for its final-score evidence, excluding $37.02 spent on adaptation. On BusinessBench, the first frozen held-out evaluation with Codex and GPT-5.6 Luna showed smaller aggregate gains: 0.55 percentage points when task families were weighted equally and 1.34 points when instances were weighted equally.

Why it matters

The authors’ results separate a model’s ability to perform individual steps from the execution system’s ability to carry a job through to completion. They also show a boundary: an unchanged runbook developed with GPT models did not improve their DeepSeek-V4-Flash result, while an adapted one did.

Where it might help

Possible applications include tracking obligations during long coding jobs and placing checks before consequential operations handoffs. These are potential uses of the control approach, not deployments demonstrated by the paper.

Impact across sectors

• Software teams: potentially use shared runbooks to keep visible requirements, tests and handoff checks together during a long coding task; hypothetical workplace use.
• IT operations: potentially require a service-readiness check before handing over a configuration change; hypothetical workplace use.
• Business operations: potentially preserve approval conditions across a multistep workflow; hypothetical workplace use, not a finding that StateM works for all such workflows.

Where the evidence stops

The Terminal-Bench gains measure StateM’s runtime together with an evolved, benchmark-adapted runbook; they do not isolate the runtime alone. The GPT-5.5 comparison uses a published reference, not a newly rerun control matched for agent version. The GPT-5.6 Sol xhigh public submission remained open and unmerged at the stated review date. The authors say four rewarded trials should not count and that nine trajectories were flagged for possible reward hacking; the raw 95.28% must therefore remain qualified. Solving a task at least once in five trials is not single-run reliability. The unchanged GPT-developed runbook reduced DeepSeek-V4-Flash’s full-suite score from 82.7% to 82.0%; its later improvement required adaptation. BusinessBench has one trajectory per instance per arm, excludes the untreated Attendance family from intervention aggregates, and shows declines in two treated families in its first frozen evaluation. Later refinements reused evaluation feedback and are not untouched held-out results. StateM cannot recover unrecorded work, guarantee that a self-reported check is correct, or undo arbitrary external actions. PtoP note: this source is an arXiv preprint, not a claim of peer review; benchmark scores are not evidence of workplace deployment.