Test whether AI agents finish professional computer work

In one sentence

The study asks whether an AI agent improves more when its practice tasks are chosen as its weaknesses change, rather than put in order beforehand. The authors built ActiveSaddler to make that choice during offline optimization, before any deployment. A harness is the set of prompts, available tools, and operating rules around an AI model. ActiveSaddler does not replace the method that repairs this harness; it chooses which training tasks provide evidence for the next repair. It groups related failures, estimates which known weakness may be worth another attempt, and sometimes tries an unseen task to look for a new one. The authors tested this approach with the AutoSaddler harness optimizer on GAIA2 and Terminal-Bench 2.0.

Key concepts

• A training scenario is a practice task run by the current agent. Its execution record shows what happened and helps the optimizer propose a harness change.
• A failure-pattern arm is a group of observed failures attributed to a shared harness weakness. It lets the curriculum revisit a weakness that appears in more than one task.
• Arm prioritization estimates whether another attempt at a known weakness might help. The estimate considers whether the weakness persists, seems fixable, could affect other tasks, or risks disrupting working behavior.
• Exploration means running a training scenario not yet tried during optimization. Revisiting means spending the next attempt on scenarios associated with a known failure pattern.
• A rollout is one task-agent execution. The authors compare ways of allocating a limited rollout budget while keeping the underlying harness optimizer fixed.

A concrete example

Hypothetical illustration, not a reported result: An assistant repeatedly omits attachments from different calendar tasks. Instead of treating each failed task as unrelated, a curriculum could group them as one possible weakness, revisit it after a proposed repair, and then try an unseen calendar task to check for other failures.

What the researchers measured

The authors report mean test Pass@1 over three test-time executions; this score is the share of held-out tasks passed on a single attempt. With ActiveSaddler added to AutoSaddler, they report 59.8% on the GAIA2 test split and 80.0% on the Terminal-Bench 2.0 test split. These are, respectively, 4.4 and 7.5 percentage points above the same optimizer using a randomly shuffled scenario order fixed before optimization. In separate component-removal tests, the authors report lower test scores when failure-pattern grouping, arm prioritization, or adaptive exploration is replaced. Those tests concern the named variants, not a different deployed system.

Why it matters

A fixed practice schedule can revisit a weakness that has already been addressed or move past one that remains. The authors' results suggest that the choice of training tasks matters alongside the method used to change the harness.

Where it might help

A possible use is planning which practice tasks should guide improvements to an agent's prompts, tools, or operating rules. The paper tests that idea in benchmark environments, not in a workplace deployment.

Impact across sectors

• Possible, not tested: Teams building workplace assistants could use recurring task failures to decide what to practice next.
• Possible, not tested: Software-development teams could consider this approach when improving agents that work through terminal tasks.
• Possible, not tested: Teams maintaining simulated mobile assistants could use changing failure patterns to organize pre-deployment testing.

Where the evidence stops

These are controlled benchmark results, not a production evaluation. The authors used public and synthetic task data, not real user data, and say the experiments do not establish safety, security, privacy, or governance readiness. GAIA2 training, development, and test tasks came from disjoint simulated environments; Terminal-Bench 2.0 used a uniform random partition because it lacks a natural grouping axis. The reported test scores average three test-time executions. ActiveSaddler also adds optimizer-side computation. The paper says a project website and code will be available, but the supplied text does not verify an available installation. General PtoP note: this arXiv version is a preprint; the supplied text does not establish peer review.

In one sentence

The study asks whether a humanoid can tackle a new object-handling task by changing what it is asked to achieve, rather than retraining how it moves. The authors’ answer is to keep its movement controller fixed and revise a reward program: a set of goals for successive parts of a task. Imagine moving a box onto a table. One part can favor lifting and carrying it; a later part can favor letting go. A completion condition tells the system when to switch parts. InterEvolve has a language-model agent revise those parts after simulated attempts, while a numerical search adjusts settings such as how strongly each goal counts. A separate, fixed verifier checks whether the attempt met the task’s requirements. Successful programs can be kept in a skill library for later tasks.

Key concepts

• A reward program describes the desired interaction in stages; it does not prescribe every joint movement.
• The fixed movement controller was trained to use information about both its body and an object. A new reward can steer its existing movements without retraining that controller.
• The agent changes the program’s stages and goals, while a numerical search tunes their settings. Simulated attempts provide feedback for both.
• A fixed verifier checks outcomes and constraints independently of the reward program being revised.
• A skill library stores selected programs and their results so later tasks can draw on them.

A concrete example

Hypothetical illustration, not a reported trial: for a request to place a parcel on a shelf, one stage could favor getting hold of it and moving it upward. A second could favor releasing it once it reaches the shelf. A verifier would separately check whether the parcel stayed there and the robot let go.

What the researchers measured

On the authors’ eight simulated box-task families, InterEvolve’s evolved programs achieved 86.5% success, compared with 34.6% for the agent’s initial program after numerical tuning. Success means every task criterion and physical constraint held in the same attempt. The authors also show a physical Unitree G1 executing simulation-evolved kicking and pushing tasks using onboard sensing and an off-board workstation; they do not give a physical-task success rate in the supplied text.

Why it matters

The authors separate task strategy from movement training. In their experiments, changing the goals and the order in which they apply let the same controller use movements that a fixed program did not reliably bring out.

Where it might help

A possible use is preparing new task instructions for a robot that already has relevant movements, then checking candidate instructions in simulation before physical execution. The paper does not establish a general-purpose deployment process.

Impact across sectors

• Possible logistics use: explore staged instructions for moving and placing boxes with an already-trained humanoid; this was not tested in a warehouse.
• Possible manufacturing use: study whether stored handling programs help with new multi-step object tasks; this was not tested on a production line.
• Possible robotics-research use: compare task instructions while keeping one movement controller fixed; this is an application of the reported experimental approach.

Where the evidence stops

The authors state that search cannot bring out behavior the controller never learned and is limited by its stored examples of body-and-object states and by available measurements. Simulation and language-model use take GPU time and add latency, ruling out real-time replanning during physical execution. The main tracking test uses held-out clips from objects represented in training; the additional box types tested for transfer were also seen during controller pretraining, though not during program search. The physical demonstration covers two tasks, with no reported success rate. In that setup, prompts prepared from a simulated run are replayed on the robot while its fixed controller responds to sensed state; the paper does not report physical test-time program evolution. PtoP note: this arXiv version is a preprint, not evidence of peer review.

In one sentence

The authors ask whether a model can generate detailed 3D shapes without losing how their parts connect. Their answer is to describe a shape through overlapping cross-sections viewed from three directions, rather than many tiny 3D cells. Imagine slicing a bicycle wheel from front to back: successive slices reveal where the open center and spokes begin and end. SILSA compresses such slices into tokens, small packets of shape information. A reconstruction model learns to turn those packets back into a 3D surface. A separate image-guided generator learns to produce the packets from a single picture. The authors also train the reconstruction model to notice connected parts and holes, while a shared 3D workspace helps slices from different directions describe the same object.

Key concepts

• Overlapping slice tokens: Each token summarizes a thin region rather than a single flat cut. Neighboring regions overlap so the model can follow a structure through depth.
• Three-direction views: Slices taken along three perpendicular directions expose different parts of the same shape.
• Topology: This means structural features such as separate parts, openings and connections. A filled wheel opening is a structural error even if much of the surface looks right.
• Slice-level supervision: During reconstruction training, the method compares holes and connected parts within slices and checks where they change between neighboring slices.
• Shared 3D workspace: During image-guided generation, slices from the different directions exchange information through a common spatial memory.

A concrete example

Hypothetical illustration, not a reported test: A designer supplies one image of a bicycle wheel. A useful 3D result would keep its center open and its thin spokes connected to the rim, rather than filling the opening or breaking a spoke.

What the researchers measured

The authors trained SILSA on Trellis-500K and evaluated using 200 randomly sampled Toys4K assets and 50 in-the-wild images, which they say did not overlap with training. In their image-to-3D comparison, SILSA had leading scores on the reported FD, PSNR, coverage and MMD measures; it tied the best KD and LPIPS scores. These measures assess aspects of generated-shape quality, image fidelity or how well outputs represent the reference set. Its PSNR, an image-fidelity score where higher is better, was 32.74 versus 30.12 for SparseFlex. SparseFlex had a slightly higher CLIP score, the paper’s input-image alignment measure. Separately, in the SliceVAE reconstruction comparison, the authors report a lower Betti error—a measure of mismatched connected parts and holes—than the reconstruction baselines. SILSA used 384 fixed slice tokens. In the reported efficiency comparison, its training memory and end-to-end inference time were lower than Dora’s under the stated test settings.

Why it matters

The distinction between a close-looking surface and a correctly connected object matters when a missing opening or broken support changes the shape itself. The authors report both structural measures and generation-cost measures, so readers can see what happened in their experiments rather than infer structure from appearance alone.

Where it might help

Possible uses, not tested deployments, include making draft 3D assets from images for creative or instructional work. The paper reports generation and reconstruction experiments, not an end-to-end workplace workflow.

Impact across sectors

• Possibility for content creation: An artist could explore image-based drafts of objects with handles, spokes or other delicate parts; this is not a demonstrated production use.
• Possibility for design: A designer could use generated shapes as starting points for inspection and editing; the paper does not test design decisions.
• Possibility for education: Instructors could use image-to-3D examples to discuss how openings and connections affect a shape; this is a hypothetical use.

Where the evidence stops

The authors say SILSA depends on the diversity and scale of its training data: shapes with unfamiliar structures may be reconstructed less faithfully. Ambiguous or low-information input images may also reduce generation quality. Their open-surface garment evaluation notes that topology measures are near zero for methods that recover the rough surface, limiting what that measure distinguishes there. The authors also identify risks of unauthorized replicas and displacement of manual modeling work. General PtoP note: this arXiv source is a preprint; the reported benchmarks are not a tested deployment.

In one sentence

An AI agent can carry out extended research tasks without taking over the decisions that set their direction. That is the pattern the authors describe in their development of Atria Dawn Preview, a model designed to use tools across research and engineering work. They trained it on tasks in working environments, keeping records of its steps and checking outcomes against evidence such as tests or file state. They then compared its performance on set tasks and examined task records and agent logs from the people developing it. In those records, agents often proposed methods and made revisions; people usually made the final choices.

Key concepts

• An agent is a model that can take a sequence of actions, inspect what happened and decide what to do next, rather than give only one reply.
• The Verifiable Experience Pipeline is the authors’ training process for linking a task, the agent’s actions, its output and an external check. For example, a program can be run and tested rather than judged only by how plausible its code looks.
• A benchmark is a fixed set of evaluation tasks. The paper uses benchmarks to compare models under specified conditions, not to measure success in every workplace.
• Proposal and selection are different decisions: an agent may suggest a method, while a person decides whether to use it.

A concrete example

Hypothetical example, not a paper result: an engineering lead asks an agent to investigate a failing data-processing job. The agent checks logs, suggests a change and runs a test. The lead decides whether that test answers the right question and whether the change should be kept.

What the researchers measured

The authors report that Atria Dawn Preview had the highest reported score on five of 16 benchmarks; rankings use the available entries, and some comparison results were unreported. In their project review, participants rated 151 of 455 completed AI-assisted tasks with usable responses as infeasible without AI under the same scope and resources. Among participants in execution roles, humans made the final choice in 85.5% of recorded method-or-parameter decisions. That figure concerns who selected an option, including options proposed by AI; it is not a measure of task success.

Why it matters

The authors’ distinction is between doing more work and deciding what work is worth doing. Their project records show substantial agent involvement alongside continued human selection and intervention. They say completing research tasks does not, by itself, show that an AI system can repeatedly discover worthwhile improvements to future models.

Where it might help

As a possible use, a research team could ask an agent to run bounded experiments and bring the results to a human decision-maker. The paper does not establish a general deployment procedure or show that an agent can choose valuable research directions on its own.

Impact across sectors

• Possibility, not a proven deployment — research labs: agents could prepare experiments and report what happened while researchers choose which questions merit further work.
• Possibility, not a proven deployment — software teams: agents could implement and test proposed changes while engineers review the evidence and decide what to accept.
• Possibility, not a proven deployment — security teams: agents could assist with diagnosis and re-testing in authorized environments, with people retaining control of scope and decisions.

Where the evidence stops

This is an arXiv preprint, not a claim of peer review. The collaboration findings come from the authors’ own development project, and the infeasibility finding is based on participants’ ratings of their tasks. The paper calls its tool-use cases selected demonstrations rather than estimates of average success. Evaluation coverage also varies: SkillsBench uses a self-contained subset that excludes multimodal tasks, and one optimization case had no complete formal mean at the end of its published trace. The authors say the weather interface does not report forecast accuracy and the displayed computer-aided designs are not mechanical or manufacturing validation.

In one sentence

The study asks whether a research agent does better when it can consult practical instructions for a task, rather than work from its general knowledge alone. Imagine asking an agent to use an unfamiliar software package: it needs to know not just what the package does, but how to set it up, check its output and recover from mistakes. The authors call this practical know-how operational knowledge. Their DisCo system turns material from repositories and papers into reusable instruction bundles called skills. An entry file says when a skill applies; supporting files can hold detail and executable helpers. A router helps the agent open only the relevant bundle. The authors compared a Codex research agent with and without these skills while keeping its GPT-5.5 model, research harness—the software coordinating its work—and downstream running budget fixed.

Key concepts

• Operational knowledge is practical guidance about which tools fit a task, how to use them and what can go wrong.
• A skill packages that guidance around a SKILL.md entry file, with deeper references and scripts where needed.
• Skill distillation turns source material into procedures through scoping, evidence gathering, construction and verification.
• Task-agnostic skills are prepared from sources for later reuse; task-oriented skills are built around a particular problem or benchmark.
• Progressive disclosure means the agent loads the relevant branch of skills instead of reading the entire library.

A concrete example

Hypothetical illustration, not a study result: a researcher asks an agent to compare two unfamiliar model-serving packages. A relevant skill might tell it to use the same workload for both, retain the commands it ran and check the measurements before reporting them.

What the researchers measured

The authors report that, on the full MLE-bench suite of 75 machine-learning competitions, GPT-5.5 Codex with task-oriented AREX-Skill guidance received an Any-Medal score of 72.89%, versus 31.11% without skills. That score reports how often the agent attained any medal. On PaperBench's 20 paper-reproduction tasks, they report an average replication score of 39.59% with the selected paper-derived skills, versus 29.45% without them; that score is assigned by the benchmark's reproduction grader. The paper also reports higher aggregate scores with skills on FrontierCS and PassNet. Those evaluations used their own task-oriented graphs, not simply the public repository collection.

Why it matters

The authors' comparison separates access to distilled instructions from changes to the agent's model or research harness. It therefore speaks to what practical guidance may add under their benchmark settings, while the separate effort of creating that guidance still matters.

Where it might help

A possible use is to give compatible coding agents inspectable package procedures and validation checks before a research task. The paper measures benchmark work; it does not establish that this approach succeeds in a particular workplace.

Impact across sectors

• Possible impact in machine-learning engineering: teams could provide agents with package-specific setup and checking guidance. This is not a reported deployment.
• Possible impact in academic research: agents attempting paper reproductions could consult reusable instructions drawn from related work. This is a possibility, not a demonstrated outcome outside the benchmark.
• Possible impact in software performance work: agents could use guidance for checking that an optimization still produces correct outputs. This is not a reported production result.

Where the evidence stops

The authors report that skills lowered the PaperBench scores on two tasks. They suggest that retrieved material may sometimes distract the agent from a better task-specific approach, but present this as a possible explanation, not an established cause. Skill construction happened before benchmark execution and used a separate budget; matched downstream running budgets do not include that construction cost. The public repository snapshot and the skill collections used in the evaluations are distinct. For PaperBench, the authors excluded each target paper and its released artifacts as skill sources, while selecting related prior work. The repository collection is described as curated rather than exhaustive. General PtoP note: this is an arXiv preprint, not a peer-reviewed finding, and benchmark scores are not deployment results.

In one sentence

The study asks whether AI agents can complete extended professional tasks on a computer. The authors built Agents’ Last Exam, or ALE, around work contributed by domain experts. In each task, an agent receives instructions and works in an environment with the relevant software and input files. The authors then check what it produced against a reference output or a task-specific set of rules. The test focuses on finished work rather than a plausible account of how to do it.

Key concepts

• A benchmark is a shared test. ALE uses task instances: specific cases with their own inputs and expected outputs, drawn from professional workflows.
• An agent combines an AI model with a harness—the software that lets it decide what to do next and use tools. For these evaluations, the authors gave the tested systems both desktop controls and command-line tools for working with files and programs.
• A deliverable is the finished item, such as a workbook or media file. ALE scores deliverables using references or rubrics, meaning stated criteria for what the output must contain. Some tasks require a basic condition to be met before any further credit is given.
• A full-pass rate is the share of tested tasks receiving full credit. It differs from a mean score, which also reflects partial credit.

A concrete example

Hypothetical illustration, not a reported test result: an agent receives source figures and instructions for a financial workbook. It opens the relevant software, fills in the workbook and saves the required file. A check of the saved cells against a reference would tell you more about completion than the agent’s description of the steps it took.

What the researchers measured

In the authors’ main ALE evaluation, Codex paired with GPT-5.5 and given desktop tools fully passed 38.1% of Near-Term tasks, the tier containing 67 task instances. The same configuration fully passed 0.0% of Last-Exam tasks, the tier containing 38 task instances. These percentages count tasks receiving full credit, not the amount of partial work completed. The authors report separate scores for other agent-and-model pairings; these findings should not be read as scores for GPT-5.5 on its own.

Why it matters

The authors argue that success on short tests does not establish whether an agent can carry a longer job through to a usable result. ALE makes the final deliverable central to the test. For a working reader, the distinction is between seeing an agent perform promising steps and confirming that it produced the requested work.

Where it might help

The authors present ALE as a way to evaluate agents on software-based professional work. A team considering an agent could use that approach as a model for specifying a deliverable and checking it, but the paper does not show that an ALE score predicts success in that team’s own setting.

Impact across sectors

• Possible use in finance: a team could define checks for a completed workbook rather than assess only an agent’s written explanation. This is a hypothetical application, not a proven deployment.
• Possible use in manufacturing planning: a team could specify required output files and safety-related checks for a software workflow. This is a hypothetical application, not a proven deployment.
• Possible use in media production: a team could check that all requested exports exist before reviewing their content. This is a hypothetical application, not a proven deployment.

Where the evidence stops

The authors identify overlap with training data or task-specific optimization as a threat to a public benchmark. They release only 150 of the 1,490 task instances publicly and hold others privately. Some checks of visual or written outputs still use an AI model as a judge rather than a fixed code-based check. The paper says repeated runs were available for only a subset of configurations because of compute limits. Each evaluation run also had a five-hour limit. General PtoP note: this source is an arXiv preprint, not a claim of peer review, and a benchmark result is not a tested workplace deployment.