Check Whether AI Agents Pass Uncertainty On to You

In one sentence

The authors ask whether AI coding agents can design brick assemblies that depict a request, pass digital buildability checks, and show the care of a human design. Imagine asking for a sea serpent beside a pirate ship: recognizable shapes are only part of the job; the chosen pieces must also fit without overlapping and remain upright in a simulation. The authors built BrickBench to compare those demands across three settings: a smaller model, a larger set, and a build limited to the pieces in one retail set. They also built BrickAgent, a workspace where an agent can choose pieces, place them, inspect views, and receive feedback about construction problems.

Key concepts

• A coding agent writes and runs a program to place bricks, then can inspect and revise the resulting assembly. In this study, agents generally used the BrickAgent workspace rather than receiving special training for the task.
• Buildability checks test whether the design obeys its piece limits, avoids overlapping parts, and stays upright in the authors’ simulation. These checks do not measure the strength of real connections.
• Semantic alignment means how much of the written request appears in the assembly. The authors turned each prompt into yes-or-no questions and had an image-reading model answer them from rendered views.
• Design quality is a separate comparison. An image-reading model viewed pairs of assemblies and chose which used its pieces better by the standard of a published set, without seeing the original prompt. The authors also compared this judge’s rankings with human ratings.

A concrete example

Hypothetical illustration, not a study result: For a request to build a bird in a tree, one digital design might show both clearly but leave a branch unsupported. Another might stand in the simulation yet use bulky pieces that obscure the bird. BrickBench’s separate checks are intended to distinguish these kinds of outcomes.

What the researchers measured

Across BrickBench’s three settings, the authors report that GPT-6 Astra with BrickAgent scored 0.954 on their prompt-question measure: that score is the mean fraction of yes-or-no questions judged satisfied for its assemblies, not a physical-build success rate. Removing BrickAgent reduced digital validity for the two agents tested in that comparison, GPT-6 Astra and GPT-5.6 Luna. In a separate Model-and-Set study, raters selected the human-designed assembly in 323 of 360 comparisons against builds from the nine reference agents and two baselines. That study did not include the later-evaluated Claude Opus 5.5 or GPT-6.1 Sol.

Why it matters

In the authors’ tests, passing construction checks and covering a written brief did not settle whether a build looked thoughtfully designed. The comparison makes that distinction visible instead of treating a recognizable image as the whole design task.

Where it might help

A possible use is comparing digital brick-design approaches before choosing which ideas deserve closer human review. The paper does not demonstrate a deployed design service or show that its passing assemblies survive physical construction.

Impact across sectors

• Possible impact for toy designers: use separate digital checks for whether a proposed build depicts the brief and whether it passes the benchmark’s construction tests; this was not tested as a product workflow.
• Possible impact for educators: use the distinction between a recognizable model and a well-constructed one to discuss design trade-offs; the paper did not study classrooms.
• Possible impact for digital design-tool makers: offer feedback that identifies troublesome parts alongside visual previews; the paper did not evaluate a commercial tool.

Where the evidence stops

The authors say their stability simulation treats connected groups of pieces as rigid bodies and does not test connection strength. A design can therefore pass while a real build might sag or fall apart. Their human-versus-agent comparisons matched assemblies by part count, not by prompt, because no human designs existed for the BrickBench prompts. The human validation of the design judge also covered the nine reference agents, not the two agents evaluated later. General PtoP note: this source is an arXiv preprint, not evidence of peer review; a benchmark result is not a tested deployment.

In one sentence

The study asks how to tell exactly where a video-description model goes wrong, rather than giving its whole caption one score. Imagine a clip in which a dog appears, disappears at a cut and later barks: a description could mention the dog and the bark yet attach the sound to the wrong moment. The authors’ OmniCapBench asks models to produce separate records for recurring people, objects and scenes; visual shots; and timed sounds. These records include identifiers and links between them. Before comparing descriptions, fixed rules reject broken identifiers, times and links, while a language model with a restricted matching role helps decide which local identity descriptions or brief visual actions correspond. Timed shots and events are matched using the paper’s specified measures. A language-model judge then compares the descriptions only within aligned pairs. The authors call these small, checkable records ‘atomic units.’

Key concepts

• Structured output: The model supplies linked records rather than only a flowing paragraph, so an error can be traced to a particular record or connection.
• Identity tracking: A recurring subject should retain its identity across camera cuts, not become a new subject each time it appears.
• Temporal grounding: A visual shot or sound is tied to a time span; recognizing an event without placing it correctly is a different error.
• Audio-visual association: A sound record must link to a compatible visual shot, rather than merely appear somewhere in the description.
• Bounded matching and judging: Rules check validity, a restricted language-model matcher helps align certain local records, and a separate local judge compares their descriptions.

A concrete example

Hypothetical illustration, not a reported test: In a workplace clip, a colleague speaks while the camera shows another colleague. A structured description would record the speech, its time and its link to a visible shot. An evaluator could then distinguish a wrong speaker link from inaccurate words.

What the researchers measured

The authors built OmniCapBench from 786 annotated videos. In its main structured-generation evaluation, Gemini 2.5-Pro scored 37.81% on cross-shot coreference consistency, the measure of reusing identities across cuts. In the same evaluation, Gemini 3.1-Pro scored 51.46% on event-shot association F1, a score combining recovery of annotated sound-to-shot links with penalties for unsupported links. These are different models and different measures, not a head-to-head comparison. The authors also report that whole-caption judging can obscure localized structural errors.

Why it matters

The authors report that whole-caption scores can hide errors in identity, timing and sound-to-image links. Their separate checks make those kinds of error visible in the benchmark setting. That is different from showing that a model will produce reliable captions in everyday use.

Where it might help

A possible use is diagnosing video-caption models during development: inspect whether their errors concern missing subjects, misplaced events, broken links or descriptions. The paper evaluates models on a benchmark; it does not test a workplace deployment.

Impact across sectors

• Possible use in media production: a team assessing captioning tools could inspect whether sounds have been attached to the right shots; this is hypothetical, not a demonstrated workflow.
• Possible use in accessibility work: people comparing video-description tools could look separately at missed actions and confused identities; this is hypothetical, not a tested accessibility outcome.
• Possible use in model development: a team could use localized errors to choose what to investigate next; this is a potential application, not a demonstrated improvement.

Where the evidence stops

The authors say the annotated references do not cover every true detail in a video, so a valid detail outside their scope may receive no credit. The videos are under five minutes; the findings do not establish performance on much longer recordings. Some reference drafts came from model families later evaluated, and the authors say possible evaluator–model bias was not fully eliminated despite validation and human review. Requiring structured output disadvantages models optimized for free-form prose; the authors exclude specialized captioning models from the main evaluation for difficulty following that format. They also caution that a low score on this specific test does not necessarily mean a model cannot write a helpful conversational summary. The source is an arXiv preprint, not presented here as peer-reviewed work.

In one sentence

A correct-looking final answer does not tell you whether an AI agent noticed a conflict or warned you about one it could not settle. To study that gap, the authors gave agents factual questions with conflicting evidence and examined their steps as well as their final answers. They also studied longer tasks where a conflict could arise during tool use. For each agent, a backbone model produced language while an agent harness managed steps and tools. The authors scored three separate behaviors: Identify the specific gap, Solve it by making a follow-up tool call and finishing, and Escalate remaining uncertainty in an incorrect final answer. Here, Solve measures an attempt, not whether that attempt found the right answer.

Key concepts

• Knowledge conflict means that evidence disagrees with an agent’s earlier answer, or that two supplied sources disagree.
• A trajectory is the record of an agent’s intermediate steps, tool calls and final response. It lets the authors distinguish noticing a problem from telling the user about it.
• Identify, Solve and Escalate measure different behaviors. An agent can receive credit for noticing a conflict without receiving credit for following it up or communicating it.
• A matched control gives a comparison without the selected conflict. In the controlled tasks, the question and number of passages stay the same; in the longer tasks, controls use different questions.

A concrete example

Hypothetical illustration, not a paper result: An agent preparing a brief finds two source documents that give different dates for the same event. Identifying means naming the disagreement. A follow-up search is an attempt to solve it. If the date remains unsettled, escalating means telling the reader which date is in dispute rather than presenting one without qualification.

What the researchers measured

In the authors’ default-run BrowseComp evaluation, Claude Code with Claude Sonnet 4.6 received a 90.0% Identify F1 score. F1 combines how often identified gaps were genuine with how often genuine gaps were identified, using both conflict tasks and their controls. For that same agent, model and dataset, its Escalate rate was 0.0%: among conflict tasks with incorrect final answers, the score counted final answers that explicitly acknowledged the unresolved uncertainty. These are different measures with different sets of scored tasks. Across agent–dataset configurations, the authors also report that conflict-relevant answer mentions tended to occur early, and that a short uncertainty prompt often raised Escalate while lowering task accuracy.

Why it matters

The authors’ results separate answer accuracy from the handling of uncertainty. Their framework asks not just whether an agent finished a task, but whether a conflict it encountered survived into the information given to the user.

Where it might help

A possible use is to inspect an agent’s intermediate steps and final answer separately when designing or selecting a research assistant. The paper evaluates benchmark tasks; it does not test this practice in a workplace deployment.

Impact across sectors

• Possible use in newsroom research: editors could ask whether a research assistant carries a source disagreement into the brief a journalist reads. This is hypothetical, not a measured newsroom result.
• Possible use in business analysis: teams comparing retrieved reports could check whether an agent names an unresolved disputed figure. This is hypothetical, not a measured business result.
• Possible use in software-tool evaluation: builders could score whether an agent follows up on a conflict and whether its final answer discloses one that remains. This is hypothetical, not a tested deployment.

Where the evidence stops

The authors evaluated five fixed agent-harness and backbone-model pairings, not every combination, and did not isolate the separate effects of model, harness and evaluation environment. Their longer-task conflict and control sets contain different questions, so differences may also reflect task composition. Identify, Solve and Escalate cover knowledge conflict, not every kind of uncertainty or honesty. Scores depend in part on model judges; the authors also report a small human-labeling study. The paper’s tables differ on the OpenHands backbone name, listing GPT-5 in the main results and GPT-5.2 in a roster, so this article does not attribute a numerical OpenHands result to either version. General PtoP note: this arXiv source is a preprint, and benchmark findings are not evidence of workplace deployment.

In one sentence

An image generator can sometimes learn from a correction and produce a better image from the original request alone, the authors report. Imagine asking for two red cups and getting three. A visual critic spots the extra cup, and a prompt writer restates the request with the count made explicit. During training, a reference version of the generator sees that revised request; the version being trained sees only the original. At successive stages of making an image from noise, the trained version is nudged toward the reference version’s next moves. The authors call this on-policy self-distillation: learning from comparisons made along the trained generator’s own image-making path. Their tested implementation updates the image generator, while the feedback components stay fixed.

Key concepts

• Critique and revision: the system identifies a visible mismatch with the original request, then turns that observation into a complete revised image request.
• Teacher and student: these are two roles for the generator during training. The reference teacher sees the revision; the student sees only the original request.
• On-policy self-distillation: the student learns at stages reached during its own attempt to make an image, rather than training on a corrected image as the target.
• Post-revision verification: in one recipe, the system makes another image from the revised request and admits the correction for training only if the critic accepts that image against the original request.

A concrete example

Hypothetical illustration, not a reported test case: a designer requests a picture of two red cups on a table, but a draft contains three. A revision specifies exactly two red cups. In the verified recipe, an image made from that revision would first have to pass the critic’s check against the original request; the revised words, not that image, would then guide training.

What the researchers measured

Verified Qwen recipe — with post-revision verification: in the authors’ paired direct-generation GenEval evaluation, the base Qwen-Image-2512 model scored 0.748 and the trained UniEvo-VL model scored 0.818. GenEval’s native score runs from 0 to 1 and averages correctness across categories of requested objects and their properties. The verified recipe also improved the reported direct-generation native scores on GenEval2 and the text-rendering task. Unverified Qwen recipe — without that second check: the authors’ separate GenEval comparison reports 0.747 for Base and 0.808 for UniEvo-VL. Its GenEval2 native score fell below Base, and its text-rendering scores were mixed or lower. These recipes used different settings and were not paired against each other, so their figures should not be treated as a direct test of the verification step alone. The authors also report a separate, unverified recipe using an external GPT-5.6-Luna critic; its results belong to that configuration, not the Qwen critic.

Why it matters

The distinction is between fixing one image after feedback and changing the generator so that its first attempt improves. The authors test both direct generation and generation with another chance to reflect. In their paired evaluation of the verified recipe, an additional reflection opportunity still improves the trained model’s reported scores.

Where it might help

A possible use is training an image generator to retain lessons from recurring corrections to object counts, placement or appearance. The paper measures benchmark images, not performance in a production design process.

Impact across sectors

• Possibility, not a proven deployment: graphic designers could investigate whether learned corrections reduce repeated revisions to image briefs.
• Possibility, not a proven deployment: teams making product mock-ups could examine whether counts, colors and placement become more reliable.
• Possibility, not a proven deployment: sign or poster makers could investigate text rendering, while noting the paper’s mixed results for that task.

Where the evidence stops

The authors report that gains varied by configuration and task. In historical, partial 500-prompt analyses, initially easier prompts had small score declines; those subset findings are not full-test-set gains. Verification accepted only some proposed corrections and added generation and critique work. GenEval uses an extensively reused benchmark, and its training prompts come from GenEval-format metadata; the authors use a separate training collection and the newer GenEval2 evaluation in part to reduce reliance on GenEval and potential benchmark-specific effects. They also note that GenEval’s automated checks can misjudge images, while their visual-quality rating is an automated proxy, not a human study. Their experiments validate the tested image-generation method, not the proposed extension to other model architectures. General PtoP note: this arXiv source is a preprint; benchmark results are not evidence of a tested workplace deployment.

In one sentence

The authors ask whether current methods can reliably spot generated videos of real crises. Their answer, within the tests they ran, is no: results changed with the video generator, the way a detector was asked to review a clip, and simulated circulation. To make the test, they started with real event footage, took the first frame of each retained clip, and gave that image and a shared description to video generators. They then compared each generated continuation with its real starting clip.

Key concepts

• A real-video anchor is an actual clip used as the starting point for a matched comparison. The generated clip begins from its first frame, as a fabricated continuation of a real still might.
• A detector is a method that judges whether a clip is generated. The authors tested conventional scoring tools, general-purpose models asked to judge videos without detection-specific training, and models fine-tuned for this task.
• Source generalization means that a detector remains useful when the generator changes. The authors report that none of the three detector families performed consistently across RA-Bench generation sources.
• RA-Bench-HumanProof is the authors’ subset of generated clips that all five assigned reviewers called real. RA-Bench-LastMile is a separate simulation of what happens to detector decisions during social dissemination.

A concrete example

Hypothetical illustration, not a study result: a newsroom receives a clip that appears to continue a genuine image of a flood. An editor compares the clip with its known source and seeks other evidence rather than treating one detector’s ‘real’ answer as verification.

What the researchers measured

RA-Bench contains 1,830 real-video anchors and 16,056 generated clips. Across its nine generation sources, the authors report that none of the three tested detector families generalized consistently. In RA-Bench-HumanProof, all five assigned reviewers labeled each of 633 generated videos real; on that selected subset, the seven traditional detectors averaged 47.5% AUC. AUC measures how often a detector ranks a generated clip above its matched real clip for a fake score. Separately, in RA-Bench-LastMile, the simulation’s Full condition reduced mean fake recall across five fine-tuned detector configurations from 46.0% to 1.4%. Fake recall is the share of generated clips flagged as fake. ‘Full’ denotes the simulation’s combined processing condition, not an individual detector or generator; the supplied text does not identify its component transformations, so their nature cannot be specified here.

Why it matters

The authors found weaknesses both in generated clips selected because people mistook them for real and in a simulation of social circulation. For work that depends on timely crisis footage, the distinction between a detector score and a verified account of an event matters. That workplace implication is an editorial inference, not a tested outcome.

Where it might help

The authors present RA-Bench as a way to study detection across generators, human judgments, and simulated dissemination. A possible use is to test how a video-checking process handles those different conditions; the paper does not report a deployed newsroom or public-agency process.

Impact across sectors

• Possible, not a tested deployment — journalism: video desks could use the findings to frame detector output as one piece of evidence when checking crisis footage.
• Possible, not a tested deployment — emergency communications: teams responding to a circulating disaster clip could plan for additional source checks before repeating its claims.
• Possible, not a tested deployment — public health: communicators could consider how to verify apparent emergency footage that has been copied or reposted.

Where the evidence stops

The authors describe RA-Bench as a snapshot of a changing field. It covers nine image-to-video generation sources and visual-only detection; audio was removed from the clips. Closed-source providers’ content-safety filters rejected some generation requests, so those sources were evaluated only on returned clips and their matched real anchors. The authors studied generation and dissemination as separate controlled stages, not an end-to-end forgery campaign. Human reviewers judged clips in a controlled interface without the surrounding online context. Public-reference detector scores came from other datasets and are context, not matched-domain baselines. General PtoP note: this arXiv source is a preprint, and a benchmark or simulation is not a tested deployment.

In one sentence

The study asks whether researchers can plan, carry out, and write up a project with an existing coding agent while keeping decisions and work easy to retrace. The authors built Dr. Claw as a workspace around such an agent, not as a new agent. For example, if an experiment cannot find a file, the researcher can inspect the recorded step, correct the path, and continue while retaining earlier files. That describes the intended workflow; the paper also shows one induced-failure recovery demonstration. The workspace connects a task plan, saved outputs, a record of decisions, and a record of execution. A researcher sets goals and decides what to accept.

Key concepts

• A bare agent is the same underlying coding agent without Dr. Claw’s added workspace. In the automated comparison, both conditions used the codex provider with model gpt-5.4 under the same permission profile.
• The task graph is a saved plan showing tasks and their dependencies. The artifact store holds outputs, while the decision log and execution trace preserve decisions and actions.
• Skills are reusable task instructions. Dr. Claw can suggest them for a stage of work, but the agent is not required to use a suggested skill.
• Human-in-the-loop means the researcher retains control of direction, checkpoints, and final acceptance rather than handing over the entire project.
• Research completeness is the share of 21 specified research components found in the produced files. It counts coverage, not whether the science is correct.

A concrete example

Hypothetical example, not a study result: a researcher asks for a literature review and an experiment plan, checks the suggested tasks, rejects an unsuitable source, and later uses the saved decision record to explain why the plan changed.

What the researchers measured

In the authors’ open-ended medical-research pilot, Dr. Claw using the codex provider with model gpt-5.4 had pooled research-completeness scores of 0.952, versus 0.873 for the bare agent using the same provider, model, and permission profile. Those scores measure the fraction of 21 components present in the output files, not their correctness. Dr. Claw scored higher on two tasks and tied on the third; with one run per task, the authors say the pooled difference was not yet statistically significant. In a separate, single-condition Derm7pt failure-recovery demonstration with the same backend, the authors report retaining all 5 pre-existing files after correcting an induced wrong-path error; there was no matched bare-agent recovery run. In a retrospective study of seven AI PhD researchers, Dr. Claw was associated with shorter completion-time bands and higher experience scores than the two reported control conditions. The authors describe those human-study statistics as exploratory, not causal.

Why it matters

The comparison holds the underlying agent fixed, so it addresses what the combined workspace adds to that agent in the authors’ pilot tasks. It does not isolate which workspace component accounts for a difference.

Where it might help

A possible use is organizing an AI-assisted research project whose plans, files, and revisions would otherwise sit in separate tools. The paper studies research workflows, not routine use across workplaces.

Impact across sectors

• Academic research — possible use: keep a retraceable record while moving from a research question to experiments and a draft; this is not a proven deployment.
• Research software teams — possible use: inspect task progress and earlier files when an AI-assisted experiment fails; this is a hypothetical extension, not a measured sector result.

Where the evidence stops

The pilot has one run per task, and its tasks come from the medical domain. The completeness measure checks for components, not scientific soundness. The comparison tests Dr. Claw’s task graph, saved state, instructions, and skill library together, so it cannot identify an individual component’s effect. Suggested skills do not always get used: the authors say a reference audit did not run on the tied task. Dr. Claw was slower in the automated comparison. Its process records are its own file format, so their presence alone does not establish better format-neutral traceability. The failure recovery is one demonstration without a matched control. The human study combines live Dr. Claw logs with retrospective reports for its controls; the authors also note possible familiarity and recall effects, a small sample, and the exclusion of model runtime from the experiment-stage timing. Transfer to other research fields remains unshown. The supplied paper is an arXiv preprint; that is a source-status note, not a claim about the study’s findings.