Live-website tests show where browser agents get stuck

In one sentence

The study asks whether web-browsing AI assistants can find short, verifiable answers when the clues cross languages and kinds of evidence. The authors built HyperBrowseComp, a benchmark—a set of questions used to test systems—with 423 questions written directly in 13 languages. Writers recorded a supporting route to each answer, and another person checked the question and evidence. The authors then tested selected models with different ways of searching the live web. A question might start with a scene in a video, require locating that scene on a map, and end in a financial report. The challenge is finding and connecting those sources, not writing a long answer.

Key concepts

• A browsing agent is an AI assistant that chooses searches and uses web tools while working toward an answer. The paper tests particular combinations of a model and its search tools, rather than a model in isolation.
• Multilingual means the questions were written in their own languages, not translated from one English question set. The authors say this preserves local phrasing and references.
• Multimodal means evidence can come in forms beyond ordinary webpage text, such as video, images, maps or scanned documents. The authors note that not every question requires non-text evidence.
• A retrieval harness is the set of tools and rules through which an agent searches and opens sources. The study compares provider-built-in search, a shared search service called Exa, and a separate multi-agent system called OWL.

A concrete example

Illustrative question from the paper, not a model result: a German prompt points to a moment in a video-game video. Following that clue leads to a real street location, a nearby bank and then an entry in a financial report. Each source supplies a piece of the route to a short answer.

What the researchers measured

On the authors’ 423-question benchmark, Gemini 3.7 Flash with its provider’s built-in search had the highest reported accuracy: 31.68%. Accuracy counts answers judged correct against the short reference answers. Across the five models tested with built-in search, 244 of 423 questions (57.68%) had no recorded correct answer. This second figure describes the group of tested runs, not a score for Gemini 3.7 Flash alone; the appendix notes that two questions in this group had unresolved model outcomes.

Why it matters

The authors report that changing the search setup changed performance for tested models. That makes the search tools part of what a benchmark score describes. The questions also bring together languages and source formats that a text-only, English-focused test would not capture in the same way.

Where it might help

Possible use: someone developing or selecting a research assistant could use questions of this kind to examine how it handles cross-language clues and evidence that is not plain webpage text. The paper evaluates benchmark questions, not workplace use.

Impact across sectors

• Possible impact for newsrooms: a hypothetical multilingual research task could check whether an assistant traces a video claim back to a document, rather than relying on a search snippet. This was not a newsroom deployment.
• Possible impact for libraries and archives: a hypothetical evaluation could probe searches that connect scanned material with other public sources. The paper does not test an archive service.
• Possible impact for AI product teams: the benchmark could help compare a model paired with different search tools. The reported scores apply only to the combinations the authors tested.

Where the evidence stops

The authors designed an unusually difficult stress test, not a representative sample of everyday searches. Live webpages, rankings, geographic access and tools can change, so an exact rerun may differ. The selected languages do not represent every community, and searchable information differs between languages; language-level failure patterns therefore do not isolate language ability. The main study covers five models, Exa was tested with only a subset, and the separate OWL setup had terminal runtime failures counted as incorrect. Evidence-format categories can overlap, and their annotations describe documented evidence routes rather than proving no other route exists. The human comparison used only a sample of questions in Indonesian, Thai and Vietnamese. General PtoP note: this arXiv paper is a preprint, and a benchmark result is not evidence of deployment performance.

In one sentence

The study asks how a video model can keep its next move connected to a final goal. The authors’ answer is to have ProAR predict a goal image while it generates each short stretch of video, and to teach it during training to anticipate what the following stretch should contain. Imagine moving tiles into place: a move can look sensible on its own yet leave the puzzle unfinished. ProAR’s goal prediction gives the current move a view of the intended ending. A rule about which information the model may use lets the predicted goal guide that move, but stops the uncertain current move from changing the goal prediction at the same step. The model revises its goal prediction as more video is generated. Separately, during training, it compares what its current internal state suggests will come next with information from the actual next stretch of training video. The small predictor used for that comparison is removed for generation; the goal prediction remains.

Key concepts

• Chunk-by-chunk generation: An autoregressive video model makes one short stretch of video after another, using earlier stretches as its history. It cannot go back and repair an earlier generated stretch.
• Outcome guidance: At each generation step, ProAR predicts an image of the anticipated final state alongside the current stretch. During training, the actual final frame supplies the target for this prediction.
• One-way information flow: The predicted goal can inform the current stretch, but the current stretch cannot inform that goal prediction at the same step. The goal can still be updated at the next step using the longer history.
• Transition guidance: During training only, a small predictor tries to make the model’s current internal description anticipate its description of the actual next stretch. This is a training objective, not a separate next-stretch predictor used during generation.

A concrete example

Hypothetical illustration, not a reported test: A model generates a video of tiles being slid into order. One slide might look reasonable but block the next move. A predicted image of the finished arrangement could help guide the slide, while training on what actually happens next could encourage attention to the intervening move.

What the researchers measured

On the authors’ selected 10-task VBVR subset, ProAR received a mean score of 0.801, compared with 0.663 for their Standard AR baseline under the same settings. VBVR’s task-specific scoring assesses generated videos against task goals and reference solutions on a scale from 0 to 1; the mean is not a share of tasks completed. On VideoRLVR’s three games, the authors report average success rates of 52.97 for ProAR and 50.97 for Standard AR. In the selected VBVR setting, they also report that ProAR exceeded the fully trained Standard AR score after 2,500 training steps, while that baseline had been trained for 10,000 steps. Their separate WorldArena evaluation reports improvements over Standard AR in a simulated manipulation setting.

Why it matters

The authors study a gap between making each short stretch look plausible and making the whole visual sequence finish a task. Their method gives the model an anticipated ending during generation, while using the actual next stretch only as a source of guidance during training.

Where it might help

Possible applications, not proven deployments, include research tools for step-by-step visual puzzles and simulators that generate videos of object manipulation. The paper reports benchmark and simulated-task results, not use in a workplace.

Impact across sectors

• Puzzle and learning-tool development — hypothetical: a developer could explore whether goal-aware video generation makes illustrated solutions easier to follow; this use was not tested.
• Robotics simulation — hypothetical: a simulation team could study generated manipulation sequences before considering real-world use; the paper’s WorldArena evaluation is simulated, not a robot deployment.

Where the evidence stops

The authors say that using the final video frame as the goal may give weak guidance when that frame says little about the important process, such as after a task has finished and become idle. Their VBVR results cover a selected 10-task subset, not all 100 tasks; because the official benchmark has five test examples per task, they used its data generator to make 50 per selected task. WorldArena tests simulated manipulation, not physical robot operation. PtoP context: this source is an arXiv preprint, and benchmark results should not be read as evidence of deployment.

In one sentence

A checklist can give an AI answer credit for something the answer never did. The authors call this failure “Vacuous Credit.” For instance, a checklist might require an assistant to ask where a patient lives, but a judge might give credit when the assistant merely says that guidance varies by region. The authors first tested whether such credit survived when required content was deleted from answers. They then built MetaRubric, a training method that checks whether each requirement is present and supported, and revises checklist wording and priorities as training proceeds.

Key concepts

• A rubric is a task-specific checklist whose requirements contribute to an answer’s training score.
• Vacuous Credit means a judge scores a requirement highly even though the answer omits the information or action it requires.
• Evidence-aware scoring limits credit to the weakest of three checks: whether a requirement is fulfilled, whether the answer contains it, and whether permitted evidence supports it.
• A counterfactual prompt changes one relevant fact while keeping the rest of the case fixed, so training can attend to requirements that depend on that fact.
• Rubric adaptation adjusts the emphasis on groups of requirements the model keeps missing and proposes clearer wording. A proposed wording change must preserve the original requirement and pass checks on held-out responses.

A concrete example

Hypothetical illustration, not a study result: A customer asks whether a service is available at their address. A checklist requires the assistant to ask for the address. Saying “availability varies by location” mentions the topic but does not perform the required action. An evidence-aware check would look for the actual request for the address.

What the researchers measured

In a deletion test, the authors report that GPT-4o-mini retained 82.0% of the target-criterion credits it had already awarded to intact responses after the content required for those criteria was removed. That figure is retention of previous awards, not the share of arbitrary answers receiving credit. On PubMedQA’s expert-labeled test set, the authors report that Qwen3-4B trained with MetaRubric gained 6.00 percentage points in answer accuracy over the Qwen3-4B static-judge GRPO training baseline. Accuracy here counts answers with the correct yes/no/maybe label. The authors also report improvements over that training baseline for their other named model families on PubMedQA, and on HealthBench-Hard, MMOral-X and MMOral-OPG under those benchmarks’ respective scores.

Why it matters

In the authors’ training setup, checklist scores shape which answers the model learns to favor. They show that credit for omitted content can distort that signal. Their method makes presence and support explicit checks, rather than relying on a judge’s overall impression of an answer.

Where it might help

The paper studies training and benchmark evaluation, not a deployed service. A possible use of its approach is designing training scores for assistants whose answers must satisfy several distinct requirements, especially where a polite generality could be mistaken for a specific answer.

Impact across sectors

• Possible healthcare use, not a proven deployment: teams designing medical-answer checklists could distinguish a requested action from general safety language.
• Possible customer-support use, not a proven deployment: teams training support assistants could check whether an answer actually requests a missing detail rather than only acknowledging it.
• Possible education use, not a proven deployment: teams building feedback tools could check whether a response supplies the reasoning a marking guide asks for.

Where the evidence stops

The authors used one seed per training configuration, so their reported results do not quantify variation across runs. They say unavailable training code for many related works prevented direct comparisons under matched settings. The two MMOral benchmarks evaluated models trained on a shared MMOral-RL training set. The paper’s auxiliary-question evaluation covers 965 of HealthBench-Hard’s 1,000 source entries; the remaining entries have no extracted targets. That auxiliary measure also uses the paper’s specified reader and is a training objective, rather than an independent measure of every aspect of answer quality. General PtoP note: this arXiv source is a preprint, not a claim of peer review; benchmark results are not a tested deployment.

In one sentence

The authors ask whether a model that reuses its processing layers can answer questions about images effectively. Their model, LoopVL, takes features from a frozen visual encoder—a component that turns image patches, or small image regions, into numerical descriptions—and combines them with the question. It then revisits that combined visual-and-language state through shared processing modules. In ordinary terms, it is more like making another pass over the same evidence than adding a new set of layers for every pass. The authors train a language backbone, connect it to image features, then continue training on image-and-text tasks before testing the resulting vision–language model.

Key concepts

• Recurrent computation means applying the same learned processing module repeatedly while the information passing through it changes. LoopVL reuses modules within one image-question response; the paper does not claim memory across separate questions.
• Module-Loop repeats a lower-level module several times while the higher-level state stays relatively stable. Model-Loop then updates that higher-level state and starts another cycle.
• A visual state is the model’s changing internal description of an image patch. LoopVL can update these states after the visual encoder has produced its initial features, without running that encoder again.
• Attention is a measure of where a model layer allocates its reading weight. The authors call a marked change in visual attention between cycles a “Visual Aha Moment”; an attention shift alone does not prove which image regions caused a correct answer.

A concrete example

Hypothetical illustration, not a reported test result: suppose someone asks how many pieces of debris lie beside a van in a photograph. An early pass might spread attention across the van and road. A later pass might give more weight to the area beside the van before producing an answer. This illustrates the kind of revisiting the authors examine, not a guarantee that the count improves.

What the researchers measured

In the authors’ comparison at the same stated training-token budget, default-schedule LoopVL received an MMStar accuracy score of 63.47, versus 55.33 for the non-recurrent Transformer-VL 1B baseline. An accuracy score here reflects answers counted correct under that benchmark’s scoring rules; it is not a workplace success rate. The models have the same number of unique Transformer layers, but LoopVL executes its shared layers repeatedly and the comparison does not hold training computation equal. The authors also report that, among separately trained recurrence schedules, the default schedule scored highest on five tested benchmarks. In a separate inference-time test of that trained checkpoint, adding more cycles did not automatically improve scores.

Why it matters

The authors separate two ways to give a model more processing: store more distinct layers, or reuse layers while its internal picture of the image changes. Their comparisons show why that distinction matters for the tested tasks. Reuse reduces the need for separate layer parameters, but it does not mean equal running time or equal computing cost.

Where it might help

A possible application is an image-question assistant for tasks where the question points to a small part of a larger picture. The paper measures benchmark answers and internal model behavior; it does not test such an assistant in routine work.

Impact across sectors

• Possible, not a tested deployment: education teams could explore image-question tools for interpreting classroom diagrams.
• Possible, not a tested deployment: document-processing teams could investigate repeated visual reading for questions about charts or text-rich pages.
• Possible, not a tested deployment: field teams could explore questions about details in photographs, while checking answers against the images.

Where the evidence stops

The authors’ visual-attention averages use a 32-example diagnostic collection, while selected image cases illustrate changes without estimating how often they occur or proving that attention changes cause better answers. They have not trained a matched model with a more general Model-Loop architecture, so they say they cannot fully establish whether the reported Visual Aha Moments depend on that loop design rather than other choices. Their tests chiefly concern image-based tasks with a frozen visual encoder; native multiple-image, video, and interactive settings remain unexplored. They also report that extended reasoning can sometimes repeat tokens or reasoning segments under their limited-data post-training setting. The broad model comparison combines in-house evaluations with selected public results. General PtoP note: this arXiv source is a preprint, not evidence of peer review, and benchmark performance is not a deployment test.

In one sentence

The authors ask whether AI agents can carry out everyday online tasks on real websites. In their test, even the highest-scoring model passed only a minority of tasks. Imagine an agent entering details for a booking: finding the page is only the beginning. It must carry the right information through each form and reach the task’s specified endpoint. The authors built ClawBench around such workflows on live sites. For each task, a person recorded a reference run. A browser tool then recorded the agent’s actions and blocked an annotated final request before it could create the intended irreversible result.

Key concepts

• A browser agent is an AI system that observes a web page and takes actions such as clicking, typing and scrolling. Here, each tested model used the same OpenClaw browser setup.
• A write-heavy workflow asks the agent to enter user-specific information, often across several pages, rather than just find an answer.
• Final-request interception captures and blocks the annotated web request that would commit an irreversible action. The authors say this protection applies to the audited task endpoint, not to arbitrary browsing.
• A reference trajectory is a recorded human run used to show the intended endpoint and field values. The agent can take a different route if it reaches an equivalent allowed endpoint.
• Agent-as-Judge is the authors’ model-based evaluator. It compares the task instructions, human reference and recorded agent run, then gives a pass or fail with a reason. A pass means satisfying the task’s specified endpoint: this can be a correct intercepted final request, or a permitted stopping point after all required preceding steps.

A concrete example

Hypothetical illustration, not a reported result: an office worker asks an agent to prepare an appointment booking using supplied details. Under a ClawBench-style task, correctly filling the form would not alone establish a pass. The agent would have to reach the specified endpoint, or a permitted stopping point after completing the required earlier steps.

What the researchers measured

The authors tested eight models on ClawBench’s 153 tasks across 144 live websites, using the same OpenClaw browser setup. Claude Sonnet 4.6 had the highest overall success rate, 33.3%, and Qwen 3.5 followed at 26.1%. Success rate is the share of tasks that received a pass from Agent-as-Judge, not the share of real-world orders, bookings or applications completed. The authors report that 68 of 153 tasks were passed by none of the eight models. On human-reviewed results for Claude Sonnet 4.6 and GPT-5.4, the model-based judge’s verdicts agreed with human verdicts at the rates reported in the paper; that check did not cover the full panel in the main table.

Why it matters

The authors’ traces show why reaching the right page is not enough. Agents encountered anti-bot checks, entered wrong or missing values, and sometimes stopped near the final action. The recorded steps let researchers distinguish those outcomes rather than treating every failure as a navigation problem.

Where it might help

The authors present ClawBench as a research tool for testing browser agents on live, form-heavy tasks. A possible use is to examine whether an agent carries supplied details through a workflow and where it stops; the paper does not demonstrate an autonomous workplace deployment.

Impact across sectors

• Possible, not a proven deployment — office administration: teams considering help with detailed online forms could use this kind of test to identify unfinished steps and incorrect entries.
• Possible, not a proven deployment — travel services: teams considering help with booking flows could examine whether an agent reaches the required endpoint on a live site.
• Possible, not a proven deployment — recruitment: teams considering help with job applications could study whether an agent preserves the applicant’s details through submission steps.

Where the evidence stops

The authors say live websites can change by layout, region or account state, so exact reruns are not guaranteed. Live runs cost more than tests on saved pages, limiting repeated trials and variations. Scores depend on the shared browser setup and prompt, not just the underlying models. Pass/fail scoring can hide partial progress. The task collection underrepresents mobile-only, non-English and accessibility-dependent workflows. Final-request blocking is task-scoped, not a guarantee that all browsing has no side effects. The authors also report a sampled run marked as a pass whose judge rationale noted a phone number absent from the supplied user data. PtoP note: this source is an arXiv preprint, and a benchmark score is not evidence of a safe deployment.

In one sentence

A video can end in the right place and still show the wrong way to get there. The authors built VGI-Bench to test both parts: whether a generated clip reaches a visual goal and whether its intermediate steps obey the rules. A model receives a starting image and written instructions, then generates a short video. The benchmark contains 27 tasks and 810 instances, including moving a toy car through a maze and packing suitable objects into a bag. Its inputs are designed to look like realistic scenes, and its tasks are chosen to require a sequence of actions rather than a plausible final frame alone.

Key concepts

• Process-sensitive task: success depends on the route taken as well as the ending. In the maze task, a car must travel through corridors rather than pass through a wall.
• Completeness: a measure of progress toward the goal. The authors’ automated judge classifies a generated video as complete, partial or failed.
• Rubric Score: a measure of whether the video obeys a task-specific checklist. For the maze, that includes preserving the walls and keeping the car in the corridors.
• Final Score: the authors combine each video’s Completeness and Rubric Score, so a convincing ending cannot by itself erase a rule-breaking route.

A concrete example

Hypothetical illustration, not a reported model result: a generated clip shows a toy car reaching the maze’s goal, but midway through it passes through a wall. A viewer checking only the last frame might miss the shortcut; checking the clip against the maze rules would catch it.

What the researchers measured

On the authors’ image-to-video evaluation, Seedance 2.0 had the highest reported overall Final Score, 51.0. That score combines progress toward a task’s goal with adherence to its process checklist; it is not the share of clips completed perfectly. Under the authors’ stricter measure, which counts a clip only when both measures are perfect, Seedance 2.0’s overall success rate was 14.0%. These results come from the evaluated portion of VGI-Bench, not from workplace use. The authors also report that changing an input’s visual style affected measured performance, especially for open-source models.

Why it matters

The authors report that models can make some progress on visually grounded tasks while often losing track of objects, physical behavior or rules across a clip. For a reader using generated video, the distinction is between a scene that looks resolved and a sequence that actually depicts the requested procedure.

Where it might help

As a possible use, someone comparing video generators could inspect both goal completion and the steps shown in each clip. The paper reports benchmark evaluations, not a tested workplace deployment or a guarantee that its scoring will suit another task.

Impact across sectors

• Possibility for creative production: a team making instructional-looking clips could use the distinction between a correct ending and a valid sequence when reviewing drafts; the paper does not test that workflow.
• Possibility for education: a teacher could ask learners to spot impossible shortcuts in an imagined demonstration video; the paper does not test classroom use.
• Possibility for model research: developers could use process-sensitive tasks to investigate where generated actions break rules; the reported experiments are on the authors’ benchmark.

Where the evidence stops

The authors designed tasks for roughly 5–10-second clips, so longer procedures are outside the benchmark. It covers a starting-image-to-video setting at a fixed 16:9 shape, not text-only, multiple-image or audio-conditioned generation. Prompts and scoring checklists are in English, and the task collection is representative rather than exhaustive. Because evaluation is costly, the main video results use half the instances for each task; a full-instance check covers only a representative slice. The authors’ separate study of synthetic fine-tuning groups tasks by structural overlap with another training set, so its reported transfer depends in part on that overlap. Their human reference study used computer-science students who were paper co-authors, with response formats that differ from generated video. General PtoP note: this arXiv source is a preprint, not a claim of peer review.