The Unstoppable Robot

01Experiment setup

Jev only reads text, so it needs someone to look at the robot for it. I used a vision-language model (VLM) to describe the scene. But if a VLM is already looking, why not let it decide? I compared three routes: send its description to Jev, ask the same VLM to judge its own description in a separate call, or ask the VLM to decide directly. Every judge also gets the task and measured robot state.

I tried five VLMs across a range of sizes and costs: GPT-6 Astra, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Qwen3.8-27B, and Qwen3.5-9B running locally. I wanted to see whether a smaller VLM plus Jev could hold its own against a larger model doing the whole job.

Images

Now

One second earlier

Each image: scene + wrist camera

Task description

Put the red mug upright so that the opening is facing upwards.

Measured robot state

Joint positions
Joint velocities
End-effector pose

Now + one second earlier

VLM → Jev

Images+ task

→

VLMDescribe

→

Description+ task + robot state

→

JevJudge text

→

Finished?Yes/No

VLM → VLM

Images+ task

→

VLMDescribe

→

Description+ task + robot state

→

Same VLMJudge text

→

Finished?Yes/No

VLM directPlay flow

Images + task + robot state

→

VLMJudge images

⟶

Finished?Yes/No

For data, we ran pi0.5 on 15 RoboLab tasks ten times each, giving us 150 recordings. We let each run continue even after success, then selected 500 moments: 250 finished and 250 unfinished, labeled by the simulator. Each check sees the current moment and one second earlier, with both scene and wrist views. The question is always: is the task finished now? Most comparisons use all 500 examples; Astra uses a matched 50-example subset.

I measured accuracy, time for the full check, and API cost per 1,000 checks, aiming for checks under two seconds. This is an offline test on recordings; the monitor has not earned the stop button yet.

02Results and Findings

Gemini and GPT-6 Astra give the most accurate completion judgments. Their direct answers are slower and more expensive than those of the lower-tier models, but the cheaper alternatives remain much less reliable. The trade-off is clear: the models that recognize completion best cost more to keep checking.

Jev helps the lower-tier models improve on their direct answers, but it does not improve on direct Gemini or Astra. Separating description from judgment can help weaker models, although that benefit is not unique to Jev: Qwen3.8 does better judging its own descriptions. For the strongest models, going straight from images to a decision works best.

Direct answers are usually fastest. Describing the scene takes most of the time in the two-stage pipelines, so Jev’s fast judgment cannot remove the main delay. Even the fastest online direct answers exceed our two-second target at the median. Local Qwen takes just 0.36 seconds, but always answers “unfinished” on this benchmark. Speed alone does not make a useful monitor.

Separating description from judgment

A completion check has two jobs: recover what is happening in the scene, then decide whether it satisfies the task. A poor score could come from either step. To investigate, we bypassed the visual description and gave each judge physical measurements extracted from the simulator, alongside the same task instruction and robot state.

These measurements describe object positions, bounding envelopes, orientation relative to gravity, motion, and measured contacts at the current moment and one second earlier. They contain no success label, task-specific success code, or future information. We supply no checklist. The judge still has to connect the measurements to the task.

The simulator facts bring Astra and Gemini close to perfect accuracy. When the physical state is explicit, they can usually decide whether it satisfies the task. This suggests that recovering the scene from images is a major bottleneck, while their task reasoning is relatively strong on this benchmark. Qwen3.8 also improves, but DeepSeek and local Qwen remain near the constant-answer baseline: clearer evidence is not enough for every judge.

Jev’s score is slightly lower than with Gemini’s descriptions, although the confidence interval spans zero, so we cannot establish a decline. One hypothesis is that the simulator’s positions, orientations, and contact measurements are harder for Jev to digest than task-focused language. Rewriting the same measurements into simpler language would help test that explanation.

The diagnostic still has limits. These simulator facts omit exact mesh geometry used by some containment checks and the earlier history required by the ordered task. They also change the format and richness of the evidence, so remaining errors cannot all be assigned to reasoning.

Could a large model tell the monitor what to check?

Perhaps the monitor needed a clearer definition of “done.” We asked Gemini 3.8 Flash to inspect the task and initial scene once per trajectory and write a completion checklist for the monitor to reuse. The checklist identifies relevant objects, required relations and counts, necessary robot states, and limits on what the evidence can establish. It uses the task and pre-action image, without future frames, success labels, or simulator success code.

We then give the same cached checklist to each judge, keeping the cases, images, task, robot state, and descriptions fixed. The describing VLM never receives it. The chart below shows accuracy with the checklist and the change from adding it.

The checklist produces no consistent improvement. Most changes are small, while Astra → Jev becomes less accurate. For the description routes, extra guidance can clarify what to verify, but cannot recover missing facts. Explaining individual errors would require inspecting the checklist and decision together. Guiding the describer to collect the right evidence remains a separate experiment.

Conclusion

For now, we still need a strong model to make the judgment directly. Jev does not push the stronger models further: judging their descriptions, it does worse than letting them judge the images themselves. It does help the smaller models, making better calls from their descriptions than they make on their own, though still well short of a strong model judging directly. Even so, the strong models are not yet fast or cheap enough to check constantly, so there is still a gap before a completion monitor is truly usable.

03Final Thoughts

A quick decision is still the goal

I still like Jev’s basic idea: make a useful decision quickly, without writing an essay at every check. The robot does not need a philosophical discussion about the mug. Jev’s own judgment is fast and cheap, but the Jev-based system we tested does not yet deliver the accuracy, speed, and cost we want together. Generating the visual description takes most of the time.

My next bet is to train a small VLM specifically for the judging job, so it can give a short answer directly from the task, images, and robot state. RoboReward already explores this direction, training 4B and 8B vision-language models to score robot task outcomes (Lee et al., 2026). The question for our monitor is how well a small model generalizes to new tasks and scenes while staying within our time and budget. I would also want more than “finished” or “unfinished.” Is the robot getting closer, getting stuck, or undoing its own work? Learned rewards and progress signals offer useful starting points (Ma et al., 2023; Liang et al., 2026). A signal like that could help us decide whether to keep going or change strategy. We did not evaluate these models here; they are promising next comparisons.

Write once, check often

Another idea is to have a strong coding agent build the stopping checks before the task starts (Liang et al., 2022; Surís et al., 2023). It could wire existing perception tools into the checking code: SAM 3 (Carion et al., 2025) to segment and track objects, or LocateAnything (Wang et al., 2026) to locate objects from text descriptions. The code could combine those outputs with robot state, then check object counts, gripper release, or whether a relation has held for several frames. Pay for the coding once, then reuse the monitor throughout the run. Perception still costs computation, but we could avoid generating a fresh written scene report at every check.

The harder part is covering what can happen during a long, complex task. An object gets occluded, a grasp slips, a completed step gets undone, or a recovery changes the order of later steps. The generated program needs to handle these possibilities and their combinations, including missing or ambiguous observations. Each individual rule may be simple, but anticipating and testing all the relevant cases is hard. A coding agent could help, but “just add another if statement” can become a very long afternoon.

When should the robot give up?

Calling an attempt a failure seems harder than recognizing completion. My Claude put it nicely: “Failed isn’t something you see, it is a decision to give up, made relative to a budget.” That captures the recoverable cases: a dropped mug may still be one good grasp away from success. “Unfinished” alone gives no reason to stop trying. Left to itself, the robot can always keep calm and carry on.

In practice, I would give the monitor a few concrete reasons to stop or ask for help: an exhausted time or retry budget, a broken object or unsafe situation, or an unproductive loop. Confirming a loop needs memory; one snapshot cannot tell us this is the fifth identical attempt. Related work covers learned failure explanations (Duan et al., 2024), recovery from execution history (Liu et al., 2023), and asking for help under uncertainty (Ren et al., 2023). And a safety check should be able to trigger intervention before we decide that the task has failed.