An Anatomy of Coding Agents for Robotics Tasks

Tianyu Li · Dexmate
CoRL 2026 Workshop on Agentic Robotics · Austin · Nov 12, 2026

Coding agents can tackle robotics tasks

2

But they can also be unreliable

3

So we investigate how coding agents address robotics tasks

Put the fruit into the basket.
Place the bowl on the rack, release it, then bring it back to the table.
Open the door and enter the room.
Claude Code · Opus 5.5Codex · GPT-6 AstraMCP toolsobserve · move hand · turn headgripper · step baseControl serviceIK · joint-limit +self-collision checksRobot→→→observations + status
Agent learningfirst meeting · builds missing toolssolves the taskAgent operatingfresh session · reuses what it learnedsame + rearranged scene→
4

Successful execution still takes minutes

Opus 5.5GPT-6 Astramin–maxMean time · minutes0481216FruitLearning4.1710.56Execution2.439.62DoorLearning6.8512.24Execution5.1110.77Mean cost · $$0$3$6$9$122.265.331.214.332.807.352.037.31
100%
fruit + door success, both models
0%
bowl-rack success,
original tools
Opus 5.5GPT-6 Astra
Clear a table of 20 objects≈ 1 h · ≈ $25> 3 h · ≈ $90
Pass 5 doors to come find you≈ 25 min · ≈ $10≈ 1 h · ≈ $40
5

Observe-think-act

6

Thinking Time > Action Time

20–36%
robot actions
64–80%
everything else
Opus 5.5Act33.2%Think (model between steps)53.1%Waiting12.2%Tool calls1.5%GPT-6 AstraAct20.6%Think (model between steps)71.6%Waiting7.0%Tool calls0.9%
Think time that ended in…Opus 5.5GPT-6 Astra010203040% of task timedeciding the next action28.038.5checking the result again6.124.2computing geometry with code8.74.2re-reading the saved skill or docs4.83.9deciding to wait · final answer5.60.8
7

126 seconds for a 6.5 cm correction

0:00 control.observe → save RGB-D data
0:17 Python check 1 · masks and 3D extents
0:45 Python check 2 · camera and target projections
1:18 Python check 3 · rim fit at four heights
1:42 Python check 4 · banana extent by image band
1:59 saved image → enlarged crop
2:06 control.set_hand_pose · ΔY = +0.065 m
4.2 s move126 s holding still → 6.5 cm correction2:16 / 2:16
0.0 s motion starts · execution started
4.2 s motion complete
6.6 s sleep returns
8.6 s next command
motion 4.2 ssleepresponse0.0 s4.2 s6.6 s8.6 s
8

Saved skills still need model judgment

Built and saved by the agent

  • geometry helpers — pixel → 3D point, rotation math
  • a planner that keeps earlier decisions — e.g. the 45° waypoint after a refused rotation
  • a skill file — steps, checks, lessons from failed motions

Still decided by the model

  • which targets to pick in the image
  • how the fingers make contact
  • what to do with feedback — retry a failed lift, re-locate the basket
−33%Opus
−10%Astra

task time when reusing a saved skill (mean)

9

Seeing danger is not stopping

0.0 s“high force on that arm …
should retract”
+16.8 sanother 15 cm forward —
arm as a “bumper”
+48.2 sretract
These motions reported normal completion.
Unacknowledged contacttable contact, never mentioned
Mistaken progresspushing a latched door
Delayed interventionarm hits the rack; operator stops it
These motions reported normal completion.
Also seen: an angle sent in degrees where radians were expected — a rotation ≈ 57× too large.
10

What we learned so far

01

The agent runs a ReAct loop: observe, think, act, repeat.

02

Thinking time > action time.

03

It builds geometry and planning helpers, but keeps the key judgments.

04

It can make unsafe moves, even when it recognizes the risk.

11

What if we empower the coding agent?

01

Give it more tools

02

Ask it to build a simulation first

12

What we added

Claude Code / CodexMCP toolsobserve · move hand · turn head · gripper · step baseControl serviceIK · joint-limit + self-collision checksRobotLocate objectsSAM 3 segmentationFind reachable posesorientation sampling + IKCheck pathssampled checks incl. table and obstaclesCoordinate motionarm + torso waypoints
13

More tools did not consistently help

SuccessTime (min)Cost ($)
Opus · original → expanded100% → 100%4.4 → 4.71.93 → 2.23
Astra · original → expanded100% → 83.3%door: 100% → 66.7%10.6 → 12.95.99 → 8.81
Expanded runs using each toolMean calls / runcheck_waypoints100%11.7move_whole_body100%9.9segment91.7%5.0reachable_poses62.5%1.4
So why no gain?
14

Added tools shifted effort into checking

ActDeciding the next actionChecking · querying · computingOther thinkWaitingOpus 5.5Original33.2%28.0%20.0%6.7%12.2%Expanded30.1%20.8%35.4%4.8%8.9%GPT-6 AstraOriginal20.6%38.5%32.6%1.3%7.0%Expanded15.7%18.0%61.1%0.6%4.6%
Tools moved the work: preparing waypoint and reachability checks alone took 14% (Opus) and 25% (Astra) of task time.
15

Build a simulation first

You can build a physics simulation of the real sceneand run your whole solution in it first,then do it on the real robot.Blender and Isaac Sim are installed.
16

How the agent builds the scene

From observationsChosen by the agent
table plane fitted from depthbowl mass 0.15–0.25 kg
rack rails and dowels measured from depthfriction 0.35–1.2
robot model (provided)gripper gain and torque cap
“Physical values assumed.” — the agent’s own notes
17

Rehearsal changes the grasp strategy

1 · grasp across the diameter — no lift
2 · adjusted approach — slips
3 · rim pinch — lifts
simulation 10×
real robot 10×
46.6 min
observe + build + rehearse — run shown
2.4 min
on the real robot — run shown
1 of 4
standard sim runs succeeded
18

What our experiments show

Coding agents, with the large models underneath, can tackle some robotics tasks.
But three issues remain:
01

Time

slow steps; the robot waits for the model

02

Safety

sees danger, doesn't stop; no undo

03

Motor skills

no reliable motor-level skills to call

19

Model latency cannot set the control rate

0.01 Hz0.1 Hz1 Hz10 Hz100 Hz1000 HzVLMGemini 4 Carbon 0.041 Hz (24.3 s)GPT-6 Astra 0.13 Hz (7.44 s)Claude Opus 5.5 0.30 Hz (3.37 s)DeepSeek V4.1 Flash 1.0 Hz (0.96 s)DeciderGPT-6 Luna Decisions 3.8 Hz (0.26 s)Jev 1.14 9.1 Hz (0.11 s)Our agent loopsAstra full agent loop (our runs) 0.08 Hz (12.1 s)Opus 5.5 full agent loop (our runs) 0.18 Hz (5.6 s)Fast-WAM ≈5 Hz (190 ms inference)π0.5 50 HzSONIC policy 50 Hz
● VLM / ◆ decider: 1 / end-to-end latency · ◇ full agent loop: 1 / loop time · △ policy: command or loop rate (Fast-WAM: 1 / inference latency)

Coding task

the world waits · many attempts in parallel

Robot task

the world keeps moving · one body, one thread

Model latency: OpenRouter, average end-to-end latency, best provider, past week (Nov 11, 2026).
20

Safety needs a local response

Coding task

  • git revert
  • mistakes caught before they matter

Robot task

  • no undo
  • asked to retract 48 s after noticing high force
  • stopping must happen in milliseconds — without waiting for the model
21

Plans depend on reliable motor skills

Coding agent's elementary skills

write a file · run the tests

precise · composable

Robot motor skills

continuous · contact-rich · few mature

  • built its own geometry helpers
  • bowl rack: 0% with the original tools
22

We need an AI-native robot system

A large model alone is not enough.
We build a robot system to support the large model, so the whole robot agent can finish tasks reliably and efficiently.
23

An AI-native robot system

cloudon deviceRobot brain< 0.5 Hzturns the user's task into an explicit plan and instructs the levels belowlarge model — GPT · Claude · Gemini · Musememorycontext managementtask trackingTask execution & grounding1–10 Hzturns language subtasks into motor-skill commandsVLAmotion plannernavigation plannersubtask monitorPrimitive skills20–100 Hzsolves local tasks — motion tracking, moving an object to a targetwhole-body control policyinteractive policyRobot service200–1000 Hzdrives the hardware safelylocal collision avoidanceself-collision checksrequestsstaterequestsstaterequestsstatedirect access
24

What should a robot remember?

Earlier: the robot moves the lemon to another table.

Later: “bring the apple that was to the lemon’s right.”

Saving images: higher success and lower token cost.
MemoryTask
success		Reasoning tokens
/ request		Read $ / success
info · action
text only39%4.2k0.031 · 0.055
descriptions (1024 tok)60%3.3k0.030 · 0.051
frames70%3.1k0.026 · 0.047
frames + scene graph75%3.1k0.028 · 0.049
More information in → less reasoning, despite a larger input.
Gemini 3.7 Flash, 53 simulated tasks; $ at $0.75/M in, $3.75/M out.
Wang, Liu, et al. What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents. arXiv:2610.12561, 2026.
25

An AI-native robot system

cloudon deviceRobot brain< 0.5 Hzturns the user's task into an explicit plan and instructs the levels belowlarge model — GPT · Claude · Gemini · Musememorycontext managementtask trackingTask execution & grounding1–10 Hzturns language subtasks into motor-skill commandsVLAmotion plannernavigation plannersubtask monitorPrimitive skills20–100 Hzsolves local tasks — motion tracking, moving an object to a targetwhole-body control policyinteractive policyRobot service200–1000 Hzdrives the hardware safelylocal collision avoidanceself-collision checksrequestsstaterequestsstaterequestsstatedirect access
26

How does a robot know it’s done?

Simulator: Unfinished8.0 / 60 s2×
0102030405060doneundone
Reaches the goal — and keeps moving.
Today’s robot models don’t know when to stop.
We need a model that tracks task progress.
Best: GPT-6 Astra86% · 4.0 s · $22.54 / 1k checks
Fast, cheap models≈ 50–64%, close to guessing

With simulator facts instead of images

Best: GPT-6 Astra98%
Fast, cheap models≈ 51–68%, no gain
Li. The Unstoppable Robot. Blog post, 2026.
27

A renaissance of robotics problems

Large models now take on the semantic side
understanding the request · planning · judging what to do next
Back to robotics
01Reliable systems around a slow, sometimes-wrong model
02Perception that sees the scene correctly
03Clear interfaces between modules
04General, dependable motor skills
05Fast, safe execution — every time
28

Thank you

Tianyu Li
ty.li@dexmate.ai
easypapersniper.github.io
29