An anatomy of coding agents tackling robotics tasks

What tools and skills do agents build?

Agents build helpers and retain key decisions

Across tasks and models, agents repeatedly build helpers for the same operations: converting camera measurements into robot coordinates, calculating poses from chosen targets and alignments, preparing command sequences, and checking what happened. Common tools include:

• Locate an object in 3D: combine depth, camera intrinsics, and camera pose to turn an image pixel into a point in the robot’s coordinate frame.
• Calculate hand poses and prepare moves: quaternion helpers convert the agent’s chosen alignment into rotations. Scripts produce command sequences from measured object positions.
• Check motion completion: watchers check whether the robot has settled and a commanded action has finished.
• Inspect geometry and outcomes: depth-cloud and gap checks help inspect a doorway. Other helpers preserve camera snapshots, compare observations, or calculate the space occupied by the robot’s body.

The details follow the task. For fruit, helpers calculate grasp and basket poses and support checks after a short test lift. For doors, they estimate lever contact and whether the whole robot will fit. Examples include a top-down depth view, a gap estimate, and a helper that uses the robot description to calculate link positions and clearance against the door frame.

Many important decisions remain with the core model. For example, in the learning stage, the agent often identifies the object and selects image pixels itself; its projection helper converts those pixels into robot coordinates. This division of work appears across four capabilities:

This distinction also separates motion completion from task success: a helper can report that the commanded movement has finished, while the model must still judge whether it achieved the intended result.

Skills combine procedures, tools, and lessons

During the learning stage, the agent develops a method and supporting tools, then records them for a fresh execution session. A saved skill combines a written procedure, reusable code, and lessons from the attempt. For example, one learning run on the fruit task saves the following files:

Inside the SKILL, the agent records the following five-step procedure (condensed):

1. Perceive. Identify fruit and basket pixels in the head-camera image, then use pixel_to_base.py with depth and camera pose to calculate their positions in the robot’s coordinate frame.
2. Plan. Pass the fruit and basket positions and the fruit’s orientation to plan_pick_place.py. The script prints a command sequence; the agent issues the commands one at a time and checks that each motion has finished.
3. Grasp. Approach from above with the fingers closing across the fruit’s width. Check the wrist-camera image to confirm that the fruit is centered between the fingers before closing the gripper.
4. Transfer. Back out through a tilted intermediate pose, then approach the basket with the hand yawed 45 degrees toward it. This records the route used to overcome the arm’s difficulty reaching sideways in the original orientation.
5. Release and verify. Lower to about 0.20 m above the table, open the gripper, retreat upward, and check in the head-camera image that the fruit is inside the basket.

The SKILL also preserves example fruit and basket coordinates, table and rim heights, and a hand orientation used for the transfer. Its lessons explain why the procedure includes intermediate moves: a large hand rotation was refused, so the agent split it with a 45-degree tilted waypoint. When packaging the skill, the agent encoded this intermediate waypoint in plan_pick_place.py and explained its purpose in the written procedure. It also notes that torso movement changes the head-camera pose, requiring fresh projection when the observation changes. These details describe the recorded attempt; the next session must check them against its own scene.

Providing tools does not reliably improve results

If agents repeatedly build similar helpers, can we save work by providing those capabilities in advance? We compare the original tool interface with an expanded interface. The original interface already provides camera images, depth, robot state, and Cartesian hand motion; inverse kinematics is handled by the control service in both conditions. The added tools expose object localization and more explicit motion planning:

To our surprise, more tools did not translate into better overall results in these runs. Mean completion time and model cost both increased in 75% of task/model/stage comparisons.

Changes are relative to the original interface; “+” means more time or cost. Means include successful runs only, and sample sizes are small. Learning is the initial attempt; execution reuses its saved skill in a fresh session. Time excludes subsequent skill packaging; model-session cost may include in-session writing, but excludes separate skill writing, judging and tool-service costs.

The agents did adopt the supplied tools: all fruit and door attempts used waypoint checks and whole-body motion, and more than 90% used segmentation. The traces show both the work these tools replace and the decisions they leave to the agent:

Perception replaces some calculations but still needs interpretation. In one fruit run, the agent used the 3D centroids returned by SAM3-based perception to set target positions, replacing manual pixel selection and projection for that localization step. It still rejected an uncertain extra basket detection as a shadow and worked out the hand’s alignment. In another run, an uncertain “banana” result prompted another query with “yellow banana” and “yellow fruit” and a zoomed crop. The tool supplies masks and geometry; the agent selects detections and judges whether the grasp worked.

Planning can expand into repeated search and revision. In one fruit execution run, the agent made five reachability queries and nine waypoint checks before its first motion, about 2.7 minutes into the attempt. Those calls occupied only about 23 seconds; the agent also inspected results, recomputed orientations, and revised the approach. Some checks caught paths too close to the table. The run succeeded in 4.79 minutes: the tools returned quickly, but using them still required substantial planning.

The supplied tools replace some helper calculations, while interpreting their outputs and adapting the plan still takes work. Agents continue to write task-specific geometry and pose code; some door runs even bypassed segmentation. These examples do not establish that SAM3 is worse than the model’s vision or that the checks were unnecessary. They show that tool use did not reliably produce faster, cheaper, or more successful task completion.

For the bowl task, the expanded toolbox was still insufficient: all provided-tools trials without simulation failed. One learning run placed the bowl on the rack, but seven attempts to regrasp the rim missed, leaving it unable to return the bowl to the table. The extra tools supplied localization and motion checks, but did not resolve the contact and regrasp decisions needed to complete the task. We next test whether building a simulation helps with this delicate manipulation.

Can building a simulation help?

Simulation gives the agent a way to test an interaction before trying it on the robot. Since the earlier bowl-rack trials had not completed the task, even with the expanded toolbox, we added an instruction to build a simulation and rehearse the solution first:

You must build a physics simulation of the real scene and run your whole solution in it first, then do it on the real robot. Blender and Isaac Sim (conda env behavior) are installed.

With simulation, agents achieved a 25% overall success rate. In a separate successful recording, shown below, observation, simulation building, and rehearsal took 46.6 minutes, followed by 2.4 minutes of physical execution. The head and wrist views are aligned by task stage.

The agent builds both the scene and the test

The agents used Blender or Isaac Sim with available robot assets, but constructed the task scene and rehearsal code themselves. Their common loop was observe the scene → construct a model → simulate candidate motions → inspect the outcome → revise. In the successful recording, the agent chose Isaac Sim, fitted the table plane from depth, and estimated the bowl and rack geometry from images and point clouds. The resulting model combines observations with assumptions:

The agent saves the scene, physical parameters, joint waypoints, and replay code as reusable files. But the test is only as complete as its model: the rack is fixed in this example, so the simulation cannot reveal whether a collision would move it. The agent chooses both the plan and the conditions under which that plan is tested.

Use simulation for rehearsal to find the correct solution

The agent did not find the successful grasp in one pass. It first tried squeezing across the bowl’s diameter. When test lifts failed, it adjusted the approach; when those changes still failed, it changed the contact strategy to a rim pinch. Play the edited sequence below to follow selected rehearsals in order, with holds on the outcomes and a final comparison of the two grasps.

The final rehearsal also placed the bowl on the rack, released it, regrasped it, and returned it upright to the table. After checking this full sequence, the agent reused the joint waypoints on the real robot and checked camera feedback along the way.

A simulated pass does not guarantee physical success

Grasp retention can be assumed too generously. In a failed trial, the simulation attached the bowl to the hand with a force-limited joint after detecting contact at both fingers. This allowed a complete simulated cycle, but did not establish that friction alone would hold the real bowl. A separate contact-only test showed slippage, and the real test lifts failed.

Geometric errors can change the first contact. In another failed trial, the real hand touched the bowl earlier than the contact simulation predicted. The agent revised the wrist geometry and approach, then rebuilt and rehearsed again. A final simulated cycle passed near the deadline, but the agent never executed that plan. The logs establish a contact mismatch; the proposed wrist-model correction remains the agent’s hypothesis about its cause.

A failed rehearsal still needs to change the next action. In another case, a recovery simulation lost the bowl. The agent dismissed this as an interpolation artifact and continued with real execution; the task remained unfinished and the rack moved.

Simulation can help an agent refine a workable strategy, but its value depends on whether contact, support, and release transfer to the real robot. These exploratory trials also allowed 80 minutes rather than 40, so the success-rate difference cannot be attributed to simulation alone.