Using vision-language models (VLMs) to control robots is not a new idea. What surprised us was how easy it had become to try. Give a coding agent access to robot tools, and it can work on commands like "put the fruit in the basket" or "open the door". We wanted to try this with Codex and see what actually happened along the way.
Developers have shown many such demos. Agents have inserted a plug after one video demonstrationGPT-Policy-Eval One-shot Plug Insertion, learned to type on a keyboardLearning to Type and Self-express on a Keyboard, painted with a brushPainting the Golden Gate Bridge from a Semantic Prompt, controlled a robot they had never seenRapid Adaptation to an Unseen Robot Embodiment, copied a mobile-manipulation task from a single videoMobile Manipulation with In-context Learning, and solved tabletop and Rubik’s-cube tasks in simulationLLM Harness for Tabletop Robot Control,Robot Hands Solve a Rubik’s Cube. Others have benchmarked models and agents on real robotsRobocurve. “GPT‑6 Astra on robot arms.” September 4, 2026. Robocurve,XDOF Ember-0,RoboQuest. Closest to our study, Agent as PolicyJia et al. “Agent as Policy for Robotic Manipulation.” arXiv:2609.12541, 2026. Agent as Policy runs Codex and Claude Code as the robot policy and shows that saved experience makes later trials faster. AA-RoboticsAA-Robotics v0.1 Artificial Analysis, 2026. reports about 30 minutes per task for most models, against about 20 seconds for a person.
Still, we wanted to look closely at how a coding agent solves a robot task. What steps does it take? What tools does it build? What does it do well, and where does it struggle? So we gave Codex and Claude Code access to our Dexmate Vega, using GPT-6 Astra and Opus 5.5, and recorded the attempts. We went through the logs and recordings to see what happened between the task description and the final result.
TL;DR — Key Takeaways
- Coding agents can develop workable robot procedures.
In our trials, agents interpreted images, wrote helpers, adapted after failed attempts, and saved what worked. Simulation also helped refine a grasp strategy, although its assumptions still needed checking against the real robot.
- A successful task can still be slow or unsafe.
Agents spent substantial time checking, writing code, and adjusting individual actions—even with a saved plan. Additional tools did not consistently improve performance, and recognizing hazardous contact did not always lead to timely corrective action.
- Making these agents work requires an integrated robotics system.
Our results suggest combining the model’s ability to learn, plan, and handle unfamiliar situations with skills and controllers that manage routine execution, verification, recovery, and immediate safety responses. Progress depends on both the model’s capabilities and how these parts work together.
Experiment setup
We picked three tasks: