RoboWild reveals the sim-to-real gap: 64.9% in simulation, 13.3% in real-world tasks
Here's a number worth knowing: Claude Fable 5.2, currently the top-performing AI model on the RoboWild benchmark, succeeds on 13.3% of real-world robot tasks. Anthropic's own launch benchmarks for the same model show 64.9% success on RoboDojo Sim.
Same model family. One task in 8 in the real world ... nearly 2 in 3 in simulation.
Can I just say ... OUCH!
That gap is what XLANG Lab, the group behind the OSWorld computer-use benchmarks, built RoboWild to measure. The benchmark, released this week, tests AI models on 341 tasks across 38 furnished scenes: homes, workshops, laboratories, stores.
What RoboWild tests, and how
The benchmark includes 4,480 objects across 23 scene types, at a density of 3.60 objects per square meter. That is nearly twice the density of the next most demanding benchmark in XLANG's comparison table, Gym-Anything at 1.82 objects per square meter. The tasks are not just demos:
- stack mail in a drawer
- hang umbrellas on specific hooks
- open a locked door
The setup puts an AI model in a ReAct-style control loop. It receives a task instruction, robot-use guidance, and 3 camera feeds: head, left wrist, right wrist. It issues movement commands one at a time. The simulator advances, returns new images, and the loop continues until the task is done or the model fails. The same interface runs across 4 robot embodiments: Mobile ALOHA, a Franka Mobile, Unitree G1, and Dexmate Vega-1.
65.7% of tasks require dependency chains of 2 or more subgoals. 61.9% require at least 2 of 8 measured skill types. 58.6% need visual disambiguation among similar objects. These are not tasks that fall apart because the benchmark is unfair. They fall apart because rooms are hard.
The Humanoid Technology Expo is where the builders, integrators, and enterprise buyers shaping this space come together. Early registration is open.
The leaderboard
Five models made the published leaderboard. All scored fairly low.
- Claude Fable 5.2: 13.3%
- GPT-6 Astra: 11.6%
- Gemini 4 Carbon: 10.6%
- Claude Opus 5.5: 6.5%
- GPT-6.1 Sol: 4.6%
The ordering holds when you compare against simulation scores. Fable 5.2 leads in both RoboWild and RoboDojo Sim. But the absolute numbers collapse. Opus 5.5 goes from 50.0% on RoboDojo Sim to 4.81% on RoboWild. Fable 5.2 goes from 64.9% in simulation to 13.3% in real-world-style scenes.
Essentially ... Opus 5.5 loses about 90% of its simulated success rate when transferred to RoboWild. Fable 5.2 loses about 80%. The more capable model transfers better, but neither transfers well.
What breaks most often
XLANG's failure analysis across 188 failed runs found multiple failures, but grasping was the worst:
- grasp failure: 42.6% of failures
- motion or execution errors: 28.7%
- functional object operation failure accounted for 16.0%
- goal or target selection errors for 6.9%
Also, 87% of runs disturb at least 1 unrelated object. That is ... extremely clumsy. For any robot operating near people, side effects are the safety concern, and the benchmark is surfacing how frequently they happen even when the primary task fails to complete.
The token economics also shift compared to flat digital tasks.
Fable 5.2, Opus 5.5, and Gemini 4 Carbon use several times more output tokens per task than GPT-6 Astra or GPT-6.1 Sol. Routing logic that picks fast cheap models for coding tasks may need a different policy for multi-minute robot episodes with 3 live camera feeds.
What the numbers say about simulation
The gap between RoboWild and RoboDojo Sim is the benchmark's real contribution.
Simulation benchmarks are valuable tools for development. They are not the same as a furnished room with articulated cabinet doors, deformable objects, and 4,480 items that can be knocked over.
XLANG says existing benchmarks used workspaces that were too small, object sets too simple, and task distributions too narrow. RoboWild was built to surface the cases those benchmarks smoothed away. By that measure, it is working. 13.3% and 64.9% are not the same number, and now there is a public benchmark showing exactly how wide the gap is.
The question the robotics industry will spend the next few years answering: how do you close it?
Sources
In this story
Follow to get an alert when they're in the news
Humanoid (almost) Daily
The humanoid robot news that matters, with context from HumanoidMegaHub's tracking. Free, almost every day.
Keep reading
Written by
I tell stories with data and build startups. I've raised angel and VC funds, led software companies and startups, and chronicled the rise of the mobile economy. Currently I write for Forbes, host several podcasts, and am starting up HumanoidMegaHub. I was TUNE's mobile economist and Singular's VP of Insights, built the research team at VentureBeat, and managed teams creating software for partners like Intel and Disney. In past lives I built websites, e-learning platforms, and mobile apps, and, a long time ago, was named to Folio's top 100 of the media industry's "most innovative entrepreneurs and market shaker-uppers.”