Originally presented as a live talk on October 28, 2026
Background
Let’s start off with some brief robotics history.
Folks may not know this, but Scale actually started as a robotics data company. Specifically, we did (and still do) data for self-driving cars. When I joined in February 2021, most of the business was about lidar and cuboids and semantic segmentation. We did some language model work here and there - including for a small R&D shop called OpenAI - but by and large we were an autonomous vehicle and computer vision data shop.
Another company active in robotics at the time was SenseTime, whose generative AI brand SenseNova is the first logo on this paper. I actually had not heard of SenseTime until reading this paper, but they’ve been around since 2014 and are publicly traded on the Hong Kong stock exchange. It’s a commercial spinout from a computer vision lab at a university in Hong Kong, so a lot of their early work was in the same vein as ours.
Now at the time we and SenseTime were contributing to robotics, the AI stack looked like this: separate models for perception, prediction, planning, and control. There’s a lot of detail here we unfortunately don’t have time for, but the big thing I want to note is the human design behind this stack. Humans decided that perception is distinct from prediction is distinct from planning is distinct from control. Humans decided what the inputs and outputs between modules should be. Humans decided that the state between modules should be human-readable.
Most of that has changed. As LLMs came to dominate the minds and research programs of most AI labs, the robotics folks took note and started playing with new paradigms.
Specifically, the new robotics models started with LLMs. Taking that raw intelligence, created by learning language but abstract enough to stretch to other domains, researchers added vision input on the front and robot action output on the back to create vision-language-action models (VLAs). Models from leading “new wave” robotics companies, like Physical Intelligence’s π0.5, are VLAs.
But it wasn’t just startups embracing the LLM-based paradigm. For example, Waymo pivoted from a traditional robotics stack to the Waymo Foundation Model, which incorporates Gemini.
The major constraint the new paradigm avoids is the boundaries between modules. Yes, the vision encoder is separate from the LLM is separate from the action decoder in a VLA, but the boundaries are learned and machine-native rather than pre-defined by and for humans, just like the individual layers of an LLM are separate but with hidden states passed between them, with no concern at all for human interpretability. In the jargon, VLAs are “end-to-end” rather than distinct models.
Now as all users of SOTA LLMs know, it can take a long time for them to respond and make decisions. That’s fine for most office work but doesn’t suit most physical work.
One approach is to just cap the size of the VLA. So π0.5 for example is ~3.5B parameters, which is ~1000x smaller than SOTA LLMs today. As a result, models like π0.5 can produce several actions per second, which you really need if you want your VLA to wash dishes or fold laundry or even walk at a reasonable pace.
Borrowing from the cognitive science literature, folks call big deliberative models “System 2” and small rapid models “System 1”. In humans, System 2 is what you use to consciously think about something, like say a math problem, whereas System 1 is what you use instinctively, like simply knowing 2 + 2 = 4 or like brushing your teeth while your conscious mind wanders.
As human cognition hints, the best results come when combining System 2 and System 1. Many robotics teams now do the same thing. Waymo actually uses this framing explicitly in the Waymo Foundatio Model. Google’s Gemini Robotics series features Gemini Robotics Embodied Reasoning (ER), a System 2 VLM, and Gemini Robotics, a System 1 VLA. System 2 does the reasoning, as the name states, and works as an agent. System 1 works as a model and does not explicitly reason.
So that gets us most of the old robotics stack back. However, there is an even lower system - System 0 - corresponding to the fastest and most automatic actions. The human equivalent here is the intelligence in your various body parts, like maintaining balance with constant small adjustments. System 0 in robotics can act tens or even hundreds of times a second, so really the control loop has to be in code or perhaps in a very small and specific model, like in the single millions of parameters maybe. In parallel to our human example, a robot may have a System 0 controller to sense changes to its center of gravity and automatically adjust its parts to compensate.
System 0 pretty much carries over from the old robotics paradigm, so we won’t discuss it further. But it’s important for folks coming from LLM land to remember how much further down the robotics stack goes.
Along with the convergence of stacks has come the convergence of potential customers for data.
The leading examples are NVIDIA and Google. NVIDIA has the Nemotron series of generative models and the GR00T series of robotics models. Google has Gemini for everything, plus some other dual-purpose models like Genie.
We also know OpenAI worked on robotics before ChatGPT narrowed its focus, and it restarted those efforts in 2024. Similarly, Meta has worked on robotics. Also, assuming SpaceXSI merges with Tesla, it will also have both sets of models under the same roof to power Elon’s robot army.
Note that Anthropic is not on my list, as robots aren’t on the critical path to ASI. However, I did read a rumor on Twitter that Anthropic considered buying Physical Intelligence at the start of 2026, and we know that Anthropic has given Claude access to a wetlab. So I wouldn’t count them out.
The Paper
With robotics papers, I like to start with the robots themselves. That will give us a good foundation to understand what tasks are even possible and what the model has to work with.
We get a good variety of embodiments here, as the table shows. Single- and dual-arm setups are pretty common in real deployments, even though all the flashiest demos have humanoids. Grippers, which simply grasp and release with their two pads, are also pretty common and surprisingly versatile. Dexterous hands, i.e. hands with fingers, are pretty expensive still so you don’t see them as much yet. Also, we haven’t totally cracked a design that matches human hand strength and dexterity, so folks really do try to squeeze as much capability as they can out of grippers.
Here are some examples of different robots doing different tasks, which we’ll talk more about when we cover the benchmarks. Note how elementary the tasks still are, despite all the recent progress on robotics.
Equally important to the interaction hardware is the observation hardware.
If you’re not familiar with robotics hardware, you may expect only an “egocentric” view, i.e. a view from the robot’s head. If it doesn’t have a head, like if it’s just an arm, maybe you’d expect a fixed camera somewhere else.
Both views are common and present in this table, but wrist cameras are actually the more common view. Keep that in mind for when we get to some of the benchmarks and failure modes.
So we have various visual input per embodiment. Now we need our physical input, and then of course our physical output. The third column here gives the outputs, the actions the model can take, but only implies the inputs. Like in the first row for example, “end-effector pose + grasp label” means the desired position of the gripper in space and whether the gripper should grasp or not. Those outputs correspond to inputs from the gripper: where it actually is in space, and whether the gripper is actually open or closed. Also, the fact that the embodiment is an arm implies the model will receive input about the arm’s position, usually described as the state of its joints.
Note the differences in inputs and outputs even between robots of the same form. Even a single arm with a gripper, the simplest setup, differs in camera count, camera type, and action interface. There’s also variety in the extra information supplied, as the last column shows.
The equivalent for LLM agents is the harness, which can vary considerably within the same domain. Like even though all coding agents ultimately produce text, harnesses like Claude Code and Codex and OpenCode all present different tools and feeds to the models. As we’ll see later on though, there are yet more components of the robot agent harness.
So we know our robots and their harnesses, which will take in observations for the model to contemplate and allow the model to act in the environment. Similar conceptually to the traditional observe-think-act loop of agents in LLM land and other areas of AI.
Now let’s focus on what exactly we want these physical agents to do. Here’s a table of all the benchmarks the paper uses, broken out by embodiment, although the last case is embodiment-agnostic because many embodiments can move around and navigate environments.
Since the names are pretty uninstructive and often pretty bad, I added a column briefly describing what they test. We’ll see examples later on that will make it more concrete. Broadly though, a lot of robotics tasks are about picking stuff up and putting it down somewhere else - “pick and place” in the robotics parlance - with a variety of object arrangements, horizon lengths, wordings, etc. It reminds me of the basic instruction-following benchmarks LLMs were stuck on in like early 2023.
The table also includes the state-of-the-art models and their scores - “success rate” (SR), also a standard piece of robotics jargon. In the middle column, we have dedicated robotics models, mostly VLAs. That includes π0.5 from Physical Intelligence, which we have covered before.
In the right column, however, are the sorts of setup we’re talking about today: a general-purpose model, usually inside a specific harness, set on these robotics tasks. The model pretty much needs to be a VLM, not just an LLM, but other than that there’s nothing special or tailored about the model itself. That’s what allows popular closed models like GPT and Claude to be here. For the ones that don’t mention a recognizable model, like PhysicalRSI or Agent, the name is the harness and there is some model within it, although in these cases the model is undisclosed.
We’ll see this table a bit later when I reveal the benchmark results for GPT-6 Astra.
Another way robotics benchmarks are like early LLM benchmarks is that they come with example data. It may sound strange in the era of AGI, but back in the GPT-2 and GPT-3 days, you had to provide examples of what you were looking for in order to test the model’s ability in that domain. Even for something as simple as grade school math (GSM8k), at minimum you had to show what form the questions and answers would take. And actually, benchmarks for base models still need example data, since without post-training LLMs can’t natively answer questions; they can only complete text.
Robotics is in the same relative place on the curve of progress. All the benchmarks we just looked at except for RoboLab come with example tasks for the robotics model to train on or to use as examples in the prompt for in-context learning (ICL). That’s what’s going on in the first two boxes of this diagram, which shows how GPT-6 Astra turns this example data into something the researchers call a “knowledge package”. We’ll look at what’s in there on the next slide, but for now the thing to know is it comes from the examples and other relevant information about the embodiment and the task. The same knowledge package then appears in the prompt for every attempt at the task, no updating based on experience.
Here’s what GPT-6 Astra decided to put in its knowledge package. Note that the researchers did not prescribe any structure, and between tasks there actually was some variation, but on average this is what the knowledge package looked like.
A couple details to note. One is the motion references, which is robotics-specific. Concretely, that’s going to be a series of data points about each physical quantity and degree of freedom over the examples. GPT writes helpers to turn the data points into something more high-level and meaningful.
Another is the executable helpers. On its own, it’s not surprising to see some software tools in GPT’s kit. Some of the tools are for controlling the embodiment, probably need those. The main thing for folks new to robotics is that “FK/IK” in there: forward kinematics and inverse kinematics. They’re fancy terms, but really it just means mapping from joint angles to hand position and vice versa.
FK is relatively easy; if you know where all your joints are and what angles they’re at, you know where the hand is. It’s much less useful though, since what we really know and want is the hand position. IK tells you how to set joint angles to achieve the desired hand position. IK is much harder, because there can be many solutions or perhaps none at all. A tool for IK is critical, and we’ll see a bit later that GPT struggles when it has to act using joint angles directly, without the benefit of deriving the joint positions from a hand position via IK.
Now let’s bring back our table of benchmarks to see how GPT + ICL does.
At the standard budget for the benchmark - a number of time steps, a number of turns, or sometimes whichever comes first - GPT is SOTA on every single one. Occasionally it’s slim, and for the second-to-last benchmark it’s a previously reported result, but it’s true.
Now if we 10x the budget, we can reach substantially higher in many cases. We’re not considering ROI yet - we’ll look at costs on the next slide - but in pure performance it seems GPT-6 Astra is a pretty capable robotics model.
Do note that asterisk for RoboDojo though. That’s actually not only GPT and ICL; it’s a deeper exploration of designs that we explore later on.
So quality is great and all, but we really need cost and time to complete the picture. And here, GPT does not shine. Or maybe it shines for the wrong reasons.
On the left is costs. For standard pricing - no batching, no super long requests - GPT-6 Astra costs $10 per million tokens of input, and $50 per million tokens of output. However, for long trajectories, most of the input will be cache hits, which are substantially cheaper. With the input-heavy mix and with the help of cache hits, the researchers estimate $1 per million tokens overall as a lower bound.
So the cost graph basically translates directly to dollars. That means one task on average for their range of benchmarks can be anywhere from a couple bucks to ~$70!
Now on the right we have time, which needs no translation. There is a similarly wide range though, from ~30 to 400+ minutes. Yes, that is about seven hours on the high end. Probably not practical, but it puts the $70 into context. Seems like GPT is making more than federal minimum wage.
Not shown here is the ratio between thinking time and acting time. It’s quite lopsided, sometimes 50 or even 100 to 1. I imagine it would be quite painful to witness in realtime, the robot accomplishing like a minute of work over the course of an hour.
We have looked at the aggregate metrics, which are good for understanding success, but for understanding failure it’s best to look at specific examples. After all, there are way more ways to be wrong than right.
One failure mode is thinking you’re done when you’re not. Like in the top-left example, from one perspective the mug with the red handle appears hung on the rack, but actually it’s not even close by. Or in the example below that, attempting small tweaks to finish a puzzle reveals it’s actually not close at all.
Another failure mode is unrecoverable errors. Spilling and dropping somewhere inaccessible are common, and sometimes the agent hits a dead end like in the middle middle example.
The last one, which I think will be more familiar to LLM-based agent users, is a long horizon causing the agent to neglect parts of the task. The token intensity of images explains a lot of this in my view.
One data-related point is the use of cameras. Like for egocentric and teleop training data, it’s rare to purposely get a different view of a scene; humans have binocular vision and thus aren’t susceptible to issues like the mug from earlier or the example here on the left, and in teleop it’s common for the teacher to be in the same room as the follower arms and thus not need to fiddle with camera views.
For a real robot though, working the camera matters. For a single arm with only the monocular wrist cam, it’s especially important to actively take other views, but for dual arm it’s also helpful to move one arm for new views while the other arm performs the target task.
Now here is that exploration design slide I promised. For RoboDojo, the researchers play with a few different designs that trade off cost with quality, broken up into two experimental blocks, A and B.
Block A covers the System 2/System 1 split we discussed earlier. The baseline is just plugging GPT-6 Astra straight in, no robotics-specific System 1 model in sight, and also with no example tasks for ICL. Already it does better than just π0.5 on its own, although at considerably higher cost.
After establishing each baseline, they start combining System 2 and System 1. The first method, which they call “open-loop”, uses GPT to break the task into sub-goals, which π0.5 then tackles. That does slightly worse than letting π0.5 run on its own, so they ditch that design.
The second method, “closed-loop”, instead has π0.5 propose an action for GPT to then accept, edit, or reject. π0.5 is a decent model on its own, so taking advantage of its strengths and only intervening as necessary seems to work.
However, providing a verdict on each individual proposed action is inefficient. Actions are usually quite small, like moving a hand a few centimeters, so “chunking” actions up to L=4 per chunk saves review cycles and may also provide clarity as a little trajectory rather than a single action in isolation.
Finally, as you’d expect, adding examples in for ICL improves performance at some cost. The best-performing version of all does ICL with closed-loop and L=2 chunking, but with by far the highest cost.
Now for block B, they look at two design axes: action representation and chunking. Chunking appeared in block A though, so let’s discuss action representation.
For whatever action we want our robot to take, we could describe it in many ways. For example, we could say “move your hand right three centimeters”. Equally though, we could also say “move your hand to coordinate (5, 19, -7)” in some Cartesian space. Or, more intrinsically to the robot, we could say “open your elbow joint by 20 degrees”. They are all representations of the same action.
The baseline action representation for GPT uses Cartesian coordinates, which seems to work fine. But directional commands, like “move your hand three centimeters right”, do better with lower cost. Unsurprisingly, joint-angle does the worst, as we expect from our brief discussion of forward vs inverse kinematics.
Finally, for chunking, they start with L=4 but later allow GPT to vary L over the course of the task. The additional question adds cost, but improves performance, which makes sense given how precise and deliberate a few key action sequences must be to succeed on many tasks.
So that’s GPT and various helpers on the current state of affairs in robotics. However, the researchers take some issue with that current state of affairs - the benchmarks in particular.
Take a look at the example on the left from the BEHAVIOR-1k benchmark, which is the benchmark that took by far the most tokens and time on average.
It’s a bit hard to parse, but if you can deal with the syntax, you’ll see it’s about cooking bacon. The bacon starts in the fridge, moves to the frying pan, and cooks. Once the bacon cooks, the task is over. The only other condition is that the fridge needs to be closed.
Seems okay, but there are a few issues. For one, it checks the goal conditions at every time step. It does not look for the model to conclude work, which is what LLM benchmarks do and what we actually want in the real world; it’s not helpful if we have to constantly monitor the work of our robot.
Also, checking for completion at every time step means whatever’s in process at that time may continue happening. In the bacon example, if the robot started cooking and just walked away, it would succeed the moment the bacon cooked enough but would end up letting the bacon burn, maybe even starting a fire!
Along similar reward hack-ish lines, the task could succeed even if the fridge was open until the very last minute, or if other stuff spilled out of the fridge, or if the bacon fell on the floor before going into the pan, or if the robot itself caught on fire. Those omissions aren’t intrinsic to benchmark structure like the continuous monitoring is, but apparently it’s quite common for robotics benchmarks to only check the happy path.
Finally, they make some observations about poor benchmark task quality, like an example from Bench2Dex that is not internally consistent. Not particularly insightful, but very likely true given how hard it is to make a good benchmark. Perhaps an opportunity for Scale.
The researchers lay out their benchmark best practices, then create examples to demonstrate them.
All of the examples occur in a simulated apartment, shown here. The agent sees only through the robot’s cameras, but a virtual camera allows observers to capture footage from anywhere in the apartment.
Also, unlike in most simulations, time passes while the agent deliberates. So if GPT takes too long to decide the bacon is done, it could end up burning, whereas in BEHAVIOR it gets to think a lot on every simulated time step.
As I noted earlier, failure cases are the big treasures of benchmarking. This graphic shows some of what they hauled up.
First we have failures in planning and proprioception, knocking over items during what should be a tidying task.
Next we have a fridge failure similar to the one in the new version of the make_breakfast scenario, showing how the agent loses track of items outside its singular area of focus.
After that we have a dead end, a robot trapped in a loop of its own making.
Then we have another forgotten action to reverse, much like the open fridge, but this time with much messier consequences.
And finally we have our old friend make_breakfast, but with egg instead of bacon. Here, GPT actually did decide to turn the burner off, but by the time it reached that conclusion the egg had burned.
My Takeaways
Robots still have a long way to go
We are still working on atomic skills, or maybe very small “molecules”
Luckily, with exponential progress in AI, it takes less and less time to cover the same distance
The paper says as much about robotics benchmarks as it does about robotics models
A robotics expert had the same thought
Given the real issues in performance (time, singular focus, irreversible mistakes) it’s surprising GPT did well at all
Are robotics-specific shops in for a Bitter Lesson?
No matter which customers win - omnimodel shops vs robotics-specific shops - they will need data
What data may depend on architecture though (e.g. VLA vs WAM, see DreamZero)
We can read benchmark lessons as data lessons
System 2/1/0 is probably the right paradigm
So perhaps Luna will be the VLA for Astra
If WAMs beat VLAs, will Anthropic have to invest in image generation?