Originally presented as a live talk on December 3, 2025
Background
A brief bit on Physical Intelligence, which is the company behind the Pi series of models. They’re one of the top names in the field, although they’ve only been releasing models since late 2024. One of the cofounders, Sergey Levine, is one of the top robotics researchers in the world.
Now let’s do a quick recap of the Pi 0.5 paper, since it was the starting point for Pi* 0.6. (Pi 0.5 deck)
A robotics model like Pi 0.5 is a VLA, a vision-language-action model. That’s an LLM at the center, with a vision encoder added on the front so it can see, and an action expert added on the end so the model can do stuff - the action expert outputs signals for the motors and such. In the case of Pi 0.5 the LLM is Gemma 2 2.6B, made by Google. The vision encoder is SigLIP, also made by Google, so the vision-language model so far is 3B parameters. Then you add on the action decoder for another 300M parameters, so 3.3B parameters total. It’s a pretty small model, but it really has to be in order to produce actions in real-time. Gotta keep that latency down, no way to do that with the parameter counts of SOTA LLMs, which are in the tens or hundreds of billions.
So one of the major contributions of Pi 0.5 was this concept of “co-training”, i.e. training the VLM part on a bunch of multimodal web data. If you’re not familiar with this work already it may be somewhat surprising that a robot could learn to manipulate the world better by just looking at images or watching videos, but of course we know humans can do the same - we can learn how to do things just by watching. Hands-on practice is better but it’s not always required.
They also throw in training data from other robots, like other embodiments that this model won’t get used on.
The other major contribution was this idea of subtasks, which they give examples of in the pinkish red boxes. Like with a chain of thought breaking down a reasoning problem, the subtask improves actions by making each step more granular. So the VLA keeps the high-level prompt in mind while also feeding itself these low-level subtasks to guide its actions. As it progresses through the task, it generates new subtasks for itself to complete, sort of like if you talked out loud as you completed some physical work.
Now on the more fundamental side, I want to briefly discuss the idea of a value model.
A value model is a way to estimate how good your current state is. In LLMs, the value model is this second model in training that judges the response so far after each token. You need that value model in PPO, which is traditionally the most popular reinforcement learning algorithm for LLMs. Looking at the graphic above, the value v turns into an advantage A, which tells you how much the last token helped your response. Then PPO takes that advantage and uses it to adjust the policy model, that’s your main LLM you’re trying to improve.
The value model requires training to get good at this estimation business, which makes PPO expensive and finicky to run. And of course the value model can’t perfectly predict the future so the values it produces are always going to be somewhat noisy. Still, empirically it does help the policy model produce better outputs to have these intermediate ratings available in training.
As an aside, you may have heard of GRPO, a more recent RL algorithm that dispenses with the value moel and instead compares lots of different responses to calculate advantages. People tend to use GRPO when there is one clear reward to give each response, like in math/reasoning. So GRPO and RLVR often go hand-in-hand.
Video
So here is their method in a nutshell. We’ll get into the details on subsequent slides but I wanted to start high-level.
The basic approach is: take an existing VLA, in this case Pi 0.6, then add a value function that predicts whether the current action will be helpful or not and teach the VLA to work with the value function, and then collect post-training data with that two-model system.
They also use a lot of pretraining data, like the robot trajectories shown on the left but also some web multimodal data. That sometimes involves the value function too, and they mention collecting “tens of thousands of hours of demonstrations from numerous tasks and a variety of different robots”. But the heart of the method is the loop of autonomous episodes and reward labels used for post-training.
Note also the “Intervention” video in the bottom-right. The frame around it sometimes turns red, indicating that the robot has made a mistake and that the human has intervened to recover. Unfortunately they don’t give any operational details about how the teleoperator intervenes or relinquishes control etc.
Now let’s zoom in on the architecture. Pi-0.6 is a VLA, like we reviewed in the background slides. They upgraded from Gemma 2 2.6B to Gemma 3 4B, a bigger and better model all around. They also more than doubled the size of the action expert. So that’s ~5.5B total parameters, apparently still manageable for real-time tasks.
The major change is of course the second model, the value function. That’s a VLM, a vision-language model, trained to output a single number between -1 and 0. That number indicates how valuable the current state is for getting to a successful conclusion. It’s a remarkably small model, not even 1B total parameters, and the 400M parameter vision encoder is most of it. That’s because the intelligence for this kind of work is mostly visual, with little abstract reasoning or factual knowledge required, so a 270M parameter LLM is adequate. Whereas for the VLA you need to, for example, recognize what the prompt is asking for and break the task down into subtasks, which requires reasoning and verbalizing information about the world. And the VLA has to make and execute a plan, sometimes covering the whole trajectory, whereas the value model just lives in the moment, it just needs to say how good or bad the current state is given the end goal.
Anyway, in training the value gets turned into an advantage, then that gets turned into a binary based on how high the advantage is in order to cut down on noise. Then that True or False value goes into the VLA as additional text, like literally the text “Advantage: positive” or “Advantage: negative”. So the VLA is basically playing hot and cold with the value function and learning what this new signal means. Then during inference, when you’re actually using the model, you always pass in True. That tells the VLA to take actions like the good ones it saw in training. A little strange but apparently it works.
Just to make it concrete, here’s an example of the value function rating the situation over time. It’s a little noisy but it does pick up on the major error around 100 seconds and the subsequent recovery around 140 seconds.
Now here are the specific tasks they’re going to evaluate the model on. There’s some generic folding task, then a more challenging one, then one where you have to fix existing errors in folds, then box assembly and of course our friend espresso making from the video.
Note that they post-train the model on these tasks, they’re not evaling right out of the box. If you’re coming from LLMs you might think that’s cheating, but actually on most benchmarks there is a train set you use to teach the model the general form of the task you’re going to test it on. Same thing is happening here - you’re teaching the “rules of the game” so to speak and then seeing if the robot can play that game well no matter how the pieces are arranged or what moves are made.
As for that eval-specific post-training data, there are some confusing notes in the appendix about dataset composition, but in general it’s in the hundreds per task.
I should also quickly note the embodiment they used. It’s a simple pair of arms with grippers, running at 50 Hz.
Now we’re about to see the results of their work, but I wanted to clear up their somewhat confusing nomenclature first.
The initial baseline is Pi 0.5, which we discussed back in July.
They also run on Pi 0.6, kind of an all-around upgrade on the Pi 0.5 formula but no radical changes.
After that are the components of RECAP added on element by element. So the first Pi* 0.6 is with the value function, which then becomes and advantage which then becomes an indicator which is a new input for the VLA.
The next one adds the first bit of post-training, for the specific eval tasks. I don’t know why they change the nomenclature from “RL pretrained” to “offline RL” but it’s the same thing.
Then the final step, the full RECAP, does online RL; in other words, it trains on its own successful trajectories, including the ones where a human intervened for part of it.
And here are the results. As you’d expect, the bright yellow bars representing the complete Pi* 0.6 model are the tallest and best, but I’d like to zoom in on a couple things.
First, let’s distinguish the top and bottom set of charts. The top is just the straight success rate, with the box assembly one further broken out into subtasks. That’s a fine metric, but the bottom one is probably more important: the throughput. That’s the number of successes per hour, which rewards fast completions. It does also reward fast failures though, so you do want to keep that success rate metric around as a constraint. Pi* 0.6 never goes below 75% success rate, which seems acceptable as a lower bound.
Anyway, the biggest “speed bonus” seems to come on the regular laundry task, where the success rates are mostly similar but the throughput nearly doubles for Pi* 0.6 compared to the offline RL + SFT version. A less dramatic version is also true for box assembly.
The other thing to note is just how big the jump is when you add that final online RL step. Like it’s not a smooth improvement as you add on each component of RECAP, you really do see a jump at the end. So it seems like RL and intervention are crucial. They also have a graph later on showing improvements over multiple iterations of online RL: collecting trajectories, training on those, collecting more trajectories, training on those additional ones etc. Even in tasks where success rate is basically 100%, you can improve throughput with one or two additional training cycles.
Overall it’s impressive work. This model got a robot to successfully fold laundry for two hours straight in a real home and to successfully serve espresso for 13 hours straight!
My Takeaways
Correction of models could be right up our alley
May require other embodiments to be acceptable
I have no idea how much labs would pay
Could better value function data help too?
The training data for the value function is labeled by rule, not by model or by human
Surely the data could be better, but open question how much it would help
I stick by my prediction that the ChatGPT moment for robotics is 1-2 years away