Paper · Website

Background

Computer use agents are what most people picture when they hear “agent” — a JARVIS sort of figure who clicks through a bunch of things for you when you give it a task.

In practice it’s much easier to give agents access to code tools like APIs and MCPs, since navigating around a GUI is more of a human thing, but some tools don’t have APIs or MCPs. So we have to use CUAs.

Now here’s the background on computer use agents:

• Sep 2022: Adept Action Transformer 1, or ACT-1. Adept saw that transformers worked by predicting the next thing in a sequence and decided computer actions like clicks and scrolls could be that thing, the native language so to speak of a transformer. ACT-1 was trained directly on these computer actions. Unfortunately they were heading down the wrong path; the CUAs we’re discussing today are LLMs with computer use capabilities added in, making use of the raw intelligence that language modeling seems to impart. They were also well ahead of their time too as you can see from the rest of the timeline. ACT-1 never got good enough and was not widely adopted after release. The company had a fake acquisition like Inflection and Character in 2024, and now they focus on agents.

• Jun 2023: Mind2Web, an early and widely adopted benchmark for “LLM-based web agents” as the description says. It has over 2k tasks across 137 real websites.

• Oct 2023: SWE-bench, the first agentic coding benchmark. This isn’t web-based, it’s command-line based like Claude Code, but it does appear in the UI-TARS paper and it’s a benchmark that’s still with us today, like our SWE-Bench-Pro leaderboard, which we recently released.

• Apr 2024: OSWorld, the gold standard of CUA benchmarks. 369 tasks for desktop and web, with multi-app workflows.

• Oct 2024: Claude Computer Use, the first CUA from a major lab

• Jan 2025: Operator, OpenAI’s CUA, now part of ChatGPT Agent which honestly I had never heard of until writing this. Also, UI-TARS.

• Apr 2025: BrowseComp, also from OpenAI, containing requests that require many steps of web research to answer. Also, UI-TARS-1.5

• May 2025: LMGame-Bench; Terminal-Bench

• Sep 2025: UI-TARS-2

Now by complete coincidence, Anthropic release Claude Sonnet 4.5 last week with significant bumps in a few agentic areas. Look at that jump on OSWorld: nearly 20%! Sonnet 4.5 already blows past UI-TARS-2, although UI-TARS-2 does surpass Opus 4.1 and Sonnet 4.

There have been lots of other papers published here recently too.

The one on the left came out on September 19th but is already a little outdated - you can see it gives results from a previous version of UI-TARS. It has a lot of similarities with the paper we’re reviewing and goes into a lot of detail about how they collected data.

ScaleCUA is the open-sourced computer use agents that can operate on corss-platform environments (Windows, macOS, Ubuntu, Android).

[2509.15221] ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

The one on the right came out only a week ago, from our friends at Apple. It’s actually the perfect bit of background, because it covers small agents like the NVIDIA paper we talked about last week, but for GUIs, like the paper we’re covering today.

Of course since it’s Apple they’re thinking about iOS use, but they test their agent on a lot of the same benchmarks as ScaleCUA and UI-TARS do. They are more buttoned-up on the details, but they also do a lot of the same work as the other two papers on unifying interfaces, creating and annotating data, doing SFT and RL etc.

[2509.26539] Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

Note that all three papers use some human annotation!

Broadly, you’ll see actions, observations, and thoughts. For example, there are “plan” and “replan” nodes for the model to think about how to accomplish the prompt, and the whole Browser State section is basically the observation.

Now CUAs cover more than just browser use, but since browsers can do so many things and are more standardized than OS-dependent apps, browser use is probably the biggest component of computer use. That’s also why Perplexity released a browser and why OpenAI is rumored to be developing one too.

At the same time, browser and computer-use agent data has drawn less commercial demand than coding-agent data. One component I think is the more mature, commercially valuable progress on coding agents. Like Anthropic clearly spent a lot of effort getting to that 60%+ result on OSWorld, but what really rakes in the tokens is Claude Code.

Another issue I suspect is standards, or really lack thereof. We do see this in coding agent world, where different scaffolds like Claude Code or Aider demand different data. For computer use agents it’s an even more basic, scientific question, where it’s not obvious what combination of vision and text like a DOM or an a11y tree is going to yield the best results, and whether one may be better in the short-term but more limiting in the long-term. And that type of indecision or lack of clarity causes commercial paralysis. But I think standards and science have progressed a lot over the last year, and the quickening pace of CUA releases may finally drive real demand for CUA data.

Now I do have to mention one specific way in which CUAs are really not ready, and that is safety.

This is a frame from a short video showing how to hack Perplexity’s Comet browser, which has an agent built right in, as you can see in the right panel there.

The user is currently viewing a Reddit post, where one comment started with all its text covered, using the no spoilers feature. Now even though the user and any purely vision-based agent wouldn’t be able to read the text, since Comet looks at the DOM or a11y tree as well, it can read this visually obscured text.

And the text is basically a prompt injection, a set of instructions for the model that the user definitely would not want. Here it’s asking the agent to post the user’s email and one-time password for their Perplexity account to Reddit, which involves navigating Perplexity account detail pages and checking Gmail for the one-time password.

If a leading product in the space can get you hacked just by reading Reddit, then clearly we are not ready for CUAs in daily life. Useful for enterprise maybe where malicious content is unlikely to appear, but then again this is just one method. Lots of work for red teamers to do to discover other vulnerabilities.

Lastly, let’s recap this lab, Seed. It’s the AI development arm of ByteDance, which makes TikTok.

As far as I can tell it’s the product of some reorganization and centralization in 2023, which I suspect was a response to the ChatGPT moment in late 2022.

It’s quite a diverse group. Here is a list of models or products the Seed team makes:

• Seed, a series of LLMs, including a multimodal variant called SeedVL

• BAGEL, which does multimodal input and generation natively

• Seedream, for image gen

• SeedEdit, for image editing

• Seedance, for video gen

• Seed LiveInterpret, Realtime Voice, and Music, for everything audio

• GR-3, a robotics model that we covered in a previous session

• UI-TARS, the CUA we’re covering today

Of course being the parent company for TikTok, the multimodal and translation models are not surprising. But the robotics and computer use agent models take a little more digging. ByteDance also owns Douyin, which is like TikTok for China, but has a heavy ecommerce component, including warehouses with robots. They also make software with workflows instead of just consumption, like various creator and internal tools.

One major reason you’ve probably never heard of Seed is they only just started releasing open-weights, permissively licensed models - right at the same time as OpenAI did with gpt-oss, and in fact they called theirs Seed-OSS.

The Paper

So here’s an overview of what we’re looking at with this paper, which again is pretty similar to the other two papers and fairly representative of CUAs.

At the top of the diagram the user provides the prompt. Over on the left we have the different environments - desktop, mobile, browser, game - and the different actions the agent can take. There are GUI actions, like clicking and typing, and there are tool actions, like using the terminal or calling an API. Depending on the environment and the task, you may have access to different tools.

Now the way the agent proceeds is with our familiar observe-think-act loop. Looking at the example in the diagram, the agent looks at the game and thinks about how to play it. It looks up instructions and strategies, then contemplates how to apply them here on this particular board, then clicks somewhere on the screen. That loop happens over and over again until the game is over. So we’ve taken one turn, since the user has only given one prompt, but there have been many steps, many iterations through the loop by the end of the first turn.

Now we see the second turn where the user asks the agent to make a website about what just happened. Pretty contrived, but it works. They don’t bother showing much more of the loop here, just showing that they added in the terminal tool really. So apparently this was a two-turn task, with N steps for the first turn and 2 steps for the second turn.

So knowing how the agent works concretely, we should have a good idea of how the data looks and how to collect and use it. That’s what these images show.

The top image shows how they use the data. They describe it as a flywheel but they show it more like a staircase, which I actually think fits better.

They start with the base model, Seed1.6, which is 230B-A23B, ~500M vision encoder, has its own post-training. Apparently on par with Qwen3-235B-A22B.

They kick off the flywheel with some data they already have on hand from prior versions. Some of it is unlabeled, pretraining-like data, which they call “continual pre-training data” - that would be stuff like tutorials or guides. The rest of it is SFT data, a combination of synthetic and human-generated prompt-trajectory pairs.

The flywheel really gets going with their first bit of RL, which we’ll cover on the next slide. But the key to the cycle is that cube labeled “Rejection Sampling”. It sounds fancy, but basically what they’re saying is to take the good trajectories, where the model got a positive reward, and use those as SFT examples in the next iteration. The remainder, the failed trajectories, go in the CT data, since they’re still valuable in showing how computer use works, just not valuable for showing what to do exactly for a certain prompt.

Now how do they get the annotated CT and SFT data?

For CT, they actually have a program installed on the personal computers of their contributors. While the contributors perform their tasks, they also think out loud and narrate what they’re doing, and the program captures that audio, which they then clean up with a LLM. So they get good trajectories and the reasoning behind each individual step or decision in the trajectory, which is really helpful for the model. They actually note in the paper that models trained only on actions, without thoughts, tend to mimic the training data rather than understanding it, which of course doesn’t generalize well.

They also tried retroactively adding reasoning to trajectories, but that didn’t work either - they said it was “nearly impossible to reconstruct the annotator’s original thought process.”

As for the contributors themselves, we don’t get any numbers, and the only detail about them is that they are a mix of novices and experts. The novices teach exploration, the experts teach efficient usage.

For diversity, they pick applications based on “industry coverage, user engagement, and market penetration” across web and desktop. Then they synthetically generate a representative array of tasks. Unfortunately they’re quite vague about exactly how they do this, but apparently there is some human input and also some input based on “normalized measures of usage frequency, user benefit, and cross-scenario transferability.” They also ensure a range of difficulties based on step count, number of pages required, and error handling.

So that’s CT. For SFT, they want the data to be “on-policy”, i.e. using model actions. On-policy is always more effective than off-policy, where a human or perhaps another model has composed the response or trajectory, but on-policy is often harder to generate.

So to get on-policy data, they have the model generate a few different thought + proposed action pairs, and then the annotator picks the best one OR writes their own. Once the annotator approves a thought + candidate action pair, the action happens and the observation follows, in this case represented by a screenshot. By the end of the task, you have a trajectory composed mostly or solely of thoughts and actions from the model, i.e. on-policy.

For RL data things were a bit different.

First off is their three different environments

1. Web browsing, to search for information

2. Web app usage, to accomplish tasks

3. Web games, to reach a certain state or score

For web browsing, they synthetically generate questions that use lots of partial or obfuscated information. Here’s an example:

““Discovered by a representative from the Music And Cabaret talent agency, this group had a founding lineup—initially under another name—that included members from Dreghorn and Irvine, plus a lead guitarist and drummer. The lead vocalist joined after being recommended by a founding member who saw them perform with a Kilmaurs-based band, and their lead guitarist left to form another ensemble before late 1975. Which record label did this group sign with?”

So as you can tell that’s going to involve a lot of searching to figure out. And the agent has to do the searching in a browser, no APIs.

For the web app one, again they’re synthetically generating prompts, this time by getting a more powerful model to look at screenshots from 690 different web apps and determine what functionality different apps have. Then they put those functions together in various ways to make sufficiently complex tasks.

Finally for the game one, it’s actually very similar to old-school RL, from before LLMs were even a thing. The model plays the game and you check the score or level or whatever.

So as you can imagine, these outcomes are varying amounts of verifiable. Like for web browsing, it’s pretty objective, and for games it definitely is. For web app usage though it’s tough, so actually they train UI-TARS to also be a good judge model for UI stuff and have a separate instance to judge outcomes.

They also occasionally reward proper formatting and penalize excessive length, but they provide no specifics and my general understanding is you take an accuracy hit by having other goals like this.

So they do this RL training on three separate models, all initialized from the same SFT checkpoint. Once they have their three RL’d models for web browsing, web apps, and web games, they then merge the specialists together into one model again. And then that new model is what generates the next round of trajectories for SFT, then that SFT’d model gets RL’d into three and merged again, in cycles until performance levels off.

Now let’s look at a couple impacts of training and scaling.

The top graph is showing the amount of thinking per cycle, per think/act/observe cycle. The “step” label on the x axis refers to the number of training steps, NOT the number of steps per turn.

Anyway, as you can see, with more training the model actually needs less thinking. That’s the opposite of what we see in so-called “thinking models” or “reasoning models” like o1 or R1; usually the more you train, the longer the chain of thought gets.

But CUAs make progress by acting more, not thinking more apparently. And it makes sense, since most actions on a computer don’t need tons of thought - it’s usually fairly clear in a well-designed program what button a user needs to press next for example. An initial plan might need an increasing amount of thought, but probably not the thoughts for each individual action.

That’s the pattern for web browsing and web app use anyway. For games, it’s actually kind of a funny pattern, where thinking goes up and down with training. The reason there is that as the model trains more, it gets better at the game, and it progresses to harder stages or levels. Then it needs more time to think, but then it gets used to that next stage or level, then it needs less time. So that inherent difficulty progression works against the baseline trend of increasing efficiency we see with more general computer use.

Now for the bottom graph, we’re scaling up test-time compute but in terms of actions, not in terms of thinking. So the x axis here shows the maximum number of steps or cycles we allow the agent to take, and the y axis shows how performance on a given benchmark improves.

For OSWorld on the left, allowing more steps slowly increases performance, but probably isn’t worth it given how much longer 100 steps is going to take compared to 20 or 50 steps where you’re getting most of the gains. And even then I somewhat struggle to imagine as a user tolerating 50 steps.

For their suite of games on the right, we see steadily increasing scores for UI-TARS-2, all the way up to a perfect 100 if you allow 1000 steps. Obviously the steps are going to be qualitatively different for games compared to browsing or apps, and also you as a user probably don’t care about a maximum number because a longer game session means your agent is doing better. Although why you’d want to employ an agent like this to play a game for you I struggle to imagine. Still, game annotation is a real if niche area, so it’s not totally obscure.

And finally, here’s where we end up. UI-TARS-2 is probably the best open-weights CUA right now, although given the pace of progress I don’t expect them to hold that crown long.

Also as a reminder, Sonnet 4.5 got over 60% on OSWorld, so even though UI-TARS-2 is better than some pretty recent Claude models, Anthropic is still king of CUA.

Also also, Gemini apparently loves Candy Crush.

So What?

• 2026 probably is the Year of Computer Use Agents

• CUAs on the edge for phones should arrive next year, at least on the research side

• More broadly, software will be made with non-human users in mind

  • Similar to how websites are now responsive to different viewports and have accessibility features

  • Eventually it may seem odd to interact with much software directly at all

• CUA red teaming could be big

Paper · Blog

Background

Since this is a paper about the mechanics of RL, I wanted to start with a low-level tour through modern-day RL.

RL is a form of model training. Training means changing the parameters of the model so that it performs better. The parameters are basically the weights in our embedding, attention, and feed-forward matrices, although technically there are other parameters.

So how do we take a model-level result - whether or not a response was good and thus rewarded - into all these individual changes to parameters?

Let’s work backwards from the end: the changes to parameters.

To change the parameter, which is really just a number, we need to know how much to change that number by. We take the existing number, θ (theta), and subtract from it the amount we want it to change. That amount has two components: the learning rate, η (eta); and the gradient, g. The learning rate is a hyperparameter you set manually during training. The gradient, however, comes from your loss.

This is the part where we go from individual parameters to the entire model. When you use a model, mostly what you’re doing is a bunch of matrix multiplications, with a few other mathematical operations thrown in. That chain of calculations is how we can trace the influence of each parameter on the final output. Sometimes people call this chain the “forward pass”.

Which implies the existence of a “backward pass” right? Indeed there is one, and it’s where information from training gets disbursed to all the individual parameters. But we need some information to pass backward, something based on the final output.

That information is the loss. I’m going to punt on where the loss comes from right now, so just think of it as a way to give a specific number or score to each token in a response. The lower the loss, the better the token is.

So we have our loss, and we have the chain of calculations that led to the tokens that produced the loss. On the backward pass, we use the chain rule from calculus to see how much a change in each parameter, each link in our chain of calculations, impacts the loss. In calculus terms, we take the gradient of the loss with respect to the parameters, which is what we have written here.

So we have our forward pass to generate responses, then somehow we calculate loss, then we use the loss to calculate gradients, which we then use to update parameters. What is that missing piece in the middle?

As you’d expect, it takes in responses and outputs loss. And how exactly you get from one to the other is where a lot of training algorithm magic lies. PPO, DPO, GRPO, DAPO - these RL training algorithms are all about collecting responses and turning them into loss.

Today’s paper is about GRPO, which stands for Group Relative Policy Optimization. Let’s see how GRPO turns responses into loss.

First, we need responses. Specifically, we’re going to get K responses per prompt. How big or small to make the group is an important part of today’s paper, so keep that K in mind.

Next, we’re going to score our K responses. Often we use GRPO with RLVR, meaning there is one verifiable, ground-truth answer. Usually the response earns a reward of 1 for a correct response and a reward of 0 for anything else. Rubrics also work with GRPO, since they can also numerically score and reward every response in a group, but today’s paper does RLVR.

That process of generating and scoring K responses is likely already familiar to you if you’ve worked with RLVR data. In fact, the other terms familiar to data folks - rubrics, RLHF, RL environments - also cover the same ground between the prompt and the rewards. However, there are more steps to go to reach our loss. These next steps are where the different algorithms play. That means GRPO and its variants like DAPO, but it also means classic algorithms like PPO and REINFORCE (yes, it’s an acronym).

Broadly speaking, the next step is to turn rewards into advantages, which are like rewards but transformed or altered according to the specifics of the algorithm.

For GRPO, our advantages depend on how well a response did relative to the rest of its group, hence the name Group Relative Policy Optimization. Specifically, we’re going to subtract the average reward from each response’s reward. In classic GRPO we also divide the result by the standard deviation of the group’s rewards, but today’s paper drops that part.

I have the advantage formula right here. For response i, the advantage is the reward minus the average of all rewards amongst the K responses.

Let me give a concrete numerical example. Say we have K=4 responses, with 3 incorrect responses and 1 correct response. Our reward is binary, so that means the reward for the incorrect responses is 0 and the reward for the correct response is 1.

The average reward is thus 1/4, i.e. 0.25. So the advantage for the incorrect responses is 0 - 0.25 = -0.25, and the advantage for the correct response is 1 - 0.25 = 0.75.

All well and good, but what is the purpose of this mutation from reward to advantage? To see the benefit, let’s examine a few cases.

In a mixed case like we just saw, the incorrect responses now have a negative advantage, not just a nil reward. We’re actively punishing the incorrect responses rather than just leaving them out.

Now, what if all the responses are correct? The rewards are all 1, but it seems there’s nothing for the model to learn. Why should we spend time and compute training when the model is already perfect here?

Advantages take care of the problem for us. Since the average reward is 1, then the advantage for each response is 1 - 1 = 0. No advantage, no learning. Same deal with all incorrect responses, although given the rewards are all zero it’s no surprise there’s no learning to be done.

For this paper, we should also consider what happens with much larger K and an extreme ratio of correct to incorrect - either a single correct answer with the rest incorrect, or a single incorrect answer with the rest correct.

For the first case, the average reward will be close to 0, since almost all the responses are incorrect. The advantage for the one correct response will be close to 1, and for the rest it will be close to 0.

For the second case, the average reward will be close to 1, since almost all the responses are correct. The advantage for the one incorrect response will be close to -1, and for the rest it will be close to 0.

Now that we have an advantage for each response in a group, we need one other quantity: the importance ratio, often just called “importance”.

But to even understand what the importance is, we first have to understand a bit about the mechanics of RL.

Earlier, when I described how GRPO gets a group of responses and scores them and turns them into advantages, I glossed over exactly where the responses come from. If I handwave and just say “the model”, it’s not exactly wrong, but it doesn’t address which version of the model the responses came from.

Remember, the end result of this big journey from initial input to final weight change is a new version of the model. During RL, we make many incremental changes to our model, many updates, which means the responses from “the model” are really responses from a model - a single, short-lived model we soon update again.

Over the course of training then, you accrue many responses per prompt, per update. Not every prompt runs on every update, but most prompts will run on more than one update.

In training, responses that came from the exact same version of the model you are about to update are called “on-policy”. Responses that came from older versions are called “off-policy”. Technically any response that isn’t on-policy is off-policy, so like a response from a completely different model is also off-policy.

Exactly how “off” we are depends on how many updates ago we generated the response. Usually it’s okay to train with responses that are a little off-policy, but past a certain point they can be harmful. Sometimes people call this split “fresh” vs “stale”. So on-policy data is always fresh, some barely off-policy data may also be fresh, then the rest is stale. There is no universal rule dividing fresh from stale; as we’ll see in the paper, it’s an empirical matter.

Now within our off-policy but still fresh data, we need a measurement of the difference between the old policy that generated the data and the current policy that we’re updating. The way we measure is per token. Specifically, for the chosen token at a position, we measure the likelihood of that token on both policies. The importance is simply the ratio of the likelihood from the current policy to the likelihood from the old policy.

Here is our importance formula. For response i and token t, the importance equals the likelihood of the current token from the current model π_θ given the prompt x and all the prior tokens in the response y, divided by the likelihood of the current token from the older model that generated the response π_θ_old, again given the prompt x and all the prior tokens in the response y.

The importance is the mathematical reason why off-policy data can be so dangerous. In the extreme case, like using a completely different model to generate responses, one likelihood can be close to 1 and another can be close to 0. 0/1 = 0, and 1/0 approaches infinity, so variance in importance is enormous. The resulting variance in loss per token can cause inappropriately small or large updates.

Of course if your data is on-policy, then both policies are the same, so your importance is always 1. It slows down training too much to get all your data on-policy, so that balance between safety and speed is important.

With importance and advantage, we pretty much have what we need to compose the GRPO loss.

For completeness, I’ll note what we left out, reading from left to right:

• The expected value operator E, which basically means we’re averaging the loss over all the prompts in our batch

• The 1/|y_i|, again a sort of averaging, this time over all the tokens t in the response y_i. Often not used in GRPO

• The min and clip, which means that we cut off the importance if it’s too far above or below 1, in this case by the amount ϵ

These aspects are not specific to GRPO and thus are not important background for today’s paper. I also left out KL divergence, another sort of safeguard against extreme updates that appears in many loss formulas, because in this paper they don’t use it.

Now that we have the mechanics of GRPO down, let’s talk about what it means for RLVR.

When we discussed advantages before, we saw that prompts are too easy or too hard for a model if every response has advantage zero, i.e. all the answers are correct or all the answers are incorrect.

As we know though, model capabilities are always advancing, so over time the balance isn’t really symmetrical; it’s always bad to generate prompts that are too easy, but at some point the prompts that are currently too hard no longer will be. Similarly, the prompts that are almost too easy now will become too easy soon, whether during the current training run, or after the next big pretrain, or from the next architecture change, etc etc. That means the long-term value is always weighted towards harder tasks. So the trick is to have a hard floor on difficulty, but a soft ceiling.

One common way to assess difficulty is to give a model some number of chances to produce a correct answer. The term for this is “pass@k”. For a given dataset, pass@k is the share of tasks the model correctly solves within k tries. And yes, that is a second “k” number to keep track of, alongside the k responses we ask for during GRPO. I don’t make the terminology, don’t shoot the messenger!

Anyway, when researchers measure difficulty, usually they pick a small value for k, like 1 or 3 or 8. The idea is to keep the number of attempts reasonable, like within what a user would actually put up with when asking the model to try again.

However, some researchers whack k up to a really high number, like 256, to see if the model in question can possibly solve the problem. That’s a different, more theoretical question, and it has produced some interesting work. I’ve covered a couple such papers before, including the one shown here. The researchers found that a base model with no post-training at all can eventually produce pretty much any answer a post-trained version can, if you use a high enough k. It’s good evidence of elicitation: RL brings forth what was already in the model, rather than teaching something new.

To be clear, RL is still valuable, since for users it’s important to receive the right response on the first try most of the time. But for today’s paper, keep in mind how important it is to dial up the number of attempts in order to finally receive a correct response. If the right answer is in there somewhere, it’s worth finding.

The Paper

On the last slide, we looked at dialing up the number of responses in eval. What about in training?

Well as you’d expect from the last slide, if you give the model more chances, it is more likely to produce a correct response. And if you have at least one correct response in your sea of incorrect responses, you have at least some advantage to put in your loss function.

So why not just whack up K, the number of responses per prompt in GRPO? Surely it will help to turn these hard prompts into training signal.

Well sometimes people do that. And this paper is about how to do that better.

The graphs above show the existing situation, using various RL settings on Qwen2.5 0.5B Instruct, evaluated on a math benchmark called GSM8k. For a fixed compute budget, minus differences in response length, you have to decrease the number of prompts if you want to increase the number of responses per prompt. That’s the Completions table, where N is the number of prompts and K is the number of responses. The left graph tells us that actually, fewer responses on more prompts is better.

But the real story is on the right: increasing K actually hurts performance on the hardest problems the most! Not only that, it hurts performance on some less hard problems too. What is going on?

It’s two things really. For one, it’s a waste of compute to get 32 responses to an easy problem. If you leave K at 4, you quickly and cheaply find the easy problems where all 4 responses are correct, drop them from the training set, and bring in new prompts that may have some learning juice in them. So small K means more rapidly reallocating fixed compute budget to harder problems.

For another, requesting more responses for the same easy prompt risks receiving a wrong answer! It is the inverse, but far less appreciated, version of what we talked about for hard prompts. With just one incorrect response, now your easy prompt goes from quickly dropped to retained, thus using more precious compute going forward, and altering the weights. Given the inherent risk of changing weights and low signal-to-noise ratio here, training on easy prompts due to fluke poor responses is not a good idea.

So on the one hand we need large K in order to find rare solutions to hard, valuable prompts. But on the other hand, increasing K wastes compute and risks regression. What are we to do?

The answer is to do both. Or rather, to adjust K depending on the prompt, rather than setting one K universally.

Start with K=4 across the board. That’s the cheap first pass, and it knocks out any prompts that are already too easy. It also gets nonzero advantages on many other prompts, where some but not all of the responses are correct. So far, it’s standard GRPO.

The trick comes for the hard prompts, where none of the 4 responses are correct. Instead of dropping them, we never give up (NGU), and keep increasing K until we get a correct response. Once we get a correct response, we stop generation for that prompt and start training.

Now it’s actually not quite never; they do introduce a small chance of giving up, like 5% here and slightly higher in other experiments, on each increase of K. Some problems really will be out of reach for a model, especially the small models they tested here, so trying forever will be a waste of compute. But the point is that NGU dynamically allocates compute to the hardest problems.

And the results are promising. The lines to compare are the blue + darkest grey on the left, and the colored + darkest grey on the right. For the same number of prompts and initial number of responses, NGU helps a bit overall, but especially helps on the hardest problems - the ones where the initial model wasn’t getting any correct responses at all in an initial test run of 1,024 responses per prompt.

That’s exactly what we hope to get out of RL. In standard RL, what we actually get is saturation on the easiest prompts and barely any movement on the hardest prompts. Like we saw last slide, standard RLVR can totally misallocate resources. The researchers call this “the Matthew Effect in RL for LLMs”, named after the original Matthew Effect from sociology, which basically says that people accrue success in proportion to their existing success. In economics, the quick encapsulation is “the rich get richer”; in RL, it’s “the successful problems get more successful”.

NGU fixes that.

Now as we discussed in the background slides, RL training happens in many steps, and sometimes the copy of the model that’s producing responses lags the copy of the model that’s receiving updates to its parameters.

With a fixed K, you can keep that drift between models small. However, with NGU, K is variable per prompt. As a result, the responses can become quite stale, with the generator model several update steps behind the trained model.

We know from earlier there is a balance between safety and efficiency, between how fresh we need our responses to be and how few times we can get away with replacing the generator model with the latest copy. However, there is an additional wrinkle with NGU.

Let’s say it takes 256 attempts until one response is finally correct. On the one hand, most of the 256 responses are probably stale by now, so we shouldn’t use them in training. But on the other hand, it’s really important information that it took 256 tries to finally get something right! How can we keep that difficulty information without risking bad behavior in training? How do we balance out the positive impact on advantages with the negative impact on importance ratios?

Well first, the researchers find that four update steps is the right cutoff, as shown on the left plot. Any response from a model up to four versions out of date is fresh enough to use still.

Second, the researchers use some simple but insightful math to rebalance the advantages.

Let’s continue with my example: 256 attempts, only one correct response. Assuming binary rewards, the advantage for the one correct response is 0.996, and the advantage for each incorrect response is -0.0039. So the advantages are basically binary too.

Let’s now say that all but 4 of the 256 attempts are stale. If we naively threw them all out, the advantage for the one correct response would be 0.75, and the advantage for each incorrect response would be -0.25. That’s a major reduction in signal for the correct response, although a nice bump up in the penalty for incorrect responses.

Now let’s take the best of both worlds: keep the original advantage of the one correct response, and adjust the advantage of the three remaining negative responses to match. So that’s 0.996 still for the correct response, and now -0.332 for each negative response. We now have strong signal in both directions.

The researchers call their approach “anchoring”, and the plot on the right shows how it improves outcomes relative to the naive approach and to one other approach called “downsampling”.

So far we’ve been looking at performance on math, which is a super common domain for training and evaluating LLMs on.

To ensure NGU is robust, the researchers test it on a different domain: programming, but in the form of puzzles. Specifically, the benchmark has the model play the game Manufactoria, where the player creates robot factories that process sequences of colored tape. It might sound completely strange, but it’s actually a somewhat literal construction of a Turing machine, which is the simplest conceptual thing you can program. So it’s programming, which is very in-distribution, but in a very out-of-distribution setting.

What the researchers find is the model needs some time to adapt to the game, to learn the harness before it can really learn how to succeed. If you try to tag difficulty early on, you get bad intel about which prompts are easy and hard. The Matthew Effect seems to disappear.

However, if you wait for the model to adapt to the harness, then bucket by difficulty, it works just fine. In this case, it only took about 100 RL steps, which is a small fraction of the 3k step budget.

It’s not a crucial part of NGU per se in my view, but it’s a nice example in the wild of how the relationship between model and harness is a two-way street.

Continuing with the Manufactoria benchmark, we see some familiar patterns in different forms.

First, a quick word about how to measure success on programming benchmarks. Usually each problem has a set of tests, rather than the one final answer from math. There are two ways to report on the test set: as a percent of tests that passed, and as a binary on whether the entire set passed or not. Here they call the first approach “combined tests” and the second approach “all-tests”.

Anyway, we see on the left that NGU helps pass more tests across all problems. In the middle, we see in particular that hard tests get the most help, which is where most of the opportunity was anyway. Great alignment of compute with need, as expected with NGU.

We also see a big jump in all-tests, i.e. in problems with totally complete solutions. In fact, plain GRPO passes all tests on zero problems! That’s arguably more important, since we expect our coding agents to build to spec and not just close to spec. It’s the same reason ProgramBench reports all-tests as the top-line metric rather than combined tests.

There’s actually some prior art on training to solve Manufactoria. In it, the researchers first gave per-test rewards, watched until performance plateaued, and then switched to all-test rewards. You can’t do all-tests to start because none of the responses will be correct, so you can’t start learning.

NGU avoids this one-two step. As the chart shows, simply adding NGU to per-test rewards pretty much matches all-tests, even when you add NGU to an existing checkpoint that has undergone significant RL without it, as they did here. Again, NGU naturally addresses issues other methods must specifically design for.

Finally, one note about easy prompts. I mentioned earlier that easy prompts leave the training set, replaced by other prompts that may still have something to teach the model, so compute can focus on the harder prompts.

However, additional training does not guarantee improved performance, and sometimes gains in one place cause losses elsewhere. In other words, models can regress in areas we’re not watching.

The chart here shows how that looks. While performance improves for a while, eventually it degrades, roughly in proportion to how much was dropped from a difficulty bucket.

The simplest fix is to never drop examples, which costs compute of course but is kind of like paying insurance on the gains you’ve made.

But the researchers think we can do better. They propose dropping temporarily, after each batch, and then bringing back the prompts at the start of the next batch. Since NGU allows K to start small and grow on hard problems as needed, it’s cheap to re-check supposedly easy prompts at the start of each batch. As a bonus, their method lets regressions show up naturally and early, rather than as nasty surprises during eval.

My Takeaways

• Clean insights that everyone should adopt

  • Never give up

  • Never risk fluke wrong answers on easy examples

  • Never permanently drop easy examples

• Data that may seem too hard can still be valuable

  • If we never give up, eventually it won’t be too hard

  • But if all data is too hard, you can’t start learning. Training data needs distributions

• Maybe there is a way to more quickly elicit correct responses

  • Or to elicit them at all, if even at huge K the model produces none

  • The former is about saving valuable compute; the latter is about saving valuable training data

Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

or, A Few Ways to Be Right, Infinite Ways to Be Wrong

Paper · Repo

Online Rubrics Elicitation from Pairwise Comparisons

or, One Lesson Ahead of the Student

Paper

Background

Let’s briefly recap the motivation for rubrics.

At the start of 2025, DeepSeek R1 made big waves by revealing how OpenAI made their reasoning models like o1 and o3. The key was a technique called RLVR, reinforcement learning with verifiable rewards. You have a problem with a ground truth final answer that is easy to check, like a number or a short expression. You let your model try to solve the problem, and then you check the model’s final answer against the ground truth final answer. Sometimes you have to deal with equivalent expressions, like “square root of x” and “x to the power of one half”, but on the whole verification is straightforward.

The reason RLVR is nice is because it’s immune to reward hacking, which is where the model gets the credit of a good response even though the response is actually bad. That’s an unavoidable problem in RLHF, where you’re training a reward model using preference rank data based on fuzzy implicit human judgments. But with RLVR, because the final answer is verifiable against the ground truth, the model only gets credit for an actually correct answer.

Of course the set of prompts with a verifiable answer like that is quite small, mostly STEM problems and code using unit tests. But surely many more prompts have many aspects of quality that are objective and verifiable if your judge has enough intelligence.

That’s where rubrics come in. A rubric is a set of objective, atomic, self-contained, and MECE criteria that describes the ideal response to a prompt. You use a LLM judge to evaluate each criterion, then you total up all the points from all the fulfilled criteria. Sometimes the criteria are weighted, sometimes a criterion is worth negative points if it’s for a bad feature. But in the end you should have a verifiable score, just like in RLVR.

Of course not every aspect of quality can fit in criteria, and some prompts may allow radically different good responses. Like a creative writing prompt for example, where two responses may take totally different tacts but both be great. In that case you’re still going to need RLHF. But rubrics have really taken a bite out of RLHF because so many open response questions do entail responses with shared, definite aspects of quality.

One other advantage of rubrics over RLHF is you can reuse the rubric any time you’re getting a response to the rubric’s prompt. That’s not the case in RLHF, where you have to make a new judgment from scratch every time you get a new pair of responses. So the reusability plus the verifiability means customers want to use rubrics wherever possible.

Now for a long time rubrics weren’t really published about - they were secrets of the frontier labs. But slowly word has gotten out, and around early summer 2025 we started seeing rubrics papers coming out.

One of them was by Scale, which I covered in an earlier Friendly Paper Review. Here are two others that came out over the summer that I’ll cover briefly now.

The first one is from Google DeepMind. They frame the main motivation as avoiding reward hacking, as we discussed in the previous slide, it’s not about reusability here. They also focus on what they call “causal attributes”, the attributes of a response that cause it to be good, as opposed to “neutral attributes”. They use a LLM and pairs of responses to suss out these causal attributes - what we would call criteria. However, this is more of a proto-rubrics paper, since they ultimately still use preference ranks. It’s just that they use their causal attributes to synthetically alter responses, adding the causal attributes to one copy of a response and then making it the preferred version in the preference rank compared to the unedited version. In the end they’re still training a reward model like in RLHF.

The second one is from Ant Group, which is an affiliate of Alibaba, so distantly related to the Qwen folks. They have some pretty exotic stuff in their rubrics, like criteria with different levels of satisfaction rather than yes/no, diminishing returns on a criterion, and interactions between criteria, like negative criteria that veto all the other criteria. If you’re on a rubrics queue it’s worth taking a look at their examples, they’re pretty different from the rubrics data I’m familiar with. I think this is a result of disconnected ecosystems, where rubrics for Chinese labs look different than rubrics for American labs because they have evolved independently. But I acknowledge that is speculation.

The second paper is also interesting because it shows rubrics can exhibit reward hacking, which I think would surprise the first paper’s authors. Really it’s because the user of the rubric is a model, and just like with a reward model, the LLM judge using the rubric can be tricked. One specific way they show models do reward hacking is “self-praise”, saying stuff like “The following answer is accurate and thorough” to bias the LLM judge.

The Paper

This graph encapsulates the main finding of the first paper, which is that distinguishing between the top responses is really what matters for rubrics.

I should warn you though that these are theoretical results, which is why the curves look so nice and smooth. We’ll see empirical results later.

Here we see what happens to models trained on various alterations to scoring. The x axis, the KL Divergence, is a measure of how much the model you’re training has changed, so a higher divergence means a more different model, separate from how many training steps or how much training compute etc. The y axis, the Win Rate, is how often a response from the trained model is going to beat a response from the initial model. That’s why at KL Divergence = 0, all the curves start at 50% - that would be just two copies of the initial model and of course that’s going to be a coin flip.

Anyway, the different curves correspond to the different alterations to rubric scores, like what errors are in your scoring system compared to the true scores that responses would get using the true rubrics. Correct means the right scores of course, and Reversed means the exact opposite scores. Correct produces the best results and Reversed produces the worst results.

The interesting bit is the other four cases. If you take the worst 50% and flip those rankings, so like the absolute worst response gets the median true score and vice versa, all the way through the bottom 50%, you see virtually no difference with the Correct case - the lines almost totally overlap. Even if you flip the worst 90%, where the absolute worst response gets the 90th percentile true score and vice versa, again you’re able to recover the performance of Correct, although it will take more training.

Contrast that to flipping the top 25%, where your absolute best response gets the 75th percentile true score and vice versa. With much less incorrect data you’re already seeing worse results. And if you flip the top 50% then your model isn’t going to improve at all!

So there are two crucial steps when writing a rubric: one, get some truly great responses so you know where the ceiling is; two, make sure your rubric can accurately distinguish between those top responses.

Then the question becomes, how do you get these truly great responses? Because if your prompt is worth training on, you’re very unlikely to get a truly great response from your model; prompts where a truly great response is even somewhat likely are probably too easy.

If you’re training a weaker model, where a SOTA model is able to produce great responses on your prompts, then you can just write a rubric based on the SOTA responses.

However, if you’re training a SOTA model, you’re going to need humans to write your great responses, possibly by editing the SOTA model responses. So that’s the opportunity for our data.

So how do they propose making these rubrics that are so good at distinguishing between top responses?

The basic idea is simple: look at the best responses you currently have available and make sure your rubric can distinguish between them. If it can’t, update your rubric until it can. Then regenerate responses and repeat the cycle until your rubric has stabilized, until it can handle any pair of responses you throw at it.

To do that you need two things: a source of new responses, and a source of rankings. New responses can come from anywhere, since the rubric is model-independent. That means you can use a bunch of models, or even humans, to generate responses. For a really tough task where even SOTA is good but not great, a human response could go a long way.

The rankings typically come from a LLM judge using the existing rubric, but it doesn’t have to be that way. For example you could have a human pick the top two, perhaps from a smaller pool of top candidates as chosen by the LLM judge with rubric.

How many great response pairs do you need, like how many cycles? Well this table sheds some light on the question. Using Gemini 2.5 Flash as a good response writer and Gemini 2.5 Pro as a great response writer, the authors see how many pairs of responses it takes to distill a maximally effective rubric out of them. They’re training Qwen3-8B Base here, then comparing it against Qwen3-8B Instruct for the win rates. Remember, the Instruct model is going to have all that post-training on it, so that’s a lot of ground for our rubrics to cover, hence the low win rates even in the best version. But it’s the trend that matters, and moving from 5% or 10% up to almost 40% win rate is pretty good.

So unsurprisingly, distilling rubrics from better responses helps more. That fits right in with this thesis about chasing the tail.

And of course more great responses is better than fewer great responses. They don’t report numbers other than 1 and 4 pairs, but I assume greatly diminished returns after 4 based on them not reporting.

The interesting twist here is that last line: great and diverse pairs. That actually mirrors a finding in reasoning data, where showing the model multiple different pathways to the same correct solution is more valuable than just training on one or the other pathway. That robustness from diversity apparently holds up in rubrics too, which makes sense to me - it gives the rubric greater coverage, and really there’s no one perfect response to most questions. So your rubric needs to be able to recognize greatness in whatever form it takes. Which is a pretty tall order, but at least it’s a north star to shoot for.

So individual response quality is the most important thing, then number of responses, then diversity of responses. But they all matter for your rubric to distinguish top responses.

Now what exactly are rubrics gaining from these comparisons? Here they show buckets of criteria, ranked by frequency, and highlighted in cases where a significant number of the refinements came from comparing either two good responses OR two great responses, i.e. not comparing good to great.

So for example in the first highlighted row, refinements that “break down complex criteria into sub-components” come from comparing two great responses significantly more often than comparing two good responses.

And then the next row, refinements that “introduce penalties, prohibitions, or negative scoring” come from comparing two good responses much more often.

So it seems great responses introduce subtlety into the rubric, while good responses highlight common errors. Given how much distinguishing the best responses matters compared to distinguishing good or mediocre responses, it seems we should spend most of our rubrics time on the frontier of performance and let some of the lower scores be noisier if we have to.

Personally I would like to see this same analysis in fewer buckets. I would also like to see a split between positive and negative features, or like criteria that came from a good feature of a response vs a bad feature. So like if a response mentions something smart that no previous one has, that to me is different from a response making a mistake that the new criterion makes sure won’t happen again.

Intermission

So the first paper shows how we can generate rubrics from pairs of great responses. This one shows how we can adapt rubrics as training goes on.

The motivation is pretty simple: as you train the model, it gets better. And as we know from the previous paper, a better model producing better responses is going to need finer and subtler rubric criteria. So if we can use responses from the most recently trained version of the model in our response pairs, we should be able to keep finding relevant criteria.

That’s really all they’re illustrating here. In step 1, you get one response from your current version of the model and one response from some other reference model, like the initial version of your model or even a completely different model from another provider, as long as the control doesn’t change. Then you compare them to find new criteria, what the authors call “elicited criteria” shown in green, that you can add to your existing criteria shown in red.

After you score your responses with the newly expanded rubrics, you train your current version and then repeat the process in step 2. They chose to show a different prompt in step 2, but the three dots above the prompt in blue means you’re doing lots of prompts in parallel, including the one from step 1.

This type of data generation, where you’re using responses from the latest version of the model you’re training, is called online. When you use data from any other source, including older versions of your model, it’s called offline. For anyone who has done an RLHF project, RLHF data is most effective when it is online, which is why it’s a good line of business for us; once you retrain the model you need to generate new responses again.

Online is more work to collect, but it’s more effective when done correctly. It’s kind of like having a personal tutor instead of being in a classroom or reading a textbook. In the lingo you’d say it has better exploration and distribution matching.

Here’s the main result. If you compare any form of rubrics to just using a good LLM judge with a simple Likert scale, rubrics is gonna win every time. But within rubrics, it seems the online approach outdoes offline. And within online, it seems to not matter much whether you pick the control model to be the original one or an earlier iteration in the online cycle. Note that these are scores on the held-out eval sets, we’ll see benchmarks on the next slide.

Now these results are from training Qwen2.5-7B, which is a smaller and older model, but I suspect the results hold up on bigger and better models.

So here’s a kind of buildup of techniques, on generalist and expert data.

They start with the initial model, then show scores for the versions we saw on the previous chart: LLM judge, offline rubrics, and then both versions of online rubrics.

The new bits are the two techniques added on to offline. The first one, Universal Requirements, is from another paper, showing that a fixed set of criteria added to all rubrics improves stability and mitigates reward hacking. The second one, Pointwise Extraction, is in contrast to the pairwise extraction of criteria in online rubrics. Here they’re just looking at the new model’s response to find elicited criteria, rather than looking at new vs control. So even though Pointwise Extraction is grouped under Offline Rubrics, really it is an online technique.

Anyway, you can see that for both domains Online Rubrics is the best or tied, within some small margin of error. I would also note that the human rubrics are also the best or tied within the margin of error, compared to synthetic rubrics.

And in case you’re wondering about the win rate: it’s against the base model, Qwen-2.5-7B-Instruct. The reason the WR isn’t 50% on the first line is because they allow ties.

As with the previous paper, they use a LLM to bucket the criteria they found by looking at response pairs.

And a lot of these line up with the other list actually, even though they are somewhat different domains; healthcare for the other list, STEM for this one. I matched up some of the buckets from the other list here on the right. Some of them are nearly identical.

My Takeaways

• Distinguishing the top responses is the most important job of rubrics

• Rubrics are an abstraction that avoids artifacts and off-policy issues

  • Human-written responses, rewrites, responses from other models - all fair game

  • But like all abstractions, they eliminate some detail in exchange for generality

• Pairwise > pointwise, specific > universal, human > synthetic (to start)

• Other fun tips:

  • Watch out for reward hacking by “self-praise”, e.g. “The following advice is the most relevant”

  • o4-mini beat GPT-5 (and all other models) as a judge, regardless of generalist/expert

Paper · Website

Background

This is an unusual paper for us because it’s not LLMs. It certainly is machine learning though. In fact, the lineage of this paper dates back to the origin of the phrase “machine learning”.

It’s a remarkable paper to read in the present day. The author, Arthur Samuel, has this to say in the introduction:

“At the outset it might be well to distinguish sharply between two general approaches to the problem of machine learning. One method, which might be called the Neural Net Approach, deals with the possibility of inducing learned behavior into a randomly connected switching net (or its simulation on a digital computer) as a result of a reward-and-punishment routine. A second, and much more efficient approach, is to produce the equivalent of a highly organized network which has been designed to learn only certain specific things. The first method should lead to the development of general-purpose learning machines. A comparison between the size of the switching nets that can be reasonably constructed or simulated at the present time and the size of the neural nets used by animals, suggests that we have a long way to go before we obtain practical devices. The second procedure requires reprogramming for each new application, but it is capable of realization at the present time. The experiments to be described here were based on this second approach.”

Did you catch that? He predicted the eventual rise of general-purpose AI based on neural networks almost 70 years ago! What we’re living through now is that first approach he described: large, generic networks for general use.

What he goes on to discuss in the paper is the second approach: a specifically designed network for a single purpose. His network learned to play checkers, whereas the one we’re talking about today learned to play Stratego, but the root idea is the same.

Fun fact: while checkers at the time had no known algorithm mathematically provable to never lose, now it is a “solved game”; a team at the University of Alberta devised such an algorithm back in 2007, after 18 years of work.

Stratego, on the other hand, is not a solved game. In fact, it is impossible to solve, because the player cannot see everything on the board and cannot feasibly game out every possible scenario. In technical terms, she has “imperfect information”.

Here’s what Stratego looks like for reference. Each player has an array of different pieces, pictured on the key at the bottom. Pieces can generally move one square forward, backward, left, or right at a time. Pieces have different ranks and functions, similar to chess, but when the player’s piece meets the opponent’s piece it’s generally the higher-ranked piece that wins. So if my captain (rank 6) moves to take your piece and it turns out to be a sergeant (rank 4), your sergeant dies. But if your piece turns out to be a major (rank 7), my captain dies. If they’re the same rank, they both die. Crucially, after a battle, the winning piece returns to facing away from the other player, so you have to remember which of your opponent’s pieces you sacrificed a piece to temporarily reveal. There are a few other mechanics, but we don’t need to get into them.

The goal of the game is to capture the opponent’s flag. The player gets to choose the starting arrangement of all their pieces, including the flag, so a big part of the game is deducing where the opponent has placed his flag and his variously ranked pieces.

So while the basic mechanic of moving pieces on a board to capture other pieces is like checkers or chess, the choice of arrangement and the secrecy of placement make Stratego a challenge to model for a learning problem.

In fact, Stratego is such a good choice that there is an existing body of work. The best and most notable entry is DeepNash, by Google DeepMind.

It’s a hefty paper, starting with the author list: 34 folks, including deep-learning heavyweights like David Silver and Demis Hassabis. It also got a hefty budget: 2-3 months of training on 1k TPU nodes, at an estimated cost of $3M-$4.5M. They needed all that compute to complete ~5.5B self-play games, i.e. games between two copies or versions of the same model that bootstrap training data. Those 5.5B games entailed 5-10T training examples, meaning each game was ~1-2k moves on average.

A few million dollars may not seem like a lot now in the age of $2T market caps, but in 2022 it was a serious commitment of resources. It’s about the same as the training cost for GPT-3.

And what did those resources get? Something good, but not something great. DeepNash won 84% of the games it played online, but all the games were one-offs against random people who didn’t know they were playing a bot. When they brought it to the world championships in 2023, it went 19-9 but lost to most of the top-ranked players. There was no equivalent of Kasparov vs Deep Blue or Lee vs AlphaGo, with one master player playing a series against the same bot.

Now you might be wondering why it matters whether a computer is good at Stratego, or chess, or any game really. Why do researchers care about it? Isn’t the goal like, superintelligence and superabundance of material goods?

Well as Arthur Samuel noted in his seminal paper, games are a nice proxy for many real-world problems. If you can make a model that perfects checkers, that can obey simple rules and learn good heuristics, maybe it can spot spam or classify expenses. If you can make a model that plays Space Invaders, maybe you can make a model that controls a robot arm. If you make a model that builds in Minecraft, maybe you can make a model that folds proteins.

In fact, before the dawn of generative AI, a lot of the same researchers and companies were working on AI for games. DeepMind, starting from before it was Google DeepMind, mastered Atari games, then Go, then chess, although they didn’t quite master Stratego as we saw. OpenAI made a model that played Dota at championship level. Before LLMs really took off as the main use case for AI, RL for games and game-like environments was kinda the main thing many AI research labs did.

Even Dario Amodei, while at OpenAI, worked on games. This gif is from his classic study on reward hacking, where the model driving the boat in the game CoastRunners learned to rack up points not by winning races, but by hitting the same buoys over and over again. It’s a good example of how games can pull out the principles of machine learning. They are clean, controlled demonstrations for ideas that could work in production. Think of them as the lab rats of AI.

Of course, a wargaming AI might be directly relevant for war. No surprise then that today’s paper got grants from the Office of Naval Research.

So that’s some of the broad-strokes history. Now I want to get into the technical background.

The foundations of the paper are transformers and reinforcement learning, which we are familiar with from LLMs. However, there are several differences between even the basic LLMs of today and the models in this paper. I’ll describe the specifics once we get into the paper itself, but here I want us to un-learn some stuff from LLM defaults.

First is the architecture. We’re going to keep the transformer, i.e. the attention plus feed-forward network composing the transformer block, but we’re going to see three different variants: encoder only, decoder only, and encoder-decoder.

The difference comes down to how attention works. In our modern LLM framing, we expect causal attention, where each token is only allowed to attend to the previous tokens. That’s decoder only, and it makes sense when your goal is to keep predicting the next token, as modern LLMs do.

The opposite of causal attention is bidirectional attention, where every token in the input attends to every other token, regardless of position. That’s encoder only, and it’s good for cases where you just need to understand the input. The classic example here is sentiment classifiers: input a tweet or a product review or whatever, output a sentiment label.

You can stack the encoder and decoder too, and have the decoder causally attend to the encoder’s outputs plus the tokens the decoder itself has output. Actually, the first transformers did exactly that. Here’s a diagram of the original transformer: encoder on the left, decoder on the right. Note how attention for the decoder says “masked”, while for the encoder it does not. That’s because for causality, you need to mask out any tokens in the future relative to the token you’re attending to.

It makes sense on its face; encode your input with bidirectional attention, then generate your output token-by-token with causal attention. So why are modern LLMs decoder only?

It turns out that decoder only is better for scaling up, since you can treat all text as training data rather than separating parts into inputs and outputs; take a big block of text, make the first token the input, predict the next token as output, tack the next token on as input, repeat. In the age of trillion-token pretraining, decoder only wins.

However, in cases where input is clearly distinct from output all the time, having an encoder still helps. The modern example here is in multimodal; since images and audio are always part of the input and not predicted in the output, multimodal models use encoders. However, the text part is still decoder only.

We are going to see all three types of transformer in the paper, so just remember: encoder = look at everything at once, decoder = look only at past and present tokens, encoder-decoder = look at all of input at once + past and present tokens of output.

Now that we have our architectures nailed down, we can talk about reinforcement learning.

High-level, I always found RL intuitive: look at outcomes, score them, use the scores to, well, reinforce what’s working. Pretty relatable stuff.

However, once you get into the weeds it gets pretty gnarly fast. The jargon in particular I find dizzying, built up and mutated as it is from decades of work. So I want to provide a glossary of sorts, to help map the abstruse onto the intuitive and familiar. That will help us immensely in the paper.

The first term of RL art is “policy”, or sometimes “policy network”. This is the model in question, the one you want to train to get better. In the context of RL for LLMs, you do hear them called “policy models”, but not really otherwise. The policy network proposes “actions”, like moving a piece on the board or producing the next word in a response. In many cases, an action is a token, but it can also be many tokens.

In contrast to the policy network, we have the “value network”, also called “value function”. As the policy takes actions, the value network provides an estimate of the value of the current state and of the value of the proposed action from the policy network. The value of the proposed action relative to the baseline state is the “advantage”, which usually is what actually goes into your training algorithm. Because of the relationship between the policy and value networks, sometimes they also go by the names “actor” and “critic”.

Two things to note about value networks that can trip up folks coming from modern LLM training. One is that the value network may or may not be different from the policy network. In the latter case, you have a shared model with two different “heads” on it: one for proposing actions, another for outputting values.

Two is that values are not the same as rewards. Rewards are definite and based on the current state of the environment, whereas values are estimates based on the expected end state. Usually rewards come at the end when there is an outcome to evaluate, but intermediate rewards are somewhat common, and process reward models exist for giving out rewards at every step of a trajectory.

That covers everything in the diagram except “TD error”, which basically says how accurate the value function’s estimate was. It can be pretty fiddly and requires balancing definite, far-off rewards with estimated, near-term state values. Important parameters, but too granular for us today.

The Paper

Let’s see how the authors finally cracked Stratego using RL.

Their system, Ataraxos, comprises three models: one for the player’s initial setup, one for guessing the opponent’s board, and one for making moves.

First is the setup network. It is decoder only, since there is no initial input; the player’s side is initially empty, and the policy’s job is to fill it, one piece at a time.

The entire network is only four transformer blocks deep, for a total of 12.6M parameters. Now that may sound shockingly small for folks in LLM land, but you have to remember, these are not general intelligences; in fact, the only “language” the setup network here can “speak” is the twelve different Stratego pieces. It doesn’t even need to explicitly note the positions, since it spits out tokens in board order, from the top-left to the bottom-right of the player’s side.

Actually, there are two other outputs: the conditional outcome and the entropy. The conditional outcome says how likely a win, loss, or draw is given everything placed so far. The conditional entropy says how random the setup is, which is strategically useful for repeat games against a human opponent who would otherwise pick up on the machine’s patterns.

We’ll talk about training later on. First, let’s check out the architecture for the remaining two policies.

Okay, now this here is the belief network. The belief network is crucial because Stratego is a game of imperfect information. Ultimately, I think it’s the most important innovation in the paper, although more mundane changes to the learning algorithm get more love from the authors.

Conceptually, after the initial setup, the flow of the game is just alternating inferences from the belief and move networks: predict the opponent’s board, make the best move given the predictions, observe any new information from that action about the opponent’s board, repeat.

Architecturally, because the belief network has both an input distinct from the output and must produce many tokens that depend on each other, the natural choice is encoder-decoder. Size-wise, it’s six blocks for the encoder and four blocks for the decoder, weighing in at 57.1M parameters total. It’s by far the biggest of the three, but still tiny compared to even the smallest LLMs.

Anyway, for the encoder component in blue, we see tokens for every square on the board as inputs. Each input token contains information relevant to that position: the current piece in it and that piece’s move history, recent events within the square itself, and some global information like the number of turns left. After the encoder is a filter that keeps only opponent pieces with unrevealed types. The keys and values then become input for the decoder.

Speaking of, the other input for the decoder is a list of all 40 opponent pieces. Each input token has two pieces of information: the type of the previous piece in the list, and the list position of the current piece. So let’s say for piece 0 that we predicted Bomb. Then for piece 1, the information is basically “piece 0 is Bomb” and “this is piece 1”.

Like with the setup network, the belief network decides on its belief of the opponent’s board one piece at a time, just like how a decoder only LLM decides on the next token one at a time. Also just like an LLM, the end result is always a distribution over all tokens.

Finally, we have our move network, which says what square to take from and move to.

I phrase it that way, instead of phrasing around which piece to move, because of how the model actually works in the end. But let’s start from the beginning.

The network itself is eight blocks deep. It contains 14.7M parameters, which is a bit bigger than the setup network but much smaller than the belief network. One possible reading is that inferring your opponent’s board is the hardest part of Stratego, which checks out to me. The authors note they needed to keep the move network small since it runs more frequently during inference than the belief network and their ultimate constraint is compute, but it’s impossible to say how big either network would be if compute weren’t the constraint.

Since the move network only needs to return one thing - the move - the natural setup is encoder only. The input is the entire current board, plus a special token at the start called the “value prediction token”. Basically, it’s a place for the move network to store information that isn’t specific to a single square. In the end that token is going to return the likelihood of a win, loss, or draw given the current state of the board and future possible moves.

Now for the rest of the tokens, again they contain information about what piece is currently in them. After the layers of the move network they’re going to combine in a particular way to recommend a move. Specifically, they’re going to form queries and keys, which will multiply in the classic transformer way to produce attention scores. The query is the source square, the key is the destination square, and the attention score is the strength of the recommendation. It’s exactly like the attention operation in an LLM, where the query of one token and key of another combine to say how relevant the key token is to the query token.

So this is normally the place I would talk about data, us being at Scale and all. But once again, we diverge from the typical LLM methods.

With an LLM, you would first pretrain on a ton of unlabeled data, from the internet and from books and from codebases and so on. Then these days you would midtrain on a ton of agentic data - trajectories and so on. And then you would post-train with labeled data: prompt-response pairs for supervised fine-tuning (SFT), prompt-final answer pairs for reinforcement learning with verifiable rewards (RLVR), prompt-rubric pairs for rubrics, prompt-preference rank pairs for reinforcement learning with human feedback (RLHF), and prompt-environment pairs for reinforcement learning environments (RL envs).

For models like in Ataraxos, we have to throw most of that out. All we are doing is SFT and RL on data from self-play, and mostly not in a form that maps directly to LLMs.

To be clear, that means no pretraining at all. Which is actually what allows this project to work in the first place for a team of academics; instead of spending millions of dollars on a months-long pretraining run using hundreds of thousands of GPUs, they spent less than $8k on like a week of RL training using 16 GPUs.

The RL training they did do is like a mix of RLVR and RL envs. It varies by model. So for the setup network, it’s pretty much RLVR: the “prompt” asks for a board setup, and the “final answer” is the game outcome. The reward is 1 for a win, 0 for a draw, and -1 for a loss, with a bonus for entropy to discourage predictability. Note that the advantage is the reward minus the expected outcome, but in self-play the expected outcome should always be close to zero. They run ~163M self-play matches, and both players are the policy, so that means ~326M examples to train on.

Here’s the result of all that training for the setup policy. For each piece type, we see how often we find that piece type in a given setup. Note that most piece types have multiple pieces - four captains, five miners, eight scouts - but for spy, general, and marshal there is only one. Apparently putting the flag in the back corner is uncommon for championship-level Stratego players, but Ataraxos strongly favors it.

Now you might think based on how Ataraxos works that the belief and move networks are part of the same training loop, just as they are part of the same inference loop. But actually, belief network training is a separate, SFT-like step, after the RL for the setup and move networks is complete. Again, working against LLM intuitions.

So let’s tackle the move network and come back to the belief network after. Move network training is most akin to RL envs, where a policy takes actions in and receives rewards from the environment. There are many turns, the environment is only partially observable, and the reward comes only at the end of the game/trajectory.

It’s a bit weird though, because the opponent’s move policy is a big component of the environment! Not only must your move effectively impact your opponent’s pieces, it must also minimize the amount of information about your board the opponent can infer.

One consequence is that during self-play, which is where all the training happens, the majority of games end in a draw due to hitting the maximum number of turns without a battle. Also, the player version of Ataraxos (as opposed to the generally frozen opponent version) barely has a winning record. Compare that to a typical LLM-powered agent training in an RL env, where you expect rewards to go up over time and to eventually nail the task.

Self-play overall is trickier than the RL env training we’re used to because the opponent is dynamic. One resulting issue is cyclical behavior rather than genuine progression, like learning to throw rock because your opponent throws scissors but then learning to throw scissors because the updated version of your opponent now throws rock. The authors mention strategies to mitigate self-play issues, like adding in randomness and making progressively smaller updates over the course of training, but we will gloss over most training algorithm details.

Overall they collect and filter down to ~50B moves with enough advantage to be worth training on. That’s over 100x more data than the setup network, but it’s much noisier; any single move in a game between two evenly-matched players is unlikely to strongly impact the outcome, and estimating that advantage in a game spanning over 1k steps is guesswork at best.

So now we have a highly competent player, comprising the finished setup and move networks. Now it’s time to learn how to predict a highly competent player’s board, i.e. to train the belief network.

Belief network training is like SFT. The “prompt” is the player’s piece positions and types, plus the opponent’s piece positions, plus some parts of the game’s history so far. For example, any opponent piece whose type was revealed in combat retains that information permanently in the history, even though on subsequent turns the type is not directly observable. Of course, any opponent piece you defeated is visible in the opponent’s graveyard.

The authors put significant effort into engineering that “prompt” - those “features” as practitioners say - rather than just dumping all available information into it. You might think that’s due to the limited capacity of a sub-1B parameter model, but actually it’s a compute issue. The more stuff in the context window, the more compute it takes to make a single move. For these academics, compute is the bottleneck, so anything they can do to reduce compute needs per move or per game, etc., means more self-play games and more training steps.

Engineering features goes against The Bitter Lesson, but the premise of The Bitter Lesson is that over time increasing compute swamps everything. If you are working just at one moment in time, where compute is fixed, hand-designing features can help. The authors are fully aware of what they’re doing and even cite The Bitter Lesson. The main results of the paper don’t depend on these non-Bitter features though, so they should still be valuable. In fact their general approach - architecture, self-play, training, search - generalizes to other games of imperfect information.

With the belief network in place, Ataraxos is complete. Setup behavior in testing is the same as in training, but move behavior is different thanks to the belief network. Specifically, Ataraxos can now use search.

Again harking back to The Bitter Lesson, search is a powerful ML method because it scales with compute. But search isn’t straightforward in a game with imperfect information; for Stratego, you can’t even complete one move if it involves a battle without knowing private information. The belief network is important because it provides a reasonable substitute for that private information and allows search to continue, simulating many moves ahead.

The graph here shows the result. With no search at all, the Elo tops out just below 2100. With increasing breadth and depth of search - rollouts and plies, respectively - Elo ranges up to 2200+. At high numbers, improving Elo increases in difficulty exponentially, so gaining 100+ points is significant. For reference, it took the last 70% of RL training to go from 2000 to the final RL score as shown by the dotted line.

Once the researchers had their completed models in hand, of course they tested against earlier versions and some other bots and random players online, but those evals are no fun.

The real fun is playing a series against a skilled human. In fact, they had Ataraxos play against the best human - a Dutchman named Pim Niemeijer, widely acknowledged as the best Stratego player of all time.

Ataraxos played 20 games against Pim. Of course, Pim learned from each game, but Ataraxos did not; its weights were frozen. Despite its opponent having a learning advantage, Ataraxos was unpredictable enough over the 20 games to avoid giving much ground. It’s true that Ataraxos only won 6 of the last 10 compared to 9 of the first 10, but losing only 1 out of 20 is still the headline result.

Just for some flavor, here are the 20 setups Ataraxos used. The flag is always in the back row and always touches at least one bomb, but otherwise not much repeats.

Qualitatively, some world champions made the following observations about Ataraxos’s play relative to elite human play:

• It does a better job disguising the identity of its pieces

• It makes more high-variance bluffs and takes more “arrogant” risks

• It stalls and employs “rude” tactics when it is behind in order to force a technical draw, i.e. a draw due to hitting the cap on the number of consecutive turns with no combat

• It excels at strategies that take a relatively high number of turns to complete

So Ataraxos plays in some alien ways. Although it cuts both ways; to Ataraxos, human play seems alien. They actually incorporate that idea when training the belief network, with a technique called “dropout” that improves performance on out-of-distribution inputs. Since Ataraxos only learns through self-play, it risks not predicting human boards very well.

I want to conclude with the final sentence from the authors about play style: “Ataraxos feels preternaturally lucky, always seeming to have the pieces it needs in the right places, to have its gambles pay off, and to have its opponents do as it wants.” (Emphasis theirs.)

That’s what ASI feels like. Ataraxos is not general intelligence, but it is superintelligence.

My Takeaways

• The Bitter Lesson is undefeated

  • More and better search produces results

  • Are you making full use of agents and the power of search?

• There is plenty of juice left outside of LLMs

  • As we saw with Jev and the ensuing spurt of decision models, some work LLMs can do is better left to specialist models

• AGIs will train specialist AIs

  • And they will be major consumers of GPUs

• We will all be Pim in our own domains sooner or later

  • Will we all pull a Lee Sedol and retire?

Paper · Repo

Background

So to really understand this paper, we have to understand loss.

“Loss” is what ML folks call the mismatch between what you wanted the model to do and what the model actually did. A perfect model would always have zero loss, and a terrible model will have really high loss. More loss means we need to make bigger changes to the model to get somewhere good.

To measure this loss, we use a loss function. Here’s the loss function for pretraining actually. It says the loss for each token to predict is just the log of that token’s probability when it comes time to predict that token. If the model is perfect and the probability is 1, then log of 1 is 0, so the loss is zero. If the model is terrible and the probability is close to zero, log of zero is a very large number, so your loss can be huge. Technically the loss would be a very large negative number, but people like dealing with positive numbers so usually you’ll see a little negative sign tacked on to this.

So if you’re training on a big pretraining corpus of like 30T tokens, you’re going to get the loss for each of those tokens and then sum it all up. And that’s what you’re trying to minimize in pretraining.

This is also the loss we use in SFT! We don’t count the user tokens in the loss function since the model doesn’t predict those, but otherwise it’s exactly the same. And this loss function is why SFT is like imitation and can lead to memorization and why it’s great for enforcing formatting and so on, because the SFT loss is lowest when the model produces the exact token that was in the training data. Even synonyms will incur a loss; predicting “amazing” when the training data says “wonderful” still hurts. Very inflexible but sometimes that’s what you’re after.

For RL things get a little bit… messier. That’s because there are many different algorithms for RL.

Let’s review: in reinforcement learning, you give rewards for certain outcomes. For RLVR, you give a reward for a verifiable answer, like the answer to a math problem or the confirmation email when a computer use agent makes a purchase. For Rubrics, you give a reward for each satisfied criterion. For RLHF, you give a reward when your reward model picks the right response in a preference rank. Simple enough, but where does this reward information actually go?

That’s where all these different RL algorithms come in. REINFORCE, PPO, GRPO, and all the other versions are just ways of turning rewards into losses.

This here is a rough and incomplete taxonomy of RL algorithms. Even if you’re familiar with RL training for LLMs, you probably have never heard of most of these. In fact the only one on here you typically will see is PPO, Proximal Policy Optimization.

And actually within PPO are many variants: REINFORCE, RLOO, CISPO and so on. The most famous version is GRPO, Group Relative Policy Optimization.

Briefly, PPO requires this thing called a “value model” to say how valuable each token in a response is. You compare that value to the reward you got for your entire response - whether from a rule like in RLVR or from a reward model like in RLHF - and the difference between the overall reward and the value at that token says how valuable the next predicted token ended up being.

So on the image above, a prompt q gets a response o. That response gets a reward, in this case from a reward model but could be rule-based like in RLVR. You also see how far away you are from your reference model, so you don’t stray too far from a known good model. That’s called KL divergence, hence the KL in the diagram. Anyway the reward and the KL divergence together produce r. You also get the value v of each token in the response with that value model. You compare r and v using this formula called “generalized advantage estimation”, and that finally produces the advantage A. You need all that stuff for your loss function.

GRPO simplifies PPO by getting rid of the value model. Instead, you give the same credit to all tokens in your response. But now you need something like the value function that provides a baseline, a way to say “this is normal” vs “this is better and should be encouraged” or “this is worse and should be discouraged”. GRPO does this by normalizing the rewards, basically taking the average of all your responses to a prompt and seeing which ones are relatively the best.

So in the image above, you get multiple responses, each one gets a reward, and you normalize them to get an advantage for each response.

Here’s a simple version of that PPO loss function. We see our friend the advantage A, and also r which compares the current model with the reference model. The CLIP part basically says we only want to make changes up to a certain amount, even if some of our signals say we should make really big changes. The 1 - epsilon and 1 + epsilon parts control how big that maximum change is.

The main ingredient here is that advantage A, which depends on the reward and also the value where applicable. So from a data perspective, accurate rewards are the only thing we can influence. If we make an RLVR problem with the wrong answer or a rubric with missing criteria, the loss function isn’t going to point us in the right direction of how to change our model in training.

Compare all this to SFT, where yes, each token in the response we write influences the loss function, but also how much simpler the SFT loss function is. RL has way more knobs to fiddle.

Finally I want to clarify two pairs of terms I often see confused.

One is on-policy vs off-policy. This is just a fancy way of saying “the data came from this model” or “the data came from somewhere else”. So for example SFT, which usually comes from a human writing the ideal response to a prompt, is off-policy. But the RL we’re familiar with, like RLHF where we get two responses from a model or RLVR where we get a response from a model then grade it, that’s gonna be on-policy.

RFT, which stands for “rejection-sampling fine-tuning”, is when you generate responses from the model you want to train and keep only the good ones, based on whatever criteria you have, and then do supervised fine-tuning on them. That’s on-policy, but it’s offline; those good examples go into your dataset and stay there, like they’re still good responses no matter what later happens to your model.

Online means you don’t keep your data after training, you regenerate after you train. The RLHF we’re used to is like this, where we need to generate new data when there’s a new version of the model to train.

Usually online and on-policy go together, which is why you hear them conflated. Same with offline and off-policy. But they are distinct.

The Paper

So with all that background, we are at least somewhat equipped to understand their main theoretical point: that SFT and many RL techniques can be viewed as different versions of the same loss function, written here in the middle. Technically their “unified policy gradient estimator” here is the negative of the loss, but all the lessons we draw will be the same.

Anyway, we have four components:

1. The likelihood gradient - this means the action we’ll take is ultimately to adjust the model weights

2. The advantage estimate - this says we adjust more based on how good the response was

3. The reference policy - this is where you say whether you’re comparing to an older version of your model or not

4. The stabilization mask - this is for clipping

Not every algorithm has every one of these, but you can put in dummy values or terms that cancel each other out to make this universal form work. The helpful part is how it pulls out each of these four universal ingredients.

Here is how they translate various algorithms into their framework. Let me touch on the algorithms first and then explain the translation.

SFT, PPO, and GRPO we already know. REINFORCE is actually an ancestor of PPO from before the era of LLMs, but the 2024 paper they attach to it here is applying REINFORCE to LLMs - either way it’s basically PPO but without the value function. CISPO and GSPO are offshoots of GRPO.

SRFT and LUFFY are not important to distinguish individually, but you’ll see they’re grouped under “Offline/Online RL Methods”. They are direct competitors to the method the authors of this paper are going to reveal, so we’ll wait until then to discuss what this title really means.

Now the reference policy is what the algorithm compares to when seeing how big of an update to make. If a correct token is predicted to be highly likely by the reference model, make a small update. If a correct token is predicted to be highly unlikely by the reference model, make a big update. Pi-zero here means the current model, pi-zero-old means an older but still recent version of the model. Using the most recent version is higher risk but higher reward; if you’ve been training and your current version is untested because you just finished one cycle of training, you don’t actually know if it’s better or worse than the last version you tested.

That’s why some algorithms use an older, fixed version that they have tested. Your updates are going to be anchored to your fixed reference model here, which means less risk but also less reward, since your fixed reference model may actually be quite far away from the global optimum.

Don’t worry about the GSPO one, it’s too complicated to get into here.

Okay, next is advantage estimate. We talked about PPO and GRPO in detail. For SFT the advantage of every token is 1 because in SFT every token of the response in the training data is supposed to be the ideal token. Again, that’s why SFT is like imitation or memorization. REINFORCE is actually similar, but you get -1 across the board for a bad response.

And when you put everything together in the right column, you can see that the gradient is indeed a function of these four components. There was no column for the stabilization mask but it’s in the final formula, like you can see 1-CLIP in PPO and GRPO.

So what is their practical innovation here? Well it’s to do with those two algorithms from the last slide that I didn’t explain, under “Offline/Online RL Methods”.

The idea is much more simple than all this fancy language makes it seem: if SFT and RL have different strengths, can you do better by mixing them? Or another way of looking at it: instead of choosing a certain amount of SFT to do and then a certain amount of RL to do afterwards, what if we discovered the right mix during training?

The top formula there encapsulates it: let’s make the loss a combination, a weighted sum of the SFT loss and the RL loss.

The second screenshot there shows the exact loss formulas they used in their experiments, but you could use a different RL loss from a different algorithm if you wanted to. Theirs is a variant of GRPO called Dr. GRPO. Hopefully you have a rough idea what this is saying by now - look at the relative likelihoods of a token for the current vs reference model, see how advantageous it was, do that for all your tokens across all your different responses to the same prompt, that’s your loss.

The third screenshot is the definitions of alpha and beta, the coefficients that determine how much of each loss goes into your final loss. The math looks fancy, but really all it’s saying is that if your model gets at least some threshold of correct responses to a prompt, where gamma is that threshold, then just use the RL loss function. If your model doesn’t hit that threshold, just use the SFT loss function. It doesn’t actually recommend a mix of both on any one problem at any one time - alpha and beta are binary, and when one of them is on the other is off. The mix that you discover is when to switch loss functions per problem, per training step.

So does this work? And how much better is it?

Well they train a couple models, mostly Qwen2.5-Math-7B, on a bunch of math benchmarks. The rows show the base model, then just SFT, then just GRPO, then SFT followed by GRPO, which are all standard patterns. Unsurprisingly, SFT followed by RL does the best, as is traditional.

The next section down is three different hybrid techniques, including theirs, Hybrid Post-Training or HPT. HPT does better than all the other techniques on all benchmarks except for one, including on the two benchmarks they didn’t specifically train for as shown on the right.

The other entries in grey we don’t need to worry about, they’re just other attempts at math post-training on the same base model. That comparison is less instructive but good to see how HPT stacks up with other respected techniques.

Now one charge they need to defend against is something called the Elicitation Hypothesis.

The Elicitation Hypothesis asserts that RLVR, using whatever algorithm you want, doesn’t actually teach the model anything new; instead, it makes hidden knowledge or abilities come to the fore, it elicits rather than teaching. So your RLVR’d model gets more things right on the first or maybe second or third attempt only because it has prioritized a correct solution that it otherwise would rarely produce.

The way that some papers have shown the Elicitation Hypothesis to be true is with charts like this, where you take a really high number of samples from your model and see if any of them have a correct solution in them. In the jargon, that would be pass@k for very large k, like 1024 in this case. If your base model hits the same pass@ large k that your RLVR’d model does, that means the base model had the solution in there the whole time and that RLVR just elicited it. So what you’d see on a chart like this is two lines, with one line starting above the other but then the two lines meeting at a big enough k.

But that’s not what you see on these charts, at least not totally. In the first chart that’s not what you see at all actually - the lines for HPT and Base Model are basically parallel, so it seems like the model really learned something new from HPT.

The second chart is more mixed, with performance leveling off for HPT while it still goes up for Base Model. So perhaps HPT is doing some elicitation but also some teaching, and we’re hitting the limit of this small model’s performance at 85% or so.

The third chart is the most Elicitation-like, although the model is getting basically 100% so there’s really nowhere else to go. Still, if the base model can get nearly 100% without any training if you just give it enough chances, then maybe HPT only could elicit, since apparently almost all the answers were in there somewhere.

Now this is a fun one. It’s showing how HPT stacks up against the traditional SFT then RL combo on the exact same set of problems. Each row is a problem from the dataset, and Level 3 and Level 5 are two different difficulty levels of problems. Blue means SFT then RL did a better job, red means HPT did a better job. The denominator on the key is 8 because at each step they’re generating 8 responses to the problem and grading each one.

So as you train more, as you go through more epochs of training, the chart generally shifts from blue to red, or at least to white which means no difference. In some cases it’s a pretty dark red, meaning HPT got the model to get 8/8 attempts right but SFT then RL made the model get 8/8 attempts wrong.

Here’s a quick look at the mix of SFT and RL. The higher the number, the higher the ratio of problems that used SFT, which in this case happened when the model got all 8 of its attempts wrong.

The weaker model needs more SFT the whole time, especially in the beginning, but in both cases it’s a small share most of the time. But apparently that small share makes a significant difference!

The last thing I want to note is how this model preserves entropy.

Briefly, entropy is a measure of the model’s willingness to explore, as in our explore-exploit dynamic from earlier. Too much entropy is unconstructive of course, but too little means you miss out on some solutions outside your exploration zone.

It’s a known issue now that naive RLVR depletes entropy and overoptimizes models, which the left chart shows; the grey line for GRPO descends and bottoms out to nearly zero pretty quickly.

But that little bit of SFT you keep injecting with HPT apparently does wonders for entropy, keeping it pretty steady on average. And that willingness to explore keeps the model more robust, better able to deal with problems outside its RL training.

My Takeaways

• SFT will remain useful

• Full worked solutions are more valuable than isolated final answers

• Data that preserves (or even restores?) entropy could be valuable, but is under-explored

  • What would entropy-informed RLHF data generation look like, where entropy guides the creation of preference ranks?

Paper · RL envs · RL envs explorer

Background

So, let’s talk about MiMo.

MiMo is the name for all the models from Xiaomi, a Chinese consumer electronics giant. In fact, the name MiMo just takes the “mi” from Xiaomi and adds “mo” on the end, which is short for “model”. The straightforward and perhaps uncreative name suits, as we’ll see in a bit.

Anyway, Xiaomi makes an incredible range of stuff: phones, computers, appliances, power tools, scooters, cars, even shoes. And they also make the software that runs it all. Well, not the shoes, but the rest of it yes.

So as many other big tech companies have done, Xiaomi has gotten into the LLM game. At minimum, they want their own AI to run their vast empire of stuff in the same way SpaceXAI does. But like SpaceXAI, they also want external users of their model, to make AI yet another line of business.

Xiaomi’s efforts started out modestly, with a 7B parameter model in April 2025. I recall no impact at all from their initial release, which is perfectly fine for a new lab catching up to the frontier. It’s just about proving you can do the thing.

Where it really starts to get interesting is later in 2025, when a major researcher from DeepSeek jumped ship to lead development of MiMo. After that the catchup got a lot quicker:

• MiMo V2 Flash, at 309B parameters, in December 2025

• MiMo V2 Pro, at 1T parameters, in March 2026

• MiMo V2.5, at the Flash size but without the Flash name, in April 2026

V2.5 in particular was notable for being nearly identical in benchmarks and in price to DeepSeek V4, which also came out in April 2026. So at that point Xiaomi had made it crystal clear what their strategy was: copy the best Chinese lab and use their size and distribution channels to muscle them out.

As you would expect from corporate copycats, the MiMo series historically didn’t generate much notable research. The lab’s approach has started to change though, and the V2.6 development process has gained respect from the AI community for being remarkably open.

The screenshot here is part of the story. Xiaomi actually livestreamed their RL, with top-level metrics for accuracy, tokens, and cost. There are graphs for many granular metrics too, like the average staleness of rollouts and the KL divergence. It’s a remarkable view into what RL actually looks like from a researcher’s perspective, and a fair show of confidence to make your ups and downs so public. They even issued notices at the top about infra problems they encountered.

And of course, like DeepSeek, Xiaomi makes its model weights freely available under a permissive license.

Speaking of other model makers, this is where V2.6 Pro and Flash land on the Artificial Analysis Intelligence Index. It’s not the only way to index benchmarks into a ranking, but it is an industry standard.

A couple things to note. One is that the US models are clearly ahead. We’ll see later on that Xiaomi has chosen only US models for comparison, albeit the previous and not current set of frontier releases. As the heuristic dictates, the Chinese models remain about six months behind.

Within the Chinese models though, V2.6 Pro apparently leads the pack! It’s a lead of only one point, and I expect the next Pro-tier model from a leading Chinese lab to surpass it, but still impressive for a lab that has only been making models this big for about a year now.

In the Flash tier, MiMo is a bit behind DeepSeek, and here I credit the science of DeepSeek for keeping them ahead. Specifically, their Engram technique, which I covered in a previous talk. The reason I credit Engram is that the other Flash-tier model ahead of MiMo, Qwen3.8-Flash-Next, also implements Engram. So when DeepSeek V4.1 Pro comes out, I expect it to surpass MiMo-V2.6 Pro for at least partly the same reason. (The other big reason is that later releases always have more data to train on and so should generally be better, without any other changes.)

Now we move into the technical background. And in fact, we will be looking at the most important scientific contribution Xiaomi has made thus far: multi-teacher on-policy distillation (MOPD).

That’s quite a bit of jargon, so let me break down the meaning and lineage of each piece.

The base word, “distillation”, means taking the wisdom of one model and transferring it to another. Historically, distillation was from a bigger model to a smaller model, which makes sense with the term; the idea is to make the intelligence of the big model as pure and direct as possible so it fits in the smaller model. The original paper on distillation, from way back in 2015, is by Geoffrey Hinton and Jeff Dean, who are both celebrities in the ML world.

On-policy distillation is a much more recent contribution by the folks at Thinking Machines. In their post they describe the problem with distillation, which is that it fails to capture the range of good responses for a given prompt. Like if you take a big model, give it a bunch of prompts, capture its responses, and then do SFT on the student model using those prompts and responses, then you risk teaching the student to mimic the big model rather than learn from it. That’s because you have presented a single, complete response as the right one, when really there are endless variations on what a good response can be in most cases; language is highly flexible, and even math problems can admit multiple different solution pathways.

On-policy distillation solves this problem by teaching the student one token at a time, on its own tokens. So the teacher and the student get the same input, then you compare the next-token probabilities from the teacher and the student, and you minimize the difference between the two distributions. The feedback is much more granular now, and also the student gets credit over a wider set of tokens. Like if we analogize to chess, it’s helpful for the student to have in mind several good next moves that the teacher can each grade, not just the one the teacher picked at that step, because a slightly different board could make the second-best move into the best move.

The graphic above shows on-policy distillation on the right actually. You can see they have a student model that generates a probability distribution at each position - Token 1, Token 2 etc. Then the teacher gets the same input and generates its own distribution at each token position. You compare the distributions using this metric called “reverse KL”, and you optimize the student model to minimize that reverse KL across all the tokens.

By now you have probably figured out the final part of MOPD: “multi-teacher”. Instead of one smarter model teaching a less capable model, you have several domain-expert teachers all distilling into the one student model. Which makes sense right? When you’re in school, different teachers have different expertise and you learn from each of them.

Notice though that we still have to train all our teachers. So why train a bunch of teachers who then train a student - all originating from the same SFT model by the way - when you can just train the one model directly?

Well there are a couple reasons. One reason is it’s easier to train on one domain than it is to train on several domains, either concurrently or in sequence. Training domain-expert teachers allows you to optimize each teacher for just its domain, so each teacher achieves better results than a jack-of-all-trades model would. Again, the human analog holds up well here; as a student, you want to learn from the best, and the best teachers have deep specialty.

The other, more operational reason is it’s faster. Notice the structure of Stage 2 in the diagram: the teachers all train in parallel! Assuming the same amount of training data and compute, it’s faster in wall time to train many teachers at once and then do distillation at the end versus training one model on all that data in serial.

Xiaomi first unveiled the technique in the MiMo V2 technical report, and since then it has gradually become a standard post-training technique, as we first discussed in the Nemotron 3 Ultra technical report where MOPD plays a major role. It’s not an incredible breakthrough, more a combining of existing techniques into a new package, but they did present and name the approach. In this technical report, we will see its successor.

The Paper

As always for new models, let’s start out with the architecture.

There’s not a ton to report here actually, since V2.6 is a minor release and generally inherits from V2.5. In fact, pretty much the entire focus of the paper is on post-training, which is right in Scale’s wheelhouse. The lead researcher at Xiaomi has also said they are saving their major innovation for V3, which I expect will come out before the end of the year.

Anyway, let’s tick through what they’ve shown us in their architecture diagram. On the left we note vision and audio inputs, broken up into frames. Then all the inputs go to this “Hybrid-SWA Backbone”, which uses a bog-standard pattern of several local attention layers for every global attention layer. There are no fancy attention tricks, nothing with the KV cache, etc.

After you predict your token x_t, you also speculate three additional tokens with the MTP block over on the right. Again, MTP is standard and expected these days, although now the more common setup is speculative decoding with a separate draft model like DFlash or DSpark. And actually, after pretraining they compared their MTP setup with DFlash and found that DFlash worked better, so they dropped MTP. But if you want MTP you have to have it in pretraining, so you might as well do it and run the cheap comparison against DFlash later. Apparently pretraining with MTP also helps the model predict x_t better too, which I didn’t know.

Here are the architectural parameters all laid out. I want to call out three quick things here.

First is that V2.6 basically is a new mid-train and post-train on top of V2.5, although strangely it’s unclear if they actually did pretrain from scratch or just reused the V2.5 base as you’d expect from this being a minor version number bump. There is little detail about mid-training, just that they add a lot of agentic data in, so this technical report is mostly a post-training report. More on that later.

Second is the MoE setup. They are in a pretty normal range for sparseness, i.e. the ratio of total experts to activated experts. The only somewhat unconventional choice is to have no shared experts.

Third is the speculative decoder, which I mentioned on the last slide. The parameters in here aren’t important, I just want to cite the change from MTP to DFlash.

There is literally a single sentence about SFT, so we’ll jump straight to RL, which is really the focus of this model and this paper.

Of course before you can do any RL, you need the data to RL on. They spend a significant amount of the paper talking about their data synthesis methods, they have a whole other paper about deriving tasks from repos, and they even released 7k environments that I’ll touch on a bit later.

Much of the focus is on code, including the graphic here. On the left they enumerate sources of tasks, which are mostly self-explanatory but come with a few fun nuggets.

For one, “everyday workflows” includes data from employees, both vibe coders and professional developers. Harvesting employee data for training is increasingly common, with a mention in the most recent DeepSeek technical report and a mini rebellion for similar stuff at Meta. Given the increasing importance and general rarity of data from real, highly skilled workers, I can only see data harvesting becoming more common.

For “long-horizon engineering”, they take existing tasks and make them increasingly complicated with scope and dependencies. I strongly suspect the result is a high share of contrived tasks, but they don’t give any examples and none of the tasks they released come with the labels given here.

And for data vendors, sadly they don’t name names.

The supervision half on the right is worth describing just to demonstrate how much the dropping cost of inference unlocks in training:

“Each task is attempted four times by a coding agent. An auditing agent receives all four rollouts in a shared workspace containing the problem statement, tests, reference patch, and each rollout’s submitted patch, test output, and full conversation log. The auditor first articulates what a correct solution requires and assesses whether the tests capture those requirements. It then evaluates each submitted solution against the specification using its patch and conversation log, and compares this assessment with the observed reward. A passing solution judged incorrect is flagged as a potential false positive, suggesting incomplete verification. A failing solution judged correct is flagged as a potential false negative, suggesting overly restrictive tests or execution failures. These disagreements identify tasks requiring further review and provide concrete evidence of possible mismatches between the intended behavior and the implemented verification.”

That’s a lot of agentic coding and inspection per task! They also use an agent to examine the trajectory for evidence of reward hacking. Intelligence begets intelligence.

Now on the general agent side, the focus is more on diversity instead of complexity and difficulty, since “general” is a wider category. Also, because of that diversity of outputs, verification takes more thought.

Start with the environments. Here you have a diversity of interfaces - MCP, API, CLI, GUI - as well as a diversity of tools and applications. Add in the fact that many applications allow multiple interfaces, and you can see how managing the diversity becomes a primary challenge.

General agent environments also have a wider array of artifacts in them, like PDFs and spreadsheets and CAD files. Here they collected real-world artifacts, either for direct inclusion or as inputs for artifact synthesis.

And as before, they threw tons of intelligence at the problem, with multi-agent pipelines for making mocks of popular software and creating tasks based on the environment, and multiple rollouts to test tasks and make rubrics from.

They don’t have dedicated graphics for anything outside of code and general, but they specify visual and cybersecurity agents as additional areas of focus. The one neat thing I wanted to mention here is their split between “open-ended design” and “high-fidelity visual replication”. The latter is mostly objective and thus easier to verify, so of course they’re going to train on it, but they do find rubrics with a vision-equipped LLM judge work for the more aesthetic matters of open-ended design. We will see a bit of that effect at the end of the talk.

You can actually go look at all the environments they made. If you’ve never looked at an environment before and find the whole thing a bit abstract, find a few where you’re comfortable with the subject matter and poke around. I recommend filtering to General, where the tasks and environments are still complex but the subject matter is often straightforward.

It’s nice of Xiaomi to release this data. I think it’s part of their strategy to gain adoption via goodwill and engagement with the research community, which isn’t as measurable and straightforward as benchmark scores or pricing, but can be just as effective.

Now in terms of what and how they’re rewarding in these environments, they have two complementary tactics for their code agent tasks: Groupwise Reward Synthesis (GRS) and Groupwise Advantage Redistribution (GAR).

On the left we have GRS, which I think will be familiar to anyone who has worked on rubrics before. The idea is to review many different rollouts, many different responses, to then have an agent abstract away into a rubric. The interesting bit here is the two sets of criteria in the rubric: solution and behavior. The solution criteria are about the end state, whereas the behavior criteria are about the way you got there. The examples of behavior criteria in the paper are “gathering relevant evidence and checking the effects of code changes,” but surely there are many more - including some negative criteria for reward hacking.

I wish they showed or discussed an ablation with the behavior criteria, because my general understanding is that intermediate rewards can hurt output quality because they add constraints. Maybe they did the ablation and just didn’t include it. Either way, they definitely kept the behavior criteria in, and even made the reward formula multiplicative, meaning a zero on behavior means zero reward total. Pretty strong endorsement.

So GRS is for tasks with a high pass rate, where distinguishing by behavior adds a layer of refinement. The other method, GAR, covers the rest and can handle more disparate outcomes between rollouts.

GAR starts with a group of rollouts and their rewards. After a sweep for reward hacks, and a zero reward for any rollouts found guilty, the remaining rollouts get a look from an agentic grader that evaluates aspects of quality not represented in the rewards - maintainability, readability, consistency with codebase conventions, stuff like that that isn’t just a matter of running unit tests.

The resulting ranking of solutions then redistributes the advantages, which are basically the rewards but on a curve. The advantages then go into your loss function so you can update your model weights.

Here’s one fun result of all this agentic grading going on: your costs go up! Between the two models, they spent about $450k just on agentic grader inference.

Of course the lion’s share of the cost is on training and inference of the model you’re developing, and related models like the teachers. I was surprised to see rollout costs match training costs, but given the rollouts can be hundreds of thousands of tokens long, it’s believable.

Here are the classic RL graphs showing scores and rollout length increasing with training. The flatline on Flash for tokens on AutomationBench is surprising, but everything else looks as you’d expect.

One other, more recent graph I’ve been seeing a lot is how different harnesses perform over the course of training.

Here, the Xiaomi folks took a stripped-down approach to harnesses, creating four barebones variants for use in training rather than taking popular harnesses like OpenCode or Codex off the shelf. They also tested performance on existing third-party harnesses over the course of training, as shown on the right.

There’s actually not much to see here; performance generally improves over the course of training regardless of harness. Apparently Codex isn’t a great fit for MiMo, but otherwise it seems harness use is a pretty generalizable skill.

Okay, now remember how we talked about MOPD earlier? Well here is MOPD2, which stands for “multi-prefix multi-teacher on-policy distillation”.

Actually it’s a bit confusing to call MOPD2 a single technique, because really it contains three: standard MOPD, Teacher-Prefix OPD, and SFT-Prefix OPD.

Let’s start with what they share, which is part A at the top. For verifiable tasks, they use all the normal RL techniques to create teachers. For “open-domain” tasks, like game development and scientific research, they use SFT instead. Either way, they make many teachers.

For the RL teachers, we can use them in the expected way, which we discussed in the background slides. However, we can also use them in this new way, Teacher-Prefix OPD. Here, we have a series of turns in a trajectory from the teacher, and we split that trajectory into different prefixes, each with a different number of turns. The student then generates the next turn in the trajectory, and the teacher grades it as normal.

For the SFT teachers, we only use them in the new way. So for SFT-Prefix OPD, we instead have some existing trajectory, not from the teacher model. We do the same splitting and next-turn completion, and then the SFT teacher grades.

As we know from before, rollouts are getting quite long and very expensive, in both compute and in time. The big advantage of MOPD2 is you can train at the level of a single turn, which makes your updates much more precise.

In fact, the MiMo team recently patched V2.6 using just this method! After the initial release of V2.6, some users reported repeated tool calls - calls with similar or identical arguments that did not contribute to completing the task. Classic reward-hacking smell.

And actually, the team had anticipated this form of reward hacking and added in a tool repetition penalty. But apparently the penalty wasn’t harsh enough.

Now they could have gone back to an earlier checkpoint and done more RL with the harsher penalty added in. However, given the length of rollouts and the relative rarity of the bad behavior to begin with, the authors estimate it would have cost over $2M to fix with standard methods.

That’s where MOPD2 came in. Specifically, they could take trajectories where the bad behavior occurred, take the prefix up to the turn with the bad behavior, and then have a teacher with the correct behavior supervise the model at that turn. All it requires is training a teacher with the right behavior and distilling it onto the student.

So that’s what they did. And it fixed the problem, for $90k instead of $2M.

Now you might be thinking, if it’s such a small problem, why didn’t they just do a little extra RL on top of the final checkpoint to fix its repetition problem? Well the post doesn’t explain why, but the likely reason is that training in one domain can degrade performance in other domains. When they made the new teacher to address the repetition problem, they turned back a few steps in their existing MOPD2 training pipeline and inserted the new teacher in with the rest of the teachers, so the last few steps now included anti-repetition in addition to all the other skills.

Here’s what we get for all that work. 2.6 is a big jump from 2.5, and 2.6 Flash trounces 2.5 Pro. For folks who joined us for DeepSeek-V4.1 Flash, it’s that story on steroids.

What’s interesting to me is what competition they picked: leading American models. DeepSeek isn’t on the original table, and neither are Kimi and GLM. The authors don’t explain their choice of competition, but to me it implies they are shooting for the absolute frontier and not just the Chinese frontier or the open-weights frontier.

More cynically, maybe they are trying to hide their losses to non-frontier models. For example, DeepSeek beats them on DeepSWE (74.2) and AutomationBench (54.8), GLM beats them on Terminal Bench 4.0 (41.8) and ExploitBench (54.4), and Kimi beats them on OSWorld-Verified (84.8).

I don’t mean to diminish the accomplishment here. I just want to call your attention more generally to framing and choice of benchmarks in launches. For all the scientific rigor around benchmarks and other capabilities measurement, release blog posts and even technical reports have some element of marketing.

It’s not cherrypicked though. Putting aside the internal benchmarks, the ones they picked are mostly legit and appear in many other announcements by rival labs. I would only note CyberGym is saturated and therefore not so telling, and that the OSWorld folks have released a successor in OSWorld 2.0 that is probably a better indicator.

One final place we do have results to look at is here, in a distillation experiment they did.

To show the efficacy and transferability of their training data, the authors take Qwen3.5 9B and train it in two stages: SFT, using prompt-response pairs distilled from their final model; and RL, using all the assets they released and that we discussed earlier.

Of course the training worked and produced much improved benchmark scores, but the interesting bit is in the qualitative results they show here.

The prompt asks for a website about the heritage of Ethiopia and provides some high-level design guidance like the color palette. There are no mocks or wireframes; the model has wide latitude.

As we can see, the starting model - which has gotten its own SFT and RL from the Qwen team by the way - is perfectly fine but not especially enticing. The tones of orange are a little too close, and the emojis at the bottom are amateurish, but it has many fundamentals right.

After some SFT from MiMo, things get more interesting. For one, the color choices make more sense. The text is also significantly more compelling, although I know you can’t read most of it. It’s a more creative and thoughtful interpretation of the prompt.

Finally, after all the RL, the best version yet emerges. The hero image is striking, the orange is a sparing highlight rather than a frontal assault, and the text has once again improved. It’s not perfect - one button has overflowing text, and they use the historically outdated name “Abesinia” with a weird alternate spelling - but it’s good.

I appreciate the authors for throwing in an eval that is more subjective but can still show clear progress. It’s something I would like to see more of in these technical reports.

My Takeaways

• It’s relatively easy to catch up

  • Especially if you can distill, which American open models that lag the Chinese open models likely haven’t been doing

  • But where are all the European and Korean/Japanese Xiaomis?

• Faster/cheaper inference means faster/cheaper training

  • Rollouts are enormous now

  • “Given enough eyeballs, all bugs are shallow”

  • The very first words of this paper are “Recursive self-improvement”, which depends on fast/cheap inference

  • Expect more Jalapeños, and more chip-design + kernel-authoring RL

Paper · Environment Demo

🌍❓

I think we have to start with, what are world models?

And right at the top I want to say they are not necessarily models of The World, like the planet or the universe as we know it, although there are models that attempt something like this, where they can take a realistic scene and extrapolate forward, like what’s down the street in a city or what’s on the other side of a mountain. The Genie series from DeepMind is a world simulator like this.

That’s impressive and all but it’s kind of on a different axis than what our paper talks about. The world model concept here is more abstract, and the worlds they choose to build are very simple.

The world model concept is about whether a model can form a coherent and predictive view of a given environment based on limited information.

That may sound like a requirement for making good predictions, which we know models can do, but it’s actually not. Let me give you an example.

Let’s say you’re flipping a coin and you don’t already know that coin flips are inherently 50-50. One way you could discover they’re 50-50 is to flip the coin a bunch of times and extrapolate from that data to predict future data. You don’t need to understand the nature of coins or the laws of physics to do that extrapolation, you just need basic data analysis skills.

Of course the way a human would conclude coin flips are 50-50 is to look at the coin, maybe toss it a few times to make sure it’s not weighted, and then reason a bit to predict the coin toss will be 50-50. That does require a world model, because it requires underlying assumptions about how objects behave in the physical world.

The point of this world model line of inquiry is that LLMs do not have world models by design - they are just trained to predict the next token, much like the extrapolation from coin toss data we mentioned earlier.

So can they acquire world models emergently in this next-token training? Or maybe with certain data? Or does world model understanding require entirely new approaches to artificial intelligence?

One early but still quite relevant attempt at world model evaluation is the Abstract and Reasoning Corpus for Artificial General Intelligence, better known as ARC-AGI.

It’s from 2019, created by Francois Chollet, a longtime Google engineer who also created an influential ML tool called Keras.

It asks models to learn and then apply a pattern, as shown here.

Now for humans this is pretty easy, sometimes laughably so. Like the first example seems incredibly obvious, just complete the 2x2 square by adding a 1x1 square in the missing corner. But actually this can be pretty hard for models as we’ll see.

One note about the modality though: the default input is textual, specifically a 2D array of color names. If you don’t know what that means, just imagine each square here is written down as the name of its color and then put in a specially formatted list. Of course you can turn that into an image and then give that image to your model instead, or you can provide both, whatever works best. But of course not all models accept images, so those ones are going to be working with the textual representation only.

Regardless, in either case the output has to be textual.

This is what the ARC-AGI leaderboard looks like. They have a traditional ranked list version, but because throwing more thinking tokens at the problem does tend to improve outcomes, they have this chart of performance vs cost. So you can see all the way out on the right, it cost o3-preview hundreds of dollars per task on average. But it did also set a performance record for a model. The two orange dots at the top with papers rather than model names are custom systems built on top of models. Whereas the model names are just from feeding the textual and possibly image input straight to the model and not providing any special tools etc.

Even with all that token budget and all those tools though, the original ARC-AGI still hasn’t saturated! It’s perhaps the oldest ML benchmark fitting that bill.

But as the name at the top implies, there is a second one as we’ll see. And even a third one under development.

So here’s an example from ARC-AGI-2. Again, humans get these right virtually 100% of the time. In this example the rule is that blue-bordered squares move as far left as they can, and red-bordered squares move as far right as they can. Squares aren’t allowed to overlap so sometimes they stack. There are other differently themed puzzles but all of them still rely on visual reasoning and forming some hypothesis about the rules of the environment.

Here’s the leaderboard. The shape is similar but look at the y axis - the max score isn’t even at 30%. o3-preview spends hundreds of dollars per task just to fail on almost every one of them!

One model I do want to shout out here is Hierarchical Reasoning Model, and its successor Tiny Recursion Model - it’s the two orange triangles just past the $1 mark and hitting around 5%. These are small small models, like tens of millions of parameters rather than the tens or hundreds of billions of parameters on leading models. But they’re specifically designed for non-verbal reasoning and solving games like this. So they’re not the generalist workhorses that modern LLMs are, but they do show how much of a brute force approach LLMs are and how patchy their intelligence can be.

Anyway, Chollet left Google a year ago to found Ndea (pronounced “Endia”). Here’s how they describe program synthesis, their main research topic: “Instead of interpolating between data points in a continuous embedding space, program synthesis searches for discrete programs, or models, that perfectly explain observed data. This allows it to achieve much greater generalization power with extreme data-efficiency, requiring only a few examples to learn.”

So they want to combine the empirical, extrapolation-based intelligence of current LLMs with the principled, model-based intelligence of program synthesis.

The website claims all the big labs are working on this, though it’s not obvious there’s broad demand yet. There has however been a recent uptick on Google Trends for the term “world model”, and Meta recently produced a Code World Model. This isn’t a brand new line of research but it’s getting attention.

Speaking of Code World Model, let’s take a look at some of their data to understand how a coding model differs from a coding world model.

So normal code data for us would be a prompt and a response, if it’s SFT, or a prompt and a set of unit tests that verify whether the model’s answer works, if it’s RLVR. But in either case we’re training the model directly for writing code.

The coding world model has a different goal: to predict how a piece of code will work. If the model can do that, then it probably has a good world model for the world of code, its “laws of physics” so to speak.

The way they train that in is to provide a piece of code and an example of using that code, then ask the model to predict the state and action at each step of the program.

Here the state is in yellow and the action is in blue. As you go down the rows, or “frames” as the paper calls them, you can see what the model is keeping track of and how it changes after each action. Like for example in the second frame, we get a new variable n, with the starting value 0. Then in the third frame we see n being tracked in state, along with its current value, 0. The two dots in quotes is just a visual shorthand that means the value of the variable hasn’t changed, so like s is “strawberry” for the first frame and then just the two dots for the rest of the frames.

The paper we’re covering is a benchmark, so we won’t see any other training data, but hopefully this little snippet helps you compare against training data you’ve seen here at Scale that isn’t focused on world modeling.

Examples

The paper opens with four qualities a world model eval should have:

1. Interactive - the model should have the freedom or responsibility to gather information at its own direction

2. Behavior-based - the eval should only require looking at final predictions, not intermediate work or representations like a description of the AI’s world model

3. Goal-free - the model should not have an idea of what the test will be when it is forming its world model, so that it doesn’t reduce its model from the whole world to just the task-relevant stuff

4. Transferrable - the test environment should be related to, but different from, the learning environment; if the environment doesn’t change, you can’t test whether the AI developed a shared underlying world model versus just extrapolating from prior data in the learning environment

They basically take ARC-AGI and adapt it to fit their four qualities.

I’ll explain more in a minute, but it will make things much more concrete if I show you one of their examples first.

[Pick Masked Frame Prediction - P-7WWW9]

So here is one simple example. You have this little grid world, some shapes in it, and some basic inputs: the four arrow keys, plus clicking anywhere in the grid.

There are two stages to the task. In the first stage you just play around, get to know the environment. And you’ll see that when I provide input there’s a frame where the input happens, along with any resulting output, then frames continue to pass. So there’s a concept of time here, or at least sequence.

I also get to reset the environment if I want, so I can play around more with other choices. And then when I’m done playing I can take the test.

The type of test I picked here is Masked Frame Prediction, basically saying what a certain part of the world will look like at a certain time. So if I scroll through the frames I see eventually they apply a mask to part of it, and now I have to say what’s behind the mask.

Now let’s check how this matches up to our four qualities:

1. Interactive - yes, it’s interactive

2. Behavior-based - yes, all I had to do was pick the right option, I didn’t have to diagram or write out my understanding of the environment

3. Goal-free - yes, in the first part I was just playing around

4. Transferrable - yes, the rules I inferred transferred to the test environment, even though the test environment had the added factor of the mask

The big miss for ARC-AGI on that list is Interactive, although it also doesn’t have a goal-free setting and so it misses on 3 and 4 is kinda not applicable. Ultimately it all stems from not being interactive though, which apparently they are changing in ARC-AGI-3. So maybe this is a little preview of that.

Here’s a diagram of their eval, which they call WorldTest. Well technically WorldTest is a method or framework, and their particular instance is called AutumnBench, which I find a somewhat needless distinction. So I’m gonna just call it WorldTest.

Anyway, WorldTest has three different test tasks, one of which is the Masked Frame Prediction we saw in the example. The others are Planning, which sets a goal for the model to achieve, kind of like completing a maze; and Change Detection, where one rule of the environment changes and the model has to report the first time step where that rule change becomes visible.

The environments vary somewhat: size anywhere from 3x3 to 25x25, 1-12 different colors, 1-5 or so different object types. 43 different environments in total, each with the three task types, so 129 different tasks in total.

For human testing there’s a GUI, but for model testing they do only textual representations, not graphical.

So they tested human along with three different models: o3, Claude 4 Sonnet, and Gemini 2.5 Pro.

For the humans they recruited about 500 people from Prolific and had them pass an initial screener. Individual quality was pretty poor I suspect, but they compensated for it by taking the 80th percentile performance on each problem across 20 different human attempts. That 80th percentile was alway good if not perfect, so the contributor ceiling did not need to be that high. And that’s what we’d expect given the human results on ARC-AGI too.

As for the models, they did pretty poorly across the board. On the left is a breakdown by task type - CD for Change Detection, MFP for Masked Frame Prediction, PL for Planning - which shows relatively little variation. On the right is a plot of the distributions across all task types, where you can see the models sometimes doing well but mostly doing poorly; there’s a long tail of occasionally good or even great performance.

It is somewhat unfair that each model only got one attempt but the human score is based on an average, but I don’t think that explains most of the gap.

One interesting bit is the differences in human and model behavior. There are only four types of action you can take in these worlds: clicks, arrows (in one of four directions), resetting to the initial state, or nothing, also called “no-op” which is short for “no operation performed”.

There are two big differences here. One is how much more clicking the models did, which is weird but not that interesting. The other is the bias for action and pressing ahead, compared to the humans who reset or did nothing much much more.

That lack of epistemic humility, that overconfidence in knowledge is a big problem for modern AIs. It’s the same reason why models halllucinate, at least according to a recent OpenAI research paper; we train them to always provide an answer, to always know or at least think they know. If you ask a question the model doesn’t know the answer to, there’s a good chance it will answer anyway, because we don’t train on question where the right answer is “I don’t know” or “it might be this but it might be that”.

I think the same thing is happening here. That bias for action, that reluctance to explore more or start over again and implicitly admit you don’t know, that’s the same overconfidence showing. And I do think it’s down to how we do training.

The authors note a couple other model behaviors they saw in the chains of thought.

One is hypothesis formation and testing, but with somewhat of a blind eye, as some informative actions or results go unused.

The other is hesitancy to make updates. On the previous slide we saw Change Detection had the lowest score by task type; the models sometimes detected the change but stuck to the earlier version of the rule.

Again, all part of the same root issue in my view.

My Takeaways

• Modern LLMs don’t have world models

• Coding and tool use are prerequisites for LLM-guided program synthesis

  • As are more diffuse qualities like reasoning and creativity

• World-model testing environments are an interesting artifact, though I’m not convinced you need a human touch to create them

• Training data for LLM-guided program synthesis could be on the horizon right now for some research groups

Paper

Background

This is a speech model right? So we gotta talk about audio. But to do that, I first want to talk about text, which we’re all more familiar with when it comes to machine learning.

Because it turns out that text is a bit of an outlier in the data universe. Unlike audio or image, or even proprioception and action in the world of robotics, text is a discrete abstraction.

“Discrete abstraction”, what do I mean by that. Well let’s break it down. Text is discrete because everything is individual words or characters. There are clear divisions between minimal units. No judgment involved to split up a sentence into words or words into letters.

That is not the case for audio and image data. And I’m going to talk about both, because image data is more intuitive than audio data but has a lot of the same structure. Anyway, when you divide up some audio or an image, you have to choose where to slice within that data. So like a single image for example, if you break it up into pixels you have to pick a pixel size, and some pixels are going to fall right on the border of two things in the image. But a pixel can only be one color, so you end up with these artificial breaks or discontinuities.

Same deal with audio. Within a sample of speech let’s say, what you’re getting is just a waveform over time. Yes, we know that usually the speech is a series of discrete words, but that is our understanding on top of what is objectively going on. And if you want to divide up that waveform into words or sounds - “phonemes” as the linguists call them - you have to pick out the border between two phonemes and split. And that border requires some judgment, because sound is continuous.

And the reason sound and images are continuous is that they are physical, they are part of the real world. Reality is almost always continuous at the macro level. Like it’s hard for the stripes on a zebra to have sharp edges, it’s hard for a hot thing to be near a cold thing with no warm zone intermediate, it’s hard for a sound to instantly start or stop. Things might seem discontinuous to us, but reality has a surprising amount of detail, and if you look closely there is usually some bleed or a curve rather than a jump.

Abstractions don’t have to be like this. In fact it’s much easier when they aren’t. That way you can mix and match them cleanly, like Lego castles rather than sandcastles. In English, we get letters we can arrange into phonemes and words that represent sound. We lose a lot of detail about that sound, but abstractions are often lossy like that. It’s the price you pay for the nice clean features.

Computers deal in discrete abstractions. At base they are electrical circuits in the physical world, yes, but we have worked to introduce abstractions on top of those circuits as early as possible, so that we can understand and program them.

So text is a natural fit for computers with their 1s and 0s. And if you know about the modern transformer-style LLM, you know it spits out text based on a list of all possible tokens and the likelihood of each one of them being next. That doesn’t work if the output has to be continuous.

Now there is ML machinery to generate continuous-seeming outputs. For example, diffusion models - which draw inspiration from the physical process of diffusion - are common for generating images. But most of the focus is on transformers these days, and transformers generally output probability distributions over discrete options.

There are ways to get transformers working with continuous data though. If it’s just input, you actually can avoid the jump to discrete if you bypass the tokenizer and go straight to the embedding space, where meaning really is continuous over the values of the vector in the embedding space.

Another option is to make the jump, to take the continuous data and tokenize it anyway, even though it doesn’t inherently have the nice clean boundaries that text does. We don’t need to get into audio tokenization for this paper, but basically you can boil down a bit of audio into what sound it represents and how to construct that sound. The graphic here gives an idea of how involved speech tokenization is.

Besides faithfully tokenizing audio, another challenge for model makers is two-way communication and fluidity, as opposed to the turn-based flow of a text chat.

The technical term here is “full duplex”. A full-duplex model allows simultaneous communication from both sides, like a phone or a Zoom. By contrast, a half-duplex model requires one side to wait for the other before replying, like a walkie-talkie.

If you have used a voice-enabled model, it was very likely half duplex. Certainly if you took my ML diagnostic back in April, you experienced half duplex, specifically with the gpt-realtime model family. Half duplex is unnatural for people and results in some clunky user experiences. It also requires the model to sense when you start talking and immediately shut itself up, a technique called “voice activity detection” (VAD). VAD is notoriously finicky and frequently leads to either the model suddenly going silent (VAD too sensitive) or completely ignoring your interjection (VAD not sensitive enough).

Full-duplex models, on the other hand, can gracefully handle interjection and overlapping speech because they can understand what the user sounds mean for the conversation. Like if the sound is just a dog barking in the background, or the user throwing in an “uh huh”, a full-duplex model can hear and understand it while it continues talking. The result is a much more natural conversation. However, full duplex is harder and thus relatively rare. It was a big part of the pitch for the Interaction model from Thinking Machines, which had these impressive demos where a user conversed with the model while the model continued working on its code or its spreadsheet or whatever.

The graphic here is from Thinking Machines by the way. It shows the model speaking even when there is audio and video input at the same time. You can see many more demos and possible use cases on the launch page.

Here is a pioneering full–duplex model from 2024 called Moshi, which the authors of today’s paper will take as the base for SteerDuplex.

Now we aren’t going to cover everything here, but a couple details different from the standard LLM will be crucial to understand.

One is the three-track thing going on at the bottom, with the user’s audio input in purple and two model streams in orange. The first orange one is just Moshi’s audio output, what it is saying back to the user - fine. The other one they call the “inner monologue”, just the text that Moshi is about to speak. So if Moshi’s inner monologue puts out a word in step s there, then in step s+1 it will start speaking the word, continuing to produce audio output tokens in these 80 ms-long steps until it has finished the current word and is ready to speak the next word. And for reference, people generally speak 2-3 words per second. That’s 4-6ish steps per word, meaning most inner monologue steps will be empty, waiting for the sound of the current word to finish so it can kick off a new word.

One interesting detail about how the model produces these sounds is at the top: the distinction between semantic and acoustic output tokens. It’s the difference between what sound the model wants to make, like what phoneme, and how it’s going to make it. They mention in the Moshi paper that for every semantic token the model produces, it also produces seven acoustic tokens, meaning it takes a lot more information to construct the sound than to indicate what sound Moshi should make. Again, you have a tidy little abstraction versus a big complicated physical phenomenon.

Also, speech is information-dense, much more so than I think many English speakers realize. Tonal language speakers may have more of an appreciation, but still, speech is so natural to us all that we take for granted just how much it can convey.

Tech companies are well aware though, and as speech understanding improves it’s opening a lot of doors for product plays.

A popular one nowadays is voice interaction with your computer, like Wispr Flow. But that’s more speech to text, also known as automatic speech recognition (ASR), with AI making it significantly more accurate.

Another popular category is wearables, which can’t work at all without some alternative to the keyboard. At Meta Connect recently, Zuck debuted audio-only smart glasses, in addition to a new version of the existing video+audio smart glasses. He also led the Muse section of the event by announcing Muse would soon support voice, and he closed his related Twitter thread with a shot of this device here, Muse Charm. If you think about it, it’s really no surprise that the socials company now named after the metaverse would go hard on speech tech; voice is a highly interpersonal medium, and typing just doesn’t jibe with the controllers or haptics of VR.

So for the version of AI that Iron Man gets with JARVIS, we really need to master speech tech.

The Paper

So for SteerDuplex, the goal is to train and measure a model for steerability, which the authors define as “the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions.” Contrast that with instruction following, which is more about conversational outcomes.

To start off, the authors helpfully orient us on the terms that fall under “steerability”, as well as under “duplex”. We will see this capabilities taxonomy crop up again, so it’s worth spending some time here.

Broadly we have three categories: text, audio, and duplex. Text and audio are about adapting to information in the relevant form, although remember that the actual input is always speech. Here we’re distinguishing between information that is most naturally text and information that is most naturally audio. So an instruction to be more sarcastic is easy to convey in text, but a sigh is a better fit for audio. I’m a little unclear on why the text inputs for Voice/Style Editing and Audio-Context Memory are in the Audio column frankly.

More clearly, anything specific to full duplex is over on the right: taking turns in a natural and timely fashion, handling interruptions from the user, adding affirmations while the user is speaking to show the agent is still there, etc.

An overlapping set of authors published an earlier paper, Audio MultiChallenge, that has overlapping skills in it. We’ll see results on AudioMC later, but keep these taxonomy entries in mind: Audio-Context Memory, Self-Coherence Over Audio, Mid-Utterance Corrections, and Dynamic Mid-Conv. Steering. I have marked them with stars for easy reference.

So that’s the benchmark side. Now here’s the model side, which will of course run against the paper’s benchmark and use some excellent Scale data.

The starting point for the SteerDuplex model is Moshi, the full-duplex model we covered in the Background slides. You can see in part A the parallel tracks for the user and the agent, and the split between semantic and acoustic tokens. They also show the agent’s text output, which Moshi called the “inner monologue”.

I’m going to skip over part B, which SteerDuplex also inherits from Moshi, so that we can focus on the data and training.

Parts C and D show how training happened: one stage of SFT, then two stages of RL, which incorporated different rewards.

For SFT, it’s about 8,500 hours of audio across more than 500k examples, along with a bit of pure text data. We don’t get a full accounting, but the audio data is a mix between natural conversations and examples specifically targeting instruction following, steering, duplex interaction, safety, and reasoning. They also note the natural variation in prosody - a fancy linguistic term for the rhythm, tone, stress and so on in speech.

As for sourcing, some of it is licensed from outside, but a lot of it is Scale data. There is some synthetic speech in here too.

Now let’s talk about RL. Sadly, there is no information about how much data they used. However, we do get some details about what they’re testing and looking for in the model’s responses:

• Verifiable timing: objective checks for appropriate pause timing (wait instead of barging in), turn timing (respond when the user finishes), backchannel timing, interruption yield and recovery (stop when interrupted, then respond sensibly), noise robustness (don't react to non-user sound), and speech-mirror .

• Transcript rubric: judging whether the text of the response fully addresses the user prompt. No audio involved.

• Response continuity: this is basically an anti-reward hacking term, to balance out the model getting points on verifiable timing with short throwaway responses like “Sure” or “Okay”. They put the minimum duration for a substantive response at four seconds.

• Continuation: keeps the model responding when the user backchannels, e.g. saying “Uh huh” in the middle of a model response, or when there is unrelated noise in the user’s background. Continuation comes in the second RL stage only, with significant weighting, because after the first RL stage the main failure mode was the model yielding the floor too readily.

They also include a few checks on the waveform itself to filter out clips of pure silence or poor audio.

So now that we have our model, we want to measure it on what we trained it for. That’s the impetus for SteerBench, a novel benchmark testing for steerability in full-duplex models.

SteerBench comprises 390 tasks across tone, persona, style/accent, and speed/length. Each audio prompt uses one of a small set of neutral synthetic voices for uniformity. The task comes with a human-authored rubric, with some criteria for the audio and some criteria for the transcript. There are just over 1k criteria, so the average rubric has 2-3 criteria, which is much lower than I would have expected, likely due to the relative simplicity of the tasks compared to something like GDPval.

For the audio criteria, it’s actually kind of a mix between rubrics and side-by-side evals; there is a text description, like “uses sarcasm”, but there is also a reference audio clip exhibiting the criterion in question. It was an empirical decision to include the reference clips; judge models agreed with human ratings almost 15% more often when they had a reference clip accompanying the relevant audio criterion.

As for the judging, they have three separate metrics:

• Audio Pass Rate (APR): the share of tasks where the model satisfies every audio criterion

• Sample APR: the share of tasks where the model satisfies every audio and text criterion

• Pass Rate: the share of criteria the model satisfied across all rubrics

Here’s where we start to see them in action. On the left we compare two full-duplex models to SteerDuplex after SFT but before RL. As the name suggests, SteerDuplex is more steerable, and the gains come from SFT. (We’ll see the benefits of RL later.)

On the right we have the same model on the other two metrics, although they call Sample APR “All constraints” instead. Not only is speed/length the hardest to get right individually, it’s also significantly harder to get entirely. There also seems to be plenty of room to improve on the benchmark, which as a benchmark author must be a relief to see.

Still on SFT results, we get a look at improvement on a few other benchmarks.

For AudioMC, first let’s recap what it measures:

• Inference Memory (IM): the ability to use information from earlier in the conversation to tailor the current response

• Instruction Retention (IR): whether or not the model keeps following standing orders

• Self Coherence (SC): sticking to earlier claims from the model itself, rather than flip-flopping due to user input

• Voice Editing (VE): keeping user context and instructions straight after the user makes changes

For smaller models in particular, which most audio models are due to latency concerns, all four present challenges. But it seems our audio data really helps. These are hardly SOTA results, but the point is relative improvement over the Moshi baseline.

For Full-Duplex-Bench (FDB) v2, the setup is GPT Realtime as the examiner and the model in question as the evaluatee. There are a few different categories of scenario, all the scenarios are multiturn, and a judge grades the evaluatee on a scale from 1-5. Here, SteerDuplex is much closer to saturating the benchmark, especially on Safety.

Finally, VoiceBench is a single-turn benchmark that basically turns other existing benchmarks into voice versions. SteerDuplex mostly doesn’t improve here, with one outlier significantly impacting the overall score, so I won’t dwell on it.

By the way, they report later on how the model changed on the above metrics after RL. Basically, it didn’t, which really drives home how focused the RL was on timing specifically.

Now we can move on to RL results, which reflect behavior more than content.

On the left we see how often SteerDuplex barges in during a pause from the other party, like if you say “um” and think for a bit, intending to continue talking soon. The final version of SteerDuplex, after both stages of RL, seems to do much better at judging when to take the floor or not. Actually there’s some evidence from FDB v2 that the model errs the other way now, yielding the floor a little too readily.

On the right we have sort of the opposite case, where the user starts to overlap with the model. In the case of a genuine interruption, the model should yield, while in the other three cases the model should continue on. They graphed success rate, so higher is always better, i.e. a higher score on Interruption does not mean the model interrupted more.

Now we get to talk about the fun bit of RL: reward hacking.

Earlier I described the rewards they used in RL, each measuring different elements of performance. You might have wondered, how did they come to that exact formulation?

Well let me tell you, just looking at their list I could tell it was an empirical process, not a spec they designed once and then implemented. A lot of what gets rewarded and by how much comes from looking at model behavior at every stage of the training process and trying to compensate.

Sometimes you’re trying to compensate for new bad behavior or weaknesses you didn’t notice before, just adding to your reward picture. But other times you are reacting to behavior you yourself induced with your earlier rewards. In other words, you are responding to reward hacking.

The chart here is a good example. For some of the rewards in this paper, silence actually earned the model just as much as an appropriate response. For example, a reward for stopping a response when the user interjects is easy to reap if the model was already silent.

What the chart shows is how bad the “empty speech” problem gets when you isolate parts of the reward. Like if you look at the transcript only for example, half your outputs are empty. The full set of rewards mostly addresses the problem but doesn’t solve it.

Of course with more rewards there often comes a tradeoff. Here for example, the authors saw noise robustness decline while behaviors around interruption improved. You can adjust weights of the various rewards to reflect the balance you want to strike, but that introduces yet another thing to optimize and can get expensive.

My Takeaways

• Audio still has a long way to go

  • Speech is more specifically human than writing

  • Full-duplex models can’t be that big due to latency. Clever engineering required to make them smart enough to talk to like a peer

• Audio is a different world of research

  • Image input is common now - photos, screenshots - and required for a lot of important benchmarks

  • Audio is more about products than RSI

• Concurrent interaction will require product changes

  • Text is easy because it is single-stream, you can’t really write or read at the same time as you do anything else

  • But it’s common to talk while doing something else, and we will expect that of agents

• You’ll know audio has made it when there is a voice equivalent of AI slop

Paper · Model · Repo

Background

To understand this paper at all we have to understand the main bit of jargon in the title: Monte Carlo Tree Search, or MCTS. And to do that we will break the term down into its constituent parts: Monte Carlo, and Tree Search.

So, Monte Carlo method, what is it? Well as the name implies, Monte Carlo originally being the name of a casino in Monaco, it’s all about chance. Specifically, how you can use random tries over and over again to get a pretty good empirical idea of something that you’re having trouble calculating theoretically.

Let’s say you’re an ancient Egyptian and you want to know how to calculate the area of a circle. You’ve worked out that it’s proportional to the radius squared, but also some other number, some constant you can’t pin down or derive purely mathematically. What do you do?

Well you can try getting a pretty close estimate by drawing this shape: a square with side length 1, and a quarter of a circle with radius 1 inside. The area of the square is 1 - that’s easy - and the area of the circle is pi over 4, since the full circle is pi-r-squared and r is 1, which means the full circle’s area is just pi, so the quarter circle’s area is pi over 4.

And that means the ratio of areas is pi over 4, over 1, so just pi over 4. So if you can empirically find that ratio of the inscribed area to the whole area, the quarter circle part to the whole square, then just multiply by 4 and you’ll have pi.

The way you empirically find that ratio is shown here. If I’m that ancient Egyptian and I’ve drawn this pattern on some papyrus or whatever, then I can get a little pebble or something and drop it from some height and mark where it lands, over and over and over. Then I just take the number of marks in the circle and divide it by the total number of marks. The more random marks I make, the better my estimate of pi.

Now this is a toy example of course because we have far more precise ways of calculating pi. But in real life there are many cases where the theoretical calculation is impractical, impossible, or not even known yet. So Monte Carlo is still a popular method.

So that’s the Monte Carlo part. Now we need the Tree Search.

To explain this I’m going to use another toy example, this time Tic Tac Toe. What we see here is the different possible states of the board, starting from blank and progressively covering every possible state until someone wins or there’s a draw, although this one only shows three levels deep. Also if you’re wondering why some versions are missing, it’s because we group cases that are the same if you flip or rotate them. So like the last board in the second row, with the X in the top-left corner, that covers any board with just X in one corner.

This structure is a tree, specifically a game tree. The search part isn’t really relevant since this is such a simple example and we can analyze every possible game state pretty easily.

But imagine a much bigger game like chess or Go, where drawing a game tree is theoretically possible but practically impossible. Or something even crazier like language or math, where again you could in theory map out every single choice of next word or character to produce, but practically it would be impossible.

What you need is some heuristic, some way to decide which nodes in the next layer are actually worth diagramming out so to speak, which next steps are most promising.

Enter Monte Carlo Tree Search: a method for expanding that tree in the most promising directions. This diagram explains it using a game between a white side and a black side, like chess, where the white circles are moves the white pieces can make and the black circles are moves the black pieces can make.

There are four steps:

1. Selection: start from the beginning, make reasonable choices along the nodes you’ve already mapped, until you reach the bottom of the existing tree - not the end of the game or the response or whatever you’re searching, otherwise there’d be nothing left to do with that node. In this example they’re always picking the lowest win ratio for white and the highest win ratio for black, which is the simplest method but isn’t optimal for finding the absolute best path - too much exploitation and not enough exploration, in the ML lingo. That’s why I said “reasonable choices” earlier.

2. Expansion: again, assuming this is not a terminal node, map out the next moves you can make. In this example there is apparently only one, which is plausible for a simple game, but for a complex game or in other domains, there will always be many possible choices. Anyway, the win ratio here is currently 0/0 because we haven’t done anything forward-looking with this new node yet.

3. Simulation: this is the tricky part. You have to have some way of estimating the value of picking this option. The standard, simplest way to do that is to keep playing from that point forward with random choices, until you reach an end state. Then you report back what happened. In this case, apparently black won and white lost in this random simulation, so since this is a white node we report back a loss and update the win ratio to 0/1. Now this probably seems silly - surely both players making random moves is not actually how any game would play out - but that’s the insight of the Monte Carlo method! If we do this loop over and over again, the randomness of any one simulation won’t matter, it’s the collection of all the random simulations that paints an increasingly accurate picture.

4. Backpropagation: now that you have more information about this option you’ve just created, you also need to let all the other, earlier options know what lies in their future. Note that the white circles have added a loss, but the black circles have added a win - the 1/6 for white went to 1/7, the 3/3 for black went to 4/4, etc. Updates all the way up the tree, that’s backpropagation.

There are many complications and variations on MCTS, but now you understand the basic idea of looking at the most promising path, picking a next step, doing a random trial for all its future steps, and updating the path based on that trial.

Now what does this all have to do with AI? Well first, MCTS is behind a lot of the AIs used to master increasingly complex games. For example AlphaGo, which handily beat the world champion of Go in 2016.

But second, MCTS was one of the techniques rumored to power o1, the first reasoning model and a major breakthrough in LLMs from OpenAI.

If you were following the rumors at the time, you may remember the name Q*. Right around the time the OpenAI board briefly fired Sam Altman, some OpenAI researchers had sent a letter to the board warning about an internal project called Q* that was unlocking significant milestones in math and reasoning. Ilya Sutskever - a cofounder, board member, and chief scientist at OpenAI - was leading the Q* project. People started wondering if Ilya believed OpenAI had dangerously advanced AI on its hands but that Sam was disregarding safety in the name of progress.

As we know now in November 2025, based on depositions released last month, Sam’s firing and Q* were unrelated. But the drama of Q* got people speculating about what it was under the hood. The name seemingly came from Q-learning, one of the many RL algorithms, and the A* search algorithm. That combination of planning and search is just like MCTS.

Anyway, later in summer 2024 we started hearing about Project Strawberry, again a supposedly super advanced model from OpenAI. This time Sam Altman played into the hype, posting photos of strawberries and engaging with a mysterious Twitter user with strawberry emojis in their username saying “welcome to level two”, a reference to a reasoning-related milestone on OpenAI’s path to AGI.

Again, people speculated that this new model used a planning + search method like MCTS to iteratively find the correct reasoning path. But now with o1 and its successors released, when looking at the chain of thought it doesn’t seem like they’re doing something so systematic. It’s more like thinking longer, maybe checking assumptions, much more organic stuff than building a search tree methodically.

This chart lays out the main advance of o1: using a lot more tokens to “think” before giving a final answer. And as the official blog post from OpenAI showed, the longer you got the model to think - the more inference you did - the better your performance got. Nowadays people call this inference-time compute scaling or test-time compute scaling.

So o1 came out in September 2024, but for the rest of the year nobody else released anything like it, and we of course got no publication or confirmation from OpenAI on the techniques they used to make this so-called “reasoning model”.

Then came the DeepSeek moment: the release of R1 in January 2025.

The Chinese lab DeepSeek, which had been quietly building increasingly capable models, dropped the weights and an accompanying research paper for their own reasoning model, R1. In the paper, they revealed reasoning did not require fancy planning + search algorithms; instead, it was plain ol’ RL, specifically RLVR, rewarding the model whenever it got math problems right. No principled method for planning and searching, just trial and error over and over and the magic of RL.

Now the DeepSeek team also released their particular RL algorithm, GRPO, which is a simplified version of PPO, and that definitely helped simplify and stabilize training. But the key insight was RLVR.

This chart is from that R1 paper, showing how the response length - the amount of reasoning, the amount of inference-time compute - grows and grows the more you train the model on their setup.

This big reveal from DeepSeek really took the air out of MCTS research for LLMs. We got a flood of o1 imitators after R1 broke things open, and all attention turned to RLVR. At least until now.

The Paper

Okay, this is going to take some serious explaining even with that background.

The first bit is of course the MCTS. They do a loop similar to the one we saw before, but there’s no simulation step where they take one node and make something like random choices until they hit a terminal state. Instead, if all the nodes in the expansion step are intermediate rather than terminal, they go back to the selection step.

So when you finally do get a terminal node, you send that information back up the tree like in the backpropagation from normal MCTS. A node has a higher score the more correct paths it’s involved in and the closer it is to the correct node. Of course some nodes can be on the path to both good and bad outcomes, but if it’s on the path to a good outcome then they guarantee the score will be at least a little positive.

Now I did want to say one thing about the selection step. They have this “frontier priority score” thing that looks mysterious on here but which they explain in the text. Basically they’re picking nodes to expand with three aspects in mind:

1. The quality potential, which looks at the quality score of parent nodes. That’s Q_parent

2. The uncertainty, as measured by the entropy, which rewards nodes where the model is less certain and therefore more exploratory. That’s H, the traditional abbreviation for entropy

3. The depth, as in how many nodes down it is, which tends to mean you’re closer to a terminal node. That’s D

So the heuristic here is: the earlier steps in the reasoning chain should be good, and the next step we choose should be somewhat uncertain so that we can look at a few different paths and learn which one is the best. But we also don’t want to just keep making intermediate steps forever and never get to a solution.

The next bit is this “replay buffer”, which collects the correct reasoning paths the model has found with MCTS.

The only tricky bit is the middle step with the red robot. The notation is dense, but all it’s saying is we only want to keep training on solutions the model is unlikely to get on its own. If you keep training on problems you already get right then at best it’s wasted effort but at worst you start to memorize and become brittle.

Then the last bit is how they train.

Basically they’re taking GRPO, which is an algorithm for turning rewards into model updates, and they’re adding rewards on the node level. Like in regular GRPO you just give one reward to the whole response, which is admittedly pretty imprecise; there might be a lot of text in the response that isn’t actually helpful. So to take advantage of this MCTS tree where we know which nodes were more or less helpful, this Tree-GRPO algorithm uses the q score at each node in the correct reasoning path as a reward and updates the model accordingly.

The bit at the bottom with the red and green tree drawings is about filtering your initial training set. We only need to do this whole MCTS process with problems where the model hasn’t gotten a correct reasoning path yet. For cases where the model is still bad at the problem, but has gotten it right at least once, you revert back to the normal RLVR thing of just passing in the problem and checking the final answer, what they call a “direct rollout”. And as we mentioned in the last slide, if a problem is too easy then we stop training on it altogether.

Now if you’ve been at Scale for a bit or have the relevant background, you might be thinking, isn’t this step-level reward thing pretty similar to process supervision? And even though the authors don’t make this comparison, I think you’d be right to.

For those unfamiliar, process supervision is when you have a human label each reasoning step as correct or incorrect. You then use that data to train a process reward model, a PRM, which you then use like any other reward model in your RL training loop.

So the two key differences are: one, process supervision directly assigns credit rather than calculating it based on correct paths and number of visits to a node; and two, process supervision is for training a PRM rather than directly training the generator. But the general idea of assigning more specific credit holds in both cases.

So this is where we net out. The starting point for DeepSearch was that Nemotron v2 one row above it in the table, and as you can see it’s not a big jump. Part of that is because there’s really not much room to move after a benchmark already hits 90%+ as with AMC23 and MATH, but even without those you’re looking at 0-3 points improvement. Real enough but not amazing.

And here I want to provide some other background and context, which I typically provide up front but I didn’t want to tip my hand here.

That bit of background is The Bitter Lesson.

In March of 2019 a famous computer scientist wrote a famous essay. That man was Rich Sutton, the father of reinforcement learning. That essay was The Bitter Lesson, describing the one constant he observed in his 35 years of work on AI.

I’m going to read the first paragraph verbatim:

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other. There are psychological commitments to investment in one approach or the other. And the human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation. There were many examples of AI researchers' belated learning of this bitter lesson, and it is instructive to review some of the most prominent.”

He mentions chess, Go, speech recognition, and computer vision as fields where attempting to program in human knowledge failed, while throwing more computation at searching and learning succeeded. Chess and Go are examples of search scaling up, with AI looking many moves ahead into many possible futures at the same time. Speech recognition and computer vision are examples of learning scaling up, with bigger neural networks, bigger datasets, and more compute.

He couldn’t have known this in 2019, but we can now add natural language to that list. Getting AIs to talk has a long history, and until relatively recently it involved programming our rules of grammar into the model. But LLMs don’t work like that. As the compute became available, we spent more on learning and gave up the human-derived rules, and now we have models that can write better than most people can.

So the question for our paper is, is Monte Carlo Tree Search bitter? Or are we programming in human methods for short-term gain, forgetting the opportunity cost of neglecting more bitter methods?

Certainly I think RLVR is more bitter. If all we do is reward models for getting right answers, we’re giving them complete freedom to be their own sorts of minds. If we enforce the paradigm of step-by-step thinking like in MCTS, even if MCTS is a method of searching and did work for mastering Go, I think we lose something.

In fact the DeepSeek R1 paper demonstrated it too. They trained two versions of their model: R1, which used a little SFT to teach a reasoning format and then RLVR to improve reasoning; and R1-Zero, which skipped the SFT and just did RLVR. R1-Zero ultimately did better BUT had some usability issues, like sometimes switching between English and Chinese. So we forced human strictures onto a machine mind and hurt performance as a result.

I’m not disputing the results of this paper - I believe MCTS did help performance - but I am skeptical that this amount of structure and method is what really breaks LLMs through to the next level.

Now by sheer coincidence ANOTHER small reasoning finetune of Qwen2.5-1.5B came out while I was making this presentation. VibeThinker drops some crazy benchmark scores, rivaling fairly recent models 10-100x its size. Just for context on this model size, you could run a quantized version of this on your phone today. In fact, the newest Pixel phones ship with a 1.8B version of Gemini. 1.5B is also the size of GPT-2, which like The Bitter Lesson came out in 2019, although The Bitter Lesson came first.

Anyway we’re not going to review that paper, but I will note there are no specialized structures in their method; they do SFT then RL, with some tweaks to which SFT checkpoints they RL on and what problems to train on in RL, both of which are bitter in my view.

My Takeaways

• MCTS seems great for games with discrete states, but I don’t buy it for languages (natural, math, code)

• Maybe we will see MCTS for agent domains with discrete states

  • Computer use?

• I do buy entropy as a guide for what you train on

• I feel like we are due for another breakthrough

  • Incremental gains are still happening on track though (see Gemini 3) - “there is no wall”

Paper · Blog · Weights

Background

Ohh DeepSeek. DeepSeek DeepSeek DeepSeek.

Of course I was always going to cover this release. We’ve covered so many of your works before:

• DeepSeek V4, the last major release from DeepSeek

• mHC, a super technical improvement to a very specific part of the model architecture

• Engram, a sort of built-in memory for common phrases

• DSpark, a way to guess at upcoming tokens and thus speed up the generation process

You always have such wonderful tricks and such tough engineering. A researcher’s paradise! You charge next to nothing for your products and seem to care not a whit about commercial appeal.

Well you have, anyway. But things have changed for you in 2026, haven’t they? You sought your first outside funding. You were late for your V4 release - rumored to be before Chinese New Year but in fact released over two months afterward - and even then you called your release a “preview”. You arguably lost a technical bake-off against Moonshot AI, the makers of Kimi, with mHC vs Attention Residuals (linked above).

I don’t count you out. Nobody should. But it has been a rough year for you.

The Artificial Analysis Intelligence Index results here tell the story.

DeepSeek V4, both Flash and Pro, were underpowered. The vibe at the time was that both models had not gotten enough post-training, which sands down rough edges and really develops the finer capabilities. They are far behind the models of today, but even contemporary models had them beat. For example, Meta’s first Muse Spark model came out around the same time and slightly edged out DSV4P. For a new-ish lab playing catch-up to beat the most respected lab in China, with years of continuity and innovation, is both impressive for Meta and shameful for DeepSeek.

By this same measure, Qwen3.8 27B - which fits on a single prosumer GPU and is not on this chart - is clearly superior. 27B, compared to 10x that for Flash and over 50x that for Pro! Now that’s a bit unfair because Qwen3.8 came out four months later and is dense rather than MoE, but it is still shocking. Even if you look instead at the non-preview version of DSV4, the ones that came out in late July and early August when Qwen3.8 did, the capabilities of Qwen and DeepSeek are still on par.

4.1 Flash is more where we expect DeepSeek to be though, approaching the latest Gemini Flash release at least. Based on past trends we can expect a few points of improvement from Pro, which would place it nicely with its real competition like Kimi K3 and GLM-5.3 and Qwen3.8 Max - the other flagship releases from the other important Chinese labs. But they are at minimum late, and I feel like a disappointing release is a real possibility.

A couple other observations to situate DeepSeek further. One, their last flagship release is over six months behind the frontier as measured by ECI. Six months is the typical lag between the best American model and the best Chinese model, so they are once again behind the relative curve.

Now we don’t have an ECI score for DSV4.1F, but if you assume it’s one point ahead of DSV4P on here, then again, the leading American lab got there over six months ago - GPT5.3, back in February.

The other thing to point out is specifically how it does against Kimi K3, which I covered last week. As DeepSeek’s fortunes have fallen recently, Moonshot AI’s have risen, in sort of a mirror image. I encourage you to read that post after this one and judge for yourself. Anyway, Kimi K3 scored 158 on here, on par with GPT-5.4 Pro, and ahead of the six-month heuristic.

So DeepSeek has been in a bit of a hole in 2026. But DSV4.1F may be the first step out of it.

Speaking of trends, we have all witnessed the shift from chatbots to agents. And that trend has encompassed a lot of different things: the improvements in model intelligence and instruction following and tool calling and so on that allowed agents to work at all; the increase in context window size that made long trajectories feasible; the birth and development of harnesses for models to inhabit and thus become agents. Now it’s to the point where the primary use of a frontier model is as an agent, not really as a chatbot.

Which raises the question: what does an agents-first model look like?

Well for one, it should be able to handle a lot of input. With a chat-oriented model, typically the input will be small and the output will be large. Like if you’re a student asking questions about calculus, or you’re a grifter getting Claude to produce slop essays, you are providing a little context and letting the model spin its yarns.

By contrast, an agents-first model expects to receive tons of information. Not from the user - that won’t change much from chat - but from the environment. Any time your agent is reading the codebase or searching the web or analyzing documents, it is processing input and not yet producing tokens.

In the LLM lingo, we call that prefill and decode, respectively. Prefill happens in parallel, since we have all the input tokens available to us at once, but decode has to happen in serial, since we need to know the previous token before predicting the next one. Prefill and decode have different bottlenecks, on both the software and hardware side. Keep the software side in mind for when we get to the technical innovations.

The other important difference between chat and agent tasks is the density of information.

Again, consider a typical chatbot interaction. The user provides relatively little input on every turn, but the information that is there is almost always relevant. Prior model responses are also likely shorter and more consistently relevant. Humans can have long conversations but often many parts are germane.

Now consider a typical agent interaction. Let’s think about a research task, say, where the agent has to do a lot of web searches and trawl through documents to get the relevant information. The user input might be just as valuable, but a ton of the prior agent and environment output will not be. Like when you do research, you do a ton of Google searches of varying effect and may see tons of web pages or documents that don’t end up mattering at all. You only need a very fuzzy idea of what was in any of those traces. And if you did find something during any of them, there’s a good chance you wrote it down somewhere to collect or underlined it or somehow marked it out.

So when looking back at all the prior context, the agent does not need a thorough, rich understanding of all of it. What it needs is a clear view of a few key areas and a general notion of everything else.

So we have wildly varying levels of attention necessary, and we need to process inputs as quickly and cheaply as possible. That’s what our agents-first model needs to work like.

Now we already have existing DeepSeek work addressing both concerns. For attention, DSV4 introduced something called Compressed Sparse Attention (CSA). We are going to see a new version of that called CSA2 in this paper, so I am going to defer on CSA for now, but you can read about it in my DSV4 coverage.

As for processing inputs quickly and cheaply, the existing work is Engram, which I described earlier as a built-in memory for common phrases. Again, because we did a whole paper review on just Engram, I will refer curious readers to that installment.

There is a completely new aspect for DeepSeek related to processing inputs though, what they call Causal Encoder-Decoder (CED). And for that we need some new technical background. Specifically, we need to discuss the KV cache.

When you are using attention, you are taking your input X and making three different versions of it: the query (Q), the key (K), and the value (V). In the context of language, the query is like the subject of a sentence, the key is like the object of a sentence, and the value is the meaning of that subject-object relationship.

To understand a token as a query or as a key or as a value, you multiply it by the relevant weights (W), which the model supplies.

Now matrix multiplication, or matmul as the kids call it, is expensive. Really every operation is expensive at the scale of a SOTA model, but matmul is so core to the LLM and so frequent that if you can reuse previous matmuls, you should.

Now the standard way to save this compute comes from how prefill vs decode works. In prefill, all your input is new, so you have all this new matmul to do. Fine, you know it has to happen at least once for each token.

In decode though, you only have one new token to deal with at a time - the one you just produced. So there’s only one token that needs to be your “subject”, your query.

However, you still have to understand how this new token relates to every other past token. That means you need all the key and value versions of all the prior tokens. And you don’t want to recompute all that.

So when you’re computing the key and value of each token, you store the results. And those stored results are the KV cache.

Now a couple details to add to our picture. One, each attention layer has its own weights, and so it has its own KV cache. People talk about “the KV cache” as one giant pool, but really it’s more helpful to think of many KV caches, each associated with a particular attention layer.

Two, the KV cache is alterable, like any other part of the model. You can’t train the KV cache - it is ephemeral and only exists for a specific input, like your current state of mind - but you can learn how to do new stuff with it. It’s not just a common way to save compute.

By the way, you may have clocked that the KV cache grows along with your input. More specifically, it grows linearly with your input, in this baseline implementation anyway. Knocking that growth rate down is something all ML engineers shoot for, not just in this age of agents, and we’ll see a bit later how DeepSeek has managed to keep the KV cache relatively tiny.

The Paper

So this is DSV4.1F. It is kind of a lot to look at, with parts we need to explore in depth before the whole picture makes sense.

Now broadly speaking, we still have the foundations of any modern model: embeddings to turn input into vectors, attention to form an understanding of that input, and feed-forward networks to process that understanding. Here, the variously colored boxes are versions of attention, and MoE is by now the standard form of feed-forward network on flagship models.

There are a few new boxes and lines to point out though. On the left is our old friend Engram, which adds itself on to a couple carefully chosen attention layers. Remember, Engram is a cheap way to attach a lot of meaning to short phrases quickly.

Right between all the embedding boxes is mHC, DeepSeek’s tweak to the standard residuals. Then all the way at the end is DSpark, which speeds up output. Again, I refer you to the past presentations if you want to dig in more.

The main show is all about the attention layers and what goes into them. For that we need to note a few elements:

1. The notation: SWA is “sliding window attention”, a very common local form of attention that just looks at the most recent tokens before the given query. CSA2 is the real star, and it comes in various flavors, indicated in parentheses. The number indicates compression, so 1 means no compression and 2 means you compress two tokens into one. The word after it indicates the mode of CSA2: Full, Reindex, or Reuse. We’ll expand on that next slide when we dissect CSA2. We’ll also see that CSA2 includes SWA, meaning SWA is part of every attention layer. So we are always doing local attention, and most of the time we are also doing global attention. Just not at the start.

2. The left-right split: the authors divide the architecture in half, with the first half called the “causal encoder” and the second half called the “decoder”, hence the term “causal encoder-decoder” I mentioned earlier. I will get to what that all means in a minute. For now, note the even split: 20 layers of causal encoder, 20 layers of decoder.

3. The hidden states: the output of the causal encoder. A hidden state is just the output of a layer, so basically the output of the final layer in the causal encoder is going to have a special role in the decoder.

Okay, now we are ready to plunge into the guts of CSA2, which took me a long time to really get. So if some details escape you now in this review, I don’t blame you.

Now remember, the goal here is to do as little computation as possible while still achieving good quality. Everything is downstream of that efficiency goal, and that switch from chatbot to agentic use cases.

So the first thing to note is we have three versions of CSA2: Full, Reindex, and Reuse. They appear in order from most to least computationally intensive.

Another thing to note is how they overlap. Specifically, they all share the grey boxes in the Reuse Mode diagram. I will explain the steps in sequence, but use the Reuse Mode diagram to ground yourself as components of CSA2 change.

Last thing before we dive into the different modes is that we have two levels of attention: global and local. Global attention is normally more expensive because the cost of attention scales with how much there is to pay attention to; with local attention, you only look back at the most recent n tokens before your query token, in this case n = 128 for our sliding window. So all the fancy engineering to wring out efficiencies is gonna focus on global attention, which the researchers called “main” here. That’s why all the action is on the right side of each mode diagram, where your job is to select what from the main KV you want to combine with the SWA KV for when you finally do the attention calculation with the main Q.

Let’s start with Full Mode.

Our inputs are:

• Main KV, which we get by multiplying the input by the weights matrix W_KV. Note that in vanilla attention, we have W_K and W_V. Here we have only the one weights matrix, meaning K and V are always equal

• Indexer K, which comes from Main KV

• Indexer Q, has its own weights matrix to transform the input

Now for folks who came to the Kimi K3 session, you may recall that in the attention formula, Q and K multiply to produce the attention scores, which basically say how important each combination of any two tokens in the input is. That same principle is at work on the right when we run our indexer, taking these special versions of Q and K to say how important each position in our current input is.

Since we want to be efficient here, we’re only going to attend to the top echelon of indices for any particular query. In this paper, it’s 512. So even if you have hundreds of thousands of indices to choose from - because your input is in the hundreds of thousands of tokens - you gotta narrow down.

So you take your 512 top indices from the global context, from that main KV. Then you glue ‘em onto the SWA KV and do the normal attention operation.

In Reindex Mode, we are doing almost the same thing. The difference is where our inputs come from.

Specifically, we reuse the Main KV and Indexer K from the most recent Full Mode CSA2 layer. So for global attention, this KV cache is not specific to this layer!

We do however need to compute a new Indexer Q. Otherwise we would be reusing all the information from Full Mode, so this layer wouldn’t be adding anything really. So the net effect is we take our same pool of index candidates but come out with a new top echelon, a new 512 we want to keep. Some computation saved, but much of it still left to do.

Finally we have Reuse Mode, which takes Main KV from the most recent Full Mode and the top echelon of indices from the most recent Full Mode or Reindex Mode. Reuse is the cheapest mode.

Now looking back at the overall architecture and where each CSA2 mode lands, we start to understand how they decided when to use each one.

The first two layers are SWA only, purely local attention. So the third layer, which is the first shot at global attention, has to be Full. Otherwise there is no Main KV or Indexer K to reuse.

After that comes five layers with Reuse Mode. To me that says, “we got a good picture of the global context already, so now let’s look at that picture from different perspectives”. Like the MoE layer right after the attention layer does some processing, so after that processing let’s reexamine our global picture with our new and altered local context. Because remember, we never reuse anything related to SWA.

Once you’ve done everything you can with that version of the global picture - after five blocks of CSA2 Reuse and then MoE - you form a new global picture. And you work with that, extract what you can, and do the Full + Reuse loop one more time. That is the Causal Encoder half.

Now note the Decoder half: there is only one CSA2 Full, and it’s right after the last Causal Encoder layer. So that is the last time you compute a new Main KV. And because of that, the Causal Encoder is the only part that runs during prefill! You only have to activate half your model when reading new input! That really puts all the savings at the individual layers into perspective.

So we have our now-familiar path of Full + a few Reuse. However, instead of cycling back to Full again, we do Reindex. Remember, Reindex is a different subset of the same global picture that Full drew from, it’s a different 512 indices to then stitch onto your local attention. But apparently it plays a pretty similar role to Full, where Reuse can draw new information from it a few times before you need to reshuffle again.

So that’s the mechanics, or at least part of it - there are additional details I left out for time. What’s the outcome?

Well one is a massive drop in the size of the KV cache per input token. Compared to the prior release, DSV4.1F drops it by nearly a factor of four. If you reach all the way back to the first release at the end of 2023, it’s a 437x drop! DeepSeek has been attacking this problem for a long time.

The upshot here is that the same hardware can support much longer trajectories, which is crucial in the age of agents.

So we’re more efficient with our memory, but we’re also more efficient with our compute too.

To produce a new token requires a certain amount of compute, measured in FLOPs. And that amount of compute changes depending on how much context there is to look at and refer back to and consider.

DSV4.1 is actually a bit worse than its predecessor at smaller contexts - up to about 100k - but is barely sensitive to context length at all. Remember, this is a log graph, so what looks like a modest rise in the DSV4 trend actually matters quite a lot, and at long contexts decode speed is gonna drop substantially. For DSV4.1, decode speed would eventually drop, modestly.

Again, this wouldn’t matter so much for chat - it might even be worse - but for agents you are regularly gonna get into the hundreds of thousands of tokens on your trajectories, so the difference matters a lot.

Okay, so where does that all get us?

To answer that, let’s take a look at how DSV4.1F stacks up against both sizes of DSV4. We are going to look at base models, i.e. models that have completed pretraining but have not done post-training. Of course it makes sense to look at V4 compared to V4.1, but also, not many shops release base models anymore, so pickings are slim for this table anyway.

Also, before we look at the results, I want to call out the split in benchmarks. Pretraining is where the model gets all its raw information and its potential, which post-training then shapes. So you’ll see World Knowledge is the first section, and you’ll also see that the other benchmarks are mostly not that hard. It’s only later, after post-training, that models can do things like patch gnarly bugs or solve Olympiad-level math problems.

Now, the results. Despite having far fewer total and active parameters, DSV4.1F goes toe-to-toe with DSV4P. And actually, if you judge on parameter count, you should really think of Flash as closer to 400B; the Engram portion, which is more like a lookup table, is 196B parameters, so if you discount them heavily that 552B total drops substantially.

Anyway, Pro is probably better based on some vibe checks I’ve heard as opposed to just benchmarks, but it’s at least close, which is impressive. I’m not surprised to see Pro generally in the lead on the World Knowledge benchmarks, since knowledge tends to correspond to total parameters, but Flash leading on some Code & Math ones is impressive.

To be fair, DSV4.1F got about 39T tokens of text in pretraining compared to 33T for DSV4P, but I don’t think 6T more tokens of pretraining would have closed the Flash-Pro gap for V4 - some of it has to be down to the architecture.

As for what’s in that pretraining data, they don’t say much, other than the standard stuff about improving quality and removing duplicates etc. The one interesting detail for Scale is “we involve more domain experts to construct fine-grained data quality evaluation dimensions”, which maybe is like rubrics for pretraining data? Hard to say exactly.

Now let’s move on to post-training. As is standard nowadays, the researchers conduct SFT on one checkpoint, then RL on several checkpoints to create teacher models, then do multi-teacher on-policy distillation (MOPD) to create the final model. In fact, they call this the “standard paradigm” and explicitly avoid innovation here, focusing exclusively on the data. Here’s a direct quote:

“We find that, under a fixed and unremarkable optimization procedure, systematic improvements in the scale, diversity, and verifiability of synthesized data and environments account for essentially all of the observed gains.”

So because they want to lean on their own model to generate training data, they start by training the model to create better training data. Each task consists of a problem, an environment, and a verification system. They evaluate generated tasks based on difficulty and correctness, although they don’t explain how. They also watch the traces when the generated tasks actually do make it into the RL pipeline, to gather real-world evidence about quality.

Speaking of real-world data, they mention collecting traces and feedback from internal and external users on a voluntary basis. They use the traces to reconstruct environments, mocking popular SaaS apps but also proprietary software and backend systems. They focus in particular on recreating failures observed and reported in the wild, targeting them for improvement.

For coding agent data in particular, they describe a multi-agent process for turning popular GitHub repos into environments. Again it’s high-level enough to be nothing revelatory - one agent makes sure the repo actually can work in a container, another scrubs it for information that could leak answers, stuff like that - but I thought it was interesting that DeepSeek landed on the same generalist-coding split that Moonshot did in thinking about agent training and capabilities.

Of course coding is where the big money is these days, and is also how labs plan to hit recursive self-improvement, so it comes front and center. Here are some RL runs on popular coding benchmarks. The discontinuities and changes in color indicate model reinitialization, basically taking concurrent RL runs with different harnesses or configurations and merging them after a certain number of RL steps. So like let’s say you start with one checkpoint and make one copy each to put in Claude Code, OpenCode, Pi, and DeepSeek Harness. You train your four copies, they all become somewhat different, then you merge them to smooth out the idiosyncrasies while hopefully keeping the real gains.

Speaking of harnesses, here’s a look at what impact they can have during RL.

On the left we have several curves for different versions of Claude Code. It’s a bit hard to see the lighter shades, but overall the trend lines for both pass rate and output length pretty well overlap. Also I’m impressed to see how the final run in dark blue cut trajectory length almost in half.

On the right we have trends for various scaffolds, including OpenCode, Pi, and DeepSeek Harness. Again, some variation but pretty similar trends.

My hunch is that because benchmarks are supposed to be diverse, you get a sort of averaging effect and any reasonable harness is about the same as any other. But in the real world, where you mainly work on one or perhaps a few projects and you have your own ways of working and so on, individual users feel a big difference on harness sometimes. That’s how I would reconcile the frequent reports of big differences in harness against this apparent similarity between harnesses.

To be fair, they do have a table later showing a nine-point spread on one benchmark depending on the harness, but with error bars around each score I don’t think it’s a big deal.

I also think Anthropic and OpenAI in particular don’t want good performance outside of their own harnesses, similar to how Apple software only works with Apple hardware and functions more as a whole than as separate components. So if folks are using Claude or GPT in OpenCode for example, and they have prior experience in Claude Code or Codex, that would explain the disappointment. The open-weights model companies on the other hand do want good performance on popular harnesses, since their models can crop up anywhere and be part of any stack.

One other fun detail from their RL training is how they added reasoning effort.

By now the user-facing terminology is basically settled: low, medium, high, xhigh, max. Not everyone has every level, but reasoning effort is a thing and users expect it to look like that.

Under the hood though, it’s not obvious how you would get the model to actually obey, and from what I’ve seen different labs do it different ways.

For this model, the DeepSeek folks decided effort would be part of the system prompt (during training and inference) as shown here: a number from 1-100. 50 means “low”, 75 means “high”, 100 means “max”. For API users you can only pick from those three, but if you’re hosting yourself you can plug in any number you want. Heck, you could even put in some nonsense value or delete it from the system prompt entirely!

It’s not just the instruction in the text that controls effort during training though. They describe how the numerical value for effort feeds into a length penalty formula, which is how the model learns what effort really means. So it’s no surprise that effort tracks output token length more than pass rate. The jump in output tokens for max reasoning effort is particularly notable, like the model is tracking down every possibility no matter how remote.

I kind of wonder if the lower scores are just the model sandbagging a bit. Like if you told me to put in a low amount of effort, I might know what the right thing to do is but take the easy option instead and pretty much know it’s gonna turn out wrong. I know what they’re actually controlling is token output length, but there is clearly some connection between effort and outcome. Maybe that’s why they didn’t test at values below 25 and why in the API you can’t get a value below 50.

Now we can finally check out the final model’s benchmark scores. Notice how dominant agentic benchmarks are, and how the first nine in the list are about code.

The competition here is also instructive. They are going up against Opus and Sol, not Fable. (Astra wasn’t available at the time.) It’s still impressive that what’s likely a Sonnet/Terra-sized model can go up against Opus/Sol, but in real usage I do not see folks abandoning Anthropic and OpenAI for DeepSeek.

It’s getting harder and harder to measure model quality with benchmarks, both because of the more subtle and subjective differences and because benchmarks are saturating so quickly. Just look at the jump on some newer benchmarks from DSV4P to DSV4.1F: almost 20 points on Terminal-Bench 3.0 and Terminal-Bench 4.0, 10 points on DeepSWE, 10 points on ExploitGym. Those benchmarks are all only a few months old.

One area that doesn’t get enough benchmark love, and where I was disappointed by the Kimi K3 paper, is multi-agent.

There are some fun details in here about how they structured the swarms. The harness lets the lead agent create teammates, assign them roles, and pick what context they start out with. There’s also a “peer mailbox”, kinda like a group text thread or Slack, plus a kanban-type board for tasks. The lead agent monitors and can interrupt any other agent.

As for training, they add a collaboration bonus to encourage teamwork and a latency penalty that encourages parallelism and discourages synchronization. I would love to know more about how they came to their design and how it may have evolved, but alas, they give only the current state.

Now neither benchmark here is multi-agent per se, but it’s instructive to compare solving the tasks with one vs many agents.

In theory there can be a quality benefit from multi-agent, along the lines of specialization and swim lanes and clean context, but of course we all know how badly group projects can go. So solve rate is going to be an empirical matter. However, multi-agent should pretty much always win on time, on how long it takes to complete a task or how much progress can happen in a fixed amount of time.

What we see here is a bit of both, and somewhat different stories. On the left, I see mostly the story I expect, where multi-agent isn’t any better initially but opens up a wider and wider gap over time as the benefits of parallelization accrue. The dip at the end is unexpected, although it happened for both cases.

On the right though, I’m surprised that multi-agent starts off better yet doesn’t increase its lead almost at all. So maybe the tasks have a lot of dependencies and can’t really become very parallel. The mythical man-month and all that.

By the way, if you’re looking at the ProgramBench results and wondering what “Almost@1” means: it’s a slightly easier threshold for completeness, specifically that the solution passed at least 95% of unit tests.

My Takeaways

• DeepSeek is still in the fight

  • I’m optimistic about DeepSeek Harness, but I’ve heard it’s not quite there yet

  • They don’t make it easy for themselves when they give away so much research! Qwen has already incorporated Engram for example

• LLMs are becoming inscrutable

  • So many fine details

  • RSI will accelerate this process

  • We will deal with this by adding a layer of abstraction on top, which is how we deal with all systems that become too complicated to understand wholly on a certain level

• Still lots to do with multi-agent

  • Hasn’t trickled down to training data or architecture etc

  • I expect team design (and eventually team self-improvement) to get just as much attention as harness design

  • Already quite relevant due to the Hugging Face incident and similar swarm actions

Paper · Blog post

Background

A brief bit on Physical Intelligence, which is the company behind the Pi series of models. They’re one of the top names in the field, although they’ve only been releasing models since late 2024. One of the cofounders, Sergey Levine, is one of the top robotics researchers in the world.

Now let’s do a quick recap of the Pi 0.5 paper, since it was the starting point for Pi* 0.6. (Pi 0.5 deck)

A robotics model like Pi 0.5 is a VLA, a vision-language-action model. That’s an LLM at the center, with a vision encoder added on the front so it can see, and an action expert added on the end so the model can do stuff - the action expert outputs signals for the motors and such. In the case of Pi 0.5 the LLM is Gemma 2 2.6B, made by Google. The vision encoder is SigLIP, also made by Google, so the vision-language model so far is 3B parameters. Then you add on the action decoder for another 300M parameters, so 3.3B parameters total. It’s a pretty small model, but it really has to be in order to produce actions in real-time. Gotta keep that latency down, no way to do that with the parameter counts of SOTA LLMs, which are in the tens or hundreds of billions.

So one of the major contributions of Pi 0.5 was this concept of “co-training”, i.e. training the VLM part on a bunch of multimodal web data. If you’re not familiar with this work already it may be somewhat surprising that a robot could learn to manipulate the world better by just looking at images or watching videos, but of course we know humans can do the same - we can learn how to do things just by watching. Hands-on practice is better but it’s not always required.

They also throw in training data from other robots, like other embodiments that this model won’t get used on.

The other major contribution was this idea of subtasks, which they give examples of in the pinkish red boxes. Like with a chain of thought breaking down a reasoning problem, the subtask improves actions by making each step more granular. So the VLA keeps the high-level prompt in mind while also feeding itself these low-level subtasks to guide its actions. As it progresses through the task, it generates new subtasks for itself to complete, sort of like if you talked out loud as you completed some physical work.

Now on the more fundamental side, I want to briefly discuss the idea of a value model.

A value model is a way to estimate how good your current state is. In LLMs, the value model is this second model in training that judges the response so far after each token. You need that value model in PPO, which is traditionally the most popular reinforcement learning algorithm for LLMs. Looking at the graphic above, the value v turns into an advantage A, which tells you how much the last token helped your response. Then PPO takes that advantage and uses it to adjust the policy model, that’s your main LLM you’re trying to improve.

The value model requires training to get good at this estimation business, which makes PPO expensive and finicky to run. And of course the value model can’t perfectly predict the future so the values it produces are always going to be somewhat noisy. Still, empirically it does help the policy model produce better outputs to have these intermediate ratings available in training.

As an aside, you may have heard of GRPO, a more recent RL algorithm that dispenses with the value moel and instead compares lots of different responses to calculate advantages. People tend to use GRPO when there is one clear reward to give each response, like in math/reasoning. So GRPO and RLVR often go hand-in-hand.

Video

So here is their method in a nutshell. We’ll get into the details on subsequent slides but I wanted to start high-level.

The basic approach is: take an existing VLA, in this case Pi 0.6, then add a value function that predicts whether the current action will be helpful or not and teach the VLA to work with the value function, and then collect post-training data with that two-model system.

They also use a lot of pretraining data, like the robot trajectories shown on the left but also some web multimodal data. That sometimes involves the value function too, and they mention collecting “tens of thousands of hours of demonstrations from numerous tasks and a variety of different robots”. But the heart of the method is the loop of autonomous episodes and reward labels used for post-training.

Note also the “Intervention” video in the bottom-right. The frame around it sometimes turns red, indicating that the robot has made a mistake and that the human has intervened to recover. Unfortunately they don’t give any operational details about how the teleoperator intervenes or relinquishes control etc.

Now let’s zoom in on the architecture. Pi-0.6 is a VLA, like we reviewed in the background slides. They upgraded from Gemma 2 2.6B to Gemma 3 4B, a bigger and better model all around. They also more than doubled the size of the action expert. So that’s ~5.5B total parameters, apparently still manageable for real-time tasks.

The major change is of course the second model, the value function. That’s a VLM, a vision-language model, trained to output a single number between -1 and 0. That number indicates how valuable the current state is for getting to a successful conclusion. It’s a remarkably small model, not even 1B total parameters, and the 400M parameter vision encoder is most of it. That’s because the intelligence for this kind of work is mostly visual, with little abstract reasoning or factual knowledge required, so a 270M parameter LLM is adequate. Whereas for the VLA you need to, for example, recognize what the prompt is asking for and break the task down into subtasks, which requires reasoning and verbalizing information about the world. And the VLA has to make and execute a plan, sometimes covering the whole trajectory, whereas the value model just lives in the moment, it just needs to say how good or bad the current state is given the end goal.

Anyway, in training the value gets turned into an advantage, then that gets turned into a binary based on how high the advantage is in order to cut down on noise. Then that True or False value goes into the VLA as additional text, like literally the text “Advantage: positive” or “Advantage: negative”. So the VLA is basically playing hot and cold with the value function and learning what this new signal means. Then during inference, when you’re actually using the model, you always pass in True. That tells the VLA to take actions like the good ones it saw in training. A little strange but apparently it works.

Just to make it concrete, here’s an example of the value function rating the situation over time. It’s a little noisy but it does pick up on the major error around 100 seconds and the subsequent recovery around 140 seconds.

Now here are the specific tasks they’re going to evaluate the model on. There’s some generic folding task, then a more challenging one, then one where you have to fix existing errors in folds, then box assembly and of course our friend espresso making from the video.

Note that they post-train the model on these tasks, they’re not evaling right out of the box. If you’re coming from LLMs you might think that’s cheating, but actually on most benchmarks there is a train set you use to teach the model the general form of the task you’re going to test it on. Same thing is happening here - you’re teaching the “rules of the game” so to speak and then seeing if the robot can play that game well no matter how the pieces are arranged or what moves are made.

As for that eval-specific post-training data, there are some confusing notes in the appendix about dataset composition, but in general it’s in the hundreds per task.

I should also quickly note the embodiment they used. It’s a simple pair of arms with grippers, running at 50 Hz.

Now we’re about to see the results of their work, but I wanted to clear up their somewhat confusing nomenclature first.

The initial baseline is Pi 0.5, which we discussed back in July.

They also run on Pi 0.6, kind of an all-around upgrade on the Pi 0.5 formula but no radical changes.

After that are the components of RECAP added on element by element. So the first Pi* 0.6 is with the value function, which then becomes and advantage which then becomes an indicator which is a new input for the VLA.

The next one adds the first bit of post-training, for the specific eval tasks. I don’t know why they change the nomenclature from “RL pretrained” to “offline RL” but it’s the same thing.

Then the final step, the full RECAP, does online RL; in other words, it trains on its own successful trajectories, including the ones where a human intervened for part of it.

And here are the results. As you’d expect, the bright yellow bars representing the complete Pi* 0.6 model are the tallest and best, but I’d like to zoom in on a couple things.

First, let’s distinguish the top and bottom set of charts. The top is just the straight success rate, with the box assembly one further broken out into subtasks. That’s a fine metric, but the bottom one is probably more important: the throughput. That’s the number of successes per hour, which rewards fast completions. It does also reward fast failures though, so you do want to keep that success rate metric around as a constraint. Pi* 0.6 never goes below 75% success rate, which seems acceptable as a lower bound.

Anyway, the biggest “speed bonus” seems to come on the regular laundry task, where the success rates are mostly similar but the throughput nearly doubles for Pi* 0.6 compared to the offline RL + SFT version. A less dramatic version is also true for box assembly.

The other thing to note is just how big the jump is when you add that final online RL step. Like it’s not a smooth improvement as you add on each component of RECAP, you really do see a jump at the end. So it seems like RL and intervention are crucial. They also have a graph later on showing improvements over multiple iterations of online RL: collecting trajectories, training on those, collecting more trajectories, training on those additional ones etc. Even in tasks where success rate is basically 100%, you can improve throughput with one or two additional training cycles.

Overall it’s impressive work. This model got a robot to successfully fold laundry for two hours straight in a real home and to successfully serve espresso for 13 hours straight!

My Takeaways

• Correction of models could be right up our alley

  • May require other embodiments to be acceptable

  • I have no idea how much labs would pay

• Could better value function data help too?

  • The training data for the value function is labeled by rule, not by model or by human

  • Surely the data could be better, but open question how much it would help

• I stick by my prediction that the ChatGPT moment for robotics is 1-2 years away

Paper · Blog · Chat

Background

Let’s start with the company behind the model: Moonshot AI.

Moonshot AI was founded in China in March 2023, 50 years after the release of Pink Floyd’s The Dark Side of the Moon, after which the company is named. For context on AI progress at the time, ChatGPT came out in November 2022. Coincidentally, GPT-4, Claude 1, and Google Bard also came out in March 2023. So a pretty busy time! I remember hearing about new models from tons of new and no-name shops in 2023 and 2024, when models were still small enough for startups and even academic labs to pretrain on their own. Now of course there are few organizations worth tracking and everyone knows who they are.

Anyway, in October 2023 the first version of Kimi came out, as a chatbot for the Chinese market. Kimi K1.5 came out in January 2025 with the same approach, not making much fanfare from what I recall.

It wasn’t until the release of Kimi K2 in July 2025 that folks here really started paying attention to Moonshot. That’s because Kimi K2 was open weights, and those weights were huge: 1T parameters, the biggest open-weights model ever released at the time. Successive releases in the K2 series - K2.5, K2.6, K2.7 - held onto that record. DeepSeek took it with their V4 Pro release in April 2026, but Kimi K3 actually took it back again for Moonshot, with its incredible 2.8T total parameters.

Publicly releasing such a big model made Moonshot the perfect partner for AI startups that had decent access to compute but not nearly enough of it to pretrain their own models from scratch. So for example, Cursor’s Composer 2 used Kimi K2.5 as the base model and then post-trained for coding and use within Cursor, like with Cursor as the harness. More recently, Harvey’s Tenet did the same thing with Kimi K3 for long-horizon legal work.

Further increasing Moonshot AI’s status is their scientific work, which has improved in taste and impact as they have grown. We’ll touch on two particular topics a bit later when we do the technical side of background.

On the flip side though, we have heard repeated allegations of Moonshot distilling outputs from more advanced models to improve Kimi. Most recently, Anthropic claimed that Moonshot was even serving Claude instead of Kimi, using over 5k fake accounts in Singapore and Japan, so users of Moonshot’s APIs and chat tools were sometimes talking to Claude but told it was Kimi. There is some genuine debate about the merits of distillation, but there’s really no defense about selling someone else’s product as your own.

Still, the weights to Kimi K3 are out there and available on plenty of other inference providers, so we know it’s not all smoke and mirrors. My personal hunch is it’s more of an economic or GPU-related tactic than trying to cover for poor quality or something.

Speaking of Kimi K3 versus other models, here’s my go-to resource for checking model rankings and progress: the Epoch Capabilities Index (ECI), a composite of 50 benchmarks from several sources. The score on the y axis is supposed to reflect ranking relative to GPT-5, which definitionally scores 150.

ECI is a product of Epoch AI, a well established, independent research organization. I’d put them in the same bucket and tier as METR, whose famous time horizons graph we touch upon so often in Friendly Paper Review.

And in fact the trend is the same here as on the METR graph: a steady, unrelenting line of progress. It’s pretty remarkable how predictable frontier progress on ECI is.

As you can see from the displayed names, it’s always US labs in the lead. Mostly it’s OpenAI or Anthropic, although I spy Gemini on there. The sole red dot, off the frontier line, is Kimi K3.

There are really two ways to situate any model not on the frontier: the gap against the best current model, and the time lag since the first time a model reached the subject model’s score.

So if you look at other models available on the Kimi K3 release date of July 16th, Fable 5 was the best, with an ECI of 163. That’s five points ahead of Kimi K3, at 158. But what does that actually mean? It’s a bit abstract on its own.

That’s where the time lag helps. If you look left of the red dot, the first model to reach or exceed 158 was GPT-5.4 Pro, at 159. (GPT-5.4 scored 157.) That’s four, maybe four and a half months before Kimi K3 came out. So if you’re used to using only the best models, then you’re travelling back in time a third of a year when you use the latest Kimi.

In fact, the rough heuristic is that Chinese frontier models are about six months behind American frontier models. People freaked out at the time of the Kimi K3 release because on some measures it was on par with the leading US models, but with a little distance I think we’ve seen the time lag implied by the ECI score bear out - there was no mass exodus from Fable to Kimi because Fable was clearly better.

A couple other data points from this graph on recent models. First, Kimi K3 scores just above GPT-5.6 Luna and just below GPT-5.6 Terra. But Kimi is 50% more expensive than Terra and over 10x more expensive than Luna, which means it’s a poor economic choice on this basis but also implies it is larger than both models. So again, the leading US labs are ahead.

On the flip side though, Kimi K3 comes in slightly ahead of GLM-5.3 and Qwen3.8 Max, its top Chinese competition as of writing.

I will also throw Artificial Analysis out there as a decent source of information on model performance, although the index here only comprises 10 benchmarks. I like this chart because it’s easy to see where all the big players land.

So of course we have Anthropic and OpenAI at the top. Note that I have removed some of their models from the chart in the interest of visual clarity and variety, but they pretty much dominate the top slots.

Apparently Meta scores highly here, which impressed me, as does Grok 4.6, which is surely due to the good work of the Cursor folks who are now officially on board there. Kimi K3 trails just behind, virtually tied with GLM-5.3, another Chinese open-weights model that has made waves.

The biggest shock to me though is the cluster of DeepSeek bars spanning the 30s. DeepSeek is probably the most well respected Chinese lab, so it’s tough to see them lagging so badly. I’ll have more to say about them at next week’s Friendly Paper Review when we cover their most recent release, DeepSeek V4.1 Flash.

Okay, now let’s touch on some technical background before we get into the paper. I want to mention two bits of Moonshot prior art: Kimi Delta Attention (KDA), and Attention Residuals (AttnRes).

Let’s start by refreshing ourselves on attention, which is the basis for both.

The purpose of attention is to figure out what stuff from the input X we need to take into account when predicting the next token. The way it does that is by looking at the input a few different ways: as queries (Q), keys (K), and values (V).

The query is the token in question, like the thing you’re going to compare other things to one at a time to determine the meaning of the pairing. Let’s say for example my query token is “red”.

The key is the other token, the one that’s going to modify the query and produce a pairing. Let’s say for example it’s “shoe”.

The query and the key combine to provide the attention score, which says how much we should care about that pairing. Don’t worry about the square root divisor and the softmax operator - those are just about keeping the numbers reasonable and bounded.

Finally, the value is the meaning of that pairing. “Red shoe” has one meaning, “blue shoe” would have another, and a nonsense combination like “red go” technically has a value but not a strong one.

By the way, the way you turn X into Q, K, and V is to multiply X by the relevant set of weights W. Those W matrices are what the model is really made of, plus the formulas that say how to use the matrices. The other parts of the model like the feed-forward network also have weights of course.

It has become conventional to present LLM architecture in diagrams, often like the one you see at the top here. Remember though, it is just math deep down - the diagram is equivalent to the formula we saw before, minus the pink “Mask” box which is optional. Keep this sort of diagram in mind for the next slide.

Now the biggest issue with the original attention formula is that it doesn’t scale well. By giving equal resources to every pairing of input tokens, the compute and memory loads grow as the size of that input squared. So-called “quadratic attention” is the highest quality variant, sure, but it’s often wasteful and certainly not practical in the agentic age where very long trajectories are the norm.

So instead of quadratic attention, where resource intensity grows as the square of the input size, people try various forms of linear attention, where intensity is directly proportional to input size.

And actually there’s a form of neural network that long predates attention and the whole transformer paradigm that already is linear in this way: the recurrent neural network (RNN).

I’m going to spare us the math and use an analogy instead. For quadratic attention, the model is keeping detailed notes and can go back and find anything. But for RNNs, the model gets a single page - a cheat sheet - that it can tweak and update as it processes more input. It’s lossy, but it’s much faster to review when thinking about what the next token should be.

But how to update that cheat sheet as each new token comes in?

That’s where the delta rule comes in. Again, putting the math aside, the basic version is to look at the difference between the predicted value and the actual value and update based on that difference, that delta. And the parameter beta, over on the right there, is a number between zero and one that controls how strongly to update based on the delta. For that reason, people call it the “learning rate”, although in the original paper they called it “writing strength”.

The graphic here shows DeltaNet, an RNN that incorporates the delta rule. Don’t worry about the elements we haven’t explicitly mentioned, like Linear and Conv and L2 norm.

Now DeltaNet came out in mid 2024, and since then folks have settled on two key additions: a “forget gate”, alpha; and an “output gate”, the line on the right that goes around the delta rule and into that circle with an x. The gates give the design its name, Gated DeltaNet (GDN).

The forget gate alpha is basically the complementary knob to the learning rate beta, although they are independent. So let’s say you’ve got your cheat sheet, i.e. your recurrent state, and you process the next token.

If alpha and beta are both close to one, then you keep as much as possible from your cheat sheet but take great pains to record the information from that token. If the input is a really critical piece of the puzzle you’ve been assembling in your prior context, you could get that behavior.

If alpha and beta are both close to zero, then you write down nothing and you basically erase your cheat sheet. Maybe if your input clearly demonstrates the prior information is all outdated or irrelevant now yet no new information has come in, this would make sense.

If alpha is high but beta is low, then you’re keeping your cheat sheet as-is, like the current input is filler or repetitive etc.

Finally, if alpha is low but beta is high, then you’re clearing out space for the new stuff you’re taking in, really changing direction or maybe updating old stuff.

So that’s the forget gate. Then after your delta rule and a quick normalization for stability, the output gate decides how much of that layer’s output to keep as the information flows on to the next layer. Like if all that updating just wasn’t that important, the output gate can effectively toss it.

Overall the GDN is strictly superior to the regular ol’ DeltaNet in quality, which is why Moonshot picked it as the basis for KDA.

Speaking of which, here it is! Looks pretty similar right?

It’s subtle, but the main difference is that forget gate, alpha: it’s gone from a scalar to a vector. So now the forget gate can vary depending on more fine-grained aspects of the input.

Now there was a ton of gnarly engineering for KDA but also for GDN earlier to make all this new computation actually feasible, which we don’t need to get into here and isn’t visible with nice architecture diagrams like this. As is often the case, apparent simplicity often belies the real complexity.

So that is KDA in our pockets. Now to add AttnRes.

Luckily for us, I already did a Friendly Paper Review on this topic! And in another bit of happenstance, that FPR also covered a DeepSeek paper, just like we will be covering a DeepSeek model right after a Moonshot model. I guess when progress is this fast, you’re bound to get coincidences.

Because I already covered AttnRes in depth, I just want to quickly define a residual and how attention plays into it.

On the left we have a standard transformer diagram with embeddings to start, alternating layers of attention and feed-forward networks (in this case a mixture of experts [MoE]), and then output. They have neglected many details, but kept one in: the arrows bending around on the left. Those arrows show the input from the previous layer skipping the current layer and adding to the current layer’s output, where that sum becomes the input for the next layer. That’s a residual.

Residuals are helpful because they keep the intermediate steps grounded. Think of a game of Telephone for example, where each person’s grounding is only the immediately previous person. That is how outputs drift badly from inputs. If players knew what the previous person heard before that person passed on the message to them, it would cut out a lot of drift.

But wouldn’t it be even better if the player heard what every previous person heard? I mean it would be a terrible game of Telephone, but it would basically guarantee accurate transmission.

That is what we see on the right. There are residuals from each previous layer to the current layer. Not only that, each residual has its own versions of Q, K, and V - the ingredients of attention. These Q K and V are different values coming from different sources though; what we’re keeping is the attention mechanism, not the attention information. So with these Q K and V specific to each layer, the model learns how much to listen to each previous layer.

Of course a model is unlike Telephone because each layer is supposed to change the input and add value. But not every layer is created equal, and for different inputs different layers may be more or less helpful.

Pretty clever in my view, and probably my favorite Moonshot contribution to the science of AI.

The Paper

So the big story with Kimi K3 is just how big they made the model. Like look at the last column, the delta between K2 and K3. Every single numerical dimension is bigger, sometimes dramatically so.

The headline is of course that 2.8T parameters, which they achieved by making the model 52% deeper and having 133% more experts, each of which is 50% bigger. So the model is thinking more deeply about each token and has more fine-grained expertise.

The activated parameters also went up, mostly from having bigger experts and selecting more of them at a time: double the experts active per token.

On more of the catch-up side, the context window is now 1M tokens - standard by now for frontier models - and Kimi now comes with vision natively trained in rather than glommed on.

Here’s the architecture, which we should mostly be prepared for at this point.

In the bottom-left we have Kimi Delta Attention. In the top-left we have the mixture of experts, with two experts always active and the rest selected based on the input. They have a couple tweaks here which I’m going to skip over, but the tweaks are why they call theirs Stable LatentMoE instead of just MoE or similar.

Now on the right we have the entire model. It starts with the vision encoder and then the embeddings, then it has a series of transformer blocks, then the output - standard stuff. There are two things to note.

One is what’s in each transformer block. They open it up in the top half, that section with KDA and Stable LatentMoE and Gated MLA. What they’re saying here is that in each block, they do KDA + Stable LatentMoE three times, then a more standard version of attention called MLA + Stable LatentMoE one time. Because MLA has the traditional KV cache, albeit a compressed version, it captures global context in more detail. It’s pretty common nowadays to have this mixture of cheaper attention with a bit of higher-quality attention, and 3:1 is probably the most popular ratio for it.

So that’s one thing to note. The other is all those lines on the right side: the attention residuals. Kimi K3 actually uses attention residuals on a block level rather than at every individual layer, as a compromise between cost and quality - similar to using a mixture of attention types as we just saw. This 80/20 rule kind of stuff is a recurring theme in LLM architecture innovation.

So here’s one big payoff due in part to their architecture changes: a 2.5x reduction in pretraining compute to reach the same level of performance for K3 compared to K2. Of course data and infrastructure efficiency had an impact too, although we don’t know the breakdown.

Of course what we’d like to know is how much better performance is for a given level of compute, since the compute is what we control and budget for, but without units on the y axis we can’t say. However, based on the stars on the two trendlines, it seems K3 got better performance for less compute compared to K2. So it’s a win on both dimensions.

I also just want to say how crazy it is that scaling laws are so predictable. Look how straight those lines are! I know log plots can hide a fair amount of deviation, but still.

Before we move on to post-training, you may be wondering where the detail on pretraining data went. Well, there basically isn’t any - three short paragraphs with no numbers whatsoever is all we get. The only interesting bit in there is their call-out of “programmatic multimodal data”, things like games and 3D renders and CAD that one can describe and generate with code. CAD in particular is having a moment right now for frontier models.

Now let’s move on to post-training. They follow the emerging standard for frontier models: some SFT, a ton of RL to create teacher models across different domains, then multi-teacher on-policy distillation (MOPD) to create one final model that effectively covers all domains.

The SFT section is small and sparse, again bereft of numbers. They do mention a focus on agentic tasks - no surprise - and verification with humans in the loop.

Thankfully they spend much more time on RL. The graphic here describes how they create RL tasks. They also have RL environments, which I’ll come back to.

So for the RL tasks, they start out with a knowledge graph - basically a map of increasingly granular concepts, built automatically by specialized agents that keep distinguishing concepts until they feel the concepts are atomic. Once your knowledge graph is complete, you can sample individual nodes or nearby groups of nodes to determine what your task will cover. In the example here, it’s mostly coding with a dash of AI, which produces some LLM jargon like “RoPE”.

The keywords seed a search for relevant materials. Searching the web and in-house databases yields source material, which yet other specialized agents synthesize into tasks.

No idea how much data they actually made, and nothing in here is revelatory, so they’re clearly keeping many details to themselves. That’s the difference between open weights and open source!

So their RL focuses on three broad domains: general tasks, general agents, and coding agents. “General tasks” includes basic stuff like instruction following and vision and tool use that we now take for granted in model capabilities. It’s only with the latter two domains that we see real work like web development and deep research.

As you might expect, they have the teachers for MOPD split across domains. The other way they split up teachers though, which I’ve not seen before, is by reasoning level: low, high, and max. So three domains times three reasoning levels equals nine total teachers.

It is these nine teachers that get the RL shown here. The RL happens in a few different environments, each with the appropriate tools and data. For example, Web Development has web dev packages and a way to render what the agent produces. It also has deterministic checks where applicable and model judgment otherwise, like for evaluating the visual design of a web game. They do mention the judge uses a rubric, but the judge is the one creating the rubric in the first place - no evidence of human data here.

They do give a bit of color on some environments. For example, the personal assistant one has mock implementations of Gmail, Slack, Notion and the like. They have another one designed for agents to progressively submit, see results, and improve upon some work, like speeding up certain GPU operations. I have a graph about that on the next slide because I think it’s a wider trend.

One other interesting bit about their RL environments is they include several different harnesses: not just their own harness, Kimi Code, but also Claude Code, Codex, OpenClaw, and Hermes. So they built Kimi K3 to work in many setups. We don’t know for a fact, but it’s unlikely OpenAI and Anthropic are putting in the same effort to make their models portable across harnesses; my guess is they do the opposite and train their model to expect their harness.

Here’s that graph I promised about iterative improvement on GPU operations. On the x axis we measure how long the agent has been operating, although number of submissions or number of tokens would also work - anything to indicate duration.

On the y axis we have the relative percent speedup for the GPU operation. Each dot is a submission, and the line connecting many of the dots is the frontier, i.e. the maximum speedup so far. The difference is particularly clear for the blue and green trends, where many submissions fall below the frontier.

This type of task and graph is increasingly common, popularized by Andrej Karpathy’s Autoresearch harness. I expect to see a lot more graphs like this, especially as more work goes into recursive self-improvement (RSI) - which is exactly what Karpathy joined Anthropic to work on.

Relatedly, the authors note in the paper that Kimi K3 actually was optimizing its own performance during development, so these results aren’t in a vacuum - they are live in production and pointing to the RSI dream.

I also think more generally we will see more focus on a few hard examples and how they demonstrate model capabilities in the way benchmarks do now. As each task becomes more difficult and longer, there will be more to analyze on each one and it may be impractical to generate a benchmark’s worth of tasks. In the extreme, you have problems with specific names like the Riemann hypothesis or Navier-Stokes.

For now though, benchmarks are still plenty useful, so let’s quickly look at them. We already know the aggregate story from the background slides on ECI and Artificial Analysis, but I always like to look at which benchmarks are “in circulation” so to speak.

General agentic ones, which they just call “Agentic”, take up the most room and reflect the variety of tasks people are using agents for: research, spreadsheets, computer use, customer service, law, finance, and a smattering of other white collar work. (I have covered Agents’ Last Exam and OSWorld 2.0 previously.)

Agentic coding, which they just call “Coding”, gets its own section and includes benchmarks on ML and scientific programming. Again, indicative of where agents are going and where labs want to focus.

I’ll also call out the Scale benchmarks I see, namely Humanity’s Last Exam and MCP Atlas.

One thing I expected to see more of was some coverage of multi-agent swarms. They mention two relevant internal benchmarks elsewhere, and one case study has a single line about using subagents, but given their past emphasis I was expecting a much deeper dive. Funnily enough, Kimi K3 came out the same day Hugging Face disclosed a massive hack by a swarm of AI agents, now known as the Hugging Face incident. Probably just a coincidence that Moonshot chose to report less on swarm performance here than in previous tech reports though.

Perhaps relatedly, the paper does mention Kimi K3 discovered 16 previously unknown vulnerabilities in important software packages, including a subsystem within Linux. Finding vulnerabilities seems to be a natural benchmark these days.

That’s it for what I want to cover, but the paper has a ton more detail on their infrastructure, which they stretched to the limit with the massive number of parameters, the many extra experts, and the octupled context window. If you want a taste of why ML engineers make such bank (and why ML companies want to automate them), just take a look at section 5.

My Takeaways

• Kimi K3 has raised the bar for open weights model size, which others are sure to follow

  • Qwen3.8-Max is already 2.4T

  • I expect DeepSeek at minimum to release something in the 2T range (current max is 1.6T)

  • GPU and RAM prices are not coming down any time soon

• Kimi models will likely continue to be the top choice for companies that want to post-train their own near-SOTA models

• Autoresearch-like graphs are here to stay

  • Already seeing work like this in the wild

  • I’m expecting to hear more about single tasks in the same way we currently hear about benchmarks

DeepSeek-V3.2: Pushing the Frontier of Open

or, Papa Bear

Paper · Release notes

DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning

or, Baby Bear

Paper · Repo

Background

The main innovation of DeepSeek-V3.2 is the DeepSeek Attention mechanism, so first I wanted to cover attention more generally.

We’ve covered it before, but to oversimplify somewhat there are three components of a transformer model, which is the basic architecture all your favorite LLMs use.

1. Embedding, which turns words into numbers, since after all we are ultimately working with numbers in the form of matrix multiplications here

2. Attention, which calculates how each individual word or token relates to all the other tokens that came before it in order to form a complete, holistic understanding of the input and output so far

3. Feed-forward, or MLP as it’s noted here, which takes that complete understanding and “thinks” or “processes” it

For the attention layer in particular, you see each input splitting into a query, a key, and a value. The thing to notice here is how the queries and the keys form this matrix, where each query gets to “look at” or calculate against each key of the previous tokens. So like “The” at the start can only attend to itself, whereas “tokens” at the end can attend to all the tokens in the example. And within each row, like on each query, the circles are colored by how strong the relation is. So on the query “to” for example, it’s attending strongly to “The”, weakly to “model”, strongly to “attends”, and weakly to itself. Sometimes those relationships are a bit inscrutable to humans but in this case it generally makes sense.

The only other thing I would briefly note here is that the attention and feed-forward steps or “layers” as they’re called form the “transformer block”, and the transformer block repeats many times before the model is finally ready to predict the next token. What that attention matrix looks like in terms of the strength of relations changes from block to block.

So in the standard transformer model like we just saw, the model pays attention to everything - up to its maximum context window anyway. That would be the first box on the left, “Full n^2 attention”. And it’s called n^2 because the number of green squares goes as the square of the number of tokens. That means if you double the number of tokens in your context window, you’re quadrupling the compute and memory needs. It’s the gold standard, but like gold, it’s pricey.

People spend a lot of time trying to find cheaper attention mechanisms that are nearly as good. One of the more popular ones is Sliding Window Attention, also seen above. Here you specify a window size that is much smaller than the full context length. At first that means you’re only getting information about very nearby tokens, but as you keep going through the layers of the transformer, you start to indirectly get information from tokens that are further and further away. Sort of like a game of telephone: if I initially get information directly from my neighbors one day, and they do the same with their neighbors, then the next day when I check back in with them they’ll have information from their neighbors to share with me too. And if you do more and more layers, more and more days in our metaphor, then eventually you can get information from the whole previous context.

To accelerate this osmosis of information, a lot of models mix in layers of full and sparse attention. Like gpt-oss, the open-weights model OpenAI released earlier this year, alternates between sparse and full, so a 1:1 ratio. Gemma 3, the open-weights model series from Google, does a 5:1 ratio, 5 sparse layers for every 1 full layer.

We don’t need to talk about the third and fourth examples here, it’s just to show there are more and more exotic versions of so-called “sparse attention” mechanisms. Again though, the goal is to greatly reduce resource intensity at minimal quality cost.

I’d also like to briefly discuss proofs and why they get special treatment compared to other types of math and reasoning.

In a normal math problem, you end up with some final answer, something relatively brief that is at least somewhat straightforward to check against a known correct answer. You hope the chain of thought is correct, since users may look at it or care about its correctness, but it’s much harder to check so in practice you just check the final answer since that’s the most important thing.

In a proof however, every step needs to be correct - there is no one final answer to confirm. That’s quite a challenge to verify because one, it’s a lot of stuff to check, and two, a correct proof can take many different forms. Can’t do RLVR like you can with normal math problems.

One way people try to fix this is by writing proofs in a formal proving language like Lean. Writing a proof is like writing code, and if your code runs then your proof works. That’s nice in theory, but Lean is notoriously difficult to write, and a working program depends on all your supporting arguments and lemmas existing already. Something you can assume in your proof because someone else proved it may not exist in Lean, so you have to write their whole proof along with yours, which can spiral. So Lean works but is limited. Of course people are also trying to use AI to write Lean as shown here. Apparently this tool solved an Erdos problem, which is one of the unsolved problems posed by famed mathematician Paul Erdos.

The other thing people try is writing proofs in natural language and training some sort of AI proof verifier. I happened to see recently an example of this tactic in the wild, from this physics professor at Michigan State. Keep that note about his generator-verifier method in mind.

So the natural language approach is much more flexible of course, but it doesn’t provide the same guarantees. Still, if you believe in the increasing abilities of LLMs, you would expect this generator-verifier system to keep improving.

The Papers

So the major innovation of DeepSeek-V3.2 is the attention mechanism: DeepSeek Sparse Attention (DSA). If you understand attention generally it’s a pretty simple conceptual change, although I’m sure it took a lot of experimentation to get right.

The core idea is this: what if instead of manually setting an attention window, we learned the best attention pattern for each token? And that pattern doesn’t have to be just a certain window size, like the last n tokens - it can be just the tokens you’ve learned are likely to be the most informative. A technical person might describe this as a “learned, per-token attention mask”.

The box on the left shows a very small example of this, where “to” skips one token, and “past” and “tokens” skip two tokens. Again though, you can imagine any pattern of skipped vs attended tokens here.

Now in the real paper they always pick the top 2048 tokens to actually attend to, which means if you have less than 2048 tokens so far you’re doing full attention. But then anything after that just looks at what the attention mechanism believes will be the top 2048 tokens for the given token you’re checking against.

For completeness I will quickly acknowledge Multihead Latent Attention, which is the other efficiency measure here, but it’s more an engineering trade-off than a conceptually different approach.

And then the bit in the middle here is the structure of the overall model. If you haven’t seen this diagram before it may be confusing, so just focus on those 61 transformer blocks, the new attention mechanism, and the mixture-of-experts layer. The other stuff like the token embedding layer or RoPE or RMSNorm are standard and relatively minor details.

Anyway, these changes from V3.1 to V3.2 apparently have no negative impact on quality. They report very similar Elo scores on ChatbotArena for example. So you’re getting way cheaper tokens of the same quality, usually faster unless the context is very short.

The DeepSeek-V3.2 paper is mostly not about data, but they do have one section about it I thought would be relevant for us.

Briefly, they care about four use cases:

1. A code agent, like Claude Code

2. A search agent, like many deep research agents we’ve seen before that can take many “hops” of reasoning and searching to find answers

3. A general agent, for day-to-day or personal tasks like planning a travel itinerary

4. A code interpreter, which uses a Jupyter notebook to solve complex reasoning problems by writing code, like if a problem requires a graph or calculating a square root or doing simulations

For the code agent they mined tons of repos and pull requests from GitHub, which is standard. They were able to set up the environments for those repos, like getting the required packages installed, totally automatically.

For the search agent the environment is just search APIs I think, plus a sub-agent that verifies its answers.

For the code interpreter the environment was just the Jupyter notebook.

But for the general agent, they synthesized environments - over 1800 of them actually. And they synthesized the tasks too. Here you can see an example of a synthetic task and its synthetic environment, which is just a list of tools specific to trip planning.

So here’s how DeepSeek with that modified attention and that agent data stacks up. They group by closed vs open weights, and I’ve also updated the table with Opus results where available.

Since the focus was not quality we shouldn’t be surprised to see other models sometimes besting DeepSeek here. Do note the tool use benchmarks though, where DeepSeek is strictly better than the open-weights competition - that agent data shining through! But apparently the big labs have a better approach.

So that’s DeepSeek-V3.2. Now we turn to the math variant built on it, DeepSeekMath-V2.

As the name implies this is not the first time DeepSeek has produced a model focused on math. Back in early 2024 they released a 7B parameter model with a lot of extra math tokens in the pretraining. The real innovation there though was GRPO, which eventually became the most popular RL algorithm for RLVR, for cases where you can easily check the final answer against a known ground truth. It’s a bit of an aside so if you don’t understand that part don’t worry about it, but if you do it’s a fun little part of DeepSeekMath’s lineage.

Anyway, this second version is bigger and better but also has a different focus, namely proofs. And the innovation isn’t on the training algorithm this time, it’s on the training data.

They start with two key insights: one, the skill of verification is a stepping stone to the skill of generation; two, real mathematicians submit their proofs to many verifications, and even verifications of the verifications. So build a verifier that can also verify verifications, then use it to train a generator, then use the improved generator to make harder proofs for the verifier to train on, in a virtuous cycle. And just like with peer review, running enough different proof attempts with enough verifications on them should generally yield the truth for any problem.

The first goal of a working generator is here. Starting with DeepSeek-V3.2, train a verifier using some human data. In this case, each example contains a problem, a proof, and a score for that proof: 1 if it’s correct, 0.5 if it’s mostly correct, 0 otherwise. The verifier earns a reward if its proof score matches the human’s proof score.

In parallel, train a meta-verifier on some other human data, but this time it’s problem, proof, verification, and a score for that verification: 1 if the verification cites only real defects., 0.5 if the verification cites a few hallucinated or unreasonable defects, 0 otherwise. The meta-verifier earns a reward if its verification score matches the human’s verification score.

Once you have a working verifier and meta-verifier, you use the meta-verifier to reward the verifier, so you don’t need human verifications anymore.

Then you pull the same move again, this time to make the generator. Specifically, you take the verifier and have it generate proofs, using self-verification steps to double-check its own work and fix errors. You also take another copy of your verifier and use it for verification, as a reward signal. With enough of that training you get a good generator.

Finally, with a generator and a verifier in hand, you can kick off the virtuous cycle.

All the details are here, but really the idea is that with enough eyes, all bugs are shallow: throw enough generation and verification attempts at it and usually the truth will emerge. It doesn’t work right away, but after each cycle they checked whether the majority vote agreed with human experts, and eventually it did. So then as long as you have problems to feed in, you should get proofs out. Very Bitter Lesson of them.

Here’s what they end up with. They’re beating basically everyone at proof generation, although GPT-5.1 and Opus 4.5 aren’t on here. Deep Think is a mode of Gemini 3 so that is on here. IMO, CMO, and Putnam are all elite math competitions, where DeepSeek is also cleaning up.

For our last paper slide we actually need to come back to the first paper, for a variant of the model they call Speciale.

Here they made two changes. For one, they reduced the length penalty, and as you can see by the token counts in parentheses the Speciale responses are all much longer than the responses from the thinking mode of the regular model.

For the other, they trained on an extra dataset: the one produced for DeepSeekMath! And indeed it ends up with the same performance on the IMO 2025 benchmark. So the DeepSeekMath paper is really just a part of the DeepSeek-V3.2 paper.

My Takeaways

• I like the addition of learning to the attention mask, but it seems more like an optimization than a breakthrough

  • Full attention is king

  • I would believe mixing local and global attention layers has no downside within certain ratios

  • If we believe The Bitter Lesson then the issue of cost or of compute scarcity is temporary anyway

• Per-prompt rubrics could improve the verifier

• Is there a limit to their verification system? If it’s so reliable shouldn’t it be able to produce new proofs and solve long-outstanding problems?

  • Math journals are in for a new wave of slop

• These tricks could be adapted into data pipelines

Blog post · Paper · Chat

Background

So what is AI2, the organization behind this paper? The Allen Institute for AI is the leading US lab doing open source AI.

And what I mean by “open source” is truly, fully open: all the code, all the data, all the processes etc are totally free and available.

That’s actually quite rare; most “open” models are open weights, meaning the finished models are free and available, but the ingredients that made the models are not. So big open weights names like Qwen, DeepSeek, Llama, Gemma - these are all open weights models, not open source models.

Don’t get me wrong, it’s great that they spent all that money on the compute for training! You and me are getting great products for nothing. But it doesn’t advance the science when many of the steps are still secret. For example, the technical reports tend to be light on post-training data details, often not even saying how many examples they used.

AI2 has been doing open source from the start. In fact, their language model, Olmo, stands for Open Language MOdel. They really run with that name by the way, like Molmo is the multimodal version of Olmo, and Dolma is the data for Olmo.

So not only are they established, but they’re nearly peerless. There are a few other outfits doing open source LLMs, like the Marin team at Stanford or the LLM360 team in Abu Dhabi, but no one has the same combination of talent and resources.

Perhaps the most publicly identifiable figure from AI2 is Nathan Lambert, the author of Interconnects on Substack, which I recommend for folks who want relatively technical but still accessible AI news.

He also wrote a book this year called The RLHF Book, which I recommend for anyone who wants to build a deep technical understanding of the topic.

The Paper

That’s actually going to be it for separate background slides, because this whole paper is basically a tutorial. So I’ll weave background knowledge in, as we go from architecture to pretraining to midtraining to post-training to evals.

There are many engineering details we’re going to skip over, like learning rate schedules or GPU utilization, but if you want them they’re all right there in the paper. I just think the science and the data are going to be better uses of our limited time today.

Let me give you a roadmap here to start. We’ll discuss architecture first, which is kind of a prerequisite for this graphic, but then we’ll go as it says:

1. Pretraining

2. Midtraining

3. Long context

4. SFT

5. DPO

6. RLVR

The base model just has the pink pretraining stages, then depending on the greenish post-training stages you get one of three different models:

1. Think, which is a reasoning model that produces a long chain of thought before giving a final response

2. Instruct, which just launches into its answer and so is better for general chat

3. RL Zero, which is more for experimental use and demonstrating the effects of RL directly on a base model. We’re not going to cover that one.

First up is the architecture, which is basically the same across the 7B and 32B variants, except scaled up of course for the bigger model.

We’re working with a fairly traditional Transformer here. It has three main parts:

1. Embedding, which turns words into numbers

2. Attention, which takes every piece of input and builds a holistic understanding out of it

3. Feed-forward, which takes that holistic understanding and processes it, “thinks” about it so to speak

On the embedding side there’s usually not much to report. I’ll point out the embedding dimension here because it shows one of many places where the 7B and 32B differ. The bigger the dimension, the more detail you can capture about a token when you turn it into numbers, specifically into a vector with that dimension. So larger models have a more subtle understanding of each token.

On the attention side there are two things to note. One is this 3:1 ratio between local and global attention. What that means is that 75% of the time, they’re only letting each token talk to a small number of neighbors, and the remaining 25% of the time they’re letting all the tokens talk to each other. Local is less resource-intensive, global is more informative, and empirically a lot of models have settled on this 3:1 ratio.

The other is this note about attention heads. An attention head is like a lens through which the model views the input. Pretty much all models provide lots of attention heads so that the model can view the input in many different ways. Like one head may pick up on sentence structures, another may look at high-level themes, a third could parse code, etc. Those functions aren’t assigned or designed in by the way, the division of labor emerges naturally over the course of training. And the function of each head is often inscrutable to humans.

Anyway, the architecture difference related to attention heads is in that note about key & value heads. In the small model, each attention head gets its own key-value pairs, basically its own set of notes about what it has observed. More notes means more memory required. In the big model, the memory demands are already higher because it’s a bigger model, so to compensate they make attention heads collaborate on notes, in this case one shared set of notes for every five attention heads.

So that’s the attention bit. Then on the last part, feed-forward, this is a dense model, it is not a mixture of experts. You can see they scaled up the intermediate projection size on the big model, basically giving it more space to think, but the rest of the details are shared and not interesting.

The only other thing I’ll note is the number of layers. The 7B model does the Transformer block, which is attention + feed-forward, 32 times. The 32B model does it 64 times. More blocks generally means better reasoning but also makes training more finicky since learning signals have to travel for longer.

So once you’ve got your architecture all coded up, in theory you can start pretraining. But of course, you’re going to need your pretraining data. And getting that in order is no easy task.

Now compared to post-training data, which is what Scale makes, pretraining data is much simpler: it’s just examples of text, including natural language, math, and code. It doesn’t have to be “right” in the sense of a test question, it just has to be a reasonable example of text you’d want your model to produce. The tolerance for errors like incorrect facts or buggy code is high.

The real challenge here is cleanup: of all the text you can find, mostly from the internet, how much of it is even remotely helpful?

Depending on the source, shockingly little! So for Common Crawl for instance, which is basically a scrape of the entire internet, 85% of it is junk: highly repetitive, super spammy content, from known bad sites, boilerplate, error pages etc. Just on that heuristic filtering step they go from 253 billion documents to 39 billion documents.

If you then aggressively deduplicate, including close matches, you reduce by another 75%! So under 4% of the entire internet basically is actually useful.

Now for the 8T English tokens they get that way, they’re able to add another 1T tokens from the other three sources: academic PDFs, GitHub repos, and a couple math pretraining sets. But even with the PDFs they lose just over half their starting pool from filtering.

Filtering takes a lot of iterative work, but after that it’s more straightforward. They tag every document with one of 24 topics and one of 20 different quality levels.

Then they take that dataset and make a mix by sampling based on topics and quality. For topics, they experiment by training very scaled-down versions of Olmo 3 with very scaled-down versions of the pretraining dataset and testing on some benchmarks. So for example they might find that if they sample documents related to programming more, their coding benchmark scores go up. Within a constraint for an overall number of pretraining tokens, you figure out a balance across topics that may look different from the natural distribution.

For quality it’s a different story of course, because you clearly want to sample more from the better documents and less from the worse documents. Through a similar set of experiments to topics, they settle on dropping the bottom 40% of documents by quality, then gradually increasing sampling by percentile, with the top 5% of documents by quality getting sampled seven times.

So here’s the final mix they end up with. Remember the 9T tokens is what they filtered down to, before any sampling decisions. So most of the tokens are from the internet, then academic documents, then code, then a bit of everything else.

One small thing I would note is the number of tokens per document in the original pool compared to the final mix. Just looking at the bottom row for instance, you have an average of 1000 tokens per document to start and about 1500 tokens per document at the end. Long documents tend to be higher quality, although sometimes model and people alike conflate length with quality.

I also wanted to briefly show what pretraining actually looks like for a researcher. The graph here shows the loss as training progresses, with lower loss meaning better predictions. So at the start you see really quick improvement, as the model goes from basically trash to something vaguely coherent. And then the rest of the time you get this slow march of progress, with lots of movement back and forth around a gradually improving trend.

And when I say gradual, I mean gradual. This pretraining run took 45 days on 1024 GPUs, consuming over 90% of their total compute and costing over $2M. By contrast, post-training takes a fraction of the time and an even smaller fraction of the compute.

So with pretraining done, we move on to midtraining. The mechanics are the same - we want the model to get better and predicting the next token - but the data shifts.

Specifically, the document look a lot more like chats, like question-answer pairs or code completions etc. A lot of the work involves taking a source document and turning it into chat-ish data, like having a model write a bunch of questions and answers based on a nice Wikipedia article. There’s also a lot of SFT-like data, like the reasoning traces in the Thinking section here, but used as pretraining rather than post-training data.

Also similar to pretraining, they collect a big corpus and whittle it down with filters and sampling. So the 2T token pool becomes a 100B token mix, focusing on certain abilities and balanced using the exploration + assessment workflow shown on the right. We don’t need to get into the details, but broadly it’s the same idea as in pretraining, using small experiments to then extrapolate into bigger results.

The last stage before we have a complete base model is context extension. Models almost always start out with a very small context window, 8192 tokens in this case. That’s enough for a normal request or a short conversation, but quite limited for tasks involving a lot of input like agentic coding or document analysis.

Mechanically it’s quite easy to increase the amount of context you give to a model, but the model needs training to understand how to deal with all that context. That’s what we do in this stage. So we take documents of increasing length and pretrain on them, which gradually extends the amount of context the model can make sense of. We also keep some short documents in there to balance the learning, otherwise the model will regress on shorter contexts.

The context window of Olmo 3 is 64k tokens, which is why they didn’t include any longer documents in the final mix. But they include them in the starting pool so that other researchers can experiment with them.

So checking back on our roadmap, that gets us to the base model. Now we enter post-training.

Again, across all stages they’re going to spend a fair amount of time gathering and cleaning the data into a mix they call Dolci. Since they don’t have a big data annotation budget, everything is either going to be open source or synthetic.

First they start with the skills they want, tending mostly towards STEM with some baseline chat abilities and that tool use that will be so helpful for agents.

Then comes filtering: non-commercial license, incomplete reasoning chain, non-English, explicit mentions of other models, irrelevant requests like image generation, etc.

After that comes mixing, again using small experiments to extrapolate. Throw in decontamination against the benchmarks and you’ve got your post-training mix.

Here’s the SFT mix. Nothing crazy on here, I would just note the relative size of each category and the overall count of about 2.3M examples.

Now here’s the DPO mix. DPO uses preference ranks, like RLHF but technically not RL under the hood. They make preference ranks by getting one response from a big model and one response from a small model in the same family, in this instance Qwen3, then assume that the big model’s response is always better.

And finally here is the RL mix. For prompts with verifiable answers, they do RLVR as shown in the top-right. For prompts without verifiable answers, they use an LLM judge to assign a score, usually Qwen3 32B.

While we’re on the topic of RL, I wanted to show another example of what researchers are actually watching as training progresses.

All these graphs are showing how the reward given by the verifier or LLM judge changes over the course of training. Each step is where we’ve updated the model and run our suite of RL prompts again to see how performance has changed. Higher reward means the model is getting more prompts right.

As you can see it’s always a noisy trend, sometimes almost more noise than signal. The overall trend for a good run is generally a curve up and then flattening out, although the code reward at the top is a little bit scary.

You’ll notice these graphs have different numbers of training steps. I’m not sure how to square this with what they wrote in the paper though. They mention that Olmo 3 got 750 RL training steps over 5 days, but that they continued up to 2300 steps for 21 additional days and released that version as Olmo 3.1. And apparently they expected further gains with more training, but they ran out of training budget.

Anyway, my guess is these exact curves are from smaller experimental models.

So here’s how each step nets out compared to the open weights and open source competition. The graph generally speaks for itself, but I would note two things.

One, this graph completely depends on the evals you choose. They talk a bit about how they chose their list, but you could easily make a good argument for a different list, and maybe all the sudden Olmo 3 is a lot worse than Qwen3. I do find it a bit suspicious that Olmo 3 is so close to the open weights competition here.

Two, some improvements don’t show up on this graph. An easy example is long context training, which visually seems to make no difference at all, but if you test specifically on long-context tasks you’re going to notice the difference right away.

Certainly this is an improvement on Olmo 2, and almost certainly it’s the best open source model out there. Otherwise I’m not sure what conclusions to really draw.

If you want to break open that aggregate score here’s what you’ll see for the thinking model. Again it seems roughly on par with Qwen3, slightly behind the minor upgrade of Qwen3 VL, but in the ballpark anyway.

Same general story for the instruct model. You’ll see a big jump on the AIME 2024 and 2025 benchmarks, but I think that’s more about Goodhart’s Law than true specific progress.

My Takeaways

• The split between SFT and RL has flipped

  • When RLHF was popular, the rule of thumb was 10x RLHF compared to SFT

  • The flip is likely due to RLVR (and Rubrics) taking over for RLHF in the areas Olmo focused on

• Rubrics are still trickling down

  • I was surprised not to see any mention of them in here

• Agents only somewhat overlap with ASI

  • The core components of intelligence do not significantly improve with agentic training

  • Agent work (e.g. environments) is an additional area of work for us, not a substitute for e.g. hard reasoning

• Data is a big deal :)

Paper · Repo of references

Background

So this is a survey paper about agent memory, but to lay the groundwork I’d like to start with human memory.

In common parlance I think most people have long-term memory in mind when we talk about human memory. Folks with a bit of psych knowledge will likely distinguish between short-term and long-term, and will probably only distinguish those two by time, maybe like remembering what you had for breakfast as a short-term memory - which is wrong by the way.

The reality is more complex. For one, short-term memory is better described as working memory, which is the number of things you can keep in your head at once. A very simple test of someone’s working memory is giving them some random numbers or words and then asking them to immediately repeat them back. Most people can do 5-7. That working memory is a fundamental constraint on how much information you can take in and process and potentially convert into long-term memory. It also gets used in basically all cognitive activity, like reading or playing video games or driving a car. Stuff outside the working memory gets lost.

For our purposes, the context window is the working memory of the agent. That includes the kv cache, which we’ll talk about in a minute.

Now for long-term memory, we have a granular breakdown here that doesn’t map directly onto agents, like there’s no one component that has to store facts vs another that has to store tasks. But keeping all the different versions of long-term memory in mind is going to help us pick which form of memory to augment our agent with.

One insight this chart doesn’t capture is how long-term memory complements working memory. The more concepts and ideas you have in long-term memory, which has indefinitely high capacity, the more powerful your working memory can be, since it can manipulate those preexisting concepts and ideas rather than constructing them on the fly from scratch.

A good example is math. If you already know how to add automatically, like without any working memory effort, then you can learn to multiply by using the working memory to process the concept and do the problem. But if you’re not good at adding, or you only know how to count, then you’re going to fill up your working memory before you can start to grasp multiplication.

In agent world, if your agent has a memory of a certain skill or process let’s say, then you don’t have to spend precious context window explaining it and giving examples. Of course if your memory mechanism is just retrieving text, then you don’t get any gains, but as we’ll see there are other forms of memory.

Now we kind of glossed over this, but LLMs and agents don’t intrinsically have memory. That’s part of what makes benchmarks and leaderboards work, you can ask the same question repeatedly to test the model - no risk of memorization, unless the benchmark is in the training data of course.

But a lot of chatbot websites do have memory. For example, ChatGPT rolled out memory to all users by default in April 2024. The way it works is there’s a system that reviews your responses for facts and preferences, then records those as text in a database. The LLM itself does not remember anything, it just gets access to this database that the memory system is building and refining in the background. The memory system is surely also ML-based, but it isn’t the GPT you’re talking to.

Then when you’re chatting in the future, it probably does a combination of system prompting and RAG. In other words, it’s going to put some memories at the top of every conversation, but for others it will fetch based on context. So let’s say you mentioned once you like concise responses and the memory system captures that. Now every chat might say in the system prompt that the user prefers concise responses, which will guide the model’s behavior and also could come up in conversation. But let’s say you also noted you’re a vegetarian, and in one chat you’re asking for restaurant recommendations in a foreign city. The memory system will find that note about you being vegetarian and provide it to GPT so you get more relevant results.

Of course that’s in addition to the memory of past conversations, where the model can look through earlier chats and get caught up. But that’s more like a Google search than proactive memories.

Anyway, this type of memory is pretty rudimentary. It’s just a certain type of tool use at the end of the day, reading and writing to this little database. And it’s only capturing facts, which as we saw from our chart of memory types is a small slice of what memory entails.

So memory in consumer chatbots is kinda new, but the idea of supplying external information to a model at inference time is not. Other than just typing in whatever information you want to provide as a prompt, the most established method for providing relevant information is RAG: retrieval augmented generation.

The basic idea is simple: take a bunch of data, put it into a form you can search, and then run a search with your prompt to get any relevant information. Kind of like doing a Google search and looking at the top search results before writing your own response to a question.

Of course if you have a bunch of relevant information in front of you, that’s going to help your response. The tricky parts are how to put the data into a form you can search, and doing a good job searching.

The first part is at the top here: chunking, which means splitting up long documents into much smaller pieces; and embedding model, which turns the chunks of text into the same type of vector a model uses when it takes in text. The original chunk, plus the vector that represents its meaning, go into the vector store.

The second part is the connection between the top and the bottom, where you turn the prompt into a vector with the embeddings and then compare the prompt vector with the chunk vectors. The more similar the prompt and chunk vectors, the more relevant the chunk is to the prompt. The most relevant ones come back as text, then the LLM responds based on the prompt and the retrieved text.

The original RAG paper is from 2020, over two years before ChatGPT came out, so the memory solutions we’ll see often descend from or still use RAG.

I also want to quickly recap agents. The term “agent” has picked up a lot of cruft and buzz, but it’s really pretty simple: it’s any system that takes actions in an environment and gets observations back from the environment, in a loop until it has completed its task. Usually when people talk about agents nowadays they mean an LLM with access to tools that has a system prompt telling it to be agentic, like to complete tasks for the user.

So in this framework, an “action” for our purposes is going to be using a tool, like web search or file access etc. Requesting or writing memories could be an action, and receiving a memory back could be an observation. However, not all agentic memory is tool use.

Now for the most technical parts, we need to get familiar with embeddings and the kv cache.

First we have to recall what a Transformer-style LLM is, which is a series of matrix multiplications. Training a LLM is just adjusting the numbers in these matrices. Using a LLM is just turning words into numbers, into vectors, that you multiply through all these matrices.

One of those matrices is the embeddings, which turns input tokens into vectors. Each token in the model’s vocabulary will have a unique place in the embeddings, a unique vector that pops out when you plug the token into the embeddings. You can also reverse the process, feeding in a vector and getting out the corresponding token, which is what happens at the end of the LLM when you’re ready to switch back from numbers to words.

While tokens are discrete though, embeddings space is continuous. So if you’re doing that reverse process and you just pick a series of numbers to go in the vector, it’s pretty unlikely it will match up exactly to one token. But that vector will still have a meaning - it will just be a mixture of meanings that doesn’t map cleanly to one single token. Through a certain lens, you could view that mixture of meanings as a kind of memory. And if you wanted to save that memory you could make a special new token that maps to that mixed vector.

I think of this sort of like how a smell can instantly evoke something deep and subtle in a way that’s hard to describe, or that words couldn’t exactly access. The meaning is there but sometimes it takes a special input to activate.

Now for the kv cache.

So in the attention part of the LLM, you take your input and multiply it into three different quantities: the query, the key, and the value. That’s what the middle section is showing, that each input token gets turned into Q K and V.

The actual formula for attention is also in this section: Q times the transpose of K, with a transformation called a softmax. That tells you how important each combination of input tokens is. Then that whole thing multiplied with V to give the meaning of that combination.

The key fact we need to focus on is that to generate the next token, you only need the query for the last token, but you need the key and value for all the prior tokens. So that means if you keep the keys and values in memory, you don’t need to compute them again. That is the kv cache.

The kv cache is like the model’s conscious state of mind, carrying over and changing with each computation. But when you turn off the model the kv cache goes away, like when you go to sleep and lose consciousness.

Since the kv cache is where information about past tokens lives and persists, if you change the kv cache you’re changing the model’s understanding of prior inputs. The normal way the kv cache changes is by continuing to process inputs, like your normal stream of consciousness changes as you continue experiencing. But if we go in there and fiddle with the kv cache in a different way, we can alter the model’s “state of mind” and “thoughts” about the past - its memory.

The Paper

So first we should clarify what agent memory entails, because it does overlap significantly with many other concepts.

First is our old friend RAG, which we covered before. The general idea of reading and writing to a database works for agent memory too, so the main difference is what goes in that database. Traditional RAG would be a static, external corpus like a bunch of company documents, whereas agent memory would be stuff the agent chose to remember or was given to remember, usually with some way of updating and consolidating the stored memories.

Next is context engineering, which is the science of optimizing the context window. Using our human analogies from before, that’s going to be the working memory. So deciding what goes in the context window, how it’s structured, when to remove it maybe, when it’s about memories in the context window that’s going to be agent memory. Of course there can be other things in the context window that aren’t memory, like tool use, so that’s in the non-overlapping part of the Venn diagram.

Finally there is LLM memory, which again will have tons of overlap because the LLM is the heart of the agent. Prompting strategies and KV cache stuff is shared, but LLM architecture stuff like linear attention isn’t related to agent memory. Basically anything intrinsic to the LLM falls under LLM memory, while any ancillary system or technique will be part of agent memory.

So as you can see from this graphic, there is a shitload of agent memory techniques. We are absolutely not going to go over every one of them, but we’ll talk about their categories and maybe mention a few of the most clever techniques.

Let’s start with the categories in red, the “functions” as the graphic calls them. We already saw these in the human context so there’s not much to add here, just reviewing. Factual memory is gonna be facts and figures. Experiential memory is gonna be principles, strategies, tactics, tips, and tricks. And then working memory will be anything in the current state like the chain of thought or the kv cache.

The other axis, forms, uses LLM-specific terms but has human analogs.

• Token-level is like taking notes, assuming for now that our model isn’t multimodal. That is by far the most common and most researched memory technique. It’s also nice because it’s legible to humans, you can go in and see what the agent is remembering.

• Parametric is like actually forming a memory in humans. It’s in the model’s weights, or the weights of an adapter or ancillary model. That’s great for memories you want globally incorporated, like a tone or a strategy etc, and it’s going to be faster than token-level. The major downside is it’s illegible, although in some cases that could be useful.

• Latent is like your current state of mind. I will spend a bit more time on this later because it’s the least intuitive and most technical of the three.

They don’t show it in this graphic but they also have categories for structure, like unstructured vs a graph or tree vs a pyramid of increasing abstraction. I’m going to ignore that one for the rest of this presentation but if you’re interested it’s in there.

So to expand on the Latent form, there are really two different mechanisms you can use.

One is the embedding space. As I mentioned in the background slides, embeddings encode meanings into vectors. That means any vector with the same size as the embeddings matrix will have some sort of meaning. In the case of tokens, we have a pretty good idea of what the vector version means, because we know what the token means, like if the token is “dog” then we know what the vector is going to represent.

So the trick with the latent embeddings is to take something more complicated, embed it, and save that vector for later. In this drawing they show an auxiliary model producing that vector or those vectors for later, the bit labeled “latent embeddings”. Then when you use your main model, you pass in your prompt AND the latent embeddings, and suddenly your model gains these memories, these extra meanings as context with the prompt.

The other mechanism is the kv cache. Again, it’s like a state of mind, except in models you can swap them in and out instantly. If all you do is pop in a saved kv cache from before, that’s what they call Reuse. If you’re somehow fiddling with a saved kv cache, they call that Transform. Either way, if you swap in a previous state of mind with all these memories, again the prompt is going to gain new context.

In both cases, you’re using the power of models to distill context into a more compact representation. That’s really helpful if your source material is highly compressible but doesn’t contain obvious stuff to trim out like boilerplate text; you let the model do the compressing once and then reuse it many times.

One other thing to note is that compressibility depends on the model. Like think about two college students taking the same upper-level math class. If one of them is a freshman who doesn’t have the prerequisites and another is a math grad student just filling in a hole in his knowledge, the freshman is going to take way more notes because he can’t rely on the shorthand of earlier or related concepts. Whereas the grad student probably will take a few really compact notes that only he can decipher because they draw on his richer background knowledge.

Here’s a bit more about the relative advantages and best uses of each memory form.

For token-level, there are a few advantages and use cases. First, a lot of data natively is token-level, like anything you’d keep in a database, or facts and figures, reference material etc. Just like how a human doesn’t need to memorize or internalize things like dates or addresses, agents can use this external memory effectively. Second, it’s going to be legible as we mentioned. And third, it’s by far the simplest to implement.

For parametric, the best fit is broad, conceptual, implicit knowledge. They mention role-playing on here, that’s an easy example since character traits are going to be broad and kind of filter everything, not something to look up at clearly defined times.

For latent, as a middle ground between token and parametric you get some of the good and some of the bad of each, but the main way to think of it is a more efficient representation of some prior input. So it’s going to shine in resource-constrained environments. And as they mention it’s a better fit for multimodal because it can fuse the modalities, whereas for tokens you need separate tokens for each modality.

Now regardless of form, you have to deal with memory formulation, evolution, and retrieval.

Formulation means turning raw data into memories, as the graphic notes. That’s probably the most intuitive of the three here and frankly isn’t that specific to agents, like there’s already a lot of best practice about data structures.

Evolution is more challenging, as anyone who has maintained a knowledge base or wiki or any form of documentation knows. They have another graphic about that so we’ll save it for the next slide.

Retrieval is the most agent-specific in my opinion. Like human memory is an unconscious process so it’s a little funny to have to design it for agents.

When to retrieve is the biggest one. Like if the user says “Hello” that probably requires no memory, but if they want help planning a vacation then clearly you want to know how old they are, whether they have kids etc. But you can’t enumerate that whole list of course. Heuristics work okay, retrieving every time and then just letting the agent ignore everything if it’s not helpful is fine but wasteful, so letting the agent decide is often the way to go. And then you can use RL to improve the decision making.

What and how to retrieve is also interesting. The naive approach is to use the prompt directly as a search query, but people have developed way fancier methods: rewriting the prompt to be a better query, breaking down the prompt into its elements to be sub-queries, writing a hypothetical document based on the prompt that then serves as a query. You can also use a sub-agent just to do the retrieving, letting it iterate and experiment until it’s satisfied. All depends on your quality, speed, and cost requirements.

You also need to process the results before sending them back to the agent. Like if there are duplicates or near-duplicates for example, or conflicts. You also may want to rerank or filter down to save on context or tokens.

Jumping back to memory evolution, there are tons of techniques for consolidating, updating, and forgetting memories. Again, all unconscious work we take for granted in our brains that needs designing for agents.

Consolidating means taking new information and cleaning it up, then putting it into the right place in the existing body of memory. After that you have to check the rest of the body for conflicts or updates, and decide how you want to resolve them: deletion, archive, timestamping etc.

Now eventually all minds fill up, and some forgetting occurs. How to forget gracefully usually depends on the case, but time limits, frequency minimums, and value filters all work, often in combination.

The paper wraps up with a peek into the future, including this somewhat confusing graphic.

One thing they look forward to is improved memory generation, like taking the base information from the memory store and enriching it on the fly, then storing it enriched later on. For example, if it’s a data analysis agent and it tends to have GTM folks as users, it might add new sales strategies to its memories about key tables.

More autonomous management is also on the table. Instead of relying on rules like a time cutoff for forgetting or a priority in conflicts, the agent can make the decision using its own intelligence and the rest of its memories.

Of course once you introduce an ability, you can start improving on it with RL. In the long-term I would expect, and the authors note on this graphic, no human prior on the memory system - something optimized for machine intelligence, not human intelligence. They call RL-driven memory “the next major stage”.

Multi-agent memory is a relatively new challenge, now that agents are actually practical. Here I do expect human priors to provide guidance for a while, since there is a lot of best practice and tooling for sharing knowledge in organizations. Again though, the optimal system for agents to coordinate will probably look strange to humans.

Relatedly, one concern now that will only grow over time is auditability, traceability, legibility - anything that allows humans to understand and debug agents related to memory.

So What?

Paper · Repo

Background

Let’s start today with a look at the lab behind this paper, namely NVIDIA. As we all know, NVIDIA is practically synonymous with AI these days. Here’s a graph from Epoch AI showing the total amount of AI compute capacity sold, starting in Q1 2020 and ending in Q4 2025. They normalize the amount of compute to equivalents of the H100, an advanced NVIDIA GPU.

While there are other chip makers in the game, all of them combined have sold just over 30% of the world’s compute. The remaining 70% is all from NVIDIA. It’s no surprise then that NVIDIA is the most valuable company in the world, and has been for some time now. It’s also no surprise that Alphabet, the parent company of Google, recently became the second-most valuable company in the world, in part due to their tensor processing units or “TPUs” starting to seriously compete with NVIDIA’s GPUs.

Relatedly, Amazon is the 5th most valuable company in the world as of writing. All those companies are worth $2-4T apiece. AMD is around $330B and Huawei is estimated at $100B, although they’re not a public company.

Anyway, that stock value is largely pricing in future sales, with a price-to-earnings ratio of around 46. For Alphabet is about 33, similar to Amazon and other tech companies like Microsoft. AMD’s P/E ratio is over 100! So the market expects lots of NVIDIA chips to be sold in the future, but also expects some other players to make a dent.

So where is all that demand for AI chips gonna come from? Well part of it is definitely training, so that’s the big foundation model labs like OpenAI and Anthropic and xAI that don’t make their own chips, i.e. excluding Google and Amazon. But a bigger part of the demand is inference, i.e. people using models. Even OpenAI and Anthropic spend a significant amount of their total compute on inference, since their revenue is roughly proportional to how much use their models get. But anyone who’s only serving models, or who maybe is doing some post-training, is using zero or nearly zero training compute, and that’s going to be the vast majority of organizations. In business lingo, the total addressable market or “TAM” of inference compute is bigger than the TAM of training compute.

So if you’re NVIDIA, you of course want to drive all forms of GPU demand, but the business wisdom here is to “commoditize your complement” - to make the other stuff in your bundle of goods as cheap as possible to drive the overall cost down, which increases demand due to lowered costs, but by definition doesn’t hurt your prices.

The most famous example of this is the PC market from the 90s. Microsoft, being the maker of Windows, wanted everybody to have a PC and to buy a copy of Windows. But PCs used to be specialty items, with differences in quality and architecture etc. In that non-commodity PC world, consumers had to spend more, and Microsoft had to make different versions of Windows to work on those differentiated PCs. So Microsoft worked hard to commoditize the PC market, making everything standard for Windows and forcing manufacturers to compete primarily on price.

NVIDIA wants the same thing. They want AI to be a commodity, they want model providers to compete on price, they want the inference market to grow. And one way they do that is by permissively releasing open-weights models, which gives anyone with a GPU a no-cost way to use AI, thus driving up usage. That setup isn’t going to compete with SOTA models on raw performance, but they can compete on performance per price.

NVIDIA has released a lot of models, including a lot of fine-tunes of other open-weights models like we’ll see in this paper, but their main family of releases is called Nemotron. They’re on version 3 of the series, all completely trained from scratch. They have released the Nano size, which is a 30B mixture-of-experts model with 3B parameters active, and plan to release two larger sizes soon. You can also see a post-train of Qwen3 here, for use as a reward model.

Now while the Nemotron series will compete with the likes of Qwen3 for the open-weights SOTA crown, NVIDIA has also released a lot of smaller models focused on reasoning and tool calling. These small language models or “SLMs” are great for enterprise work where the logic is custom but often not complex. For example, it doesn’t take PhD-level reasoning to understand how to join a couple of tables in a company’s database, but it does take understanding and implementation of that business logic, and familiarity with SQL or perhaps a specific analytics tool.

NVIDIA made waves in certain circles last year with a paper called “Small Language Models are the Future of Agentic AI”, laying out this exact case theoretically but not running any experiments. Here’s an excerpt from the abstract: “Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice.”

In case you didn’t catch that, what they’re saying in layman’s terms is that LLMs are usually overkill in enterprise work, and that if you really need that SOTA LLM level of intelligence, you can just call it when you need it.

The featured diagram in their paper shows two approaches to agentic systems. On the left is one language model for the user to interface with that is also calling some tools directly and also employing another language model as needed. On the right is a language model for the user to interact with, which then offloads coordination work to a controller, which then uses tools and the other language model. Keep both examples in mind.

I also should quickly describe the key benchmarks the paper uses. One is HLE, which hopefully we’re all familiar with, but if not it’s basically hard reasoning with optional research tool use.

The next one is FRAMES, by folks at Google. It involves reasoning but leans much more on tool use, for checking facts and doing research. Basically a RAG benchmark.

The last one is Tau Bench, or more specifically its successor, Tau-squared Bench. This one simulates conversations between users and agents in retail, airline, and telecom settings. You can see a telecom example here and some stats from each domain below. This benchmark is by some folks at Sierra, a big customer support agent company, and includes one of the authors from SWE-bench.

The Paper

Okay, so here’s what they’re up to in this paper. They’re basically applying the theoretical argument they made in the Small Language Models paper and training Qwen3-8B to be an orchestrator, which will talk to the user and use tools plus other models to provide an answer.

I want to note a few details here. As the graphic notes, they give a positive reward for correctness, a negative reward for cost, a negative reward for wall time, and a positive reward for any additional user preferences. In the example the user’s preference is also about cost, which makes it kind of a lame example honestly, but you could imagine other preferences like sticking to basic tools for maximum determinism. They use GPT-5 as a judge and GRPO as their RL algorithm.

The second detail to note is the three categories of tools and their partial lists of examples:

1. Basic tools, which is what you normally think of when discussing tool use with LLMs - any sort of command or API or MCP etc. If it’s an external service, the orchestrator will get information about its cost per query

2. Specialized LLMs, which are models either trained or prompted to do specific stuff. Like the Qwen Math series or Codestral for example, those are versions of Qwen and Mistral that have received extra training in math and code, respectively. In my view this is somewhat of a fake category, except for coding specialist models, which a lot of folks still regularly release

3. Generalist LLMs, which are just other models the orchestrator can use like you or I can. These “tools” are going to be the most expensive

The researchers made prompts that naturally require multiple steps to address, giving the orchestrator many opportunities to pick the optimal tool. We’ll see more about that on the next slide.

So being NVIDIA and having basically infinite access to compute, they made a lot of synthetic data. Here they track an example of creating synthetic data for a single domain, “movie booking”.

First thing they do is generate a database, including the fields like movie time and total price, and of course the data within the database.

Then they generate tools that would operate on that data. Here they give examples like a cancel tool and a refund tool.

Then for a random sample of tools, they generate “intentions”, or what we would call “prompts” - things the agent should do with the tools and the data.

After that they turn intentions into tasks by randomly sampling database rows to go with the tools and intention, and then solving the task. That golden set of functions calls and key information, plus the prompt and data and tools, is one sample in the dataset.

They also do some enhancing and filtering, like adding constraints and checking that the task actually requires at least one tool call to solve.

Once they have that complete eval dataset, they can check an agent’s trajectory against the golden tool calls and key info, in addition to checking the final response or database change etc.

One last thing they don’t show in here, but which factors in at test time, is a random pricing schedule for all the tools. That’s not “golden” info since prices may change in the real world, so it doesn’t go in the eval dataset directly.

So with this dataset and training in mind, they RL Qwen3-8B into Orchestrator-8B, with results shown here.

It’s a little bit silly, but they show comparisons between Qwen3-8B and more powerful models with no tool access or just basic tool access, i.e. no LLMs available to call. Obviously the bigger models blow Qwen3-8B out of the water generally.

The real results are in the bottom row, with the full suite of tools and with Orchestrator listed. Here we see the power of SLMs. Orchestrator has the best performance, the lowest cost, and the lowest latency. On the next slide we’ll see how and why, but I want to call out a couple other results in the bottom section.

First, Qwen3-8B is already a good tool-user. The clearest way to see that is the Tau-squared Bench results, where Qwen beats GPT-5 and isn’t far off Claude Opus 4.1.

Second, the cost is the biggest improvement from Qwen to Orchestrator. So Orchestrator is using better tools, but the stronger lesson it apparently learned is to incorporate pricing information. And latency I view as a consequence of that, since cheaper tools are also likely to be faster tools.

This here is a good summary of how Orchestrator beat out the other models. Basically, it learned to use the right tool for the job. I put the cost numbers relative to Qwen3-8B at the top for easy comparison, so for example Orchestrator is only a third the cost of Qwen3-8B. And remember, Orchestrator wins on quality against every other model.

Since Orchestrator is a Qwen model, let’s compare relative changes from Qwen3-235B-A22B since they don’t give this same breakdown for Qwen3-8B:

• GPT-5 reduced by over half, mostly replaced with GPT-5-mini - this is probably the biggest cost saver. GPT-5 also employs this strategy. In fact, OpenAI already knows this strategy - that’s what the router in ChatGPT is for!

• Far more Qwen3-32B use, unclear why but if I had to guess it’s because Qwen3-8B knows Qwen3-32B is its bigger brother and thus should be able to help. Certainly it’s well known that models prefer outputs by other models in their family.

• More local search, using cheap lookup instead of thinking longer themselves or asking another model - both expensive in tokens!

  • Note that this is not at the expense of web search

  • This is very relevant for FRAMES since it’s a RAG benchmark, and fairly relevant for Tau-squared Bench since it’s in a corporate environment

• No Llama use, although the real question is why the big Qwen uses Llama at all - none of the other models use it really

So that’s the formula for success: use tools instead of models when possible, use small models instead of big models when possible, and always use the best model within the size you pick.

My Takeaways

• We will continue stacking models - “It’s models all the way down”

• Very cheap and small models will be ubiquitous, similar to how very cheap and small chips are in things as simple as USB cables

  • It will be rare to interact with a bare tool

• Models still aren’t that business-savvy

  • As demonstrated by Vending Bench

• ToolOrchestrator is a cousin of Claude Code

  • I don’t touch a command line anymore, Claude Code does it all. Let it decide what resource to use.

Paper · Blog Post · HF

Background

Let’s start by recapping recent Chinese model releases and progress.

The top two labs in China are DeepSeek and Qwen. DeepSeek produces the most original work, like with R1 at the start of 2025 or with recent hardcore engineering papers. Qwen produces the most work overall, with models in most varieties you could ask for: small or large, dense or MoE, general or coder, text or image, etc. Qwen models are the default choice for most work involving training.

Below that S tier is the A tier, companies putting out highly capable models but not quite matching the originality of DeepSeek or the volume of Qwen. Here I would put Z.ai (the makers of GLM), Moonshot (the makers of Kimi), and MiniMax. We’re talking about the recent Kimi release today of course, but by sheer coincidence both Z.ai and MiniMax have released major updates today: GLM-5 and MiniMax M2.5. Really incredible pace by all the big Chinese labs.

Now we’ll see some Kimi benchmarks that compare it to recent SOTA, but the vibe I get is Chinese models are only a few months behind the best American models. And for an increasing share of use cases, “recent SOTA” is good enough. So I think it will become increasingly common to be using Chinese models, although more often through a third-party host like Fireworks rather than directly calling the China-based servers.

Now in terms of focus, I think the lightning success of Claude Code non-code use and then more recently OpenClaw gives a good idea of where we’re heading and what the underlying models in agents will need to train for. We’ve reached an inflection point where models have enough smarts and skills to do most generalist computer work: browsing, researching and the like. Domain-specific programs, spreadsheets, decks etc I think need a lot of work or specialized harnesses and training data for now, but everyday computer use is happening now. As we’ll see in this paper, those particular skills but also the general approach to agentic work are hot areas of research and progress.

Finally, I’d like to harken back to a paper I covered last month called ToolOrchestra. The good people at Nvidia took a pretty lightweight model, Qwen3-8B, and trained it to tackle complex queries by calling other models and some dedicated tools. Even a simple model trained for the specific task of orchestrating bested all other models, even far larger and smarter ones.

So it seems orchestration per se is a skill, not something we can take for granted as part of general intelligence, and that it has significant advantages for cost and latency in addition to at least marginal benefits for quality.

The Paper

Let’s talk about architecture first. This is a big mixture of experts model, as all frontier models are nowadays. 1T is on the high end, maybe the biggest open weights model available today. 32B active parameters is proportionate. 61 layers is pretty normal, the more layers you have the more thinking you do but also the longer latency is. There’s also a practical limit on depth because signals you get during training get attenuated over all those layers, so if you have too many of them you basically never teach the earlier layers anything.

Another stat or ratio to check out is the number of experts compared to the number of selected experts. That’s the sparsity, which is 48 here and is on the pretty high end. I also want to call out the 256k context length, also on the high end - 128k is more common and is also where Kimi K2 topped out at. So really big, super sparse, pretty long context - that’s Kimi K2.5 from an architecture POV.

Now Kimi K2.5 is actually an update to Kimi K2, meaning they took the K2 base model and trained it more.

Specifically, they got a vision encoder working first, then added it to K2 and trained the new system for 15T tokens. That’s on top of the 15.5T tokens Kimi K2 already got, meaning we’re just over 30T tokens total. The third stage here is about lengthening the context out from 32k to 256k, which happens in steps and adds another few hundred billion tokens. So all told it’s like 32T tokens to make this beast.

In the paper they talk about how the conventional wisdom is to train only on text for most of the time, then throw in a heavy dose of vision tokens at the end, like a 50/50 split. In the extreme, you can even take a fully trained LLM and add a vision encoder on afterwards, onto a model that never knew it was going to be multimodal.

What the Kimi folks did instead is to start training both vision and text from the start, but with a smaller dose. Specifically, they do a 90/10 split between text and vision tokens the whole time. They also don’t group their training data by modality, just by skill, so in post-training you’re getting a sprinkling of vision throughout.

Also note the types of data in each stage. First off, you’ll note they do video as well as image, although no audio. Also they’re looking at a lot of computer use imagery, definitely angling for a computer use agent here.

For the SFT after this pretraining, they only spend a single paragraph on it, but they mention lots of synthetic examples from K2 and some “in-house expert models”, and also some human annotation. No sense of numbers here sadly.

Now let’s talk RL. First we’ll look at vision, then the agent stuff.

They’re training in a lot of capabilities: OCR, captioning, segmentation, counting, grounding etc. The higher-level goals of understanding and processing sit on top of that.

In terms of how they reward, they do RLVR where possible, which covers many cases for vision. Where they can’t do RLVR, they do rubrics, although they don’t use that word. I believe the rubrics are generic rather than prompt-specific, but they don’t say. Kimi K2 is the LLM judging the outputs.

Now again, they’re not sharing anything about their data, but they do show the effect of RL. They see improvement across the board on their vision benchmarks, in pretty classic-looking curves of training vs performance improvement.

So that’s vision taken care off, which was mostly par for the course other than their insight about when to start vision in pretraining. Now we move to the more forward-looking stuff, which is agent swarms.

Here’s their overview of an agent swarm. The examples are not especially inventive but you get the gist: an orchestrator inventing subagents to give tasks to. The orchestrator manages while the subagents work. The user is only ever talking to the orchestrator.

Now for training, we’re only ever rewarding the orchestrator; it’s going to take credit for the final outcome, as well as some other decisions we’ll see in a bit. Because the subagents are “frozen” like this, we can think of them as tools in the environment rather than as part of the main agent, the orchestrator. Of course below it all every subagent was also Kimi K2.5 in this case, but like with the ToolOrchestrator paper you can imagine the subagents using different models, including API-based ones that you don’t have the ability to train anyway.

As an aside, I find it amusing in a Dilbert sort of way that the manager at the top is the only one who gets rewarded. I suspect we will discover many organizational principles and tropes don’t depend on having humans in the loop.

Now as for those rewards, I came into this paper not knowing what to expect. I think we’re just at the beginning of multi-agent RL, so I take this as just one idea rather than gospel.

Anyway, as you can see at the top they reward three things: the outcome, r_perf, which is the most important thing; the creation of subagents, r_parallel, which they call “instantiation”; and the completion of subagent tasks, r_finish. R_parallel and r_finish together are a good example of how tough reward design can be. On the one hand, you do want the orchestrator to spin up subagents rather than doing work itself, but on the other hand it’s not helpful to spin up subagents and just let them hang there.

By the way, even OpenAI can make mistakes in reward design. It recently came out that due to over-rewarding of tool use, a prior version of ChatGPT would open the calculator tool, do 1+1, then close the tool and answer the user’s query. Apparently that was happening on something like 5% of all queries at one point, which is huge in absolute terms.

Anyway, how much of each reward you need in order to balance the other out and avoid reward hacking is an empirical matter, hence the weights lambda_1 and lambda_2. Over the course of training, lambda_1 starts at one and goes to zero as the orchestrator durably learns to call subagents. Lambda_2 starts at zero and goes to one as the model gets better at giving work to subagents, so the tolerance for poor subagent use goes down.

The second formula calculates how parallelizable a task is by measuring the critical path. If you’re not familiar with that term, the critical path is the set of steps that determines the length of an overall project. So let’s say I want to make dinner, and my entree needs 45 minutes in the oven. The critical path is likely just the time it takes to prepare the entree, then that 45 minutes in the oven, then any cooling and plating time after. Other courses can happen in parallel, like making the salad while the entree is in the oven, so those are not on the critical path.

Bringing it back to agent swarms, you have hit maximum parallelism if the overall time is equal to the critical path time. To measure the critical path time, for each round of decisions, you find the longest-lasting individual subagent task. Then you add those up for every round to get the critical path of the entire task. If your time matches the critical path time, you get full points on parallelism.

Note that the prompts they’re going to train on don’t say anything about parallelism per se. There are tools implying parallelism is a good idea, but the system prompt or whatever doesn’t say like “make sure to divide up your work”.

So here’s the payoff for all this reward design. On the left we see accuracy on their training set increasing with compute. They don’t give exact amounts of training data but they hint it’s in the tens of thousands of examples. It’s all synthetic prompts, emphasizing “wide search” (many independent tasks) or “deep search” (multiple branches and then aggregation).

On the right we see the level of parallelism, which they measure as the number of overall steps divided by the critical steps we saw before. So if all your steps are critical, your parallelism would be 1. Note that different tasks vary in their maximum parallelism, like some can be totally run in parallel while others may have only a few parallelizable steps. But since we’re seeing the same tasks many times, overall you can accurately draw conclusions about changes in parallelism during training.

Anyway, two things to note: one, the x axis does not start at zero or one even; you’re getting good parallelism right out of the box. Two, the trend has two phases: flat–ish in the first part, up in the second. The way I interpret that is the model starts its learning on more generic things like designing better individual agents or understanding agents results better. Once it hits a ceiling on correctness, it then has to learn to improve parallelism if it wants to get every available reward. Like it only learns later on how to reason about parallelism, which it is not naturally going to be skilled at since it’s not naturally part of the pretraining data.

Of course parallelism per se isn’t valuable, but the effects are. You could expect better quality as the orchestrator maintains clean context and abstracts tasks away to subagents. You could expect lower cost from using cheaper models, or from more efficient thinking and thus fewer tokens to pay for. But I think the most sure effect is savings in time, “wall clock time” as they say. There’s gonna be two reasons for that: one, obviously parallel agents can do multiple things concurrently; but then two, models get slower as their context length grows.

Here’s a graph quantifying that speedup, with single agent vs agent swarm performance on the WideSearch benchmark, which requires gathering information from a wide variety of sources. Swarms are always significantly faster, but the trend seems to grow superlinearly. So it seems like if we want agents doing all our work for us in the future, they’re gonna be working in swarms, just like how most work is done by companies instead of individuals these days.

Then just a couple other quick remarks here. They have a word cloud on the left showing all the different subagents the orchestrator made, which are all pretty similar in my opinion and also have a lot of duplicates, like the most common one is “Biography Researcher” but they also have “Biograph Investigator” and “Biographical Researcher”. Kind of a lazy chart honestly, and they don’t provide a table of the data. I asked Gemini to extract and dedupe and group, and it gave four high-level themes: research, technical/coding, content processing, and orchestration.

On the right they compare swarms with a technique for managing a single agent’s context. Basically they’re trying to see how much of the benefit of swarms is just due to subagents abstracting work away and keeping the orchestrator’s context clean. If you only have one agent, you can basically have it remove the intermediate work from its own context. Cleaning up context like that helps for sure, but the orchestrator-subagent setup is still superior. The authors claim it’s because the orchestrator can proactively plan which context to keep and which to give away to the subagent, whereas retroactively wiping intermediate work may erase stuff the model was planning to refer back to.

Personally I think the swarm results are kind of a lowball measurement, and that the potential for swarms is much higher, whereas the context management stuff for a single agent is more like an optimization closer to the inherent ceiling. Both valuable but not comparable in the long-run.

Finally, here’s where we net out. Kimi K2.5 is, according to benchmarks, on par with Opus 4.5, GPT-5.2, and Gemini 3 Pro. That’s an extraordinary claim, and while it’s increasingly difficult to compare SOTA models, I think it’s at least arguable. I have seen one respected ML researcher say he has switched to Kimi from Opus, although he always uses one in combination with GPT 5.2. And who knows now with Opus 4.6 and GPT 5.3 Codex already out.

On the data side, one unfortunate but likely true theory as to how Kimi pulls even with SOTA on benchmarks is that the Chinese models are distilling from the American models. In other words, the Chinese labs are creating synthetic data from the American models and then training on it. The easiest evidence for this is if you ask the Chinese models who they are, you often hear they are GPT or Claude. Personally I suspect that’s part of why we don’t hear much about the data in many of thee tech reports.

Anyway, I encourage you all to experiment with Kimi. It’s available on OpenRouter. If you want it for coding you can use Opencode, which is like Claude Code, or you can hack it into Claude Code directly and it works pretty well. But it’s also supposed to be good at creative writing, so don’t restrict yourself.

Last note on this table: I’ll call out HLE and SWE-Bench Pro as Scale benchmarks, and GDPVal as basically identical to our Remote Labor Index benchmark, which I think is better and seems to be harder for models to crack.

My Takeaways

• Number of agents is another axis for scaling

• Swarm training aka MARL is in its infancy

  • Setups, rewards etc

  • But also data

• Home-runnable models may be good enough for generalist work by EOY

Paper · Leaderboard

👂

We need to start with an overview of audio in AI, because there are some unique considerations and we haven’t really touched on the modality before.

First off, I want to distinguish the different types of audio input and output. For input, I think of it as three categories: speech, music, and ambient - like sounds from the environment, “noise” you might say, but without the negative connotation.

Music presents a fairly distinct set of challenges from speech and ambient, so I’m going to put that one aside. For the rest of this talk, we’ll be focusing on speech but also allowing for some ambient sound, like someone asking to identify a bird’s chirp or noticing that the person is at a crowded venue.

So sound in general has a few interesting properties. For one, it’s continuous and analog, like vision. That means we have to sample sound, with the equivalent of frame rate for video, and we have to capture it with dedicated hardware. That’s all very different from text, which is natively digital and doesn’t lose anything from compression or sampling or hardware limitations etc.

Audio is also hard to tokenize. Like with video, the raw data is too information-dense, and a naive tokenization strategy would blow out the context window in an impractically short amount of time. We don’t need to get into the details, but it involves compression and mapping to the closest token the model builders have built in.

Lastly for audio in general, there’s just less work overall in the domain: less literature, fewer models, fewer companies working on it etc. I think that’s mostly for the reasons above, that text is just much easier to work with than audio, and that text is the universal interface.

🗣

Now for speech in particular, there are further challenges.

One is the nature of spoken interaction: it is realtime, and it is dynamic. Like when you’re speaking to someone else, you generally expect them to understand as you’re speaking, to understand when you’ve finish, and to respond in short order. You also have to be prepared for them to jump in early, or to jump in early yourself if needed. And you expect them to understand easily when different people are speaking without relying on visual cues.

This is all very different from text-based interactions. If I’m chatting with a model, I can tolerate some thinking before a response. If I need to interrupt I can stop the response from processing, but conversely I know the model cannot interrupt me. And I’m pretty much only chatting one-on-one, although even if it were multiplayer like in a Discord, the model would get my username when it got my message.

So realtime responsiveness, interruption, and diarization (the term for tracking who is speaking) are all new challenges for speech models. And those challenges present constraints. For example, a realtime speech model cannot be the size of a SOTA text model - the latency is just too high. That’s why you usually see speech on the smaller versions of models, like Gemini Flash instead of Gemini Pro or GPT-4o instead of GPT-5.

But a lot of that is on the engineering side. On the data side, the biggest challenge to address is the stuff in speech you can’t really capture in text: the paralinguistics. That’s gonna include stuff like tone, pacing, volume, pronunciation etc. All the stuff that makes speech so much richer than text.

In all multimodal data, the emphasis is on the aspects that are inherent to the medium. The anti-example I often give for vision is a picture of a quantum mechanics problem - the difficulty of quantum mechanics has nothing to do with the difficulty of visually parsing the text, which is a very easy problem these days.

It’s the same thing for speech data. Like audiobooks are gonna be poor speech training data, because the text already lacks the paralinguistics. What helps is natural conversation, with all the variation we expect any time we talk to someone. It takes a lot of data for native speech models to get what all the paralinguistics mean.

We should also talk briefly about the two different architectures for speech models: native and cascaded.

Native means that the model goes directly from audio to latent representations, the inner stuff of the AI mind basically, and then back out to speech. Sometimes you hear this called speech2speech, voice2voice or audio2audio. Actually in this paper we’re going to hear “E2E” or “end-to-end”, which isn’t specific to audio, but does describe a model where there’s no intermediary or change in modality.

Cascaded means that the model is actually three models in a trenchcoat: one to turn speech into text, another to process the text into a response, and a third to turn the response back into speech. The first step is called ASR, automated speech recognition. The second step is usually an LLM these days. And the third step is TTS, text-to-speech.

Native is typically the best quality because it preserves the paralinguistics. Cascaded has to compress the speech into text, so you lose the richness of speech. And then also the TTS model often isn’t good at adding paralinguistics to the response. If you’ve ever talked to a phone menu, you know what I’m talking about.

Native is also typically faster than cascaded, because it doesn’t have to translate from speech to text and then text to speech.

However, native is also much less common as we’ve discussed, and it also has less language support. If you’re working in English, you have good options for native models, but for most languages in the world you probably have to go cascaded.

Cascaded can also be better if you need the smarts of a SOTA model. Since native speech models need to be approximately realtime, you can’t have parameter counts in the 100 billions or trillions. But with cascaded, each part is independent. This can also make debugging and guardrails easier too. So there’s room in the world for both architectures right now, but ultimately we’d like native speech to be more common, since we’re built for speech.

And actually I do think we’re headed in this direction. Bit of a spoiler, but as you can see from our Audio MC leaderboard, Google recently released an updated realtime speech model, Gemini 3.1 Flash Live, that may now be best in class.

We’re also seeing more adoption of speech with AI. A friend of mine works at Wispr Flow, which does dictation for AI, and they’re growing fast.

And I think when people imagine working with agents, they think about something like JARVIS from Iron Man or Cortana from Halo. Part of that is the autonomy and natural learning we discussed a couple weeks ago with our OpenClaw papers, but part of that I think is speech. So long-term I’m bullish.

Finally, I want to briefly touch on MultiChallenge, which Audio MultiChallenge is based on. MultiChallenge is a multi-turn benchmark that evaluates four skills:

1. Instruction retention - continuing to comply with earlier instructions

2. Inference memory - recalling details from earlier in the conversation

3. Versioned editing - iterating on a shared work product

4. Self-coherence - avoiding contradiction in the absence of new facts

The areas of investigation are still valid, and Audio MC is going to take inspiration from them, especially since speech is naturally multiturn.

Also, it’s quite common to make audio versions of text benchmarks. For example, the knowledge and reasoning benchmark MMLU has MMAU. We’re going to be working in that tradition here.

The Paper

So high-level, this is what we’re looking at for the benchmark: about 450 conversations of 3-8 turns each, totalling 15 hours, split across four different skills to probe and a wide variety of topics.

Each conversation ends in a failure, from either GPT-4o Audio or Gemini 2.5 Pro. All the prior turns form the input for whatever model you’re benchmarking. And then there’s a rubric for the final turn, with criteria that fit both the skill being evaluated and the final turn, not just one or the other.

To give you a more concrete idea of the four skills, here are some examples.

For Inference Memory, they have two types: semantic and audio-cue. Semantic would be memories of the words, of the meaning conveyed by them. That’s something you could get from text as well, as the blue highlights show. But for audio-cue, a non-word sound has to provide the memory. So here the model has to understand that the meowing implies a cat is present, and the words about Mochi are likely in reference to that cat.

For Instruction Retention, it’s the same as in the original MultiChallenge, where the model has to keep following an earlier instruction. So in the first turn the user gives an instruction, and then later presents an opportunity for the model to follow the instruction.

For Self Coherence, it’s not obvious just from the highlighted text, but in this example the human asks about a fourth component of shoulder bones even though the model said earlier there are only three. So the human is teeing up the model to choose between hallucinating to please the user or sticking to the facts from before.

And then for Voice Editing, they again split into two types. Mid-Utterance is when you revise within the same turn, as the first blue highlight shows in changing the number of weeping willow trees. You can also have Previous-Turn, where a turn changes something “in memory” so to speak from a previous turn.

And because this is a Scale paper, you know there’s gonna be some good content about how they produced the data.

High-level, the main things we want are natural conversations, producing the given failure mode. So right away we know we can’t script out the whole thing, but also it probably wouldn’t be very efficient to just put contributors in a metaphorical room with a model and let them poke away.

So instead they synthetically generate conversational blueprints, which the human will then roughly follow, improvising the details and probing for that failure.

There are a few different models involved in the process:

• The planner is o3, which was the best model available at the time - that’s what you want for finding these likely failure patterns

• The planner then gives a candidate blueprint to the tester, which is going to basically do a synthetic attempt, to make a conversation from the blueprint and see if it induces a failure. That’s GPT-4o

• Even though GPT-4o has an audio version, they need a text transcript at the end of the loop for o3 to review. So they have the tester agent output text, and then they use a separate TTS engine to speak, in this case gpt-4o-mini-tts

• The conversations happen with one of two target agents, either GPT-4o Audio Preview or Gemini 2.5 Pro, it’s random which one you get. The idea is that if there’s a failure on one of those models, it’s a case worth testing against all the models you want to put on your leaderboard

• The conversation history goes back to o3 for review against the initial blueprint, to generate a revised blueprint for the human contributor to use

Now you have your blueprints. The next phase is to turn those blueprints into the conversations that will be the benchmark. So we’re going to pass these vetted blueprints to humans and have them test against the same target agents. Although the human doesn’t have to use the synthetic blueprint, and in fact in 35% of cases they don’t, which I still think is a decent hit rate. Anyway if that works and induces a failure on the desired skill, then the humans write rubrics.

After that there’s a standard quality-control process. And at the end you have a dataset of multiturn conversations + rubrics for judging the final model turn, which you can use to run your eval.

Here’s how those evals work. We feed in the entire history except the failing final assistant turn, with audio from the user and text from the assistant, so that they can also evaluate models that only do speech input, not speech output. Then we ask the model we’re evaluating to produce a final assistant turn.

Once it does, we score it using an LLM judge and a rubric. We already know that LLM judges work well given a good rubric, but they do spend a bit of time showing how LLM judges generally agree with human judges when supplied with the same rubric.

Anyway, the scoring results in two different metrics:

1. Average Pass Rate (APR), which is per-rubric rather than per-criterion. This indicates whether or not the response was acceptable, which is ultimately what the user cares about. In the example shown, some criteria failed, so the APR is zero.

2. Average Rubric Score (ARS), which is per-criterion. This shows how close the model gets to acceptable, although personally I think it’s not that helpful without knowing the distribution. Like you could have the same ARS for a totally bimodal distribution and for a normal distribution, but the user experience will be quite different

And here’s where we land. I want to call out a few things.

First, we’re looking at both text and audio outputs. Every model here natively takes audio input, but only 6 of the 20 can talk back.

Second, the best scores vary considerably based on the output modality, but that’s at least partly due to model size. Like Qwen 3 Omni is only 30B parameters. GPT-4o isn’t publicly disclosed, but GPT-4o Mini is around the same size as Qwen 3 Omni. Whereas Gemini 3 Pro is more like 1T.

Relatedly, if you look at the text vs audio output scores for the same model, they’re often pretty close. Like GPT Realtime, GPT-4o Audio, GPT-4o Mini Audio, Kimi Audio, most of the scores are close overall and per skill. Text scores a bit better on average, but not always.

Finally, if you cut by axis and see which skill is strongest and weakest per model, Voice Edit is clearly the weakest skill, and Self Coherence is clearly the strongest skill. We’ll see a bit more about Voice Edit later.

—

1These models previously report scores on a TTS set of MultiChallenge (referring to it as MultiChallenge Audio) which we distinguish from.

On the topic of text vs audio, they do a test of human speech vs TTS output to see how some of the natural hesitations and disfluencies of human speech impact performance.

Broadly speaking, the cleaner, slower, and more even voice of TTS makes the tasks easier, which is no surprise. What is surprising is how the output modality impacts performance. Like if you look on the left, the only models that do worse with TTS input compared to human speech input are the audio-output versions of the GPT models. And then on the right, you see the same trend for the Instruction Retention and Voice Editing skills. However, you see an improvement on the Inference Memory and Self Coherence skills!

It not clear at all why the assistant’s output modality would impact quality so significantly. Even the authors note that they require “further investigation”. They speculate that at least for the left chart, the models are optimized for human speech rather than TTS inputs, but to me the whole thing is a big question mark.

Now digging into the skills a bit more, in figure 5, we see that audio cues are more challenging than semantic cues - audio models do better when the audio is more text-like, unsurprisingly.

And in figure 6, they examine fixed vs conditional instructions, where again the results line up with intuition: conditional instructions are harder to follow.

Now looking at that Voice Edit skill, which was generally the toughest one, they give us the same breakout we saw in the example tasks: previous turn edit vs mid-utterance edit. I actually was surprised to see how similar the scores for all three bar were on many of the models, like for Gemini 3 Pro the error rates are nearly identical. It seems like editing skill degrades faster than overall performance though, as seen by the steep changes on weaker models like Gemma 3n and GPT-4o Mini.

Still, I would have expected worse performance for mid-utterance compared to previous turn edit across the board, and that just wasn’t true.

One last thing: the authors note that “most failures on Voice Editing occur when users introduce multiple… edits throughout a conversation, with the final prompt typically requesting a summary or revision that integrates all previously specified changes.” So I guess one tip for using these models is to recap the current state regularly, rather than waiting until the end of your thoughts to review the final output. That’s good practice for text models too.

So because this is a multi-turn benchmark, we can look at the impact of both turns and duration on quality.

If you look at turn count on the left, you see kind of a funny trend - or really no trend at all. Like For Gemini 2.5 Flash Thinking for example, the second row, 3 turns is less than half the score of 6 turns, but is virtually equal to the score of 7 turns! Conversely, the score at 3 turns is almost double the score at 6 turns. So apparently turns per se doesn’t strongly influence quality. I think the real issue here is that the distribution of turns wasn’t designed, only bounded. So like humans had to induce a failure between turns 3 and 8, but within there it just depended on the flow of the conversation and when humans felt was the right time to strike.

For total duration though, which we see on the right, there’s kind of a trend downwards with duration, but the very small sample size for the longest buckets makes the whole thing a bit questionable.

Overall, the authors conclude that audio perception is the quality bottleneck, and that length doesn’t really have a chance to impact quality much yet.

One last note on length: longer model responses did correlate generally with quality.

Now that initially is a red flag, since length bias is a well known issue for human and machine judges alike. For rubrics in particular, models learn to just throw everything at the wall and see what sticks, to maximize their chances of hitting criteria even at the cost of readability or wrong or contradictory information. That’s classic reward hacking - abusing the scoring system while producing trash.

But the researchers did two things here. First, they just looked at some examples by hand and saw that yes, a lot of the longer responses actually were better. They focused particularly on the Gemini family since it clustered at the high end of this trend, and indeed the responses were good.

Second, they added negative criteria for common errors. That prevents the flooding approach from before where the model just says everything it can think of. That’s a good tip for all rubrics by the way, not just for this benchmark.

My Takeaways

• Native speech models still have a long way to go

  • No wonder Siri and Alexa failed

• It’s probably a good long-term bet for Scale

  • People are built for talking over writing

  • “We are still early” as they say

  • Evals will have a paralinguistic component that humans will be best equipped to judge

Paper

Background

So the obvious place to start is going to be Gaia, the original benchmark that Gaia2 follows. I have a few things to say about it.

First let’s just look at what it is, what it’s testing. We’ve got 466 prompts split into three levels of difficulty. The prompts all require internet research, and sometimes require an additional skill: multimodality, reasoning, or file reading.

All of that is bog standard today, and even looking at the toughest prompt here, your gut is probably telling you a SOTA model can handle that easily. I actually tested all three of these on Gemini Pro. It got Level 1 right, was slightly off for Level 2, and completely off for Level 3. So Gaia was pretty ahead of its time, which you can see in the results section of the paper; the one model that could really do internet search and multimodal etc at the time was GPT-4, and in a few different setups, it got basically 0% of Level 3 questions.

There’s actually still an active leaderboard, and the best system gets 87% of Level 3 questions right. At that point I suspect many of the remaining examples are actually wrong or somehow broken, like the internet resources they require are gone or something. Also on the topic of leaderboards, our old friend ToolOrchestra from a paper review back in January is actually in 4th place.

Now looking at the ground truth of each example, you’ll see they’re all verifiable: easy to check, format and even decimal places specified in the prompt. This paper came out before RLVR was a thing, but you could definitely use it for RLVR today if you wanted.

Second thing to note is how little the paper mentions about agents. The word does appear a couple times, but it’s not the dominant framing. We’re firmly in model territory, not much extra conceptual or code structure on top of them. This benchmark came out only a year after ChatGPT, only eight months after GPT-4, and using tools was still pretty exotic; GPT-4 only got tools and code execution in June 2023. Now everything is agents and all the models are tool-native. Gaia2 even has the word “agents” in the title of the paper!

Finally, I will note the one thing that hasn’t changed, which is the need for tough, creative, and objectively verifiable evals. Every benchmark has an expiration date, but benchmarking in general has a long life ahead of it - basically until we reach ASI. In a way benchmarks actually set the path that the big labs then walk.

Continuing on benchmarks, about six months after Gaia came the standard-bearer of computer use agent benchmarks, OSWorld. It has 369 tasks across dozens of apps on three different operating systems. They intended most tasks to require using a GUI, since that’s mostly what people think of with the term “computer use agent”, but as anyone with Claude Code experience knows, you can do a surprising amount of stuff just from the terminal and with Python. Actually some tools are leaning into that fact now, like Obsidian recently released a command line interface. Of course there’s a long tail of legacy software that will not be adding these agent-friendly interfaces any time soon, so being able to navigate a GUI is pretty helpful.

Gaia and Gaia2 both do not require GUI use, but I wanted to show OSWorld for a few reasons. One, it’s a good barometer of CUA progress, regardless of the modality, which is really taking off in 2026 and I think will become part of our daily lives soon. Two, the part about surprisingly powerful command line interfaces is relevant to Gaia2, which lets agents interact with apps via command line, via tool use, as we’ll see. And three, the environment bit in the bottom third of the slide is quite relevant. This setup of an agent in an environment with an initial state is how agentic benchmarks like this work. And all the way on the right you’ll see the environment checks for completion in the final state, looking for signs the agent completed the task. It’s not asking for a certain text output like in the original Gaia benchmark, or most other benchmarks for that matter. So in this age of agentic benchmarks, you’re going to see this final state thing more and more, and a final answer much less.

The Paper

Let’s start with the environment Gaia2 is testing agents in, which I think will make the tasks in the benchmark and the results easier to understand.

So they have this open source package called Agent Research Environments, ARE, that lets you create specific environments with this setup here. The user and the agent deal with it through some tools, and they get feedback from those tools plus also some notifications, which can come from other sources like an app saying you got a new email or whatever.

Now inside the environment, we have a few things. The biggest is the apps, things like email or spreadsheets that the agent could interact with. Keep in mind this is all by API, nothing graphical. Anyway, the tools the agent uses correspond to APIs in the app. The app also has a state, all the data in it basically, like what’s in the inbox or what’s in the spreadsheet and all the settings and filters that are currently on.

Inside the environment you also see this thing called event queue, pointing to event loop and then event log. That’s because their environments have a concept of time. For example, you could have an email appear at 9 AM with the day’s work agenda, or you could have a food delivery notification appear 30 minutes after you put in the order, or if you got two texts from a friend the notification for the first text would appear before the second notification. That temporal aspect is actually pretty rare in these types of evals, it’s a key differentiator for Gaia2.

The last bit is the Scenario box up top, kinda floating above the environment. The scenario is basically the particular way you’ve set up the environment. So like in one scenario you might be starting your work day with a busy inbox, in another scenario you might be submitting health insurance claims and texting your wife for details, and those are going to require different initial states for your apps. The different goals of course are going to require different verifiers, like responding to an email and providing certain verifiable information, or submitting the claims with the correct information typed into the form. In Gaia2 they are checking specifically for changes to state, rather than a message from the agent to the user with a final answer.

The scenarios can get pretty complex actually. Almost like being a dungeon master, where you have contingent plans and events you initiate, rather than just a passive environment that only moves because of the agent’s actions.

So that’s environments in general. For this benchmark specifically, they decided to make and use a specific environment they called Mobile, which of course reflects a smartphone OS. They have 12 apps: the basic stuff, so Messages, Chats, Emails, Calendar, Contacts, Files, System; then general consumer apps they made up, called City, Shopping, Cabs, and RentAFlat. The final one is called AgentUserInterface, a very unwieldy name. That one is for the user to text with the agent. Again, this is all text-based, nothing graphical, so you could translate a lot of this stuff to an environment you call Desktop or whatever. You’d probably just take out some of these apps and add in others.

The apps have tools like search_emails or book_cab or wait_for_next_notification. All together there are 101 tools scattered across the 12 apps. All the apps and tools get fed into the system prompt so the model knows how to use them.

The data in the apps is of course synthetic. But it’s coherent across the many apps, reflecting a specific persona. So for example, if you have messages with a person, that person will appear in your contacts, and maybe you’ll also have an email from them. That collection of data across all the apps is usually between 400k and 800k tokens, way bigger than almost any model’s context window, so you can’t just feed all the data into the model at once and have it work from memory - it has to use the tools and read from state.

As for the agents themselves, they’re actually set up in kind of an old-school way.

They use something called ReAct, which is from this paper from October 2022, right before ChatGPT came out. I view it as kind of a stepping stone to full-on agents, and one of the main authors of this paper actually has been on the forefront of agent progress for a while: he was on the SWE-bench paper, which came out in October 2023; and he was on the Tau-bench paper, which came out in June 2024. Those are both important agent benchmarks, SWE-bench in particular, which kind of established the genre alongside the original Gaia paper. The first real agents from the big labs didn’t come out until late 2024 and early 2025, stuff like Claude Code and Gemini Deep Research and OpenAI Operator, so in a way those benchmarks set the target for the labs to aim at.

Anyway, back before agents really worked, people tried different ways of getting chatbots to do more than just chats. ReAct was probably the most successful effort, I think because it’s so straightforward. All they do is prompt the model with a few examples of this thought/action/observation loop, and they have a special format for the actions that some helper software picks up on to execute the action and return the observation.

You can see it in action for these two example questions from the benchmarks Hotpot QA and AlfWorld. For the first question in particular, they show just asking, then asking with CoT prompting, then asking with access to a search tool, all of which yield the wrong answer. Over on the right you get the full ReAct paradigm of think, act, observe. When they lay it out like this it’s a pretty natural marriage of CoT and tool use, which is still what you see when you’re using something like Claude Code today - that think-act-observe loop is at the heart of all agents.

Remember, at the time they published ReAct, tool use by models was not really a thing, like ChatGPT couldn’t do anything except return you text. Of course now tool use is second nature for all models, so you don’t have to hammer on that in the system prompt. But it does help standardize things to all use this same basic prompt, rather than relying on more complex or model-specific scaffolds. That type of variability is a killer for benchmarks.

Now for the tasks. They used human annotators to make 800 of them, split evenly across the first 5 capabilities in this table. So that’s 160 each.

These tasks all happen within 10 different “universes”, basically a user persona and their collection of user-specific data. In the paper they give the example of a persona named Helena Mueller, a 43-year-old Marketing Manager living in Berlin. So then the contacts, emails, calendar etc all make sense for Helena and her imagined life - individual and group messages, calendar events for the week, a ride hailing history, pretty detailed stuff. That’s separate from the Mobile environment, which details in general the apps, the event triggers etc. You fill up the environment with the data to get a universe.

The tasks are all verifiable. Specifically, the verifiers are mostly going to look for write commands that show the agent actually took the necessary actions. So like the first one, which is kind of a dumb task, that’s going to look for writes to all the contacts. Unfortunately, making them verifiable usually means being weirdly specific, like the Search example where the user anticipates there could be a tie and says how to break it. I think the authors realized that and made the Ambiguity capability to compensate, but I suspect real users will be ambiguous much more than a fifth of the time.

Anyway, if the agent takes more than 200 steps or overflows its context or misses a deadline for a temporal task, it automatically fails.

The tasks specify an initial state and also a chain of events, in some cases based on earlier events, in other cases based on times.

Now for the final two capabilities, they’re actually tasks from earlier capabilities, but with a twist. For Agent2Agent, that’s going to allow the agent to call a sub-agent, either using the same model or using a different model, via Google’s Agent2Agent protocol. For Noise, they’re going to throw in some errors and see how robust the agent is.

One other note about capabilities: while each task has a predominant capability for categorization, it’s really impractical to have a task with just one capability and not any of the others. I know that’s a problem we often encounter when we’re writing prompts, that we can’t categorize them cleanly, and I think the authors made the right choice here to prioritize realism over clean categorization.

Putting all of this together, here’s what their annotation and reviewing software looks like, it’s that open source package I mentioned earlier. They have user input in blue, some agent actions in purple, events in green, all in a flow chart. Then further down they have outputs from the model in a couple different forms, like the thinking and final response in the bottom-left, the list of steps and their components in bottom-middle, and the raw contents of a particular component in the bottom-right. Also on the left is the list of apps for the Mobile environment.

So here’s where we net out in terms of raw performance. A few things to note here.

One is, we’re using older models. Putting aside the fact that it’s a Meta paper so they have to throw in Llama 4 and even Llama 3.3, we’re only seeing GPT-5 and Claude 4 and Gemini 2.5. Every one of those has meaningfully more skilled versions available today, with particular focus on agentic work. So I wouldn’t be surprised if SOTA models now did meaningfully better on at least the first two capabilities.

Second, you’ll notice Time has universally poor performance. I think that’s a bit artificial, because by definition all tasks except Agent2Agent only used one agent. For time-sensitive work, you’d definitely want one orchestrator keeping an eye on the time while sub-agents work. What they observed with the Time tasks is that because models can only process one thing at a time, they’d often be in the middle of something else when some crucial event came and went. It’s especially tough for the thinking variants of some of these models.

Overall though, I like that there’s room to move on most of these capabilities, I like that the prompts are human-made, and I like that they’ve made the tasks more real by incorporating elements like time and ambiguity. It’s a real test of agent utility.

Continuing on the realism trend, I like that they reported out cost and wall time too; if you’ve been following any conversations about Openclaw and the like, you know people are complaining about burning through tokens or taking too long. Cost is also quite relevant for enterprise, wall time less so since jobs may run in the background. But for the Mobile environment that reflects consumer usage, wall time is definitely going to matter, maybe as much as quality.

So on cost, there’s a pretty clear trendline, which only Grok is really below. Not a surprise.

On time, it’s not surprising that models can beat humans on a lot of tasks, but I do take issue with this graph. Notice the title is “Time per Solved Gaia2 Scenario”. It’s defensible to exclude the failures, since a lot of them probably involved looping and artificially long times, but if you only report on the successful tasks then you’re biased towards the easy ones. By contrast, since humans are able to solve pretty much all these tasks, you’re going to get the full spectrum of difficulty or inherent length in those human results. This is also why some of the most successful models seem to take the longest, like GPT-5. So the most I would take from this is that models definitely can be faster than humans - nothing more.

Now for the bottom graph, calls and tokens of course are the drivers of cost and time ultimately, so it’s good to look at both; you want to know what to change in order to alter your end metrics.

For LLM calls, for a given amount of performance you want fewer of them, because each call is going to add time, even for the same total number of tokens. One trend here is that thinking variants make fewer calls and also do better. They’re going to use more tokens per call, since using extra tokens to think is by definition what thinking models do, but apparently the thinking actually does result in better thoughts and actions. If you look back above, you’ll see thinking variants cost the same or less and take similar amounts of time.

One related point they mention is that which apps the agents use and how often is pretty similar across all the models. So the difference really is the quality of thought, how wisely you’re using the apps and what exactly you’re doing with them.

Do note that the x axis is logarithmic on the output tokens graph on the bottom-right. Also it’s not shown here, but they note all models flatline past a certain point, like scaling up tokens indefinitely is not going to bring better performance.

So we looked at scaling the LLM calls and scaling the tokens for the one agent. Now we look at scaling the number of agents.

They’re using the Agent2Agent protocol, shown top-left. Multi-agent stuff is still pretty new so I haven’t seen much of this protocol in the wild. I will say though, I prefer the terms “orchestrator” and “sub-agent” compared to “Main-Agent” and “App-Agent”. Anyway, if you haven’t been on any paper talks where we’ve discussed agent swarms, there are two advantages: one, you can do work in parallel and thus save on wall time; and two, having a sub-agent do a task keeps the context of the orchestrator clear. Given the push on long-horizon tasks, keeping the orchestrator focused for longer is a big deal.

Now they’ve done two experiments here on multi-agent setups. For the first one, the table on the top-right shows what happens when you upgrade the orchestrator model and the sub-agent model. In this case, they’re making a major upgrade, from Llama 4 Maverick to Claude 4 Sonnet. In both cases, performance approximately doubles, and the effect seems to be independent. My hunch is that you always want the smartest orchestrator and that you should pick the right model for the sub-agent’s job, like if it’s simple you can pick a cheap model but if it’s tough you pick an expensive one.

The second one, shown in the graphs on the bottom, is the more interesting one. The goal here was to control the amount of delegation the agent could do, with this ratio “r”. r = 0 means it’s just the one agent, using all the tools directly. r = 1 means the agent must use sub-agents, it can’t call any tools directly. And then r = 0.5 means half the apps can be used directly, half have to be through sub-agents.

They found two things. One, the more you can delegate, the fewer tokens get used overall. You might think more communication overhead would lead to more tokens, but really what happens is that keeping context clean and work compartmentalized allows all the agents to focus and work more effectively. For example, since the orchestrator doesn’t have to juggle as many tools, it makes fewer syntax errors and enters fewer loops. It’s a stronger effect in weaker models, and the communications overhead is still there, so it can become a more marginal trade. But it still holds true with Sonnet at least.

The other finding is that it’s not always obvious what the impact on quality will be. For Llama it helps to delegate in terms of pass@k, but for Claude it hurts, at least for this task in this framework. I do wonder how generalizable this finding is, since models are trained with certain scaffolds and expectations about sub-agents now. Like if you open Claude Code right now, Opus 4.6 is going to expect certain sub-agents available. And modern models likely have their own preferences on how much they want to outsource vs handle themselves, whereas when Llama 4 came out agents were only just becoming a thing, so they wouldn’t really have figured into training.

My Takeaways

• Swarms are the future

  • As the authors point out, a variety of subagents is optimal, for cost but also for latency

• How agents work together and communicate is nascent

  • First time seeing Agent2Agent in the wild

  • Authors point out that many models are trained to work alone, maybe not good at organizing other agents (just like many humans are bad managers)

  • On the flip side, models trained to work in a swarm of other agents with models in their family may be harder to benchmark in neutral tooling or with other models as agents

• There is room for a GUI version

Paper · Website

Group Relative Policy Optimization

+

Reinforcement Learning with Verifiable Rewards

Okay, the first thing I want to do is revisit two foundations of modern RL: group relative policy optimization, or GRPO; and reinforcement learning with verifiable rewards, or RLVR.

GRPO is an algorithm that turns rewards into model weight changes. RLVR is how we get those rewards from responses.

Chronologically, GRPO came first, so we’ll talk about it first. As with many other LLM engineering breakthroughs, GRPO is the work of the DeepSeek team. If you’ve been to some of the recent paper reviews, you know what I’m talking about, but if not I’ll give a quick recap: DeepSeek is one of the top two labs in China, the other one being Qwen, and they have a very R&D-focused culture. They don’t crank out models, but the ones they do put out are always respected, and they often contain new techniques that long outlive the models themselves.

A perfect example is in this paper, nominally about a math-focused LLM called DeepSeekMath, but really a vehicle for this algorithm, GRPO. Like other RL algorithms such as PPO, it’s a method for extracting signals from model outputs, basically turning them into values we can then use to adjust the weights of our model. In PPO, we have to use a special model to estimate that signal, which is complicated and sometimes messy or fuzzy. With GRPO, there is no extra model to train, as you can see by the decrease in yellow boxes in the right-hand graphic. That makes GRPO simpler and often more precise than PPO.

Looking at the GRPO graphic a little more closely, you can see there’s a reward model that turns outputs o into rewards r, that then turn into advantages A, which is ultimately what we need to adjust the weights of our model. But as we’ll see on the next slide, we actually don’t need a reward model per se - we need a system to turn outputs into rewards. The rest of GRPO is just math, completely deterministic.

The other prime example of DeepSeek magic is this paper, nominally about DeepSeek-R1, the model that reproduced the reasoning behavior of o1 and o3 from OpenAI and briefly crashed the stock market.

In this paper, they used GRPO to train their reasoning model, but they didn’t require a reward model. Instead, they calculated rewards deterministically by checking the model’s final answer against a known good answer. So like for a math problem, if the known good answer is sqrt(x), we can easily check if the model’s final answer is also sqrt(x). We may have to check for equivalent forms, like x^½, but there’s no real interpretation or judgment like with a reward model. We know that technique as RLVR.

So just like you don’t need RLVR for GRPO, you also don’t need GRPO for RLVR. Like you could do RLVR instead of having the reward model in PPO. But in practice, these two go well together, because they are basically the simplest possible setup for RL.

Anyway, the DeepSeek-R1 paper found two effects of GRPO + RLVR: you can greatly improve reasoning, and you generally end up using way more tokens. This is exactly what o1 and o3 did. Sometimes people call it “test time compute scaling”, meaning you use more compute when generating your response, and that extra compute is supposed to improve quality.

This combination of GRPO + RLVR is everywhere nowadays. Any prompt where there’s some verifiable final response, like a math problem or some instruction following, is likely trained on with GRPO + RLVR. And rubric training is basically just stacked-up RLVR training, like instead of one final answer you can verify against, you have multiple criteria you can verify against. It’s the same core idea of a method for giving a response a score, which can then go into GRPO to turn into signal on how to adjust the weights.

RLVR has two shortcomings though. For one, you might have incorrect steps but still get the final answer right. With RLVR, you’re going to get full marks, even though the response overall is not correct. For the other, you don’t know why or how you got something wrong. Or if you do, it doesn’t factor into the score, into the training signal. So you’re rewarding at a response level even though parts of the response may be wrong, and you’re leaving a lot of valuable information on the table by just calling a wrong final answer “incorrect. Keep those shortcomings in mind.

Another, more recent finding of RLVR is that it may not be teaching anything. Instead, what some researchers have found is that RLVR merely brings forth solutions paths that are already present in the model, although they may be highly unlikely ones. This is called the elicitation hypothesis.

The image on the left shows what I’m talking about. For Problem A, the base model has all these possible solution paths for a given problem, but only a couple of them are correct, so when you run the model the chances of getting a correct response are low. What RLVR does is make the correct paths much more likely, so that you as an end user are generally getting a correct answer now.

Of course it may be that those other solution paths are involved in other problems, like Problem B here. To oversimplify somewhat, if through RLVR we greatly reduce the likelihood of exploring the left side of this tree, then we can’t reach the correct answer for that problem, even though it actually was there in the base model.

The chart on the right is generally how people show the elicitation hypothesis is true, that any correct solution a model can give is in there from the beginning and not taught wholly new via RLVR. On the x axis is the number of chances you give a model per question to get a set of questions right. As long as any of the k responses is correct, the model gets credit for that question. So that metric is called pass@k.

For end users, we really only care about pass@1, since a person using a chatbot expects the right answer the first time. And you can see there, on the left side of the chart, doing GRPO + RLVR really helps - you’re getting 4x higher accuracy for the yellow line compared to the black line.

However, if you keep giving chances, up to 256 in this case, you see all the lines converge. So it seems that the base model knew the answers all along, it just didn’t know how to produce them efficiently.

Again, elicitation is still good for end users. It just shows RLVR doesn’t teach new things.

Okay, now we need to talk about distillation.

Very broadly, distillation is a family of techniques for transferring some quality or information from one model, the “teacher”, to another, the “student”. Usually the teacher is a more advanced model than the student, which by the way puts a ceiling on how far distillation can take you.

So the naive way to do this is to make training data synthetically, using the teacher model. Like imagine we made the same kinds of post-training data — SFT, RLHF, rubrics — but with GPT and Claude instead of humans.

If all you have access to from the teacher model is the tokens it outputs, that’s the best you can do. That’s true of the Claude API, so if you’re trying to distill off Claude, this is the way to go.

If you have access to the per-token probabilities, sometimes called the logprobs, you can do better. In this setup, you pass the same input to the teacher and the student. Then for each token you need to predict, you compare the probability of each possible token in the student’s distribution with the probability of the same token in the teacher’s distribution. The differences between the probabilities are your loss. So your goal is to teach not only the most likely token, but also how likely all the other tokens are too.

That’s a much richer and more nuanced understanding the teacher is providing to the student, and it’s going to be more effective at passing on the teacher’s wisdom.

Let me draw a human parallel. Let’s say you’re learning to play chess from a master. You’re both playing black, and some opponent is playing white. The prompt is white’s first move. Now you and the master each take the list of all possible next moves and assign each one a probability of being optimal. Then you compare probabilities, then you make the master’s top-rated move.

By doing that over and over, you’re going to get an idea of what set of moves the master realistically considered, what moves he kept in mind but thought were unorthodox, what moves he knew were trash. That’s a lot more wisdom to pass on than just watching him play the opponent!

Of course you need that level of access in the first place. OpenAI actually does offer logprobs through their APIs, which is partly why they’re such a popular target for distillation. And of course any open-weights model includes logprobs too.

So the teacher’s logprobs depend on its innate knowledge of course, but they also depend on the context, the input the teacher has received. In one sense this is obvious - the next predicted token clearly depends on the prompt you’ve given the model - but for other forms of context it’s more subtle.

One easy example is the chunks you provide in RAG. Even if the core answer is the same with and without the extra context RAG provides, you could get some additional details, or smaller effects like a change in tone or vocabulary. That’s all going to show up in the logprobs, the different distributions.

Even more subtly, you could provide different instructions or system prompts to the teacher. That’s what some researchers did here, in a paper called “Learning by Distilling Context”. This paper came out in September 2022, two months before ChatGPT, which explains why they’re focusing on relatively simple queries here.

Anyway, you can see in this example that the teacher receives the schema and four examples before receiving the question. Those four examples help it get the right answer. So then when you do soft-label distillation, even though the student only gets the schema in its context, through distillation it also gets the wisdom of the four examples the teacher got.

This is a very simple example, but in theory you could fill the teacher’s context window for distillation. You won’t be transmitting 100k tokens of information or whatever, but you will be transmitting some compressed version of it, compressed by the teacher model’s intelligence into this low-dimensional space. And that intelligent compression is what makes soft-label distillation generally more effective than just making synthetic data from the teacher.

I think of soft-label distillation as a way to teach process knowledge, stuff that is hard or at least inefficient to just write down as rules. Like yes, you could try to write out the mechanics of SQL and your particular business logic in the student model’s system prompt, but when you use distillation, you’re taking advantage of the teacher’s compressed wisdom and having it guide the student directly. Again drawing a human parallel, I think of it like a golf teacher standing behind you and putting his hands over yours, guiding you through the entire motion of driving or putting. Sure, he could try to explain how to do it, but the hands-on feedback and guidance as you complete a swing is going to be so much richer.

There’s a decent body of work out there about context distillation. I’ve heard Anthropic uses context distillation to take very long prompts about what Claude’s character should be and distill it into the weights. I also read a paper late last year about distilling lots of documents into a kv cache that you can then pop into a model, to basically give it memories of those documents when you then want to ask about related matters. So this general idea of using a model to effectively and intelligently compress context down into the densest possible signal is pretty robust.

Self-Distillation Policy Optimization

+

Reinforcement Learning with Rich Feedback

So this paper coins two new terms: self-distillation policy optimization, or “SDPO”; and reinforcement learning with rich feedback, or “RLRF”. Those are the parallels for GRPO and RLVR, which we discussed earlier.

The key insight for RLRF is that RLVR feedback is incomplete. Yes, if you’re only going to provide one bit of feedback, it’s going to be whether the solution was right or wrong. But there’s no reason we have to limit ourselves! We can provide more, richer feedback.

RLRF says, let’s use that feedback. Maybe we have runtime errors when our program fails. Maybe we have written feedback from an LLM judge. Maybe this is real user data and they yelled at the agent about why its actions were wrong. That’s all crucial stuff! It’s important to know how you were wrong, not just if you were wrong. And there are way more ways to be wrong than right. So just returning a reward of zero leaves something on the table.

Of course some environments are more feedback-y than others. They pick coding here because you do get those runtime errors and so on. But this is not a code-specific technique - anything you can put into tokens can be feedback here.

This here is where the feedback goes. This is the prompt for the teacher, which we bolster with as much context as possible so that it can guide and distill into the student. Remember, the goal of the teacher is to make adjustments to probabilities at each token, so we want the teacher to always be right.

So we start with our prompt of course, whatever problem we’re trying to solve.

Next, if we have one, we can provide an example of a correct solution from a previous rollout. That’s definitely going to help the teacher guide the student if it exists. If not, we skip this part.

After that is our feedback, from the compiler or the LLM judge or whatever. We should basically always have that, except for on the first run, but again if it doesn’t exist we skip it.

Then we provide the student’s original response. And you might be surprised that we’re providing an assistant response. Like, isn’t the model supposed to provide a new response?

But they’re using the teacher model differently. They’re not using the teacher to give its own answer, they’re using it to give its own probabilities at each token of the student’s response. You never get a complete response from the teacher. We’ll see more about that on the next slide.

So that’s what they explain in the paper, but I want to call out two things. One, you may be wondering why we’re doing any more training if the student was able to produce a correct solution. Well as we saw earlier with those pass@k graphs of increasingly high k, there’s a difference between “can produce a right answer” and “probably will produce a right answer”, and that difference may be enormous. We want to shrink that gap so that end users get better pass@1 results.

Two, why can’t we use someone else’s correct solution? The theoretical answer is that it will change the teacher’s guidance to bias toward the style of the correct solution as well as the contents of the correct solution. So if the model naturally is verbose, but our correct solution is concise, then the teacher will punish the student for being verbose, even though that doesn’t impact correctness. We want a clean signal from the teacher on correctness, which can only happen if the correct solution is from the same model we’re using for the teacher and the student.

However, practically they don’t test it. You could give them a pass and say it’s out of scope, but I think it would have been nice to try having another model family write the solutions and see what the penalty really is. If it’s small, it could be worth having humans write solutions to really hard problems and then having a SOTA model do this self-distillation thing. But then again, if we really believe in the elicitation hypothesis, you can’t teach new things with GRPO + RLVR or related methods. Still, I wish they had done the test.

Okay, so here’s how it looks when the teacher provides feedback to the student. In this example we have a question, then an answer from the student, then some text feedback. We’re gonna take all that and give it to the teacher.

The teacher is going to go token by token through the student’s answer, that bit in step 2, and compare its likelihood of that token vs the student’s likelihood of that token.

So in this example, for most of the program, the teacher and the student gave similar probabilities for the tokens: for the three tick marks, for the word “python”, for the function name etc.

But then you get to this plus symbol and you see the teacher disagree. Specifically, the teacher gives a very low probability for “+”, but the student gives a relatively high probability. Since the teacher knows the feedback and the student didn’t, the teacher is probably right, and we penalize the student.

Here’s a closer look at that same example. At the top they have the baseline approach of GRPO, which assigns equal credit to all tokens in a response. Red means less likely here, so because the initial response was wrong, GRPO gives low credit to all the tokens.

By contrast, SDPO assigns credit per token. We can see that for each token the student actually picked, in the first row, we also have alternative tokens it considered. So like in the first position, we have parenthesis + “range” as the selected token. But the student also considered a leading space and then “range”, “range” and a space afterwards, and a dot then “range”. The teacher agreed with the token the student chose, so the selected token has a white background.

Now if we look at the blue tokens, where the teacher assigned more credit, we see two things. One, in the second position, the student’s second pick gets high credit from the teacher, so it seems like the student was close to making the right decision.

Two, we see that plus in the sixth position is really the main problem, and that removing it and the “1” after was the right choice. The most blue token, with two close parentheses and then “\n”, is actually the final token that the student actually selected.

So again, the teacher is not generating a correct response we compare to; instead, the teacher is evaluating the student’s choice of the immediate next token given all the previous tokens chosen.

So that loss, that mismatch between teacher and student probabilities, is signal for SDPO. SDPO is the algorithm that turns the signal from the mismatch into changes of the model.

And because the teacher and the student are the same model, you can keep doing this over and over until performance saturates. So let’s say on the first round you did eight rollouts and got none right. But you got feedback, so you give that to the teacher and measure that mismatch between teacher and student probabilities. You have your loss, which you plug into SDPO, you train your model with that feedback, then you try again. Now two of your eight rollouts are correct. But the rest are still wrong. So now you take the feedback and the correct solution, give it to the teacher, calculate loss etc. Ideally you end up with all eight rollouts right, but maybe you top out at seven.

With standard GRPO + RLVR you just have to keep trying and hoping that the answer is in there somewhere, or that changes to the weights due to good signal on one problems helps you on this other problem.

With SDPO + RLRF, by contrast, you’re getting this extra feedback that can nudge you closer to a good solution. So it’s at least more likely that if you keep training like this, you’ll get there.

Here’s their first test of SDPO vs GRPO, with the Olmo3-7B model, working on a chemistry benchmark.

A few things to note. First, of course the accuracy improved, and improved more than with GRPO. They probably wouldn’t have published this paper otherwise.

Second, the x axis is in wall time, how long training took. That’s not usually a metric we see in these papers, they’re usually measuring performance against training steps or perhaps amount of data. Frankly I think it’s a little bit suspicious, like the accuracy improvements don’t stand on their own so they have to tout some other benefit.

Finally, and perhaps most underratedly, the response length went down rather than up. It went way down actually, 11x compared to GRPO, which as we saw incentivizes more tokens. “Longer thinking” as some would say, or “test-time compute scaling” - the idea that more computation to produce more tokens means more accurate answers.

But if you actually look at the thinking, it’s often repetitive or looping, and sometimes contradictory. When DeepSeek R1 came out people made a lot of observations about GRPO + RLVR getting models to say things like “Wait” or “Let’s try again” or “On the other hand”. Of course it’s good to confirm your answer from a different perspective, fox-like thinking as Isaiah Berlin would say, but doing the same calculations three times or ping-ponging between the same two points repeatedly isn’t productive.

So perhaps SDPO + RLRF is a more direct way to bring forth these reasoning paths than GRPO + RLVR.

Now here’s an experiment where they narrow their training set way down, just to hard problems, to see if that rich feedback really can guide you to the right solution.

One is exactly what they’re measuring. They’re taking a couple sets of hard tasks, where the initial model gets the answer right almost never for “very hard”, or where it gets the answer right less than half the time for just “hard”. That’s only 9 and 19 tasks, respectively, which is why the error bars are so big. Anyway, what they’re measuring is basically pass@k for increasingly large k, but with a twist: for SDPO, since they’re training after each round of a certain number of rollouts, what the x axis really measures is how many rounds of SDPO it takes to get a solution. That’s why they call the y axis “discovery@k” instead.

Two is what they’re comparing. We know SDPO of course, and best-of-k just means we take k samples all at once and count it if at least one of the k samples is correct. The other trendline, “multi-turn”, is like a hacky version of SDPO: instead of training after each round, they just keep adding feedback to the prompt. So each incorrect attempt and its feedback get added to the prompt, which could be like hundreds of examples. They do have to cap the number of examples due to context limits, but that’s still a ton of information they’re providing. Also they run a variant where they kept the attempt and the feedback, not just the feedback, and that did much worse. So that’s a practical tip for your prompting: try to boil down the lessons of failed examples if you can.

And three of course is the results! SDPO outperforms the best-of-k baseline, and also outperforms multi-turn. It’s really not surprising that sequentially trying and then updating outperforms just making more and more parallel attempts, but I was surprised how bad multi-turn was. To me that really highlights how much more effective feedback becomes when you train on it. Similar to how in theory you can carefully follow documentation to get the same outcome as someone with lots of experience, but in practice there’s something about learning that makes the knowledge fit into your own sort of mental schemas in a way that reading external text misses.

One other piece to remember about the results is that from an end user perspective, the model at the end of SDPO is going to be much more useful, because it’s much more likely to get you the correct result on the first try. Pass@k for a huge k is important scientifically, but it’s irrelevant for end users.

So one complaint you might raise is that SDPO + RLRF requires additional feedback compared to GRPO + RLVR. Remember, the “RF” in “RLRF” stands for “rich feedback”.

To address that, they separate out the RLRF part and just test SDPO vs GRPO. So now we don’t have to make any changes to our training data or our environments, just to the training process. Remember, GRPO and SDPO both depend on there being at least some correct rollouts to get some signal, they just use that signal differently. In the case of no correct rollouts, both algorithms do nothing.

The story here is again not a clean victory for SDPO. It wins most of the time, sometimes by a lot, but other times it is basically tied or even loses by a few points. Also keep in mind the scores are average accuracy across 16 attempts. I would have liked to see pass@1 to reflect the user experience.

Another way you might compare SDPO to GRPO is by making the binary more of a spectrum, like a smoother transition from GRPO to SDPO.

The authors do that here when they make the SDPO feedback increasingly granular. Recall that GRPO is feedback over the entire response, what researchers often call the “sequence”. So sequence-level SDPO is basically GRPO but with feedback given to the teacher prompt, which apparently helps, given the separation of the grey and blue lines.

The next step in granularity is token-level, just a binary about whether the teacher thought the student’s token was the best choice or not.

And then the final step is what we saw earlier, the logits, which are basically the probabilities of any possible token for the given position. That’s the full SDPO. We called that “soft-label distillation” in the background slides.

So intuitively you might think bigger models are gonna be better self-teachers, because they can take feedback into account better. Like anyone who has used AI consistently over the years knows models used to be terrible at taking feedback in conversation, missing out on subtleties or getting confused etc.

Turns out that does hold up in SDPO, at least for the range of sizes they looked at. SDPO is no better than GRPO at 0.6B, but consistently widens the gap up to 8B. Unfortunately they didn’t test the 32B model, which would have really made for a nicer story, but my guess is it does keep working at least up to a certain point. One wonders what the big labs with 1T parameter models can achieve.

My Takeaways

• RLRF could be a good use of human qualitative feedback

  • RLHF already uses human quantitative feedback (the Likert scale)

  • Justifications from existing RLHF data could become RLRF data

• Seems like a straightforward replacement for GRPO

  • There was some Twitter conversation about big labs using a better version of GRPO, perhaps this is it

  • There is a sense we are “due” for a new technical breakthrough (although maybe that was Engram)

• This all depends on the raw intelligence being in there from pretraining

  • Elicitation hypothesis

  • Does that mean we need to focus more on pretraining to get the most out of post-training?

• RSI is coming

  • Anything you can verify can self-improve

  • You don’t need a smarter teacher, just a student with good feedback looking over their old results