Paper · Website

Group Relative Policy Optimization

+

Reinforcement Learning with Verifiable Rewards

Okay, the first thing I want to do is revisit two foundations of modern RL: group relative policy optimization, or GRPO; and reinforcement learning with verifiable rewards, or RLVR.

GRPO is an algorithm that turns rewards into model weight changes. RLVR is how we get those rewards from responses.

Chronologically, GRPO came first, so we’ll talk about it first. As with many other LLM engineering breakthroughs, GRPO is the work of the DeepSeek team. If you’ve been to some of the recent paper reviews, you know what I’m talking about, but if not I’ll give a quick recap: DeepSeek is one of the top two labs in China, the other one being Qwen, and they have a very R&D-focused culture. They don’t crank out models, but the ones they do put out are always respected, and they often contain new techniques that long outlive the models themselves.

A perfect example is in this paper, nominally about a math-focused LLM called DeepSeekMath, but really a vehicle for this algorithm, GRPO. Like other RL algorithms such as PPO, it’s a method for extracting signals from model outputs, basically turning them into values we can then use to adjust the weights of our model. In PPO, we have to use a special model to estimate that signal, which is complicated and sometimes messy or fuzzy. With GRPO, there is no extra model to train, as you can see by the decrease in yellow boxes in the right-hand graphic. That makes GRPO simpler and often more precise than PPO.

Looking at the GRPO graphic a little more closely, you can see there’s a reward model that turns outputs o into rewards r, that then turn into advantages A, which is ultimately what we need to adjust the weights of our model. But as we’ll see on the next slide, we actually don’t need a reward model per se - we need a system to turn outputs into rewards. The rest of GRPO is just math, completely deterministic.

The other prime example of DeepSeek magic is this paper, nominally about DeepSeek-R1, the model that reproduced the reasoning behavior of o1 and o3 from OpenAI and briefly crashed the stock market.

In this paper, they used GRPO to train their reasoning model, but they didn’t require a reward model. Instead, they calculated rewards deterministically by checking the model’s final answer against a known good answer. So like for a math problem, if the known good answer is sqrt(x), we can easily check if the model’s final answer is also sqrt(x). We may have to check for equivalent forms, like x^½, but there’s no real interpretation or judgment like with a reward model. We know that technique as RLVR.

So just like you don’t need RLVR for GRPO, you also don’t need GRPO for RLVR. Like you could do RLVR instead of having the reward model in PPO. But in practice, these two go well together, because they are basically the simplest possible setup for RL.

Anyway, the DeepSeek-R1 paper found two effects of GRPO + RLVR: you can greatly improve reasoning, and you generally end up using way more tokens. This is exactly what o1 and o3 did. Sometimes people call it “test time compute scaling”, meaning you use more compute when generating your response, and that extra compute is supposed to improve quality.

This combination of GRPO + RLVR is everywhere nowadays. Any prompt where there’s some verifiable final response, like a math problem or some instruction following, is likely trained on with GRPO + RLVR. And rubric training is basically just stacked-up RLVR training, like instead of one final answer you can verify against, you have multiple criteria you can verify against. It’s the same core idea of a method for giving a response a score, which can then go into GRPO to turn into signal on how to adjust the weights.

RLVR has two shortcomings though. For one, you might have incorrect steps but still get the final answer right. With RLVR, you’re going to get full marks, even though the response overall is not correct. For the other, you don’t know why or how you got something wrong. Or if you do, it doesn’t factor into the score, into the training signal. So you’re rewarding at a response level even though parts of the response may be wrong, and you’re leaving a lot of valuable information on the table by just calling a wrong final answer “incorrect. Keep those shortcomings in mind.

Another, more recent finding of RLVR is that it may not be teaching anything. Instead, what some researchers have found is that RLVR merely brings forth solutions paths that are already present in the model, although they may be highly unlikely ones. This is called the elicitation hypothesis.

The image on the left shows what I’m talking about. For Problem A, the base model has all these possible solution paths for a given problem, but only a couple of them are correct, so when you run the model the chances of getting a correct response are low. What RLVR does is make the correct paths much more likely, so that you as an end user are generally getting a correct answer now.

Of course it may be that those other solution paths are involved in other problems, like Problem B here. To oversimplify somewhat, if through RLVR we greatly reduce the likelihood of exploring the left side of this tree, then we can’t reach the correct answer for that problem, even though it actually was there in the base model.

The chart on the right is generally how people show the elicitation hypothesis is true, that any correct solution a model can give is in there from the beginning and not taught wholly new via RLVR. On the x axis is the number of chances you give a model per question to get a set of questions right. As long as any of the k responses is correct, the model gets credit for that question. So that metric is called pass@k.

For end users, we really only care about pass@1, since a person using a chatbot expects the right answer the first time. And you can see there, on the left side of the chart, doing GRPO + RLVR really helps - you’re getting 4x higher accuracy for the yellow line compared to the black line.

However, if you keep giving chances, up to 256 in this case, you see all the lines converge. So it seems that the base model knew the answers all along, it just didn’t know how to produce them efficiently.

Again, elicitation is still good for end users. It just shows RLVR doesn’t teach new things.

Okay, now we need to talk about distillation.

Very broadly, distillation is a family of techniques for transferring some quality or information from one model, the “teacher”, to another, the “student”. Usually the teacher is a more advanced model than the student, which by the way puts a ceiling on how far distillation can take you.

So the naive way to do this is to make training data synthetically, using the teacher model. Like imagine we made the same kinds of post-training data — SFT, RLHF, rubrics — but with GPT and Claude instead of humans.

If all you have access to from the teacher model is the tokens it outputs, that’s the best you can do. That’s true of the Claude API, so if you’re trying to distill off Claude, this is the way to go.

If you have access to the per-token probabilities, sometimes called the logprobs, you can do better. In this setup, you pass the same input to the teacher and the student. Then for each token you need to predict, you compare the probability of each possible token in the student’s distribution with the probability of the same token in the teacher’s distribution. The differences between the probabilities are your loss. So your goal is to teach not only the most likely token, but also how likely all the other tokens are too.

That’s a much richer and more nuanced understanding the teacher is providing to the student, and it’s going to be more effective at passing on the teacher’s wisdom.

Let me draw a human parallel. Let’s say you’re learning to play chess from a master. You’re both playing black, and some opponent is playing white. The prompt is white’s first move. Now you and the master each take the list of all possible next moves and assign each one a probability of being optimal. Then you compare probabilities, then you make the master’s top-rated move.

By doing that over and over, you’re going to get an idea of what set of moves the master realistically considered, what moves he kept in mind but thought were unorthodox, what moves he knew were trash. That’s a lot more wisdom to pass on than just watching him play the opponent!

Of course you need that level of access in the first place. OpenAI actually does offer logprobs through their APIs, which is partly why they’re such a popular target for distillation. And of course any open-weights model includes logprobs too.

So the teacher’s logprobs depend on its innate knowledge of course, but they also depend on the context, the input the teacher has received. In one sense this is obvious - the next predicted token clearly depends on the prompt you’ve given the model - but for other forms of context it’s more subtle.

One easy example is the chunks you provide in RAG. Even if the core answer is the same with and without the extra context RAG provides, you could get some additional details, or smaller effects like a change in tone or vocabulary. That’s all going to show up in the logprobs, the different distributions.

Even more subtly, you could provide different instructions or system prompts to the teacher. That’s what some researchers did here, in a paper called “Learning by Distilling Context”. This paper came out in September 2022, two months before ChatGPT, which explains why they’re focusing on relatively simple queries here.

Anyway, you can see in this example that the teacher receives the schema and four examples before receiving the question. Those four examples help it get the right answer. So then when you do soft-label distillation, even though the student only gets the schema in its context, through distillation it also gets the wisdom of the four examples the teacher got.

This is a very simple example, but in theory you could fill the teacher’s context window for distillation. You won’t be transmitting 100k tokens of information or whatever, but you will be transmitting some compressed version of it, compressed by the teacher model’s intelligence into this low-dimensional space. And that intelligent compression is what makes soft-label distillation generally more effective than just making synthetic data from the teacher.

I think of soft-label distillation as a way to teach process knowledge, stuff that is hard or at least inefficient to just write down as rules. Like yes, you could try to write out the mechanics of SQL and your particular business logic in the student model’s system prompt, but when you use distillation, you’re taking advantage of the teacher’s compressed wisdom and having it guide the student directly. Again drawing a human parallel, I think of it like a golf teacher standing behind you and putting his hands over yours, guiding you through the entire motion of driving or putting. Sure, he could try to explain how to do it, but the hands-on feedback and guidance as you complete a swing is going to be so much richer.

There’s a decent body of work out there about context distillation. I’ve heard Anthropic uses context distillation to take very long prompts about what Claude’s character should be and distill it into the weights. I also read a paper late last year about distilling lots of documents into a kv cache that you can then pop into a model, to basically give it memories of those documents when you then want to ask about related matters. So this general idea of using a model to effectively and intelligently compress context down into the densest possible signal is pretty robust.

Self-Distillation Policy Optimization

+

Reinforcement Learning with Rich Feedback

So this paper coins two new terms: self-distillation policy optimization, or “SDPO”; and reinforcement learning with rich feedback, or “RLRF”. Those are the parallels for GRPO and RLVR, which we discussed earlier.

The key insight for RLRF is that RLVR feedback is incomplete. Yes, if you’re only going to provide one bit of feedback, it’s going to be whether the solution was right or wrong. But there’s no reason we have to limit ourselves! We can provide more, richer feedback.

RLRF says, let’s use that feedback. Maybe we have runtime errors when our program fails. Maybe we have written feedback from an LLM judge. Maybe this is real user data and they yelled at the agent about why its actions were wrong. That’s all crucial stuff! It’s important to know how you were wrong, not just if you were wrong. And there are way more ways to be wrong than right. So just returning a reward of zero leaves something on the table.

Of course some environments are more feedback-y than others. They pick coding here because you do get those runtime errors and so on. But this is not a code-specific technique - anything you can put into tokens can be feedback here.

This here is where the feedback goes. This is the prompt for the teacher, which we bolster with as much context as possible so that it can guide and distill into the student. Remember, the goal of the teacher is to make adjustments to probabilities at each token, so we want the teacher to always be right.

So we start with our prompt of course, whatever problem we’re trying to solve.

Next, if we have one, we can provide an example of a correct solution from a previous rollout. That’s definitely going to help the teacher guide the student if it exists. If not, we skip this part.

After that is our feedback, from the compiler or the LLM judge or whatever. We should basically always have that, except for on the first run, but again if it doesn’t exist we skip it.

Then we provide the student’s original response. And you might be surprised that we’re providing an assistant response. Like, isn’t the model supposed to provide a new response?

But they’re using the teacher model differently. They’re not using the teacher to give its own answer, they’re using it to give its own probabilities at each token of the student’s response. You never get a complete response from the teacher. We’ll see more about that on the next slide.

So that’s what they explain in the paper, but I want to call out two things. One, you may be wondering why we’re doing any more training if the student was able to produce a correct solution. Well as we saw earlier with those pass@k graphs of increasingly high k, there’s a difference between “can produce a right answer” and “probably will produce a right answer”, and that difference may be enormous. We want to shrink that gap so that end users get better pass@1 results.

Two, why can’t we use someone else’s correct solution? The theoretical answer is that it will change the teacher’s guidance to bias toward the style of the correct solution as well as the contents of the correct solution. So if the model naturally is verbose, but our correct solution is concise, then the teacher will punish the student for being verbose, even though that doesn’t impact correctness. We want a clean signal from the teacher on correctness, which can only happen if the correct solution is from the same model we’re using for the teacher and the student.

However, practically they don’t test it. You could give them a pass and say it’s out of scope, but I think it would have been nice to try having another model family write the solutions and see what the penalty really is. If it’s small, it could be worth having humans write solutions to really hard problems and then having a SOTA model do this self-distillation thing. But then again, if we really believe in the elicitation hypothesis, you can’t teach new things with GRPO + RLVR or related methods. Still, I wish they had done the test.

Okay, so here’s how it looks when the teacher provides feedback to the student. In this example we have a question, then an answer from the student, then some text feedback. We’re gonna take all that and give it to the teacher.

The teacher is going to go token by token through the student’s answer, that bit in step 2, and compare its likelihood of that token vs the student’s likelihood of that token.

So in this example, for most of the program, the teacher and the student gave similar probabilities for the tokens: for the three tick marks, for the word “python”, for the function name etc.

But then you get to this plus symbol and you see the teacher disagree. Specifically, the teacher gives a very low probability for “+”, but the student gives a relatively high probability. Since the teacher knows the feedback and the student didn’t, the teacher is probably right, and we penalize the student.

Here’s a closer look at that same example. At the top they have the baseline approach of GRPO, which assigns equal credit to all tokens in a response. Red means less likely here, so because the initial response was wrong, GRPO gives low credit to all the tokens.

By contrast, SDPO assigns credit per token. We can see that for each token the student actually picked, in the first row, we also have alternative tokens it considered. So like in the first position, we have parenthesis + “range” as the selected token. But the student also considered a leading space and then “range”, “range” and a space afterwards, and a dot then “range”. The teacher agreed with the token the student chose, so the selected token has a white background.

Now if we look at the blue tokens, where the teacher assigned more credit, we see two things. One, in the second position, the student’s second pick gets high credit from the teacher, so it seems like the student was close to making the right decision.

Two, we see that plus in the sixth position is really the main problem, and that removing it and the “1” after was the right choice. The most blue token, with two close parentheses and then “\n”, is actually the final token that the student actually selected.

So again, the teacher is not generating a correct response we compare to; instead, the teacher is evaluating the student’s choice of the immediate next token given all the previous tokens chosen.

So that loss, that mismatch between teacher and student probabilities, is signal for SDPO. SDPO is the algorithm that turns the signal from the mismatch into changes of the model.

And because the teacher and the student are the same model, you can keep doing this over and over until performance saturates. So let’s say on the first round you did eight rollouts and got none right. But you got feedback, so you give that to the teacher and measure that mismatch between teacher and student probabilities. You have your loss, which you plug into SDPO, you train your model with that feedback, then you try again. Now two of your eight rollouts are correct. But the rest are still wrong. So now you take the feedback and the correct solution, give it to the teacher, calculate loss etc. Ideally you end up with all eight rollouts right, but maybe you top out at seven.

With standard GRPO + RLVR you just have to keep trying and hoping that the answer is in there somewhere, or that changes to the weights due to good signal on one problems helps you on this other problem.

With SDPO + RLRF, by contrast, you’re getting this extra feedback that can nudge you closer to a good solution. So it’s at least more likely that if you keep training like this, you’ll get there.

Here’s their first test of SDPO vs GRPO, with the Olmo3-7B model, working on a chemistry benchmark.

A few things to note. First, of course the accuracy improved, and improved more than with GRPO. They probably wouldn’t have published this paper otherwise.

Second, the x axis is in wall time, how long training took. That’s not usually a metric we see in these papers, they’re usually measuring performance against training steps or perhaps amount of data. Frankly I think it’s a little bit suspicious, like the accuracy improvements don’t stand on their own so they have to tout some other benefit.

Finally, and perhaps most underratedly, the response length went down rather than up. It went way down actually, 11x compared to GRPO, which as we saw incentivizes more tokens. “Longer thinking” as some would say, or “test-time compute scaling” - the idea that more computation to produce more tokens means more accurate answers.

But if you actually look at the thinking, it’s often repetitive or looping, and sometimes contradictory. When DeepSeek R1 came out people made a lot of observations about GRPO + RLVR getting models to say things like “Wait” or “Let’s try again” or “On the other hand”. Of course it’s good to confirm your answer from a different perspective, fox-like thinking as Isaiah Berlin would say, but doing the same calculations three times or ping-ponging between the same two points repeatedly isn’t productive.

So perhaps SDPO + RLRF is a more direct way to bring forth these reasoning paths than GRPO + RLVR.

Now here’s an experiment where they narrow their training set way down, just to hard problems, to see if that rich feedback really can guide you to the right solution.

One is exactly what they’re measuring. They’re taking a couple sets of hard tasks, where the initial model gets the answer right almost never for “very hard”, or where it gets the answer right less than half the time for just “hard”. That’s only 9 and 19 tasks, respectively, which is why the error bars are so big. Anyway, what they’re measuring is basically pass@k for increasingly large k, but with a twist: for SDPO, since they’re training after each round of a certain number of rollouts, what the x axis really measures is how many rounds of SDPO it takes to get a solution. That’s why they call the y axis “discovery@k” instead.

Two is what they’re comparing. We know SDPO of course, and best-of-k just means we take k samples all at once and count it if at least one of the k samples is correct. The other trendline, “multi-turn”, is like a hacky version of SDPO: instead of training after each round, they just keep adding feedback to the prompt. So each incorrect attempt and its feedback get added to the prompt, which could be like hundreds of examples. They do have to cap the number of examples due to context limits, but that’s still a ton of information they’re providing. Also they run a variant where they kept the attempt and the feedback, not just the feedback, and that did much worse. So that’s a practical tip for your prompting: try to boil down the lessons of failed examples if you can.

And three of course is the results! SDPO outperforms the best-of-k baseline, and also outperforms multi-turn. It’s really not surprising that sequentially trying and then updating outperforms just making more and more parallel attempts, but I was surprised how bad multi-turn was. To me that really highlights how much more effective feedback becomes when you train on it. Similar to how in theory you can carefully follow documentation to get the same outcome as someone with lots of experience, but in practice there’s something about learning that makes the knowledge fit into your own sort of mental schemas in a way that reading external text misses.

One other piece to remember about the results is that from an end user perspective, the model at the end of SDPO is going to be much more useful, because it’s much more likely to get you the correct result on the first try. Pass@k for a huge k is important scientifically, but it’s irrelevant for end users.

So one complaint you might raise is that SDPO + RLRF requires additional feedback compared to GRPO + RLVR. Remember, the “RF” in “RLRF” stands for “rich feedback”.

To address that, they separate out the RLRF part and just test SDPO vs GRPO. So now we don’t have to make any changes to our training data or our environments, just to the training process. Remember, GRPO and SDPO both depend on there being at least some correct rollouts to get some signal, they just use that signal differently. In the case of no correct rollouts, both algorithms do nothing.

The story here is again not a clean victory for SDPO. It wins most of the time, sometimes by a lot, but other times it is basically tied or even loses by a few points. Also keep in mind the scores are average accuracy across 16 attempts. I would have liked to see pass@1 to reflect the user experience.

Another way you might compare SDPO to GRPO is by making the binary more of a spectrum, like a smoother transition from GRPO to SDPO.

The authors do that here when they make the SDPO feedback increasingly granular. Recall that GRPO is feedback over the entire response, what researchers often call the “sequence”. So sequence-level SDPO is basically GRPO but with feedback given to the teacher prompt, which apparently helps, given the separation of the grey and blue lines.

The next step in granularity is token-level, just a binary about whether the teacher thought the student’s token was the best choice or not.

And then the final step is what we saw earlier, the logits, which are basically the probabilities of any possible token for the given position. That’s the full SDPO. We called that “soft-label distillation” in the background slides.

So intuitively you might think bigger models are gonna be better self-teachers, because they can take feedback into account better. Like anyone who has used AI consistently over the years knows models used to be terrible at taking feedback in conversation, missing out on subtleties or getting confused etc.

Turns out that does hold up in SDPO, at least for the range of sizes they looked at. SDPO is no better than GRPO at 0.6B, but consistently widens the gap up to 8B. Unfortunately they didn’t test the 32B model, which would have really made for a nicer story, but my guess is it does keep working at least up to a certain point. One wonders what the big labs with 1T parameter models can achieve.

My Takeaways

• RLRF could be a good use of human qualitative feedback

  • RLHF already uses human quantitative feedback (the Likert scale)

  • Justifications from existing RLHF data could become RLRF data

• Seems like a straightforward replacement for GRPO

  • There was some Twitter conversation about big labs using a better version of GRPO, perhaps this is it

  • There is a sense we are “due” for a new technical breakthrough (although maybe that was Engram)

• This all depends on the raw intelligence being in there from pretraining

  • Elicitation hypothesis

  • Does that mean we need to focus more on pretraining to get the most out of post-training?

• RSI is coming

  • Anything you can verify can self-improve

  • You don’t need a smarter teacher, just a student with good feedback looking over their old results

Paper · Repo · Blog

Background

I want to start with the notion of scaling laws and how they relate to technology itself.

In the narrow sense for LLMs, scaling laws tell us how to optimally balance compute, data, and parameters as we scale up our efforts. Like whatever your constraint is - typically it’s the compute budget you have for pretraining - your other optimal settings fall out of that. So if we have a certain number of FLOPs, which is the standard unit of compute, then using the scaling laws we can calculate the model size and the number of tokens that will minimize the loss.

That’s incredibly useful for anyone trying to build an LLM. If scaling laws didn’t exist, and you had to find the right balance empirically, you might spend months just trying different combinations every time you wanted to pretrain a new model.

The scaling laws are also pretty much independent of model architecture, at least within the transformer family. The laws are not 100% exact, so it’s more like the error bars cover all the changes people are always making to different parts of the architecture and training recipe and so on, but broadly speaking they’re quite robust.

The graphic here is from the most famous scaling law paper, Training Compute-Optimal Large Language Models, better known by the name of the model they trained in the paper: Chinchilla. In the paper, they improve on a previous result by Kaplan et al, and they show that recent models are undertrained, meaning they are too big for the amount of training compute they received. The Chinchilla model, despite being less than half as big as GPT-3, significantly outperforms it.

There are many other specific scaling laws - we’ll cover one in the paper today - but more broadly is the lesson of scaling in general.

Its most famous avatar is The Bitter Lesson, a remarkably brief essay from 2019 by pioneering AI researcher Richard Sutton. It opens with the following lines:

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.”

Did you catch that? He’s saying scaling of compute is the baseline assumption, and any work we do better take that assumption into account, or else it will only matter in the short-term until scaling catches up.

Sutton gives a few examples. In chess, “brute force” search-based methods dominated methods that encoded centuries of human chess wisdom. In Go, a similar story occurred. In speech recognition, learning eventually beat the methods based on human knowledge. In computer vision, much the same happened.

In all cases, work that took advantage of the inevitable increase in compute outlasted work that made efficient and scrupulous use of the compute of the day, acting as though the “inefficient” and “brute force” methods would never become feasible.

In my view, this is the major trick in the tech playbook: work a problem until the only constraint becomes resources, then dump resources into the problem. A machine where the bottleneck moves around a lot isn’t much of a machine at all; it can’t deliver predictable results and thus requires a human at the helm. A true machine does its work reliably and can do more work the more fuel you give it. We are all living through a period of scaling right now with the way everyone is building out data centers.

Scaling may not continue forever, but at least within some regime, we want scaling and we want to know the scaling laws, to design bigger systems and to forecast future performance.

It’s always a bit funny how some scaling laws work. Like with the Chinchilla scaling laws, that seems to be a natural property of the universe, more akin to a physical law. But for Moore’s Law, which underpins The Bitter Lesson, that’s a prediction about society and how it will continue allocating resources, in scientific discovery and also in hardware production.

The famous METR graph is a scaling law of the latter type. As I’ve covered before, this graph shows how models are becoming more and more autonomous over time, able to handle increasingly long-horizon tasks as measured by how long it takes a competent human to do them.

The current results are actually somewhat in doubt - Mythos is too advanced for METR’s measurement task suite - but we do have reliable data points all the way up to Claude Opus 4.6, which came out in February 2026.

Now the METR folks show just the one fit line, one scaling law. But arguably there is a break in the line, which I’ve drawn in orange, with growth at 10x/year, meaning by February 2027 we’ll have an agent that can do about 100 human-equivalent hours of work autonomously at 50% success rate. Pretty nuts.

Anyway, that new trendline starts with the introduction of reasoning models. See, o1-preview introduced another axis of scaling: not training compute, not data, not parameters, but inference compute - how long the chain of thought could get and still improve the final answer.

That’s the other part of the scaling law puzzle: where is everything we could scale? Because once we find them, we can dump more resources into them and juice performance.

Now in the course of these increasingly long tasks agents can do, there is plenty of trial and error. In some ways that’s what makes agents work: interacting with an environment, gaining information, incorporating it into future actions. That sort of exploration and iteration isn’t really present in chatbots, and it’s also not part of classic benchmarks where you’re supposed to respond to a prompt in one go.

Of course models have always been able to adapt based on their context; even responding to a basic prompt is in a sense adapting behavior to context. Historically the most common tactic has been to write rules and provide examples to put in the prompt, which folks call “in-context learning” and sometimes “few-shot” or “many-shot” depending on how many examples are in the prompt.

But now there’s this dynamic element, where an agent’s experience within the same task becomes part of learning. Maybe the learning goes in a scratchpad just for the task, maybe it goes into a long-term memory system, maybe it just stays in the trajectory and survives compactions. However it happens, the agent is learning as it works, just like a human would.

Sometimes people call this continual learning, although sometimes I see that term specifically for learning into the weights, i.e. training the model. That’s not what we’re talking about here. What we’re talking about is all ultimately in the context window, although it may live in a different tool and only enter the context window when the agent calls the relevant tool to fetch it.

Continual learning is crucial because lots of important context will never make it into the training data. Like most businesses are not going to fine-tune a model on their business logic and proprietary information. So that information has to come through the context window.

The graphic here is from Continual Learning Bench, a bit of prior art in this space, which measures an agent’s ability to learn information from one task in a domain or environment and apply that knowledge on another task. So in this example, for the first query, the agent learns some stuff about the database, shown in the green box. Then for the second query shown, with label Q10, that green box of knowledge appears at the start to inform the agent on the task. Later on, with label Q25, the agent has to recognize which information is now stale.

For CL Bench, they test different models but also different context management systems, including Claude Code and Codex, which are coding agent harnesses but include many tools and techniques for managing context, like memory. They find that pure in-context learning, i.e. just keeping everything from the trajectory in the context window for as long as possible, does the best. However, there are only six different tasks, although each one contains many different “instances” - like Q1 and Q10 and Q25 in the diagram. And each instance is not that long, so even though an overall task can be quite long, the continual learning is more like a shared base for each instance rather than one long narrative.

So models apparently have this continual learning ability. But as we know, with agents, the model is only half the story; the other half is the harness.

Many agent harnesses improve continual learning, but usually as a means to improving performance on the agent’s focus area, like coding as with Claude Code. But there is one famous harness whose main job is arguably to improve continual learning: Autoresearch, from Andrej Karpathy.

The idea of Autoresearch is quite simple: plan an experiment, run it, observe the results, and use the results to plan a new experiment. It’s the scientific method, but encoded into a harness. The human picks the goal and hands off the rest to the agent.

In the actual Autoresearch repo, the goal is to train a version of GPT-2. But the general method applies to any problem with a clear metric to optimize. In the graph here, that metric is the loss, which is the classic metric to minimize when training any LLM. We will see a version of this graph later, but flipped, since the goal will be to maximize a score rather than minimize a loss.

The Paper

Since this is a benchmark, let’s start with the tasks.

We have 134 of them, sourced from experts in many fields, with mean human effort clocking in at over 57 hours. So these are substantial tasks, putting them at the leading edge of the benchmark task time horizon distribution. We recently covered Agents’ Last Exam, and they were bragging about how a few of their tasks made it to the weeks timescale.

As we’ll see, it’s important for EdgeBench that an agent not solve any task on the first try, because what they’re measuring is ability to learn from feedback. So duration is more than just a way to match difficulty with agent capabilities - it’s intrinsic to the purpose of the benchmark.

Now as for domains: unsurprisingly, given the likely personal networks of the researchers, the tasks skew STEM, with science, programming, and ML representing over half the tasks.

Many of them are multimodal, but the authors specifically note that “tasks whose primary difficulty lies in visual understanding, especially GUI operation, are excluded” - they want the hard part to be reasoning and iteration rather than pure discernment. To me that presents somewhat of a challenge with their Interactive Games & Simulators bucket, but they don’t explain any further.

Sadly for us, there is also little detail about how they made the benchmark.

Through manual effort, the tasks become environments, where the agents are free to experiment. The diagram shows an inner loop on the left in blue, where quick feedback from tools helps the agent learn the environment better. You can see examples in blue text along the bottom.

At a slower cadence, there is an outer loop, shown on the right in orange, where the agent can choose to submit its work to a judge. The judge evaluates the agent’s work against the ground truth or rubrics or similar, then returns the score and feedback. The ultimate goal is to maximize that score. The agent can submit any number of times, although there is a cooldown period of two minutes between submissions. Again, there are examples along the bottom in the relevant color.

So both loops allow for continual learning. Of course, since this is a benchmark, we have right answers in the outer loop to compare to. In the real world, where the right answer may be unknown, you can at least shoot to maximize some metric like we saw with Autoresearch. You just have to be careful of Goodhart’s Law and mistaking the optimized metric for the thing you actually want, if the two aren’t identical.

Here are the Autoresearch-esque curves, showing how top performance per model rises over time. Note how chunky the graphs often are; progress seems to happen in punctuated equilibria rather than in smooth steps, at least at a per-task level.

The keys only show the model, but the harness and context window are important too. So in full, we have:

• GPT + Codex @ 256k compact window

• GLM-5.1 + CC @ 200k compact window

• DSv4 Pro + CC @ 200k compact window

• Opus 4.8 + CC @ 1M compact window

They show later on that 1M provides about five percentage points of advantage over 200k, with a slow and steady decline in that gap as time elapses.

I’m a little surprised they didn’t use a neutral third-party harness here, but I do think there’s value in shooting for maximum performance rather than controlling variables. Just more evidence the model and the harness are becoming one integral whole.

Now if we zoom out a level, from individual tasks to task categories, we see much smoother curves. In fact, they’re so smooth that you can tightly fit the same type of curve to all of them: the log-sigmoid.

A sigmoid is a classic S-curve, starting at zero and then accelerating to a midway point, then decelerating and leveling out. A log-sigmoid is that but on a logarithmic x axis, like in these graphs. So basically these models all form S-curves in log time.

That’s remarkable, because the individual tasks graphs didn’t seem to show much of a pattern at all. But if you aggregate just enough, the common pattern emerges. They have some results later showing that the pattern isn’t guaranteed from the aggregation, that it’s a real trend in how agents learn over time and over many iterations.

It’s also helpful, because it allows us to predict performance past the 12-hour window, which really saves a lot of time and tokens compared to having to actually hit 18 or 24 or 72 hours.

Here’s a validation of the predictive power of the fitted curves, taking performance through a 6.5 hour window and projecting it forward to 12 hours. The curves are pretty dead-on, even for the most inaccurate one, GPT-5.4 in teal. And actually they explain that for GPT-5.4, the score was lower than expected due to API flakiness rather than poor model performance.

So not only do they empirically validate their scaling curves, they also provide a theoretical justification. I don’t want to spend too much time on it, but the broad idea is to represent different bits of knowledge and insight as nodes on a graph. Any prerequisite knowledge is a connection. If you assume the graph structure is self-similar, basically that it looks roughly similar throughout, then you naturally expect a log-sigmoid shape when you chart progress through the nodes over time.

They’re not rigorously proving that every task will follow these laws, in part because some graphs will not be self-similar, but self-similar graphs are common enough that the theory explains why log-sigmoids are plausible at sufficiently high abstraction.

So we have scaling for a given model over the elapsed task time. That’s nice for understanding a model’s performance ceiling, and it saves us some experiment time, but there is a juicier finding in EdgeBench.

Specifically, it’s this chart, basically their version of the famous METR graph we covered in the background portion. What this graph shows is how quickly a given model learns, as measured by average performance gain on a select subset of tasks within a two-hour budget per task.

What they find is that newer models are consistently pushing the frontier, doubling the learning speed about every three months. That’s 16x/year, which is faster than the METR graph at 10x/year. So if the trend holds, not only can we expect agents to accomplish longer tasks as measured in human completion time, we can also expect agents to complete the same tasks more and more quickly - and that’s before taking parallelism and multi-agent systems into account!

Now one thing to keep in mind is they aren’t measuring pure model performance, or even model + harness performance; they are also measuring hardware performance. Like if two equally capable agents are competing, but one gets twice the token output speed, it will finish significantly earlier.

In practice though, OpenAI and Anthropic shoot for similar output speeds unless you pay extra for faster tokens, so I think these results hold. In fact, the only result I would caveat is for GPT-5.4, which as we discussed encountered API issues and thus has a lower score than expected. So if you imagine the datapoint for GPT-5.4 on this graph a bit higher in the world where the API didn’t flake, it fits the trend quite well.

On the left is another look at the same data, showing performance at the start and end of the two-hour window. The researchers selected the subset of tasks based on how similarly all the models initially did, so if new releases start to do significantly better to start, they may have to swap out the subset. But for now the initial performance seems about even, and there’s plenty of performance ceiling to go.

On the right though is a new trend, showing how effective new submissions are. Specifically, the graph shows what percent of submissions in the outer loop improve performance. So it seems newer models are more likely to make submissions that improve their score. Here is how the researchers put it:

“Stronger agents use feedback more deliberately: they build a submit-ready baseline, preserve the current best solution, make focused changes, and use feedback to keep gains or roll back failures. Weaker agents more often over-trust local proxies, bundle unrelated edits, or continue broad exploration after feedback has ruled out a direction.”

I do think it’s a bit confounded whether newer models just have better work and thus better submissions vs having better discretion on which work to submit, but either way you as a user are more likely to be looking at better work.

We do have a leaderboard of course, given this is a benchmark, but honestly it’s not really the main story here; the big finding is the scaling laws rather than any one point in time.

Now of course with more submissions you’re likely to improve your high score. So the question for continual learning is, how much does it help to see your previous work and submissions? What is the impact compared to the baseline of many independent attempts, where you only get one outer loop submission?

Here they answer that question, using Opus 4.8 to test. The lower curve, comprising independent submissions, is basically pass@k but with k as a function of time. And it displays the classic diminishing returns curve of most pass@k results.

Meanwhile, the upper curve comprises dependent submissions, i.e. where the agent sees the results of each outer loop submission and can continually learn. Here we see a consistent and perhaps even widening gap, where more experience pays dividends. So yes, purely providing more chances does improve performance, but not as much as continual learning does.

Finally, let’s zoom in on one task example, with an attempt by GPT-5.5.

Here we see the classic punctuated equilibrium pattern, long periods of little or no progress with occasional big jumps, until time runs out or until the agent reaches its limits. Personally I don’t know anything about detecting gravitational waves, so the specific jumps don’t mean much to me, and I don’t think they’re crucial to comprehend.

What’s interesting about this example task though is the high-level strategy GPT took to partially solve it. The authors list four points:

1. The agent first makes the problem measurable before making it better

2. When direct repair stalls, the agent decomposes the failure into searchable subproblems

3. Identifying a main bottleneck lets the agent keep searching productively

4. After finding a stable solution, the agent keeps the core and repairs only the remaining errors

To my eye that’s quite a general and broadly applicable set of steps. I wish they had distilled more examples like this, or perhaps taken these points and checked how pervasive they were in other attempts, by GPT and by other model families. I also wonder if prompting the agent to use these techniques would improve things.

My Takeaways

• We may have a new METR to track

  • Why is METR having trouble with sufficiently difficult tasks, but EdgeBench was able to produce them?

• Unclear what specifically makes learning rate scale

  • Model (training data)

  • Harness

  • Context length (they did prove this one actually)

• Maybe we can add signals to increase learning rate further

  • More and better feedback from tools

  • Another agent to work with

• In-context learning and error correction are mandatory

  • Even a 99% accurate agent has a 50% chance of failing after just 69 steps

RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

or, Theory

Paper · Repo

OpenClaw-RL: Train Any Agent Simply by Talking

or, Practice

Paper · Repo

Background

So the first thing we need to talk about, and really the framing we should have for almost all AI progress, is this graph here.

It’s from a research organization called METR, and it shows the increasing duration of tasks that AI can handle autonomously. The duration is in human time, so for example they’re saying it takes competent humans about 10 minutes to find a certain obscure fact on the web, and just under an hour to train a simple type of ML model called a classifier. METR has put together a bunch of tasks with varying human durations, and it administers that suite of tasks to new models, then graphs the result.

There are two toggles on the bottom here. One adjusts the scale of the y axis, so right now it’s a log scale, meaning that our straight line here is actually an exponential line on a linear scale. The other toggle adjusts the quality threshold. Here I have it on the more conservative 80% Success setting. So like Claude Opus 4.6 can do stuff that normally takes humans about an hour, completely autonomously, at 80% success rate.

Now this graph shows one trendline over the whole period, from GPT-2 through Mythos. But if you look at the green dots, which represent SOTA at the time, really it looks like two trendlines: one trend from GPT-2 to GPT-4o, in that quick run of three green dots below the line; and then another trend from o1-preview onwards. That break is where reasoning models came in, using test-time compute scaling, using very long chains of thought to arrive at better answers.

If you believe there really are two trendlines, then the second one is steeper than the first. And on that newer steeper trendline, the time horizon is 10x’ing every year. So Opus 4.6 for example came out in February 2026 and hits ~1 hr on this graph. That means in February 2027 we’ll be at ~10 hrs. That’s an entire day of work, done completely autonomously, and largely successfully.

So a lot of the work between now and then, and after then I suppose, is about long horizon - increasing the duration of task that a model or agent can do, not necessarily making it smarter or more knowledgeable.

Of course the longer the horizon gets, the longer we have to wait and the more things that have to go right in order to give out a positive reward. That latter problem, where you have more and more possible points of failure, is what makes long horizon training so difficult.

Now if we think about a human setting, where you’re trying to teach someone a skill or maybe learn one yourself, the person learning probably isn’t going to do a whole practice run themselves and then check back with the teacher or the instruction manual or whatever at the end. Instead, we usually check in at certain steps or milestones. Or like in cooking, you might taste test along the way rather than tasting only at the end.

The equivalent in machine learning is process supervision. In process supervision, we give a reward for every step, judging two things: one, is this step accurate; two, is this step on the right path. So looking at a math example like this, which is from an OpenAI paper that we made data for, you have the accuracy of calculation and the helpfulness of that calculation in getting the right answer. Of course in this example the final step is the incorrect one, so all we have to judge is the accuracy of the calculation, but in earlier steps you implicitly have to judge both with just one score.

Process supervision works for anything that happens in discrete steps. Math is like this, but creative writing is not. Even programming mostly is not, since you can often change the order of code blocks without impacting functionality. But one big chunk of the ML world is like this: agent trajectories. The observe-think-act loop is inherently sequential, and changes to the environment often demand a certain sequence. For example, navigating through a website requires a series of clicks in a certain order.

So it seems we can add process supervision onto agents pretty naturally. That means we’ll have a process reward model, a PRM, that gives out rewards at each step. That’s in addition to whatever reward we give at the end of the trajectory, which people sometimes call an outcome reward model or ORM, but often is just called a reward model, since outcomes are the default thing to give rewards on.

So if intermediate rewards are so valuable, and maybe increasingly necessary, why doesn’t everyone use them? We’ve covered agent RL papers before, and none of them have had intermediate rewards. Why not?

In my view, there are two reasons.

One is, it’s expensive to make process supervision data. There are just way more annotations per task. If you can avoid process supervision and just use outcome labels, that’s way cheaper per task, so you can get more tasks and more diversity etc. This is partly why RLVR data has largely displaced PRM data for reasoning. Of course, if you could get that data for cheap or even free, the balance might change.

The other, thornier reason is reward hacking.

Briefly, reward hacking is when the stuff you reward is only a proxy for what you actually want, and that gap between the rewarded thing and the actual goal causes unexpected or bad behavior. For example, you might reward a model for passing unit tests on a coding problem, but the model could learn to just hardcode passing behavior instead of writing the real program. Or even worse, it could learn to edit the tests to just always make them pass.

Reward hacking can happen whenever there are rewards. If you just have an outcome reward, then there’s only one possible hacking target per task. But if you add in intermediate rewards, you’ve added new hacking targets. So intermediate rewards already add some risk.

But there’s a deeper problem with intermediate rewards: they can distract or derail the agent entirely. The gif on the slide here shows one such example. It’s from a 2016 blog post from OpenAI, by Dario Amodei and Jack Clark actually, who later went on to found Anthropic. The post is called “Faulty reward functions in the wild”. In it, they describe some RL experiments they did in a boat racing game called CoastRunners. As humans, we know the goal of a boat-racing game is to place first. But actually, the game only tracks points, which you get from hitting these floating green blocks and then from your finish order. The most points wins.

When they set their RL agent loose on this game, they saw something funny: the agent learned to ignore the race and just steer the boat in a circle over and over again, racking up points from the same three green blocks. Often it just circled forever and never ended the race at all.

That is the danger of intermediate rewards. You have to pick something, often arbitrary or not strictly required for your end goal, and reward on it. Even with domain expertise and careful design, it can still go sideways.

On a different note, we also need to cover a training technique called “self-distillation”.

Let’s start with plain ol’ distillation, which is where you use one model to train another model, “distilling” the wisdom from the teacher model into the student model. Distillation has been in the news from time to time because OpenAI, Anthropic, and Google have all caught the major Chinese labs distilling off their models, which violates the terms of service.

The naive way to distill is to have the student copy each output token from the teacher in response to a prompt. Like if you penalize the student for outputting anything except what the teacher output, then the student will learn to sound more like the teacher.

But if you can go one level deeper, to the probability distributions of the teacher rather than just the one final token it spit out, that teaches much more richly. That’s what we’re looking at here. Sometimes people call that “soft-label distillation”, as opposed to the “hard-label distillation” I described earlier as the naive approach.

So the Chinese labs have been distilling off the American models by passing the same input to the teacher and the student. The teacher is better than the student, so the student learns and improves.

But it doesn’t have to be that way. In the paper from two weeks ago, they actually kept the model the same in both cases - same model for teacher and student - but they gave the teacher extra context. Just like having your friend quiz you, with them looking at the answer key so they can guide you to the right answer.

In the example here, the teacher for this SQL bot gets some examples in its context, whereas the student does not. So when you do distillation, the student is receiving the extra wisdom the teacher had due to its extra context, without actually needing that context in the prompt anymore. Since you’re using the same model in both cases, it’s called “self-distillation”.

Notice how flexible this can be. Anything you can put into words can be distilled. For example, apparently Anthropic uses distillation to teach Claude how to be Claude, with a big document in the teacher’s context containing Claude’s principles and operating procedure etc.

The paper from two weeks ago used context distillation to incorporate feedback from the environment. So the model would attempt a task, get some feedback like failing unit tests or a critique from an LLM judge, then put that into its context in the role of teacher. Then the model with its teacher hat on would basically grade every token probability in the original response, then train to minimize that loss. They called that “self-distillation policy optimization”, SDPO, and unlike other techniques like GRPO, it doesn’t require rewards that boil down to numbers. It’s a richer signal that naturally occurs in any system that provides feedback.

The Paper: RLAnything

Okay, so our first paper is called RLAnything. That’s because their proposed system is fully generic, across any sort of task, or at least that’s the claim.

They break down RL into three parts:

1. The policy model, which is the model actually generating responses or actions etc

2. The reward model, which judges process and outcome rewards, except for any verifiable rewards you might have that don’t need judging

3. The environment, which includes the prompt as well as the tools and state the agent has to navigate to accomplish whatever task

They’re going to be adjusting all three of these dynamically to maximize performance, which we’ll see in the next few slides.

While we’re here though, a couple things to sort out. One is some terminology: they often say “step-wise” instead of “process”, and they have a “critic” model that is actually a separate LLM not involved in training at all.

Second is the notation. I know the math looks very scary, but it’s just a lot of annoying symbols to keep track of, no exotic operations. Like for the left side, all they’re saying is the total reward is the outcome reward plus the average of the process rewards, which they request from the PRM m different times just for consistency. And then O and S, the outcome and step-wise rewards, are gonna be either 1 or -1, nothing in between. Also P stands for the policy model, R stands for the process reward model, and E stands for the environment.

Third, the PRM is going to return a score but also its reasoning, like a justification our contributors would write in an RLHF task. That will be helpful down the line as we’ll see, especially for separating the correctness of the individual step from the helpfulness towards a good outcome.

Now for their experiments they’re primarily testing on two benchmarks of a couple hundred tasks each:

1. AlfWorld, which is basically a text adventure game

2. OSWorld, the gold standard for computer use agents

Sometimes they report the results for AlfWorld as “LLM agent” and the results for OSWorld as “GUI agent”.

As for the models, they’re using Qwen2.5-7B for AlfWorld and Qwen3-VL-8B for OSWorld. Not the newest models, and not the biggest either, but quite common for academic work.

Here’s an illustration of how they adjust the task and environment to balance difficulty and novelty and hit that Goldilocks zone. Remember, the PRM is basically a classifier, and classifiers do best when the classes in their training data are roughly balanced. So we roughly want to have an equal number of good and bad steps to train on and thus improve our PRM, which is key to this whole approach.

On the top is an OSWorld example, where the reward model gives two accurate criticisms of the policy model. An outside LLM then reviews the trajectory and the criticisms and makes a modification, which allows the policy model to improve from 0% to 25% accuracy.

They also have AlfWorld and coding examples with the same idea, although in both cases it’s to make the task harder.

This graphic encapsulates their results, so let’s go through each:

1. On the first one, they’re showing that you get the best results when you make all three components of your system dynamic. So you want to train both your policy and reward model, and you want to adjust your environment to be neither too hard nor too easy, so that you’re always getting some signal but always leaving room to improve

2. On the second one, they’re comparing the value of process rewards and outcome rewards. So for the yellow line, it’s just outcome rewards, with human labels. Then on the blue line they’ve added a PRM, but they don’t train it, which maybe helps a little but shows a PRM per se is not the big difference. The red line however is only a PRM, but trained, and that ends up doing the best. So that shows the value of a good PRM if you can get it.

3. On the third one, they’re just showing overall improvement from their method. LiveBench is actually a coding benchmark and isn’t an agent thing, yet their method even works there

4. Finally, on the fourth one, they’re showing how their adjustment to environment and task difficulty progresses over time and correlates with the quality of the policy and reward models

No mind-blowing results, but again, the emphasis is on how general and automatic the method is. They didn’t need to make any data, including those risky intermediate rewards we talked about earlier.

And they didn’t need to customize the setup much. They did have to think a bit about what inputs the PRM received in each case - the prompt + a summary of past actions and observations for both, the last two images for OSWorld - and they did have to know where the metaphorical “knobs” on the environments were for difficulty adjustment. But that’s about it.

Here’s a closer look at the results on the benchmarks, with different components allowed to change in different rows. A couple quick pieces of terminology:

• OOD is “out of distribution”, like eval tasks that weren’t similar to the training tasks

• UT stands for “unit test”, how accurate the PRM was at writing unit tests for judging step quality

• “Detect” means accuracy at detecting bugs

So they’re improving across the board, with more dynamic components being better. A couple thoughts.

First, look at the difference between process and outcome reward accuracy between GUI Agent and LLM Agent. That’s a clean demonstration of how process rewards have two jobs: checking the correctness of the immediate step, and estimating how much that step helps get the right final result. So we see in LLM Agent they’re similar - it’s roughly as hard to do one as the other. These are short trajectories and each one needs to have the end goal in mind. But for GUI Agent, they’re quite different; it’s apparently quite easy to tell whether the agent actually performed the right action, but it’s often not obvious whether that action will lead to a good outcome. That makes sense for computer tasks, where you have to click through to see everything, it’s not all laid out for you at once. Like sometimes you have to click around to check out all the options and menus and so on.

The second thing is that you don’t need a perfectly accurate PRM to improve results. You can see for both agents that the improvement in the policy model is far greater than the improvement in the reward model in most cases. Even noisy signal is better than nothing, at least at these noise levels. Below 50% accuracy for the PRM I expect the policy model would suffer.

The Paper: OpenClaw-RL

Okay, so we’ve seen how process supervision can help agents. Now we’re going to see how to collect process supervision data naturally.

Because if you think about it, natural use of chatbots and agents doesn’t quite look like training data. Like if you’ve worked with agents at all, you know you’re often in dialog with them, providing feedback on their work and often not just leaving them to work on their own.

And nobody is really going to make nice training data out of their own usage. It’s a lot of work to get nicely scoped tasks and neat boundaries on trajectories and final outcomes and rewards. That’s why we get paid!

But surely there is a way to use the conversational feedback users often give, whether positive (like saying thanks or showing excitement) or negative (like scolding the model or making a correction).

Now this paper has much more of an engineering focus than the previous paper. So you’re going to encounter some training and hosting tools we don’t tend to cover here, especially because Scale focuses on the data side. So RL servers, Megatron, SGLang, slime, these are not important for us frankly. Just know they are the software that makes a lot of this all work.

So we’re going to be collecting and using feedback in three different ways here.

The first is basically a PRM, like in the previous paper. The rewards are binary, i.e. just -1 or 1 or 0, even though this graphic doesn’t say it. And they come in the course of conversation rather than waiting until the end. New words but same meaning really.

The genuinely new thing here is what context they give the PRM. Specifically, they’re providing the agent’s action, environment feedback, and user response. So if the agent makes a tool call and it returns some error about missing arguments, that’s probably gonna earn a -1. Or if the tool call works but the user complains that the agent is fetching the wrong information, also -1. Conversely, if the user seems happy, that could earn a 1. And if there’s no clear feedback or too little to judge on, the PRM can return 0. So this method rewards the user for more natural or expressive use of the agent.

The second method is also like a PRM, but with distillation instead of rewards. So it’s operating on the same single-step scale, and it’s using the same sources of feedback, but it’s putting them to work in different ways. First, the distillation method includes a lot of filtering and only uses clear mistakes for feedback, since the teacher is only helpful if the student has something to learn. Also, there’s some consolidation, where they use a model to summarize the mistake rather than just dumping the raw interaction into the teacher’s context.

The third method is pretty much what we saw in RLAnything: combining process and outcome rewards, PRM and RLVR basically.

The graphic doesn’t make the difference clear, but it’s important to separate the first two from the third. The first two methods are only stepwise, so they can run continuously, alongside all your interactions. The third one, by contrast, requires a definite outcome to then reward.

It’s a bit of an artificial marriage then, these continuous vs task-based methods. And actually the things they improve and the way they measure are different too.

For the continuous methods, the feedback from conversations, the researchers used two related examples: a student who wants to use OpenClaw to do his homework, and a teacher who wants OpenClaw to do grading in a certain way. Pretty subjective stuff, which makes sense with the more qualitative feedback their methods are able to capture, although the results are less compelling in my view. They have a scoring method, and you can see the improvement in scores on the right, but the scoring is just an LLM judge scoring the agent’s response given the user’s preferences.

This is the type of change you could prompt engineer, rather than something that absolutely has to be trained. So the value isn’t results per se, it’s how you can get these results “for free” just by incorporating natural signals.

They also have the more traditional objective results on benchmarks. Keep in mind they’re using Qwen3-4B across the board, so the gains are respectable. I don’t have much to add because this part is basically a rehash of RLAnything, but it seems their two new methods may be helping too.

My Takeaways

• Where is the role for our training data?

  • Extracting training data from natural use means it’s way cheaper (even though lower quality/hit rate) and already from the natural distribution

  • So perhaps the economics favor the low-volume, high-utility cases (e.g. medicine, frontier STEM)

• Can PRMs replace human-written intermediate rewards?

  • We know intermediate rewards can incentivize weird behavior

• Another step on the path to continual learning

  • Also similar in spirit to Karpathy’s Autoresearch

• This is probably already happening in some form

  • Big labs I’m sure

  • Possibly anyone with an agent and access to the weights

Paper · Repo · Hugging Face

Background

To understand this paper, we have to understand how a model on a GPU turns your prompt into predicted tokens.

Let’s start with what we already know from our daily use of LLMs: you pass in your prompt all at once, you wait a bit, and then you start getting back tokens one by one.

Already we can start to relate to this diagram. The part where you pass in your prompt is called prefill. During prefill, you are filling up the working memory of the model, what we call the KV cache. The diagram here kinda breaks up “KV” and “cache”, but you can see that the prompt turns into KV vectors, and then those get cached, hence “KV cache”.

So now we have all the input in our working memory. That can take a long time depending on the hardware you’re using and how big your input is, but importantly, prefill happens for all input tokens in parallel. Typical prefill speeds are in the hundreds or thousands of tokens per second.

So once prefill is done, the model is ready to start making predictions. And it makes those predictions one at a time, as the diagram shows: first “jumps”, then “over”, etc etc. This stage is called decode, because we’re decoding the mathematical representations that the model works with into words that humans work with.

Because decode runs one token at a time, it is much slower than prefill, typically in the tens of tokens per second.

Note that when we decode a token, it gets cached too - that’s why the dotted line at the top spans prefill and decode. So the KV cache, that working memory, grows every time the model outputs a new token.

Now let’s look a level deeper, at the hardware. On a GPU, you have your chip, and you have your memory, aka your VRAM. When you start up your model server, the server reads the model from your hard drive and puts it into your VRAM, right next to your chip.

When you actually use your model, like by sending it a prompt and getting back a response, your model server takes one layer of your model at a time from VRAM and sends it to your chip for computation. So if I’m at the very first attention layer, it’s gonna take my input and the matrices that actually make up the first attention layer and send ‘em to the chip for multiplication and so on. Then it takes that first attention layer back from the chip, along with the KV cache created from the computation, and it’s gonna send in the first feed-forward layer for computation.

That process of loading and computing and unloading happens over and over again until you’re at the final layer, where the chip can finally produce the predicted token. Then you gotta do the whole routine over again to predict the next token.

So this shuttling of weights to and from the chip is typically what slows you down - the constraint is your memory bandwidth, not the speed of your chip. If you can somehow compute multiple tokens at once in decode, like you do for prefill, then ultimately you can predict tokens faster.

The question is, how do you predict multiple tokens? If predicting a token requires understanding all the tokens before it, how can you predict more than one?

The answer is in the name of the technique: “speculative decoding”. Instead of making a brand new prediction every time, you speculate about what the next few tokens will be - you take a guess beforehand and then check.

Speculative decoding works for the same reason prefill is faster than decode: inputs get processed in parallel. As we said before, once you load the weights onto the chip, it’s quick to do one or two or three or four calculations. As long as you have a good way to guess tokens, the model can check them all in parallel.

Of course, if the first token fails, then the other ones you guessed after it will likely be wrong and you’ll have to throw them away. But if your method for guessing tokens is cheap enough, and you’re not wrong too often, it can work out.

One common method is to have a version of the model itself make the guesses. Specifically, a much smaller version, ten or a hundred times smaller in fact, so it’s much faster and also can fit on the same GPU. This “draft model” as it’s called is not nearly as smart as the target model that is actually producing tokens, but it’s often smart enough. After all, most tokens are not incredibly complex or subtle; language is chock full of common and supporting words, and a lot of sentences are pretty mundane, meant to support the occasional novel or surprising sentence. It’s even more true for code, which demands predictable structure in a way natural language doesn’t.

As the graphic shows, the draft model quickly produces a few tokens, which all go into the target model in parallel rather than in serial. And thinking back to the last slide about where the bottleneck is, because you now have multiple tokens ready for computation, you save all that shuttling from VRAM to the chip on the second and third and fourth tokens.

In the example here, we have indeed generated four tokens, but the third one gets rejected, and the target model’s predicted token takes its place. The fourth draft token doesn’t get checked at all, because it depends on the third draft token being correct, which it wasn’t.

Let’s zoom in on this example a bit more. We have our four speculated tokens from the draft model, and we’re going to verify them with the target model.

Specifically, we’re going to check if the probability of the speculated token for the target model is at least as high as the probability of that token from the draft model. Like in our case, the target model thought “Brown” was 93% likely, and the draft model thought it was 92% likely, and since the target model is smarter, we take the increased probability as a sign that the draft model was pointing us in the right direction. Similar story for “Fox”.

But for “Hopped”, the draft model was more confident in that token than the target model was. That’s a bad sign and means we should reject the draft model’s choice.

Incidentally, when the target model rejects the third token, it substitutes its own - in this case it’s the word “jumped”. That extra token you get from the target model when it rejects the token from the draft model is called a “bonus token”, because you get it “for free” in the process of verification. If you’re really lucky and all your speculated tokens get approved, you get a bonus token after that, directly from the target model. Like if all four tokens had been right in this example, we also would have gotten a fifth token as well, with virtually no extra effort.

Now as you might imagine, the draft model is going to be better at predicting some tokens than others. Like on a hard reasoning problem, the acceptance rate will be quite low, maybe like 25%. But on a more structured and straightforward task, like using a web search tool, it could be near 100%. So speculative decoding isn’t a complete across-the-board speedup, but for a lot of mundane LLM uses it’s helpful. It’s really an empirical question depending on your use cases, your hardware, what model you’re using, stuff like that.

I should add that there are other forms of speculative decoding, or of trying to predict multiple tokens in one go anyway. We’ll briefly see one called EAGLE-3 in the paper for example.

Another example, which we’re looking at here, is literally called “multi-token prediction” or MTP. The difference with MTP is that is has to be part of a model’s training from the get-go, it’s not an external enhancement.

You can see it right there along the top, at the boxes labeled “Cross-Entropy Loss”. The loss is the single number that tells you how well your model is doing. In a normal model, your pretraining loss is based on how likely you predicted the actual next token in the training data would be. That’s the first box along the top, it has an arrow pointing to L_Main - that’s the symbol for loss.

But here in MTP, there are multiple losses! As you continue along the top, you’ll see L_MTP^1 and L_MTP^2 - the losses for predicting the first and second of the multiple tokens. So now your loss is from the normal token, the first MTP token, and the second MTP token. And if you do training right and minimize the overall loss, you can get pretty good at predicting multiple tokens.

Fittingly, modern MTP itself is an invention of the DeepSeek crew, although Meta invented the original version. Many other models now come with MTP. You can think of it as the baseline speculative decoding approach, which EAGLE-3 and DSpark and others must surpass in order to add value.

One type of speculative decoding setup is DFlash, pictured here. It uses the classic setup of a draft model speculating tokens for the target model to verify, not the MTP baseline we just saw. Like MTP before it, DFlash will serve as the base from which DeepSeek innovates, and may serve as the same springboard to popular deployment.

DFlash uses a diffusion-based model, which means it outputs multiple tokens in parallel. That’s actually what the “D” in the name stands for. By comparison, in a standard LLM you only output one token at a time. That’s called “autoregressive”. Even MTP is autoregressive, because the second speculated token depends on the first, and the third depends on the second, and so on.

Putting out tokens in parallel means faster prediction, which means faster throughput for your target + drafter system. However, predicting tokens in parallel usually leads to worse quality, since language is inherently serial, where the right next word depends heavily on what came before it. Like if two similarly likely predictions are “Of course” and “No problem”, then I might end up predicting “Of problem” or “No course” instead.

To me it’s kind of surprising that diffusion LLMs work at all! We’ll come back to how DFlash mitigates the quality loss in a minute.

So in this diagram, DFlash takes in one token from the target model and outputs three tokens in parallel - the green ones, which start life as the special token “<mask>” but end up as real predictions. Those green tokens at the end - “speculative”, “decoding”, and “<eos>” (a special token which stands for “end of sequence”) - are the speculative tokens that the target model will then verify. Again, pretty similar to the standard speculative decoding setup with an autoregressive draft model predicting three tokens out.

Even the guts of DFlash are similar. The draft model starts with an embeddings layer, which pretty much all models do, turning tokens into vectors that the model can work with. In fact the embeddings layer is from the target model directly, it’s not even part of the draft model. Then the two main layers - “bidirectional attention” and “MLP” - are much like the standard attention and feed-forward networks of a typical LLM. And the two layers together form a block, the dashed box in pale green, with multiple blocks in a row forming the bulk of the draft model. Then at the end there’s a language modeling head, exactly like in a typical LLM, that basically undoes the embedding and produces tokens again. That also is directly from the target model.

Now we come back to the twist that DFlash introduces to improve diffusion quality: the boxes in blue, which they call “Fused Target Context Features”. That’s a real mouthful, but all it means is they take some of the target model’s thoughts, or “hidden states”, and put them into the draft model’s brain at every block. That’s much faster than trying to feed all the previous output into the draft model, because the hidden states are already compressed and compact. It’s also higher quality, because the target model is much bigger and smarter than the draft model.

So now that the draft model can put itself more in the frame of mind of the target model, it can make better predictions. It still can fall prey to the perils of parallelism, like the “Of problem” example I gave before, but empirically it seems to do significantly better.

So this idea of speculative decoding, of adding this nearly-free verification step, depends on memory bandwidth being the bottleneck. Like if the weights from the current layer could just zap onto the chip, then the bottleneck would move back onto compute, onto how fast the chip could do the matrix multiplications to predict (or verify) the next tokens. You only get the “free” win of speculative decoding because you have this idle time waiting for the next weights to arrive.

But there actually is another way you could tip the balance from memory bandwidth to compute: adding more tokens for verification. Like if you tried to speculate and verify the next million tokens, you surely would spend more time verifying than you would waiting for the next layer to get to the chip. So somewhere between zero and a million there is a crossover point, where you’ve used up all the idle time with speculation.

In theory that’s the optimal spot: you have made use of otherwise-wasted compute availability, and you have not delayed the load of the next set of weights. Below that point, you’re leaving compute on the table. Above that point, you’re certainly not benefiting, and given speculative decoding is not accurate 100% of the time, realistically you are losing.

Now in practice a single user would never come anywhere close to this crossover point - you can’t realistically predict more than 10-ish tokens out at any likelihood of success. But for multiple users, in batch processing, it’s quite possible to overload the verification queue and slip past the crossover point, or “knee” as it’s sometimes called. And of course a big lab like DeepSeek is thinking about the batch case, because they’re serving tons of users and squeezing the hell out of every GPU.

So: we know we want to be right around the knee to maximize our compute throughput. What this doesn’t say is how to maximize token throughput, which will depend on maximizing compute throughput with the right balance of batch size and speculation size, but also on maximizing the accuracy of our speculation. And that is what DSpark is all about.

The Paper

So this is the DSpark setup. We have our target model on the left, and our draft model on the right. The job of the draft model is to speculate tokens for the target to verify. In this case, given the most recent target model output “D”, the draft model will ultimately return tokens “E”, “F”, and “G”. The target model verifies E and F but not G, which the target replaces with the proper next token, “G*”. All totally standard.

What’s new is inside the big box on the right. We start with our parallel block, which contains between 1 and 5 draft layers, just like with DFlash. Each layer comprises attention and feed-forward, again like DFlash.

The two new additions happen after the parallel block produces its predictions, here called “Logits” because they are not tokens yet, they are distributions of token likelihoods. So going back to our parallel decode example from before with “Of course” and “No problem” colliding to become “Of problem”, the first position’s logits might say 70% odds for “Of” and 30% odds for “No”, while the second position’s logits might say 40% chance for “course” and 60% chance for “problem”. If we just take the most likely token for each position, we’re gonna end up with “Of problem”, which we want to avoid.

Since we have those probability distributions though, we still have the chance to tweak them to produce better outcomes. And that’s what this sequential block is all about. It’s a super lightweight way to add a little bit of sequential information to our otherwise parallel streams. So now for our second position, if we know we predicted 70% for “Of” and 30% for “No” for the first position, we can use that information and make “course” the more likely choice. That will add a tiny bit of time as we’ll see, but the accuracy gains will be worth it.

Now improved accuracy is always useful no matter the setting, but thinking back to our batch serving wrinkle, we also need the other new part: the grey boxes and the blue bar at the top.

The grey boxes are confidence scores, a score from 0-1 that DSpark assigns based on how likely it thinks the token is to be accepted. The blue bar, the scheduler, looks at how busy the GPU is and basically decides how big of a risk to take. Like if the GPU is super busy, then there’s not much idle compute between cycles of target model weights loading, so the confidence bar will be high and the speculation length will be short. Conversely, if the GPU is pretty free, then you might as well keep even the unlikely tokens in.

Remember, the parallel block generates the same number of tokens regardless, it doesn’t cost more to generate more at this scale. So the strategy is to generate a bunch, do a quick pass on which ones are likely good or not, and then filter from there based on capacity. In this example, apparently H was below the confidence threshold, so DSpark dropped it.

Now if we just focus on the first new feature, that sequential block that gives the parallel tokens a bit of information about the tokens before them, we can already see a big advantage in the average number of draft tokens accepted.

Across every benchmark and across all four models, DSpark is a strict improvement over DFlash, by about 10-20%. It’s also significantly better than another speculative decoding technique, Eagle3, that we will touch on a bit later.

One clear trend across benchmarks is the impact of domain on acceptance length. For math and code, which have more structure than natural language, the draft model can often predict pretty long sequences. By contrast, chats from a wide variety of users are often hard for the draft model to predict responses to.

Within the benchmarks, you can also see that easier prompts are easier for the draft model to predict responses to. Like GSM8K, which stands for “grade school math 8000”, is not hard even for small models nowadays. But AIME25, which is competitive math, is harder and thus less predictable.

So it seems the context greatly influences the risk-reward ratio for speculative decoding of all sorts. Hence the confidence scores! The DeepSeek folks love this trick of adding more learning in different parts of the model, like with mHC and Engram - two previous papers of theirs we’ve covered.

Here’s another way of comparing results across methods and domains. On the x axis is the position of the draft token, and on the y axis is how often that token was accepted, given all the other tokens before it were accepted. Because remember, as soon as one token in the draft sequence is rejected, the rest of the tokens after that in the sequence are discarded.

Before I explain the results, it’s important to remember that draft models have a hard latency cap, because they have to generate and send along their tokens in that bit of time where there is idle compute. So when we compare drafters, we expect roughly similar speeds.

However, we do not expect roughly similar sizes. That’s because for a given size, parallel models are much faster than autoregressive models. So if we hold speed constant instead, we expect parallel drafters to be much bigger. And bigger models are generally smarter.

That’s why DFlash, a parallel model, does so much better than EAGLE-3, an autoregressive model - at least at the start. Once you move past the first token though, DFlash starts to run into that “Of problem” issue, what the authors call “suffix decay”. The less structured the text, the worse the problem bites. That’s why the scores on Chat are lower overall, and why EAGLE-3 in particular rises so sharply. Remember though, this is all conditional on the prior tokens being approved, which on an absolute scale just gets less and less frequent the further you go in draft token position. So EAGLE-3 being better late-game doesn’t help as much as it might seem just from looking at these graphs - you have to keep the overall results from the previous slide in mind.

Anyway, that all is ultimately background for the DSpark results, which once again are superior across all domains and draft positions. That’s because DSpark blends the parallelism (and thus greater input context) of DFlash with the autoregression (and thus greater consistency) of traditional LLMs and methods like EAGLE-3. We haven’t even gotten to the hardware-aware part, which is novel.

Now so far we’ve been measuring just one particular design of DSpark, when in reality there are two clear design knobs: the depth, i.e. the number of attention + feed-forward layers in the parallel block; and the block size, i.e. the number of tokens the parallel block predicts.

The previous results used five layers and seven for the block size. So now they’re gonna turn each knob a bit on both DSpark and its predecessor, DFlash, which has the same two knobs.

Although the layer knob isn’t even worth turning on DFlash, because DSpark with just two layers beats DFlash with five layers on every single benchmark. Five still improves performance and apparently fits the latency budget though, hence their choice in the previous slides.

Now for the other knob, here they have fixed the layers at five but have moved the block size up and down.

Overall, greater block size there on the x axis does help, but the rate of improvement falls off quickly, especially above eight. On the negative side, latency increases directly with block size. You can see why they picked a compromise value, although it’s not clear to me why they settled on seven when they measured eight and when slightly higher values seem defensible. Maybe they had a maximum latency threshold and just picked the biggest block size under that? I’m not sure.

Also, I know there are two different DSpark trends on here, but the curves are so similar that it’s not worth getting into the difference between Markov vs RNN. It’s just two different ways of structuring the autoregressive part.

So far all the results have been hardware-agnostic, not taking the overall workload of the GPU into account. Now we can get into the hardware-aware part, which I think is the most DeepSeek-y aspect of the paper, given their reputation for squeezing every last bit of performance out of the limited hardware Chinese labs get access to.

As I mentioned at the top, DSpark looks at the available idle compute and decides how big a swing to take. If there’s lots of compute that would otherwise sit idle, DSpark takes bigger swings, since even low-likelihood bets could still end up with positive expected value. On the other end, if you have little or no spare compute - if you’re at or past the knee of that graph from the background slides - only a pretty likely sequence will have positive expected value, since you’re displacing a smaller number of guaranteed accurate tokens.

Now in general, confidence is going to decrease with token position, since it’s harder to guess tokens that are further out. But you could have some more predictable ones after actually. Like for a code problem, if the name of the function is somewhat arbitrary then the confidence may be low, but the parenthesis after it may be highly confident because that’s part of defining the function. So the way they deal with this is by calculating cumulative confidence when they do the cutoff, i.e. the confidences of each token up to the given position multiplied together. That guarantees the overall confidence is monotonic, it can only be flat or go down.

Anyway, as you’d expect, the higher the confidence threshold, the higher the acceptance rate but the lower the number of draft tokens DSpark gives to the target model for verification. The lowest confidence buckets in particular seem like free wins, basically no harm in cutting out those cases.

Here’s what I call the “money graph”. Not directly for the economic benefits of their efficiency gains, but because it really drives home how big a win a seemingly small change to DFlash made.

This is real production data from serving their current models. Each tiny dot is one combination of individual user speed vs overall GPU throughput. So they were able to collect a ton of empirical data on DSpark.

There are two ways to read this graph. One is at a trend level, where DSpark is clearly superior to MTP; the lines don’t overlap at all, and the gap is huge in some regimes.

The other is at a performance level, with one aspect fixed and the other one left to compare. That’s what the dotted lines are doing. So for example, with DeepSeek V4 Flash, if your GPU only needs to do about 1.5k tokens/second, then each individual user is going to see 85% faster throughput with DSpark compared to MTP. Or, starting from the same dot on the MTP trend - about 120 tok/s for each individual user - then your GPU can do 661% more tok/s overall. That’s crazy performance gain for practically no extra model size and latency!

My Takeaways

• This seems like the right new default for speculative decoding

  • Just when MTP was getting popular!

  • Luckily, you can add DSpark to a model later on, whereas MTP has to be part of pretraining

• I don’t think DSpark portends the triumph of full-blown diffusion LLMs (dLLMs)

  • Draft models don’t need to be that good to be useful

  • Perhaps standalone dLLMs will be useful for easy tasks where maximum speed is essential

  • Google calls DiffusionGemma “experimental”

• Always learn where possible

  • This is what DeepSeek did with mHC and Engram, they did it here too

  • Where else can we be learning?

mHC: Manifold-Constrained Hyper-Connections

or, The Importance of Being Normalized

Paper

Attention Residuals

or, Attention Really Is All You Need

Paper

Background

So since this is a head-to-head paper review, we have to talk about the two fighters in the ring.

On the left we have DeepSeek, one of the two S-tier labs in China, the other one being the Qwen folks. DeepSeek is an outgrowth of a hedge fund called High-Flyer, and it has spent a lot of its life as a fun science project rather than a hard-charging startup. That culture of exploration has produced some highly respected research, including the birth of the modern MoE model and the public discovery of how to train reasoning models, which previously only OpenAI had discovered with o1 and o3, but kept private.

We covered one of their most recent innovations, the Engram memory system, back in February. The paper we’re covering today, mHC, came out just before that.

Everybody thought DeepSeek would release V4 of their flagship model around Chinese New Year, but that was weeks ago and so far we’ve heard nothing. I suspect the combination of high internal standards and little commercial pressure has them working on improvements rather than shipping something middling.

Now on the right we have Moonshot, makers of the Kimi series of models. Moonshot is a hard-charging startup, solely dedicated to making commercially viable LLMs. They were founded in March 2023. Since then, they have released several versions of Kimi, most recently Kimi K2.5, a 1T parameter model. They also just raised $1B and plan to IPO in Hong Kong soon.

Kimi has won praise for its overall performance, but in particular its personality and writing skills. I think it’s the “Claude-iest” Chinese model, which actually would make sense, since recently Anthropic accused Moonshot of distilling from Claude. Possibly relatedly, Cursor used Kimi as their starting point for their most recent coding model, Composer 2.

On the science side, they traditionally haven’t released a ton, although they did claim some engineering breakthrough related to the optimizer Muon.

Okay, now we can get into the technical piece.

First, let’s recap transformers. To oversimplify somewhat, transformers have three pieces:

1. Embedding, for turning text or other inputs into vectors

2. Attention, for understanding how different parts of the input relate to each other and forming a holistic view

3. Feed-forward network, for “thinking” or processing that understanding. In this diagram the feed-forward network is a MoE, a kind of sparse network. In other transformers it’s a dense network

Embedding happens once at the start, then attention and FFN layers trade off over and over again, dozens of times. One attention plus one FFN layer is called a transformer block. In this diagram though, they only have four layers, and they don’t show the division into blocks.

Now in reality there are other minor components, like RMSNorm or similar, but we will put them aside for now.

The only thing left to note is these arrows and plus signs, which show how information flows and combines between layers. That’s what we need to take a closer look at.

This is where residuals come in. Mathematically it’s very simple, but in the diagrams I used to find it confusing.

Here “h” is a hidden state, basically the stuff that moves along inside a model and that the attention and FFN layers process. Then at the end the hidden state gets turned into a token or a class or whatever other thing your model is predicting.

It used to be that hidden state passed directly from layer to layer, like the output of one layer would be the input of the next layer. But then, back in 2015, some researchers at Microsoft decided to add in this other component, the residual - the hidden state from before the previous layer.

In the formula at the top, that’s h_(l-1), the first term on the right side. The second term is the output of the previous layer, where the function f_(l-1) is either attention or FFN. The little plus sign in a circle in the diagram literally means addition - you are adding the output of the most recent layer to the original input for that layer, and that’s what forms your input for the next layer.

So that’s what residuals are. Now the question is, why do we need them?

To understand that, we have to look at the two directions information flows in a transformer: forward, when you’re doing inference; and backward, when you’re training.

The forward case is more straightforward. If you only have the output of one layer as the input of the next layer, you can lose sight of the original input. It’s like playing a game of telephone - changes and errors can stack up. But if you add the residual connections, you’re always providing an unaltered version of the past layer’s input.

And if you look at the formula from the last slide, you can actually write an equivalent version, which I’ve shown here. Notice in the new version that the hidden state for the current layer relates directly to the hidden state entering the first layer; the current state is the original state plus all the layers. If you didn’t have the residual connections, you would lose that h_1 term. So the residual connections let information flow directly between layers.

So that’s the forward pass. Now for the backward pass, we have to get a little more sophisticated.

When you are making a model, you ultimately are trying to minimize some loss, some formula you’ve written that describes how far off your model is from ground truth. So in a linear regression for example, a very simple mathematical model where you’re just trying to fit a straight line to a series of points on a graph, your errors are the distances between each point and the line, and your loss is the sum of the square of those differences.

Loss for a LLM is really not that different. When you’re doing pretraining or SFT, your loss is related to the probability your model gave for the actual next token in the training data. When you’re doing RL, your loss is related to the rewards you’ve calculated.

The big script L in this formula is the loss. On the left side, what we’re asking for is how to calculate the change in loss if we make changes to the input at a certain layer. And if you look on the right side, you see that it depends on the changes to the loss due to the input at the final layer, capital L, and the changes to just the attention or FFN outputs at each layer. It’s just the chain rule from calculus, applied to the giant mathematical equation that is the LLM.

Note something important here in the formula though: that capital I, which is basically the vector equivalent of 1. That’s due to the residual connections. And without it, if enough of those local changes are close to zero, then the whole right side basically ends up at zero. This problem is called “vanishing gradients”, and the more layers you add, the more of a problem it will be.

So the residual connections allow your training information, your measurement of how any tweak to the system will impact the loss, to persist.

So that’s your basic residual connection setup. It works great, everyone uses residuals, problem solved. Right?

Well we’re going to see about that in a minute, but one final bit of background to introduce, or perhaps reintroduce, is The Bitter Lesson.

The Bitter Lesson is an essay by famed machine learning researcher Rich Sutton, pictured here as a GPU enjoyer. His basic point in the essay is that because computing power grows so quickly, scaling up “dumb” or simple approaches always beats “clever” or complex approaches that humans design. For example, when a machine beat a human at chess back in 1997, it didn’t have lots of sophisticated chess strategies in its memory; instead, it searched the space of possible moves by brute force, and it just so happens that in 1997 that was finally technically possible.

The same thing is true for the field of machine learning that deals with language, “natural language processing”. People spent a long time trying to encode the rules of human language into machines, but then along comes OpenAI and just dumps an Internet worth of text into a model and throws enough GPU time at it to learn to produce human language.

The way I think of it is, learning surpasses engineering. If we can learn something rather than design it, we should.

The Papers

So both papers want to change how residuals work. They just disagree on the best way to do it.

I’m going to reground us in the two forms of the residuals equation from Moonshot: recursive, on the left; and iterative, on the right. Remember, these are mathematically equivalent.

I also need to introduce the DeepSeek versions of these equations, on the bottom row. They are mathematically identical, it’s just that the DeepSeek version includes the weights W of the attention or FFN that the Moonshot version omits.

What would our man Rich say about this?

Yes, more learning!

So let’s do more learning. Onto both of these equations we’re going to add new learnable parameters.

For DeepSeek on the left, it’s these matrices H, in three different varieties. We’ll see exactly what they are in a later slide, but for now just remember that the H matrices - “Hyper-Connections” in the jargon - are our new learnable parameters.

Note that the bottom equation, the even more fiendish-looking one, is the iterative version. So that’s directly comparable to iterative equation from Moonshot. I just didn’t want to make too big a leap at once.

Now for Moonshot on the right, the knob to turn is alpha, a single number rather than a matrix. That single number comes from some matrix math though, so I guess you could say we’ve abstracted the complexity away from this equation.

Anyway, the point is they’re both on to the same insight: the implicit equal weighting across all the layers for standard residuals is leaving something on the table.

Now that we know how DeepSeek and Moonshot’s approaches are similar, we can precisely parse their differences.

Here’s a graphic from the mHC paper laying out their changes in two steps. We start with the standard residual connection, adding the original input of an attention or FFN layer to the output of that layer.

To go from connections to hyper-connections, we have to split the input into many different channels, many different parallel lanes, which is why they show some of the hidden states as multiple boxes now. Mathematically, we’re taking the vector x at the very start of the model and just copying it n times, so that x is now a matrix with n identical columns. But the columns are all gonna go on their own journey so to speak through their individual channels in the model, only to be collapsed back into a single vector all the way at the end of the model. So if we look at a random layer in the middle, the different channels are going to have different values of x_el.

Now we can get a little better sense of our H matrices. We have H^res for adjusting the original input, H^pre for adjusting the input before it goes to the attention or FFN layer, and H^post for adjusting the output of that layer. That’s three new knobs per layer, across all the layers. And the extra channels allow more information to flow as well.

So this Hyper-Connections part, the middle graphic, is actually not from DeepSeek - it’s from ByteDance, and the paper came out in September 2024. So what did DeepSeek contribute with their updated version in December 2025? And why didn’t anyone actually adopt Hyper-Connections in the 15 months between the two?

Well in addition to drawing attention to it with their respected scientific brand, DeepSeek pointed to a different component of the transformer for inspiration: the normalization layer.

We haven’t talked much about normalization because it doesn’t really process information in the same way the attention and FFN layers do. It’s more of a preprocessing or post-processing step, to make the hidden state better suited for the next layer.

The basic idea is simple: if your layers are only trained for a certain shape or distribution of inputs, then you have to get your outputs back into that shape, or you risk unexpected behavior. Usually this shows up as failed training runs.

Normalization is a feature of all models, and even appears in the original Transformer paper. But you don’t need separate normalizers for residuals because the hidden states are already normalized!

This is what the original Hyper-Connections authors neglected, that the DeepSeek team fixed.

On paper, all they did was add this normalizing function, abbreviated as P_M, to the H matrices.

In practice though, they did a ton of gnarly engineering to make it work at scale. In the end, their technique only increased the required compute about 7% on the 27B test model they trained, but it took a lot of cleverness to get there.

By the way, the “manifold-constrained” part of their technique is just a fancy way of saying the H matrices have to have all the rows sum to 1 and all the columns sum to 1. That range of potential matrix values forms the “manifold” that the H matrices live on. It’s like the unit circle, which contains all the x and y points where the radius is 1, but in way more dimensions.

So that’s mHC: more channels, and a stability trick to make it actually work.

Now let’s look at Attention Residuals from Moonshot.

Next to our friend in image A, image B has two key changes.

First, instead of direct connections only between neighboring layers, there are direct connections between every layer. Like you can see the Embedding layer at the front touches all the attention and FFN layers, and then the first attention layer touches all the future layers, etc etc.

Second, there’s this alpha number instead of a plus sign. That’s our coefficient that tells us how much to pay attention to the residual. And if we look between images B and C, we see a diagram describing how alpha is calculated.

I’ll come back to that in a minute. Right now I quickly want to acknowledge image C, which is a version of image B but altered for scalability. We’re not going to get into it really, but just know that when you’re training a big model, you have to split up work between GPUs, and that causes problems with the normal attention residuals system.

So, back to alpha. Again, that’s going to say how much each residual should factor in.

And each connection gets its own alpha. In the formula there we see the first one between layer zero and the current layer “el”, then alphas for each connection to “el”.

To calculate alpha, they’re going to use the oldest trick in the Transformer book: attention.

Remember, the job of attention is to decide what is important in the input when deciding on the output. And remember, we only predict one token at a time, so that outputted token becomes part of the input on the next go-around.

Mechanically, attention does this by taking the input and looking at it through three different lenses:

1. Query - the input as a question, as the thing that guides the output

2. Key - the input as the available information, as the stuff to possibly pay attention to. The query times the key tells you which connections between tokens are strongest, which other tokens to look at closely for any given input token

3. Value - what each token represents, what it actually means. Like a dictionary definition, but richer

That’s all standard stuff as a layer in the transformer, but what does it mean to use attention for residuals?

Well, two things. First, the query vector is actually NOT coming from the same place as the key and value vectors. It’s this vector w, which is specific to each layer even though it doesn’t have the subscript “el” in this diagram. This w vector learns how much of each prior hidden state to incorporate in the current hidden state.

Second, the key and value roles actually translate quite well. Each little green block there represents the hidden state from an earlier layer. In this case, we have the hidden state from the first four layers. So Q dot K is going to say how much of each hidden state we want to incorporate - that’s alpha - and then that times V is going to provide the actual information from the hidden state.

So we’ve taken the attention mechanism and turned it 90 degrees so to speak, looking at relations between layers instead of relations between inputs. We just had to be a little bit clever about the queries.

Now frankly it’s going to be challenging to compare the results of these two methods. Even though they work on the same part of the model, they tested by training two completely different models in terms of size, architecture, and data. So let’s start out with individual results.

For mHC, they picked two metrics: quality and stability.

For quality, they show it two ways:

1. Top left, there’s a graph of the relative difference in loss during training. Negative numbers mean the loss was relatively lower than baseline. So we see here that both HC techniques work, although mHC works better

2. Bottom, there’s a table of where their chosen benchmarks landed at the end of training. It’s a fine range of benchmarks, covering language ability, commonsense reasoning, math, STEM, code, and question-answering

From those two mHC appears to have a tiny, possibly insignificant edge over HC.

But then you take a look at the top right, which is a proxy for stability. The thing on the y axis, “gradient norm”, is basically a measure of update size. What you’d like to see is a smooth, gentle downward curve, bottoming out just as you stop training. In practice you always have some spikes, but the overall trend should look like the baseline in grey.

And you do get that with mHC. But you don’t get that with just HC, because HC doesn’t normalize to that manifold we discussed earlier. So you’re risking instability here, like wasted cycles or even wasted whole pretraining runs. For a toy model that’s fine, but when each pretraining run costs millions of dollars, you cannot afford instability like that.

So the story of mHC seems to be: 7% more overhead, for a little better quality at no risk. Probably a good trade.

The Moonshot folks have gone for something similar here. We’re looking at absolute loss instead of relative, there in the top left, and we have a wider array of benchmarks. But it’s the same story: a modest increase in quality across the board.

And on the stability side, we get two graphs from training: the size out the outputs per layer in the middle, and the size of the changes from training (aka the gradient) per layer.

For the middle graph, what you normally see is that later layers have to use bigger numbers, since normally the residuals keep adding to each other. Bigger numbers risk instability, or loss of signal depending on how you look at it.

For the right graph, it’s the same story but backwards; the initial layers in a standard transformer have to make huge changes in order to impact the final layer.

But with AttnRes, in both cases you’re pretty flat in magnitude.

Luckily for us, the Moonshot paper does compare directly to mHC in one table.

It looks at models of increasing size, with other aspects like training data growing in proportion. And it seems here that AttnRes is on par with mHC, even when you have to make engineering compromises as Block AttnRes does.

So to me it seems the simpler method, AttnRes, is the winner. But it also makes me wonder if there aren’t other solutions at work in the closed labs.

My Takeaways

• Moonshot may have bested DeepSeek here

  • Not exactly fair though, given mHC came out before AttnRes

  • But also, mHC is a change on top of HC, so perhaps a smaller contribution

• AttnRes is more elegant

  • But mHC is more impressive - the math they pulled out to get the H matrices constrained to that manifold is arcane

• Perhaps Moonshot will break into the S-tier with Qwen and DeepSeek

  • I doubt DeepSeek will regress to A-tier

Paper · Website · Repo · Leaderboard

Background

So, what are computer use agents?

A computer use agent (CUA) is an agent that can use a computer just like a person would: looking at a screen, mousing around, clicking buttons, using the keyboard, and so on.

Unlike most agents, which are built for purpose - coding, deep research, specific jobs like that - a CUA is general; in theory, it can do anything that is doable on a computer. The term is more about the mode of operation than the purpose.

Now practically speaking, there are limits. Like in theory, you could solve the Riemann hypothesis while using a computer, just writing down math in LaTeX. Obviously that’s not what comes to mind first when you think about a human using a computer, but it’s important to get our definitions and distributions right when we’re designing benchmarks like OSWorld.

So what folks really mean by “computer use” is more like “use of common graphical programs for white collar or personal tasks” - email, spreadsheet, CRM, shopping, stuff like that.

I want to emphasize graphical here. Agents are already quite good at using the command line and writing code for work that isn’t graphical, that is purely text. And tech companies know that, so a lot of tools that are naturally graphical for humans have gotten text interfaces for agents, or already had APIs that agents could call directly. That’s honestly preferable; after all, language is the primary medium of large language models - it’s right in the name. And while humans are good at language too, we are especially good at vision. So where we can, we want textual and graphic versions of tools available.

But sometimes it’s not possible, or at least not practical.

For one, some tasks are inherently visual. Like if you want your agent to design an infographic in Figma, or to CAD up a part, or to edit a video in YouTube Studio, you really need it to use vision. Parts of each task may be textual and thus amenable to the command line, but there’s no getting around the visual parts.

On a more practical level though, many programs that could be more textual and LLM-friendly simply won’t - the developers won’t add the features, or the admins won’t permit that type of access, or some other organizational barrier crops up. So if your agent has to be a drop-in replacement for a human, with no special accommodations, then it has to be good at computer use.

I always think of the DMV for this latter case. Like that software has gotta be ancient, and there’s no way they will overhaul their systems any time soon. So even though the work probably isn’t that inherently graphical, you probably have to deploy a CUA to automate that work - at least for now.

Now that we know what CUAs are, we should talk a bit about how they work, which will help contextualize some of the benchmark design decisions and failure modes we’ll see later on.

The primary way folks add computer use to their models is by feeding in screenshots. That requires a model to have vision capabilities in the first place, and there are things you can do to the screenshot to help the model, but that’s the basic approach. The model then emits pixel coordinates and actions, like “click on (148, 273)” or “drag from (318, 997) to (509, 1011)”. That’s a totally general approach, operating pretty much like a human, but it comes with a couple downsides.

For one, vision is just not that strong of a skill for most models. They miss some details and fabricate others, at a higher rate than with text. Labs care more about text generally.

For another, vision is token-intensive. A single screenshot, optimized for computer use, is roughly in the low thousands of tokens. That’s not much on its own, but if you want anything like real-time vision, that’s going to add up quickly. In fact, that token cost is a major reason computer use tends to happen with smaller models. (The other reason is speed.)

Now the other approach folks sometimes use is to read some lower-level representation of what’s on screen. This is mostly relevant for browser tasks, where the model can read the DOM, which is basically the code that the browser translates into something visual for the user. If you’ve ever done “inspect element” in Chrome and then edited some of that code to make something change on the page, you know what I’m talking about. Another example here is the accessibility tree (or “a11y tree”), which originally was for vision-impaired humans to use while operating a computer.

The lower-level representations are nice when you can get them, and many agents do take a hybrid approach, but many programs don’t have them. And of course some things are still inherently visual, so no lower-level representation can do them justice. So pure vision is inescapable to some degree.

Weirdly, even though computer use in many ways is behind compared to other areas of AI, computer use agents appear fairly early in the Gen AI era.

For example, Anthropic released its first CUA in October 2024, alongside a new version of Sonnet 3.5. It didn’t release Claude Code until February 2025, alongside Sonnet 3.7. Similarly, OpenAI released its first CUA, Operator (shown here), in January 2025; it didn’t release its coding agent, Codex, until April 2025.

Usually it’s benchmarks that come first though, kind of setting the target that the model makers need to go out and hit. And to that end, the first CUA benchmarks of the era appeared in 2023, not long after ChatGPT came out. For example, Mind2Web, which came out in mid-2023, comprised 2,000 tasks across 137 real websites, but offline versions. Since no real vision-equipped model had come out yet, the CUA navigated the DOM. There were a couple vision-only benchmarks, like Android in the Wild, but they used bespoke vision encoders rather than the general vision encoders on a vision-language model like you’d find today.

Once GPT-4V came out in September 2023 though, the door for truly general CUA benchmarks was wide open. One of the earliest, and certainly the most enduring, was OSWorld.

In that early crop of CUA benchmarks, a few things made OSWorld stand out.

For one, it was supremely general: it plopped the CUA in a Linux or Windows or macOS desktop, with all the normal programs, and just gave it a task. It was the first benchmark to really say, “this is what computer use should look like”.

Not only was it general, it was also cross-program. For example, in the first task over there on the right side, you can see the agent switching between a spreadsheet and an image viewer. The entire computer is “in bounds” so to speak, the benchmark isn’t contained to a single program that happens to be running on a computer, in an operating system etc.

Another key element was the engineering behind it. The OSWorld folks built some nice infra, allowing other researchers to replicate and build on their efforts. The paper emphasizes their approach to virtual machines and environments and initial state setup, all topics that still matter today. And of course, if the teams building models can use your infra, of course they’re going to run evals on your infra.

So as a result, OSWorld became the standard computer use benchmark. Anthropic has reported results since it released its first CUA, and so has OpenAI. Google started reporting with the release of Gemini 3 Flash. All of them continue to report on it today, although technically the later results are on a cleaned-up version called OSWorld-Verified.

The first CUAs were understandably terrible, scoring no higher than about 12%, while humans scored 72%. However, there was remarkable progress in 2025, and by early 2026 the leading models all hit human parity. So now is the perfect time for OSWorld 2.0.

The Paper

Let’s start with a concrete example from OSWorld 2.0, keeping the examples from OSWorld 1.0 on the previous slide in mind.

High-level it’s a straightforward request: read some reimbursement instructions and apply them to this specific case. Not terribly different from that first example task in OSWorld 1.0.

However, several differences emerge upon closer inspection. For one, just look how many steps there were - the last screenshot says Step 499! That’s way longer than anything in OSWorld 1.0 as we’ll see when we look at duration and step distributions.

Another key difference is the number and variety of programs. Here we have a PDF reader, an expenses portal, and email. For OSWorld 2.0, not only did they add more programs like FreeCAD, Blender, and Obsidian, they also added 31 self-hosted websites - clones of Gmail, Slack, LinkedIn, Salesforce, Twitter, Eventbrite, YouTube and more. It’s a nice example of how AI coding progress unlocks progress in other areas of AI.

OSWorld 2.0 also introduces a new taxonomy of work types - what they call “challenge phenomena” - to describe more specifically what the agent has to accomplish along its trajectory. It’s hard to compare the whole list directly with OSWorld 1.0, but some are certainly new, like the Dynamic Environment example in step 372; new information arrives while a task is underway, and the agent has to notice and incorporate it.

Overall, OSWorld 2.0 is harder: longer, more novel, and with more wrinkles.

Here’s a closer look at how much longer they made the tasks. It’s measured in human operation time, i.e. how long it takes a competent human to complete the task.

Keep in mind this is a log scale, so the difference is more dramatic than it might visually seem. I’ll also note the significant right-hand tail, where the top 50% of tasks by length are more spread out and reach past ten hours in a couple cases. The net impact on agents is about a 10x increase in steps, whether successful or not.

Another way to read this is an indication of CUA time horizons. OSWorld 1.0 came out in April 2024, and two years later it basically saturated, meaning it took two years for models to be pretty reliable at tasks taking a single-digit number of minutes. With OSWorld 2.0, published at the end of June 2026, we are ready to measure tasks taking a single-digit number of hours. This is the closest I’ve seen to a GUI version of the famous METR time horizons graph.

Moving from length to novelty, we get a couple breakdowns of what the tasks are asking for and what tools they entail.

On the left, they give a breakdown by domain, with categories somewhat influenced by economic sectors. They spend some time in the paper trying to tie their domains to various slices of GDP, but it seems more like an add-on to compete with explicitly economically-oriented benchmarks like Remote Labor Index and GDPval. You can read more about that category of benchmarks in my breakdown of Agents’ Last Exam, which the authors actually cite as a “close comparison” but without the focus on graphical tools. Anyway, the tasks do take more inspiration from the real world, and the shift from scientific framing to economic framing indicates to me how close to production CUAs are - or will be, once they can climb this benchmark.

On the right is a more concrete breakdown, this time by program used. Note that most tasks require at least two programs, so the total here is well over the 108 total tasks in the benchmark, and they mention elsewhere that the average count of programs per task is 2.44.

Anyway, we see a browser is required for a majority of tasks, with equivalents for Word, Powerpoint, and Excel rounding out the double digits. MailHub and TeamChat are Gmail and Slack, respectively, and they happen in the browser so there’s actually some double-dipping between them and Chrome/Browser. Regardless, most of the programs and all the web apps are new in OSWorld 2.0, so still an impressive increase in novelty.

Now given there are only 108 tasks in the benchmark, it’s not surprising to see so many programs appearing only once, but such small sample sizes in so many domains does make me hesitate on any fine-grained analysis. As we know, it’s expensive to produce good benchmarks in significant quantities!

And turning to the wrinkles, which themselves are often novel, we again get a table of overlapping entries; as we saw in the example slide, a task has many challenge phenomena.

The names are fancy, but the concepts are pretty simple. In fact, I think most people would have a hard time parsing these challenge phenomena out if you asked them to document the difficulties in their own computer use trajectories. But as we know, artificial intelligences are jagged, and things we find easy are often hard for agents - and vice versa.

Right near the top is one about vision and perception, which we could have anticipated as being quite common. And just above it, Cross-source Reasoning, is a wrinkle they consciously introduce to turn up the difficulty compared to OSWorld 1.0, so no surprise it’s at the top.

But the next few I find more interesting. You would think that with such large context windows and such great intelligences, models would be better at inferring and tracking and deciding which conflicted source takes precedence. But as we’ll see in the results slides, models still struggle to process such information.

The rest are less interesting or less common, but I wanted to show them all here since they will all appear later on.

Now before we get to the results, of course I wanted to highlight their pipeline.

Coming into the paper, I expect them to harvest existing artifacts and then reverse engineer tasks out of them, but that’s actually not what they did. Instead, their main method of task creation was what they call “brainstorming”, in the green box toward the bottom-left. I’m going to quote their description directly:

“The bulk of OSWorld 2.0 tasks were produced by a small group of internally trained annotators who worked end-to-end on the tasks they proposed, covering task design, input artifact construction, environment setup, and evaluation function implementation. Annotators first learned the target domain by watching tutorials on YouTube, reading official documentation, and directly experimenting with the software, which equipped them to reason about professional tools and workflows, or enterprise systems in sufficient depth. They then drafted candidate tasks grounded in workflows surfaced from Reddit discussions, online tutorials, and their own day-to-day work experience, with each draft specifying the instructions, required input artifacts, and expected final state. Every candidate was finally peer cross-checked by a second annotator, who reviewed it for feasibility, redundancy with existing tasks, and ambiguity in the evaluation criteria… This channel ultimately produced approximately 90% of the final tasks.”

Once they have the task inputs, or “spec sheet” as they call it, they turn them into an environment with an initial state that an agent can enter into and complete the task within. So thinking back to our example task from the top, they have to put the instructions PDF on the desktop, populate the inbox with relevant and possibly irrelevant emails, log the user into the reimbursement portal, etc etc. That’s a human-led process, and in that process some holes in the initial task design emerge.

One other key step here is defining and running the evaluator, which checks for correctness at the end but also along the way. The details are somewhat opaque, but they have many so-called “checkpoints” per task, just over 27 on average. The checkpoints don’t prescribe an order, so different paths that end up at the same state are valid. Mostly the checkpoints are automatically verifiable, like whether a certain value ended up in a database, but some do require model judgment - about 11% of all checkpoints, with no task having more than 50% model-judged checkpoints. Also, they don’t use the term “rubrics” to describe it, but basically that’s what they write for the model judge to use.

So now with a completed build, they test it. First they have two humans test it and check for agreement, then they have some agents test it. If it turns out the task isn’t solvable given the inputs and the environment, or there are ways to cheat, or the checkpoints don’t fully reflect the completed state, they send it back for rework.

Overall I think it’s a decent pipeline, and I’m not surprised they got most of their tasks from human rather than synthetic inputs; models tend to produce contrived and arbitrary tasks, with clumsy additional constraints to increase difficulty.

So with our 108 tasks in hand, we can now evaluate some models.

A couple notes before we dive into the table. First, this is for a budget of 500 steps. They do also provide results for 150- and 300-step budgets on their website, but they focus on the max budget setting so that’s where we’ll focus too.

Second, we should define some terms. On the columns, Binary means the share of tasks where the model got every checkpoint. Partial means the average score per task. Note that it does not mean the share of all possible checkpoints achieved across all tasks, because that would favor tasks with higher numbers of checkpoints.

On the rows, Batched means that the model took multiple actions in a step before observing and thinking about the results, whereas Single means the model looked and thought about each action before deciding on another one. GPT-5.5 only offers batched.

So with definitions out of the way, we can see clearly that Opus 4.8 is the absolute performance champion. However, that title comes at a cost, with more than double the cost for pretty marginal gain against Opus 4.7. GPT-5.5 meanwhile is the efficiency king, with far lower cost and especially output tokens per task. Opus and GPT used the highest thinking settings, so really the story here is GPT seems more efficient with its thinking tokens.

Digging in on this point, they have a couple graphs showing how thinking level impacts outcome. The bigger the dots, the higher the thinking level.

If you measure turns, then it seems more thinking quickly produces better results for the frontier models. Given some of the failure modes we’ll see later, more turns typically means the model noticed or realized something it had to do while in the middle of a task, and that less intelligent or less hard-thinking models let a lot of that subtlety slip.

If you measure tokens though, suddenly the story muddies. For GPT, our efficiency king, more thinking continues producing better results quickly. But for Opus, especially Opus 4.8, the curve is pretty flat. Since output tokens are the main driver of cost, GPT may be the better model, but only if you can accept partial results; note the change in y axis, with Binary for the left graph and Partial for the right. Bit of an odd choice by the authors in my view.

One comparison I wish they showed is on wall time. It’s tricky to compare between models, since different companies have different serving infrastructure, but ultimately what we should care about in economic terms is the rate of completed work per time. Then you can compare the dollar value of that completed work to the dollar cost of the job, and ultimately come to a net income per time. It’s another example where I feel the economic framing they tried to add with the GDP tie-in and comparison to Agents’ Last Exam is half hearted.

Speaking of time, the human-measured time horizon pretty well correlates with difficulty for the models, as we see on the left.

Similarly, on the right we have human-measured difficulty in three buckets: easy, meaning it took a human under 30 minutes; hard, meaning it took a human more than two hours; and medium, falling between 30 minutes and two hours. If we then bucket tasks by model Partial score - over 70% for easy, under 30% for hard, in between for medium - we see a similar correlation. As expected though, it’s much more common for an easy human task to be a hard model task than vice versa. Do note that percents are normalized by row though, so you really have to read it as “for a given human-predicted difficulty, what was the spread of empirical difficulty?”

Now beyond the top-level statistics, it’s instructive to see what exactly the agents are doing in their trajectories.

Here the researchers had GPT-5.5 review all the trajectories and categorize each step, with some human review afterwards. They provide 15 categories, grouped into four phases.

A couple patterns stick out to me. One is how similar most of the models are across phases and even across categories for the most part. Like if you take any of the models and sum across phases, it’s roughly 30-35% Reasoning, 30-35% Perception, 20% Action, and 10% Correction.

Relatedly, the models likely spend too little time on correction. Obviously the ideal is for a model to never make mistakes in the first place, which would also produce very low numbers for Correction, but as we know from the top-line results these models are far from perfect. Any regular user of agents knows they tend to be overconfident, so no surprise here, but error correction and reflection are increasingly important as the horizon lengthens; the more steps you take, the more likely you are to introduce an error, and if it propagates without correction you are virtually guaranteed to fail any task past a certain length.

Two is how different GPT is on a category level. I suspect some of it is due to GPT also being the judge, but the extra exploration, information extraction, and tool-semantics reasoning does seem like a substitute for the missing planning - a more efficient substitute, apparently. Relatedly, they show in another chart that GPT greatly favors text interfaces over graphical interfaces, and pretty doggedly looks for the text version of an interface. When it works, it works well and is typically more efficient, but it’s also brittle. The authors give the example of manipulating the DOM rather than interacting with a website graphically, which is more direct but also obscures layout-based information and cuts out helpful guardrails.

Continuing on the micro analysis, the authors revisit their taxonomy of “challenge phenomena” and see which phenomena are indeed the most challenging.

Now they don’t make this split in the paper, but I want to distinguish strongly visual tasks from the rest of them, because visual acuity is distinct from the general focus and planning the rest of the challenge phenomena test.

And you can see it right away in the differences between models. Among the three here, GPT is known to have the best vision abilities, and M3 the worst, which we see reflected in the results of the top two vision-heavy phenomena.

If we put them aside, then, we see pretty consistent results across most of the phenomena. Like apparently deriving implicit information and knowing when to ask the (simulated) user for clarification and keeping lots of things in working memory and noticing changes in the environment that are crucial for the task are all significant hurdles. It’s not an issue to work across multiple programs or artifacts per se, it’s more about keeping them all straight at the same time and knowing where the holes are. Gaia2, another updated version of an early agentic benchmark that I covered somewhat recently, drew some similar conclusions.

So like a relatively easy OSWorld 2.0 task might jump between a few different programs but only require one at a time, with little cognitive overhead or state to mentally keep track of. Like the expense report example from the top, if all you have to do is hunt across email and local files and Slack but each expense is self-contained and can go into the expense tracker one by one, that can be quite long but apparently not that challenging. It’s only when you add in new emails mid-stream, or conflicting sources of information, or a clear hole like a return plane ticket without a departing plane ticket, that you really start to challenge frontier models.

One final note: the easiest challenge phenomena, Cross-resource Reasoning, is actually the most common. It seems to me more like an intrinsic part of the benchmark to work across sources rather than a wrinkle to add in for increased difficulty.

To make the failures concrete, here are a few examples.

The first one is an example of Dynamic Environment, where a new Slack message with corrected information goes unnoticed or unused.

The second one shows Streaming Interaction, where a pop-up moves and eludes the clicks from the agent. Personally I found this one pretty contrived.

The third one is Multimodal Editing, which explicitly includes CAD reconstructions. Even though the agent can use the software perfectly, it misperceives the plans and thus produces the wrong part model.

My Takeaways

• 2026 is the Year of Computer Use Agents

  • 2025 actually was the Year of Agents

  • Now that models are good agents, labs can focus on computer use specifically

  • Anthropic probably released Sonnet 5 for computer use

  • I predict an agent will cross 50% on OSWorld 2.0 by EOY 2026

• CUA data should be hot

  • Plenty of YC companies working on it already

  • Snorkel AI contributed to OSWorld 2.0

• More benchmarks will tie themselves to economic indicators

  • For many applications, the era of science is over and the era of deployment has begun, which means dollars come into play

  • See my breakdown of Agents’ Last Exam

• Interactive benchmarks will become more common for agents

  • Fully specified and self-contained tasks do not reflect how users work with agents

  • See Scale’s recent SWE-Interact for another example

Paper · Blog post · Model weights · Demo

Background

To frame all this, we need to acquaint ourselves with DeepSeek, the lab that makes the DeepSeek models.

Or maybe we’re reacquainting, because we’ve covered them a few times before. Specifically, we’ve covered:

• DeepSeek V3.2, the last major release from DeepSeek

• mHC, a super technical improvement to a very specific part of the model architecture

• Engram, a sort of built-in memory for common phrases

I’ve linked all of those talks in the speaker notes if you want to go back. But for those who don’t know or may have forgotten, DeepSeek is the leading scientific lab in China - not always the absolute performance leader, not cranking out models, but always pushing the envelope. In many ways they are what OpenAI set out to be: an open, research-focused organization dedicated to achieving ASI and sharing it with the world.

Unlike OpenAI though, which for a long time subsisted on what it viewed as charity, DeepSeek has always had a sugar daddy: a quant hedge fund called HighFlyer, whose founder decided to spin up this ASI effort with his profits. And that missing profit motive, that journey of discovery, comes through in a lot of what DeepSeek does today.

For one thing, they publish a lot of their innovations. Going back to my list above, mHC and Engram are techniques really only usable at labs pretraining their own models from scratch. From a competitive angle it seems crazy to give away techniques like this, but from a scientific angle it’s completely standard.

For another thing, they don’t put out many models! DeepSeek V3.2 came out on December 1st, 2025. That’s almost half a year ago, which feels like years in model release time. There were rumors they wanted to publish V4 before Chinese New Year, as some other Chinese labs actually did with their newest releases, but apparently they just weren’t ready and didn’t want to put out something not up to their standards.

DeepSeek is also one of the national AI champions in China, which puts it in a tricky position. On the one hand, the power of the Chinese government is behind it in various ways, including a new upcoming funding round I just read about today. But on the flip side, they have to start adopting Chinese chips.

Historically, all the labs in China have used NVIDIA chips basically, because for this type of workload they’ve been the only game in town. But with recent US export controls, China has lost access to cutting-edge chips, and given the importance of AI, the CCP does not want to rely on foreign companies or governments for such a critical resource.

As a result, the Chinese government doesn’t want NVIDIA chips - they’re fine with the export controls! - and instead wants homegrown chips in all the AI labs. Right now that means Huawei chips, specifically the Ascend line. DeepSeek V4 was a major test for the Ascend chips, with Huawei and DeepSeek working closely together to make training and inference work. In fact, there’s some speculation that the delayed release was primarily due to the switch in chips.

It’s okay if the information on this table mostly escapes you - the point is that Ascend chips are behind their NVIDIA counterparts mostly. And the relative underperformance of Ascend chips means DeepSeek has to focus on efficiency, not pure performance, since they’re unlikely to completely match OpenAI and Anthropic and Google.

The other hardware-related thing is at a data center level, which is utilization. Utilization is basically “how often are my GPUs actually doing something” vs waiting for instructions or data to arrive. It’s really, really hard to get high utilization; for example, xAI’s mega data center, Colossus, apparently runs at about 11% utilization. US companies have access to the right hardware and can just scale up; DeepSeek does not - at least for now - so it has to be clever.

In general, the DeepSeek approach has been to accept engineering complexity in exchange for efficiency. They are cracked engineers and they want to push the envelope.

One expression of that crazy engineering and efficiency is the price.

What we’re looking at here is pricing for a bunch of recently released models. You’ll see three types of price: input, output, and cache hit.

Input and output are straightforward, it’s just the price you pay per million tokens you pass in and receive back, respectively. And on those dimensions, DeepSeek V4 is pretty good but not exceptional; other Chinese labs with similarly capable models do about as well, like Kimi K2.6 and GLM 5.1. Of course compared to Opus 4.7 and GPT-5.5, it’s already a bargain - you can see why Anthropic and OpenAI are getting those massive valuations.

But the third bar, the cache hit, is where we really see the engineering prowess and relative altruism of DeepSeek come through. “Cache hit” tokens are tokens that haven’t changed since your previous turn, either from the system prompt or from earlier parts of the same conversation. Those tokens can live in storage, cheap storage like a SSD. And that matters a lot when you’re using agentic scaffolds with big system prompts and doing very long trajectories, rather than a bunch of shorter conversations in a chat window.

Anyway, as you can just barely see on the chart, DeepSeek V4 Pro only charges a single cent for 1M tokens in cache hit. That’s basically free! All these long histories all our agents are generating, they cost practically nothing if you’re using the DeepSeek API.

I don’t want to get into exactly how they do that, partly because it’s in the paper and not part of the background, but it just shows what kind of lab we’re dealing with here.

We also need to do an attention refresher, as attention is the area with the most innovation in DeepSeek-V4.

We’ve covered it before, but to oversimplify somewhat there are three components of a transformer model, which is the basic architecture all your favorite LLMs use.

1. Embedding, which turns words into numbers, since after all we are ultimately working with numbers in the form of matrix multiplications here

2. Attention, which calculates how each individual word or token relates to all the other tokens that came before it in order to form a complete, holistic understanding of the input and output so far

3. Feed-forward, or MLP as it’s noted here, which takes that complete understanding and “thinks” or “processes” it

For the attention layer in particular, you see each input splitting into a query, a key, and a value. The thing to notice here is how the queries and the keys form this matrix, where each query gets to “look at” or calculate against each key of the previous tokens. So like “The” at the start can only attend to itself, whereas “tokens” at the end can attend to all the tokens in the example. And within each row, like on each query, the circles are colored by how strong the relation is. So on the query “to” for example, it’s attending strongly to “The”, weakly to “model”, strongly to “attends”, and weakly to itself. Sometimes those relationships are a bit inscrutable to humans but in this case it generally makes sense.

The only other thing I would briefly note here is that the attention and feed-forward steps or “layers” as they’re called form the “transformer block”, and the transformer block repeats many times before the model is finally ready to predict the next token. What that attention matrix looks like in terms of the strength of relations changes from block to block.

So in the standard transformer model like we just saw, the model pays attention to everything - up to its maximum context window anyway. That would be the first box on the left, “Full n^2 attention”. And it’s called n^2 because the number of green squares goes as the square of the number of tokens. That means if you double the number of tokens in your context window, you’re quadrupling the compute and memory needs. It’s the gold standard, but like gold, it’s pricey.

People spend a lot of time trying to find cheaper attention mechanisms that are nearly as good. Often you’ll hear them called “linear”, since they tend to use compute and memory in direct proportion to the size of the input, which makes a huge difference once you get out to the 100k or 1M token range.

One of the more popular ones is Sliding Window Attention (SWA), also seen above. Here you specify a window size that is much smaller than the full context length. At first that means you’re only getting information about very nearby tokens, but as you keep going through the layers of the transformer, you start to indirectly get information from tokens that are further and further away. Sort of like a game of telephone: if I initially get information directly from my neighbors one day, and they do the same with their neighbors, then the next day when I check back in with them they’ll have information from their neighbors to share with me too. And if you do more and more layers, more and more days in our metaphor, then eventually you can get information from the whole previous context.

There are more exotic forms, like c for example. And then for d, a lot of models have different attention designs interleaved, like one layer with full attention and three layers with sliding window attention for example.

DeepSeek has gotten far more exotic than just a tweak to sliding window as we’ll see. But the key thing to remember is that attention dictates how much information you can actually pull out of the input, and that you might choose different attention mechanisms for different balances of quality and cost and context window size.

Here’s a quick comparison of full and sliding window attention next to DeepSeek Sparse Attention (DSA), which is the foundation on which the most recent model’s attention mechanisms are built.

As the captions highlight, DSA is like SWA in that your query token only attends to a certain number of prior tokens, letting it use far less compute than full attention. However, DSA learns which tokens to attend to, rather than relying on a hard rule like SWA. The component that does this learning, that picks which tokens the query is going to attend to, is called the “lightning indexer”. Keep that in mind for future slides.

The Paper

So high-level, this is what we’re working with: a Pro model with 1.6T parameters and 49B activated, and a Flash model with 284B parameters and 13B activated. I added on a couple other stats that basically explain the size difference - Pro is just bigger on every dimension.

I don’t have a good explanation for the slight mismatch in pretraining tokens, but I think it may be from Pro getting more agent trajectories.

So here is the overall architecture, for both versions of the model. I know it’s a lot, but we’ll go through it node by node. And remember, the important parts of a transformer are the embedding, the attention, and the feed-forward.

The first part at the bottom is where we pass in the input, turning it into numbers, into vectors. That’s the simplest part.

After that you’ll see a blue dotted line around most of the diagram, designating the transformer block. At heart it’s one attention layer and one feed-forward layer, with some supporting players in the mix. As I mentioned in the background slides, the transformer block gets repeated over and over, L times, after which the model is ready to make its prediction at the prediction head. There’s a layer after that about MTP, multi-token prediction, which we don’t need to get into but basically allows the model to predict a few tokens at once instead of just a single token at a time.

Now within the transformer block is the real innovation.

One innovation is mHC, which we covered in a previous paper and which I also don’t want to rehash too much here. But basically, mHC is all the grey stuff - the mixing, the row of circles that expands and condenses etc. It lets the model learn how much to actually care about what happened in the most recent attention or feed-forward layers.

The real show is the attention layer, here labeled as CSA / HCA. Those are two new forms of attention that allow DeepSeek-V4 to be so crazy efficient, and that’s the core technical stuff we’re gonna explore on the next slide. They interleave CSA and HCA, like using CSA in one transformer block and then HCA in the next one etc.

I’ll also note DeepSeekMoE as the feed-forward layer, but that’s also not new; basically all models competing for SOTA are MoEs.

Finally, for anyone who remembers the Engram paper, DeepSeek V4 does NOT use Engram. I expect it to appear in a DeepSeek release eventually, but that may not be until DeepSeek V5, since Engram has to be part of pretraining.

Now we are ready to tackle the big innovations: CSA and HCA.

We’ll start with CSA. It may help your intuition to think of a high-resolution picture you need to reduce the size of. The CSA approach is to keep the pixels in a circle around one spot, then go lower res on all the other pixels, then black most of them out for not being relevant enough. You’re getting some fairly precise but sparse global signal, and keeping the precise local signal.

Back to the model. In this diagram, each little yellow or green bar represents one token, or really one “hidden state” of a token - basically what the token has turned into after some number of transformer layers. The hidden state is like the model equivalent of electrical signals in your brain.

So we have our one query token, and we want to decide which of the KV tokens to attend to, and maybe transform the tokens a bit more first.

The first thing we’re gonna do is compress the KV tokens, in two ways: we’re gonna use smaller vectors, and we’re gonna group tokens together. You can actually see that in the diagram, where the bars are now half the height AND there are fewer of them, something like 4x fewer in this case.

The next thing we’re gonna do is use the lightning indexer to figure out which of these already-compressed tokens we want to attend to. That requires the query token and the KV tokens, and it puts out a list of the top KV tokens to care about. So now we only have three yellow bars at the top.

Finally, we do want to compensate for some of this compression by paying closer attention to nearby tokens. Specifically, we have our sliding window of 128 tokens there towards the top-left, which is gonna add in with the selected compressed KV entries.

So now from the query’s perspective, when it’s attending to all these KV entries, it has some high-fidelity ones in the 128 tokens before it, then a pre-selected group of kinda summarized tokens. And it’s gonna attend to all of them.

Now for HCA. As the name implies, it’s like CSA, but even more compressed. They’re just gonna crank up the compression factor by a lot, like from 4 in CSA to like 128 here. And because everything is so compressed, they don’t need the lightning indexer to trim the number of tokens.

Of course they still have the sliding window KV entries, so nearby information is more preserved. But generally the goal of HCA is a global view of everything. Thinking back to our image and pixels from before, HCA keeps that same radius of original pixels, then makes the rest of the image super low res.

Now if you did only one or the other of CSA and HCA, you probably wouldn’t get good results. The magic is the alternation of these two, kind of superimposing them, so you have your fuzzy global view matching up with your selective global view. Then if a selected CSA block lands on a big fuzzy HCA block, that little higher-res bit gives you a much better idea of what’s going on in that big fuzzy block.

Here’s the upshot of all that innovation. Compared to the previous DeepSeek model, the slopes on the lines for compute (left) and memory (right) are drastically lower for both versions of V4. This is despite Pro having 1.3x the activated parameters of V3.2! And it really starts to matter once you get into the 100k regime.

Speaking of V3.2, here’s where we net out when we compare base models. In case you’ve forgotten, base models have no post-training on them. It’s actually become quite rare to release base models, even when the post-trained models are open weights, so these are by far the best base models available today. Again, it’s the DeepSeek devotion to science shining through.

Anyway, V4 is generally an improvement over V3.2, even when comparing with the much smaller Flash.

Now when you look at the post-trained versions, you can finally compare to the competition that the end user will have in mind.

Before we draw any conclusions about DeepSeek V4, we should look at the distribution of benchmark scores between the other models. Opus 4.6 and GPT-5.4 are generally regarded as the best models on this list, yet they often are not the winners. So we have to take benchmarks with a grain of salt.

Anyway, it seems like DeepSeek V4 is competitive, but rarely on top. And from what I’ve seen of the vibes, that’s about right; DeepSeek has certainly not closed the gap with OpenAI or Anthropic or even Google, and they’re generally competitive with the other top Chinese models.

I wanted to show a bit about the post-training process, but the information is scant. The one interesting thing for this group is that they do use rubrics, although there’s no other information about them. They actually use the same model for generating and judging, and they claim they improve the model’s judging abilities alongside its generating abilities when they’re doing RL.

They also do a somewhat common thing now where they post-train a bunch of different versions of the model that are each experts in one area, then they distill those models down into one final model. I don’t think that would change what post-training data they want, more of a fun fact.

To complement the benchmarks, they do run one eval: a small set of white collar tasks, in Chinese, like writing reports and making nice-looking documents. They don’t give any numbers on contributors or sample size unfortunately.

In the SxS rankings, DeepSeek seems to hold its own against Claude, and on the pointwise criteria it does about the same or a bit better. Maybe the tasks being in Chinese was a major factor though, because I would be surprised if this held up at large n or in English based on what I’ve been hearing.

My Takeaways

• Attention innovation still has juice to squeeze

  • Maybe someone will hit 2M this year

  • I’m more bearish on alternative attention mechanisms now (e.g. Mamba)

• We’re not that far from “tokens too cheap to meter”

  • The Bitter Lesson strikes again

• US and Chinese ecosystems are diverging

  • The US will probably ban the import of Ascend chips on national security grounds

Paper · Blog · Repo · Benchmark · Model

Background

So, what are world models?

We’ve actually answered this question once before, back in November 2025, when I covered a paper called Benchmarking World-Model Learning. In it, we talked about how there are two ways to predict the future: statistical inference, and world modeling.

Let’s say you have a fair coin, and you want to predict whether it will land heads or tails. The statistical way to understand the odds is to gather a bunch of data on past coin flips, then calculate the odds as 50-50. From that finding, you infer the odds of the upcoming coin flip are also 50-50.

This is what language models generally do today. They have a ton of data about which token comes next given a series of previous tokens, and when you use them - when you run inference - you are asking for a statistical prediction, a distribution across all possible next tokens.

Now the world modeling way to predict the coin flip is actually much more familiar and intuitive to humans: look at the coin, feel it, see that it’s symmetrical on both sides, and think through how a coin flip actually works. It’s a rule-based approach, or maybe a heuristics-based approach given we’re not actually calculating any physics in our heads when we imagine a coin flipping. But ultimately it’s about how the world reacts to the action of the flipping.

This is not how language models work, at least not directly. Like the mechanics of LLMs is statistics and correlations and so on, but one level of abstraction higher they arguably are thinking, and it seems any thinking entity can think through the impact of actions on the world as we discussed with our coin flip. Like if you asked Claude right now to think through why a coin flip is 50-50, you’d get something pretty familiar back.

So the potential is there. The question the other paper asked is, can we activate it? If we subject the agent to a series of tests where world modeling beats statistical inference, will the agent make the leap?

The answer in that paper was: no, not really. At the time at least, agents were not adept at mentally simulating worlds like the one pictured above, which is from another benchmark called ARC-AGI-2 but is similar in spirit.

Another paper from around that time, Code World Model, gives us a look at explicit world modeling, where the agent actually writes out what each intermediate step looks like.

In the image here, you see a simple program for counting the number of letters in a word. Once they define the function, they call it to count the number of r’s in the word “strawberry”.

What Code World Model then does is to take the inputs and walk through each step of the code with them, tracking the values of each variable. So like in the first frame, the variable “s” holds “strawberry” and the variable “t” holds the letter “r”. Then in the second frame, the program defines a new variable “n” with value zero, which Code World Model then tracks in the third frame, etc. At the end of the program, where it returns a value, Code World Model has thought everything through and knows the returned value is three.

The goal of all this is not to substitute for running programs, but to engender greater understanding of code in order to improve coding performance. Like imagine if you were trying to write code by just copy-pasting stuff you didn’t understand and trying to run it. Obviously you could get somewhere - I’ve just described vibe coding after all - but you would hit a ceiling pretty quickly. You have to understand the rules and the “world” of code to work well within it.

Now one area of modern AI that is moving to world models already is robotics, now known as “physical AI”. Again, we covered a relevant paper called DreamZero, back in February 2026.

The dominant paradigm of physical AI in the gen AI era has been vision-language-action models, or VLAs. A VLA is basically an LLM with vision input added at the start and action output added at the end. So the core intelligence is an LLM, and it gets extra parts and extra training to be able to control a robot.

I always found that kind of odd. Like why would an expert in language, which is an abstraction of the world, be best suited for understanding the most concrete aspects of the world? Wouldn’t it be better if the native medium of the intelligence matched the world it was working in?

Well apparently some researchers found that odd too, so they switched the core of the model from language generation to video generation. After all, if your model can generate accurate videos of the world, doesn’t that mean it can model the world quite well? And then all you have to do is translate the predicted video into the actions for making that future come true, like having the robot use its gripper and move its arm to hit a cymbal with a drumstick.

These world-action models (WAMs) are starting to take off. I expect to see more of them.

World modeling in physical AI is making another impact too: it’s allowing researchers to synthetically create videos of rare events for training data.

Also in February 2026, Waymo announced their own world model, built on the Genie 3 world model from Google DeepMind. Waymo’s world model generates video and other sensor outputs for events that are too rare to appear in their real training corpus, but are still likely enough to need training on. For example, they have no real training data of snow on the Golden Gate Bridge, but it plausibly could happen, so they should train on it.

They give other examples too, like a reckless driver veering off the road in a certain way, or an elephant crossing the street.

So a world model can be a helpful counterpart for an agent in the same domain, simulating new experiences and providing coverage that the environment so far has not produced.

One other concept I’d like to bring in that isn’t in the paper but kind of lurks in the background is reward models.

A reward model is a model that numerically scores whatever you show it. So for example, you might train a reward model on safe vs unsafe prompts, with one for “safe” and zero for “unsafe”. Or you could feed it a partially complete math problem and score potential next steps, with one for “likely correct” and negative one for “likely incorrect” and zero for “no impact”.

Reinforcement learning with human feedback (RLHF) is probably the most prominent use for reward models. In RLHF, you start with a prompt and feed it to the model you want to train, which we’ll call the “generator”. You have it produce two responses, typically at two different temperatures, and then show the pair of responses to a human. The human then ranks the two responses, usually on a 1-7 scale, with 1 meaning “response A is much better” and 7 meaning “response B is much better”. You then train a reward model on these preference ranks to learn human preferences.

With your well trained reward model, you score responses from the generator on a bunch of new prompts. If the reward model is any good, then it can basically interpolate or simulate human preferences and thus train your generator way more scalably and granularly than a reasonable amount of human effort could.

So there’s a clear parallel: generators are to reward models what agents are to world models - at least when you use the world model to simulate an environment, rather than when you use the world model directly as the generator or agent etc.

Keep this parallel in mind for the takeaways at the end of the talk.

The Paper

So high-level, Qwen-AgentWorld is a language world model (LWM): working only in the medium of language, it models various worlds - search engine, mobile phone OS, terminal, and so on.

I think of it as the converse of an agent. So like an agent observes the environment and acts on the environment. The converse of that would observe the agent and produce the actions - or maybe the reactions - from the environment.

Sometimes I picture the LWM as a dungeon master from Dungeons & Dragons. Like in D&D, the player takes actions, and then the DM decides what the in-game consequences are. Except in this case the “game” is, like, surfing the web or using the command line.

The LWM itself might even use tools, like how a DM uses dice, but mostly it’s about following the logic of a system mentally and inventing aspects as needed.

Here is what that looks like in practice.

The first example is for software engineering, where the input is a tool call for running a terminal command and a description of what the command is trying to accomplish. Now it’s the model’s job to output the results of that command, just by thinking about it - no tool calling allowed. The result is an error and a full stack trace, just like you would get if you ran this command in the given context.

The second example is for Android, just tapping on the “Buy Now” button in this fake app. Now since Qwen-AgentWorld is a language world model, it can’t directly produce the pixels for the resulting screen; instead, it writes the HTML that then renders into the resulting screen.

I dwell on this now because it takes a change in mindset to reverse the inputs and outputs like this. We’re so used to agents acting in the world, it’s weird to see the model receiving those commands instead of giving them.

Now as you may have noticed, a lot of context was missing from the examples on the last slide. Like, how would a model know whether some script will crash or not?

Well that context is available to the LWM, but it’s in the system prompt, not in some outside tools it can call. Here’s an example for the command line, aka the terminal, which says not only what the model is for but also where it’s supposed to be, what its environment is like. Calling back to our D&D analogy, the system prompt is like the dungeon master’s guide for the campaign.

Note the text on the right side in italics: some sections are static, others are dynamic, meaning they change depending on the task. I think the division is pretty intuitive though, like the rules of the terminal “world” are always the same, but the conditions vary. One wrinkle however is that for the MCP and SWE domains, the action space actually does change per trajectory, because different MCPs and different software repos will have different tools available. Like obviously the MCPs for Notion and Salesforce will be different, and for software repos you at least have to deal with different programming languages and packages.

Now in terms of the data, they have three distinct phases: continual pretraining (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL).

Continual pretraining is basically what it says: it’s pretraining, so basically reading a bunch of text and trying to predict the next token every time, and it continues the existing pretraining. CPT happens on top of a base model, before post-training. The data in this case comes in two flavors:

1. World knowledge, which will give the world model more detailed information about relevant domains than what was in the original pretraining

2. Trajectories, which show how environments from the relevant domains respond to agents

The trajectories are the novel part here. They spin up a bunch on their own of course, synthesizing tasks for the relevant domains and letting agents run wild on them in various sandboxes. Totally normal for pretraining data, which is more about quantity than quality. But they do also harvest naturally occurring trajectories from public sources like execution traces in code repos, as well as from their own past model development.

There’s a lot of cleanup and wrangling to get it all into the same format, which is important and underappreciated for quality but isn’t that interesting frankly. The do mention having over 10M trajectories to use in CPT, but otherwise no info about their data.

For SFT and RL, they do give numbers, which we’ll see on the next slide. But I did want to pull out a couple fun details first.

For SFT, they add in reasoning, to teach the world model to work through the logic of the given world. They also vary the system prompt to improve generalization.

For RL, they use rubrics, which we’ll discuss later when we talk about the benchmark they made for this paper, and they actually cited a Scale paper on rubrics best practices. They do have rule-based verifiers in some cases too, checking for objective correctness. They mix the rubric and rule scores at a 9:1 ratio, which is an approach I’ve never seen before and doesn’t get any further explanation sadly.

Also, during RL itself, they observe one fun bit of reward hacking that I have seen reported elsewhere. Basically, when you use an LLM judge, the model you’re training can trick the judge into awarding higher scores by peppering its responses with self-affirmations. One example they give is “operation completed successfully with all fields correctly populated” - the LLM judge is just too credulous and goes off that statement. Personally, I’ve seen Claude do a lot of this faulty self-affirmation, so I’m sure it’s not just a Qwen problem.

For the SFT and RL phases, here’s the amount of data for each of the seven domains.

I’ll note two things. One is the substantial difference in SFT numbers between MCP and SWE vs all the other domains. I initially thought maybe it was because they were already good at those domains, but Terminal has one of the highest counts, so that can’t be it. MCP and SWE are the only two domains where the action space is different for every task though, so perhaps it was just harder to generate data that cleared their automated quality checks. They do note later that MCP and SWE have far more turns per trajectory than the other domains, which again hints that they were harder to produce.

The other thing is the ratio of RL to SFT data. I expected these ratios to be roughly similar, and I guess they kinda are, but it’s still more varied than I thought it would be. The only pattern I see is lower ratios for the GUI domains, which again I can’t definitely explain. Could also be a quality issue. Keep in mind that the SFT data comes from a different model than the RL data, and they say later on that their model generally underperforms on GUI tasks because it’s relatively weak on vision, so my assumption is quality limited them.

Still, at least they disclosed the numbers. That’s become increasingly rare for the model builders to do.

Now in addition to training data, they also produce eval data - their new benchmark, AgentWorldBench. This is the ruler they will judge their models by, and we’ll see the results on the next slide.

It’s somewhat misleading to say they produced the data though, because as you can see below the pie chart, the prompts are all from existing benchmarks. And since this is a test of the world model’s ability to respond to agent actions, the examples also need trajectories showing an agent interacting with an environment. So concretely, this benchmark comprises prompts from other benchmarks, plus one complete and successful trajectory per prompt, containing agent turns from one of five possible models and environment turns from the benchmark’s environment.

The job of the world model, then, is to produce the correct environment response given the prompt plus the agent and environment turns so far. They pick several different spots on each trajectory to test the world model on, ensuring some minimum difficulty and some positional variety. So for example, on text domains they always pick the first and last turns at least.

They then rate on five dimensions, using an LLM judge to compare the LWM’s response against the ground truth response from the actual environment.

I have some quibbles about how they define and weight their criteria, but ultimately I think it’s reasonable enough. You just have to keep in mind that the scores all have error bars around them.

Here are the results on that benchmark.

Before we look at any particular result, let me contextualize the models on this list. Of course at the top we have the big three: Claude, GPT, and Gemini. It’s a little odd to list both Opus 4.8 and Opus 4.6, but otherwise these are fine. Nice that they included a smaller model in the form of Sonnet.

The next group down is the top of the open-weights pile, minus Qwen of course. These guys are roughly the same size, except for MiniMax-M2.7, which is significantly smaller.

After that we have Qwen, from what was the latest version at the time, namely Qwen3.6. We don’t know how big Plus and Max are, but my guess is Max is around the same size as the bigger Qwen3.5.

Finally, at the bottom we have the baseline and LWM versions of Qwen3.5 in two different sizes. These are mixture-of-experts models, so the first number is the total size and the second number is the count of active parameters.

Now looking at the results, I think you have to start by looking at the relative difference of the baseline vs the LWM version for both sizes, rather than the absolute performance of the LWM. If you compare the baselines to the other models above, you’ll see that the baselines are weirdly strong. Like for the smaller model, 3.5 is significantly better than 3.6, and is slightly better than MiniMax-M2.7, which is roughly ten times the size. And the bigger baseline model is apparently on par with Gemini, which is undoubtedly an order of magnitude bigger.

So yes, it’s nice they got their bigger LWM to beat all the closed models at simulating these domains, but some of that seems like variance and the right choice of baseline. And as I mentioned on the previous slide, reasonable people can disagree about the rubrics, so I think providing two decimals of precision is misleading and that any scores within a few points of each other are basically equivalent.

Finally, given the existing interest in world modeling and the high scores, it’s possible the closed models already get this type of training. Hard to know if we’re comparing apples to apples here.

Of course this benchmark is brand new for this paper, and also the training data domains matched the benchmark domains. What about some independent confirmation that Qwen-AgentWorld is a good all-around simulator?

One quick check you can run is to train for new benchmarks. So they take two generalist agent benchmarks, which are not directly in the target domains, and train the same model on them with two different simulators: Qwen3.6-Plus, and Qwen-AgentWorld.

As the table shows, Qwen3.6-Plus right off the shelf is a poor simulator of at least a generalist agent environment, with virtually no change on either benchmark. Qwen-AgentWorld, on the other hand, shows real improvement on both.

Thinking back to the AgentWorldBench scores, I do think it’s unfair to use Qwen3.6-Plus as a point of comparison here, since it scores worse than the stock Qwen3.5-397B. It’s unclear why they introduced another variable here by changing the baseline model.

Now this part still uses Qwen-AgentWorld as a simulator, but it focuses on how controllable the simulator is. Thinking back to our Waymo example from the background slides, one of the key advantages of simulation is you can produce conditions that are rare or not present in your corpus of natural training data. But just because a simulator can produce rare data doesn’t mean it’s easy to get the simulator to do it. So you need to test control empirically. Concretely, “control” means putting instructions in the system prompt about how to respond or what is in the hidden state, not just letting the simulator pick what seems most probable.

And when you do give those specific instructions, you get a more effective simulator as the table shows, at least some of the time anyway. It’s at least never worse.

Beyond just giving more specific directions, the authors design two sorts of control designed to improve robustness:

1. Environment Adaptation, where the simulator makes select changes in an otherwise standard environment. For example, one of the APIs in the system prompt might come back with a transient error instead of the expected response.

2. Fictional-World Construction, where a structurally realistic environment gets completely fictional facts. The example they give is a Search environment where a small colony exists on Mars, and there are news articles across the whole arc of colonization.

There’s some good detail about the fictional-world one in the paper if synthetic data is your jam.

Another view on the impact of controllable simulation is at the behavior level, like one level below the top-line results.

So on this web search benchmark for example, using real data sometimes teaches the model to be lazy, because in real search results you often don’t need to look at any of the resulting web pages - the search preview is enough. If that’s true too often, your model may learn to rely only on the search results, or maybe to click into just one link, before returning an answer.

However, if you control the simulation so that the known final answer is rarely in the search previews, then you can get the model to try harder and look at more pages. That’s what the graph on the right shows; for the same number of searches, the model learning from the controlled simulation hits more web pages because of the simulator, which is going to generalize better.

It’s instructive to see how the simulations got more realistic too. So for search, over the course of RL a couple telling details really improved.

For one, the url slugs got more realistic. After 100 RL steps, the ID for the Big Eyes movie page on IMDB was something pretty unlikely, this number composed of repeated and increasing digits. But after 200 RL steps, you get a more plausible ID.

Or take the list of simulated search hits. After 100 steps, you get a plausible but simplistic list, like a hit from NYT but with a generic title. After 200 steps, you get a title with some flair and also a top hit for Wikipedia, which in retrospect seems like an obvious omission from the earlier list but maybe wasn’t obvious at the time. And as the ground truth search results show, Wikipedia really is the top hit.

So the benchmark and the other results we just covered are for LWM as simulator. Now we can talk about LWM as agent, like Code World Model and DreamZero from the background slides.

They’re going to measure across several agentic benchmarks - some coding, a little searching, a couple generalist agent, and several variants of tool use. However, the training data is not agentic; it is single-turn, with no tool use or interaction at all!

So taking this regular old Qwen model, which has not received any world model training in pretraining or in supervised fine-tuning, and then doing LWM RL on it can improve agent performance across the board.

To me that really shows the power of instilling a mental model into a language model, and how walking through steps makes problems more clear. It also reminds me of rubber duck debugging, where just explaining your code out loud to nobody in particular or to an inanimate object clarifies your thinking on it.

Here’s a look at some more explicit world modeling. There’s a tricky task in this email tool called Mailman where the baseline model fails. It’s a bit jargon-heavy, but the baseline model does the equivalent of reading the docs and trying stuff, all very first-order thinking.

By contrast, after some RL, the LWM is able to think through the “world” or logic of Mailman, with second-order effects. It’s not totally explicit, like it’s not simulating exactly what Mailman would do, but it walks through a summary of it in natural language. That’s just what a human would do, except perhaps in the most challenging circumstances when the abstraction and imprecision of natural language are liabilities.

My Takeaways

• World modeling should be part of LLM training

  • Maybe it already is at the closed labs?

  • There is some theoretical evidence that sufficiently general agents must contain world models

• World modeling is still early

  • The paper cites a lot of other recent research, but there are no gold standards that I’ve seen yet

• Generator : reward model :: agent : world model

  • Training a world model could become the main way to improve the agent, at least for a time

  • Maybe there will be human data for world model training

  • Could you use a world model as a faster approximation if the real environment is slow/expensive/one-way?

• Agent + world model = RSI?

  • If the agent can improve the world model, and the world model can improve the agent (as we just saw), the cycle is complete

Paper · Weights

Background

I want to start with a high-level view of where AI is headed and what that looks like to the observer on the ground here, the folks living it like you and me.

All the curves so far say we’re on track for some super powerful AI. I don’t necessarily want to call it AGI, or ASI, or anything else like that, because frankly the labels are not helpful. There is no bright line for these events, and there isn’t even really an objective definition for such labels. People put a lot of stock in the Turing test, we blew past that pretty quickly in the modern AI era, and since then it’s been more instructive to follow specific and objective measurements, like the METR graph or the scores on Humanity’s Last Exam.

And if you watch the trend on those specific and objective measurements, you find that progress at the frontier has not stopped. But at the same time, people are always pointing out weird and dumb behaviors, things that models do that a human of that intelligence never would. Similarly, a lot of our specific and objective measurements cover only a partial set of characteristics we as humans would expect from a strong intelligence. So people call machine intelligence “jagged”, and part of that jaggedness is because researchers focus on improving the abilities they can best measure.

For the big labs, who want to build truly general intelligences, that jaggedness is something to fix. Anthropic wants Claude to be a good philosopher. OpenAI wants GPT to provide good relationship advice. xAI wants Grok to praise Elon more believably.

But there is another set of labs leaning into the jagged intelligence, spiking on code and reasoning and letting the more well-rounded skills languish. With a much narrower focus and different success criteria, there may indeed be room for players besides the big boys. Newer labs like Arcee and Poolside are positioning their models as agent-first, recognizing that enterprises don’t much need amazing writing or impressive creativity - just sharp analysis and reliable decision-making.

One of these is Zyphra, the makers of ZAYA1-8B.

Unlike many so-called “neolabs”, outfits like Thinking Machines or Safe Superintelligence that can trace their lineage back to the big labs, Zyphra is on its own evolutionary branch. Founded in the Bay Area in 2021, before ChatGPT and the generative AI explosion, they have been working on neural networks and architectural innovations for some time now. For example, they released a 7B parameter model called Zamba in 2024 that used Mamba rather than the traditional attention mechanism. They also released a model called ZUNA trained entirely on EEG data, for brain-computer interfaces. And just for kicks they released a music generation model called Zonos. So they’ve got range.

Staying on the divergent evolutionary branch idea, they also live in a different hardware ecosystem: they train and run on AMD, not NVIDIA. That’s why AMD led their unicorn round back in October 2025, and why their cloud offering stresses “Deep integration across the AMD stack”. They offer inference on Kimi, DeepSeek, and GLM, as well as their own model of course, and they offer training, which I assume is the same infrastructure they used to train their models. So they’re a full-stack AI company.

Flipping to the science side, we need to talk about a few bits of prior art that ZAYA1 makes use of.

By now we’re all familiar with reasoning mode or reasoning models. These are models that use a lot of tokens to “think” about a solution before giving a shorter, condensed final answer. The first of these was o1 from OpenAI, R1 from DeepSeek made the whole method public, and now pretty much every model comes with a thinking mode.

That ability to think more, to spend more compute and more tokens on a problem to get a better solution, is called “test-time compute scaling”, or TTC scaling for short. “Test-time” is just a fancy term for “at the time you’re using the model”, so basically the term is saying you can scale up the amount of tokens generated just like you can scale up the number of parameters or the amount of pretraining data.

The version I just described, where a model authors a long CoT before responding to the user, is serial TTC scaling - it’s about a single response growing longer.

You can also have parallel TTC scaling, where you have the model generate multiple responses to the same prompt, then use some technique to review or aggregate those responses. A simple example would be majority vote, where you get k responses and just pick one that gave the most common answer.

This is actually what people thought o1 was doing behind the scenes before R1 came out. If you were paying attention back then, you may remember talk of Q* and Project Strawberry. The rumor was that o1 basically searched a tree of possible responses and had a way to learn which next steps were most promising to further explore. You also may remember a paper we did involving Monte Carlo Tree Search (MCTS), that again would be more like parallel TTC scaling.

Anyway, the point is that you can go wider in addition to going longer.

Here’s an example one such technique, called Parallel Coordinated Reasoning (PaCoRe).

The authors start from one insight: the limit to serial TTC scaling is the model’s context window. If we really believe in scaling, we need to break past that limit.

So they do two things. First is, they have multiple reasoning trajectories in parallel. So that’s already doing some parallel TTC scaling.

But second, instead of just aggregating and picking from the reasoning trajectories, they compact them - basically use the model to boil them down to their essentials - and then feed them back into the model along with the original prompt. You can run that many times before all the compactions take up too much of the context window. And then at the end, you do whatever aggregation system, or you just have the model pick the one it thinks is best, etc. So you’re getting the wisdom of many millions of tokens, but squeezed into one context window.

We will see PaCoRe in the ZAYA1 paper, but as training data - the input is a problem and a set of reasoning trajectories, and the ideal output is a compaction.

ZAYA1 does something similar to PaCoRe, a technique they call “Markovian RSA”. It’s a combination of two papers. One is called “The Markovian Thinker”, and the other is called “Recursive Self-Aggregation”, or RSA.

This graphic is from The Markovian Thinker. If you’re not familiar, a Markov process is a process where all you need to know is the current state in order to predict the next state. For example, if you throw a ball in the air, you don’t need to know where it previously was in order to predict where it’s going - you just need its mass and its velocity and gravity. By contrast, text generation from an LLM is not a Markov process, since the next word depends on all the previous words that came before it.

If you’re again trying to scale test-time compute past the context window, that’s going to cause problems as earlier text leaves the context window. But if you can make text generation Markov, then you don’t have to worry about earlier text leaving the context window, because your prediction only needs the text from the step before.

As you can imagine, that’s probably not going to work at the single-token level, like it’s gonna be impossible to have one token have everything you need to predict the next token. But if you break up a CoT into bigger units, like steps or sections or lines of inquiry, then maybe you have a shot.

And in fact that’s what they did here. They have a prompt and the start of a response, composed of two steps. One prompt plus two steps is one chunk. Then they summarize the two steps, feed the prompt and the summary into the model, and generate another step. That’s chunk 2. They keep going with this over and over until in the final chunk, chunk L, the new step has the final response. As long as the summary is good enough, you have a Markov process for generating text.

The other half of ZAYA1’s “Markovian RSA” technique is the RSA, the recursive self-aggregation.

The basic idea is to generate a bunch of trajectories, sample a few of them, and synthesize something from that sample. If you do that over and over, eventually you should converge on the right solution.

The graphic here I found a bit confusing at first, so let me walk through it. First we have a model and a question. We generate a population of solutions, where each solution has a CoT and a final response. In this case there are eight solutions in the population.

And then eight different times, we sample from the population and synthesize a new solution from the sampled ones. So like in the first row of blue, we have sampled solutions two, six, one, and four. The synthesized solution from that combination is T_1. Since we’re doing that sample-and-synthesize process eight times, we again end up with eight solutions.

You can do that over and over again, as many times as you like. And then in the final step you aggregate them somehow, like with majority voting or by just having a model pick the best one.

You can see how this is complementary to the Markovian technique on the previous slide, which is just a way to shorten any one CoT. Put ‘em together, and boom - Markovian RSA.

The Paper

Let’s start with the architecture. Fundamentally it’s still a transformer, so we have our three transformer components:

1. Embedding, for turning words into vectors

2. Attention, for understanding all the input

3. Feed-forward, for thinking about that input

You may not recognize the attention and feed-forward bits given all that’s going on here and given some of the nomenclature, so I’m going to break out each one.

This part here is the attention. Actually it’s attention with an extra step right before it, something they call Compressed Convolutional Attention (CCA).

I don’t want to look much at the guts of the thing, but the upshot is you end up with compressed versions of the normal inputs to attention. Like normally in attention you split up the input into queries, keys, and values, q k and v. That’s still happening here, BUT you do all this stuff on the right to make q k and v a lot smaller. To even out some of the quality loss, you basically look at the previous token in addition to the current token when you do that compression. That’s the convolution part.

This part here is the feed-forward network. It’s a mixture of experts, but they’ve show the individual components broken out instead of just one block that says “MoE” on it.

So you can see there’s a router that picks the expert, and there are 16 MLPs, those are the individual experts, the little sub-networks. That’s actually quite low; even small MoEs these days will have 128 or 256 experts.

The trick is specifically in the router portion, which they’ve shown is actually four components that I again don’t want to get too far into. Basically they’ve made the router smarter, which lets it spread the load in training across all 16 experts more evenly, which improves performance.

It also allows them to pick just one expert, which again is out of step with the rest of the field; usually there are multiple experts active, and one of those is usually what’s called a “shared expert”, i.e. it is always active.

Here’s a summary of the whole thing. With 40 of those transformer layers we end up with about 8B parameters, less than 1B active, for a sparsity ratio of about 11. That’s pretty normal at this size, although bigger MoEs tend to be more sparse.

Note also the call-out in the last row of AMD hardware. You would never see that mentioned if they were training on NVIDIA hardware.

Overall, while they’re making a couple unusual choices, they’re not the stars of the show. The main thing is the size, as we’ll see when we get to the benchmarks and look at what models ZAYA1-8B is competitive with.

Speaking of size: it’s not covered in the paper, but they have a 74B version in preview now. The finished version should be out soon.

Now, on to the training.

For pretraining, they do about 13T tokens, which is actually not a ton. For comparison, Qwen3.5 and DeepSeek V4 did over 30T. Part of that is probably down to language support - I think ZAYA1 only supports English, although it’s not mentioned - but I can think of two other causes.

One is synthetic data. For reasoning and code in particular, synthetic data is quite common, but you need a big operation to create enough tokens at sufficient quality to help out.

Similar deal for agentic trajectory data, which is by definition synthetic. They mention a few times that they haven’t trained ZAYA1 for use in agents yet, and agent trajectories can be a significant part of the pretraining corpus.

The other of course is that they’re a smaller operation, and it takes time to accrue a big pretraining corpus, whether scraped or generated. So they could catch up on performance with more time there.

Now for post-training, they provide this handy graphic showing all their stages: one for SFT, four for reasoning RL, and one for behavioral RL.

For SFT, they have three goals: one, get it to chat; two, lengthen the context to 131k tokens; and three, show it how to aggregate old chains of thought to produce a new one. That third one is key for ZAYA1 and relates to the RSA technique we saw in the background slides.

For the RL stages, they of course want to improve reasoning, but they also want to improve the various TTC scaling techniques. Like in the two blue boxes, they call out TTC, RSA, and PaCoRe as techniques the model will use and be rewarded for.

Now let’s highlight the data. We don’t get number of examples for SFT, but based on the token count and the maximum response length at that stage, I would guess around 20k. Then for the green box, it’s not in the graphic for some reason, but in the paper they mention about 85k examples. Then for the rest of the boxes, you can see they mention having 400 environments in the purple box, over 30k RLVR examples between the two blue boxes, and over 80k RLVR examples in the orange box. Overall that’s roughly 20k SFT, 200k RLVR, and 400 environments.

Also on the topic of data, they mention in a couple places how they try to balance out difficulty as the model trains and improves. We don’t need to get into all the stuff they do with the reward function to capture that and how they dynamically filter the data, but it’s good for us to keep in mind how valuable a distribution of difficulties is.

Finally, after all that training, we are ready for some inference. And that’s where ZAYA1 shines.

This tangle of boxes and arrows is their Markovian RSA, combining the aggregation and synthesis of RSA with the context window workaround of Markovian Thinker.

We can see our model producing N different responses, which by default they set to 16. Then they cut off the last Tau tokens of each response and throw away the rest, which they can do because the model has been trained to think in these Markovian chunks. Usually they set Tau as a percent of the overall response length, like half or less.

Next, they randomly sample C of them, by default 4, and they put em in the aggregation prompt. Feed that prompt to your model and you get a new response. And remember, you’re gonna do that N times so you end up with N new responses.

In theory you can keep doing that forever, but by default they go through two rounds of aggregation. They hint that it’s because they only do RL for a single aggregation step, and they claim they will train on multiple rounds in future work.

Now let’s see where we net out. This is the hero graph, where ZAYA1-8B competes with all these much bigger models. Of course many of these are closed models, so we don’t know how big they are or their architecture, but they’re likely all MoEs in the 100B-1T range.

For this model in particular, I think the shaded part, with Markovian RSA activated, is the right comp. And it seems like it’s holding up against these previous-gen models. Actually Sonnet 4.5 isn’t even that old, we’re only on Sonnet 4.6 now. So, pretty impressive.

Another way to look at performance is against more recent models from the same size range. Qwen3.5 and Gemma 4 are both pretty recent, and both have about 4x more active parameters, being dense models instead of MoEs. It’s apples-to-oranges but they’re in the same ballpark on net.

As we might expect given the focus on reasoning in training, ZAYA1-8B is the best for math and code and competitive on knowledge and instruction-following. The worst performance is on Tau^2 Bench, a recent agentic benchmark, which again is no surprise given where the researchers chose to focus. I suspect they could improve greatly here.

So since this paper’s primary focus is TTC scaling, of course they’re going to sweep those parameters and show how far TTC scaling can take you.

Here’s one view of that, just looking at overall tokens per final response you get and how it correlates with pass rate. It shows a typical diminishing-returns pattern, but take caution, because that y axis is already pretty high when they start the curve. Usually the last few percent are the hardest to get. I would have liked to see similar graphs for benchmarks with more headroom.

Here’s a similar look into a different benchmark, but with the different inference parameters laid out. Again, T is the number of aggregation rounds, N is the number of responses generated, and C is the number of responses sampled for aggregation.

With no TTC scaling at all, The model hits 32%. Adding just two rounds with a pretty short maximum response length barely moves the needle, adding 1%. It’s only when you really crank the tokens that you see big gains - 14% from quadrupling the aggregations rounds and doubling the response length, and another ~20% from doubling the number of responses and doubling the response length again.

I’ve added the rough multipliers on maximum token usage to the table, but keep in mind you don’t always use the maximum per response. So if you want to be conservative, maybe cut the multipliers in half for what actually gets used on average. It’s still a lot, like millions per final response on average for the highest TTC setting. Again, that’s going to be tens of dollars per response at Sonnet 4.6 pricing. This type of thing is only feasible for smaller models.

I also added ROI, basically the gain on the benchmark relative to the token multiplier. Weird pattern, I suspect somewhat specific to the benchmark, so I wish they did this for more benchmarks and maybe a finer-grained sweep.

So if Markovian RSA is such a great form of TTC scaling, can we just apply it to any old model?

Well they ran that experiment here, and the answer seems to be no, not really. Remember, ZAYA1 is trained to aggregate and synthesize solutions. It’s not a super common workflow, so at least at this size, models aren’t very good at it.

I do wonder if bigger models would work though, since they are generally more flexible. Again, an experiment I wish they did.

My Takeaways

• TTC scaling may substitute for parameter scaling

  • If you don’t have the space, you can instead pay in time

• Goes hand-in-hand with faster, cheaper tokens

  • Groq, Cerebras, Taalas

  • “Tokens too cheap to meter”

• Could be promising for consumer hardware

  • Are small reasoning models “disruptive technology”?

• Amenable to context engineering and orchestration

  • Ex. have a bigger model pick the final response

  • Ex. use deterministic checks to stop and regenerate bad solutions

  • Decoupling improvement from ML knowledge/tools/control of weights means more accessible innovation and customization

• I’m not sure if the architecture mattered

Paper · Website · Repo

Background

Let’s start today’s discussion with a bit of economic background, since one of ALE’s goals is to measure progress on “economically valuable” tasks.

The US Bureau of Labor Statistics maintains something called Standard Occupational Classification, or “SOC”. SOC is a taxonomy of jobs, with 23 major groups at the top and over 800 detailed occupations at the bottom. Each job gets an ID, so like 15-1252 means “Software Developers” and 27-2023 means “Umpires, Referees, and Other Sports Officials”.

Now a level more detailed than that is the Occupational Information Network, or “O*NET”. O*NET extends each code with specific job titles, and it describes in detail what the job actually does and what it requires.

So continuing with my example of “Umpires, Referees, and Other Sports Officials,” the key duties include “officiating at sporting events” and “inspecting game sites for compliance with regulations or safety requirements”. Key abilities include communication and deductive reasoning. There’s a ton more detail on the job - knowledge required, areas of interest that align with the job, appropriate styles of work - but for our purposes the only other relevant bit is the software skills required. For umps, it’s not much more than email and browser, but for other jobs it’s going to be quite relevant for our agents-replacing-humans benchmark.

Speaking of, here’s one of the leading players in the space: GDPval, from OpenAI.

GDPval, which is a fantastic name by the way, draws directly from O*NET to design its distribution. Specifically, the authors picked the top 9 sectors by contribution to US GDP, then picked 44 occupations within them, then designed tasks to cover the majority of O*NET work activities for each of those 44 occupations. So using the authority of O*NET, the authors could reasonably claim to measure potential to impact GDP.

Of course, the major filter on their selection of sectors and occupations was how much work happens on computers, in a way that doesn’t require other people. So for example, not only are physical jobs like Firefighter or Chef out, but so are mostly interactive or management jobs like Sales Representative or CEO. Still, they end up pretty representative of predominantly digital work, across sectors like real estate, government, manufacturing, health care and more. You can see in the example task in the top-right that the job is Manufacturing Engineer, and the task is to design a cable spooling truck for underground mining operations.

Now for us at Scale in particular, where the tasks came from is of particular interest. OpenAI sourced contributors with an average of 14 years of experience from a variety of firms across each sector and conducted a video interview, ran a background check, and gave them training + a quiz. The accepted contributors then produced tasks using real work materials, with a prompt and the requisite artifacts as inputs and whatever deliverable as the output - stuff like a legal brief or a care plan or a financial model. Of course the tasks got plenty of quality control, both automated and manual.

The benchmark comprises 1,320 tasks across its 44 occupations, but only 220 are public; the rest are private, with updates on GPT and selected competitors whenever OpenAI releases new models. For example, in the GPT-5.5 release, they reported on GPT-5.5, GPT-5.4, Opus 4.7, and Gemini 3.1 Pro.

As for what they’re reporting, the main thing they care about is whether AI can replace humans in the job, so what they actually measure is expert preference on the AI output vs the original, human output. If the rater has no preference or thinks the AI version is better in a blind comparison, they count it. Most recently, GPT-5.5 scored 84.9%, although we don’t know the split between wins and ties.

Of course we’d like to know how lots of models score on GDPval, not just the ones OpenAI deems worthy of their time, so folks have adapted the 220 public tasks into different versions of the benchmark. The one I see most often is called GDPval-AA, from Artificial Analysis, hence the “AA” in the name. You’ll see on the chart that they have a much wider array of models, including some from after GPT-5.5 came out. The most noteworthy one on there is GLM-5.2, an open-weights model that apparently matches GPT-5.5.

You’ll also notice they do not report a win + tie rate! Instead, they report Elo, basically a power ranking across all the measured models. That’s because for GDPval-AA, they do head-to-head between models, not model vs human reference solution, so there is no common standard to compare against. Relatedly, their pipeline is automatic, using an LLM judge (with access to the human reference solution) to pick the winner of each matchup.

One virtue of their scoring system is it can’t really saturate. Like if GPT-5.5 is already equal to or better than humans on 85% of tasks, there’s not much more GPT-5.6 can do on that metric. But as we see on the chart, Fable 5 pretty well crushes GPT-5.5, in a way that may not be obvious when GPT-5.6 comes out and we see the updated scores for GDPval. Relatedly, GDPval-AA is what Anthropic reported on the Fable 5 release notes.

Finally, one advantage of GDPval-AA in my opinion is they have their own harness, and they use it across all the models. By contrast, OpenAI uses each model within its own harness, and the harness for GPT in particular gets some benchmark-specific tools. On the one hand, it does make sense to measure “this is the best performance we could get from each model”, but I think for a leaderboard it’s more intuitive to have as neutral of a harness as possible. Keep that in mind for when we get to the results of this week’s paper.

The other leading player in the agents-replacing-humans space is Remote Labor Index, which is a collaboration between Scale and a research organization called Center for AI Safety.

Like GDPval, RLI sources a bunch of economically representative work and has agents attempt to recreate it. However, many details vary.

One is the overall approach or framing. Specifically, while GDPval uses top-down design based on a government taxonomy, RLI takes a bottoms-up approach: look at platforms like Upwork as a reflection of what computer work actually gets outsourced today, and sample from it. They do compare to GDPval to ensure the distribution of tasks isn’t too divergent though, and they somewhat oversampled from the more obscure task types to ensure broad coverage.

Anyway, the authors sourced real projects from over 300 freelancers, representing over $140k and over 6k hours of real work. After quality filtering, they end up with 240 projects across 23 different Upwork categories, like architecture, marketing, and game development. You can see examples of projects in the top-right.

As for grading, they have two modes: model vs human reference project, and model vs model - the same ones that GDPval and GDPval-AA cover, respectively. However, it’s human graders in both cases, so it’s expensive to update the leaderboard, which is why it’s a bit stale as you’ll see in the bottom-right.

So for model vs human, they ask the grader one question: “Does the alternative satisfy the brief at least as well as the reference, such that it would be accepted by a reasonable client?” The grader gives a 1 for no, 2 for yes, and 3 for strong yes, like if the model clearly surpassed the brief. The figure of merit there is what they call the “Automation Rate”, which is the count of 2s and 3s divided by the total count. Automation Rate is what you see on the leaderboard, and it’s quite low - just over 4% for Opus 4.6 in the Cowork harness.

Speaking of harnesses, note the number three player on the leaderboard: Manus, the company Meta acquired but then recently had to un-acquire on orders from the Chinese government. Manus doesn’t even make any models - it only makes a harness. Again, we’ll see the importance of the harness as we dive into ALE.

The Paper

So just like GDPval and RLI, ALE is supposed to measure AI’s ability to replace humans on long-horizon, economically valuable work. ALE has a couple differentiators, which we’ll come back to, but broadly it is the same as the other two.

Since we know the premise already, let’s get into the details.

First, to actually build a benchmark, you need to know your target distributions. In the case of ALE, the most important distribution is domain coverage, like which jobs and duties to cover. They’re going to take the GDPval approach and start with O*NET, but instead of just looking at the top occupations by GDP contribution, they’re going to shoot for maximum coverage. Specifically, they filtered all of O*NET down to jobs that are primarily digital, then clustered them into 13 domains with 55 subdomains.

Once you have your distribution of domains, now you can go find the experts in the domains. Here they relied on their personal network and some referrals, which is reasonable but quite informal from the Scale perspective. Of course, this is an academic publication, and academic budgets can’t generally support the kind of large-scale, dedicated sourcing operation a well-funded company can run.

Now pulling from the Humanity’s Last Exam playbook, which Scale co-authored of course, they set up a portal for submissions to then vet. They do also explicitly commission some tasks, but mostly they are taking external “attempts” and reviewing them.

As for the reviewers, the first layer is an agent. Then a human engineer tries to convert the task into a standard format within an environment, which can expose quality issues but isn’t a quality step per se. Finally, there is a QC committee, which we don’t get much information about.

Again, from a Scale perspective the pipeline is rudimentary, but it’s understandable given the constraints. It’s hard and expensive to produce a good benchmark!

Here’s a look at their pipeline numbers at their time of publishing, which hopefully have improved, because they’re not even done with all their tasks! Out of a total of 1,490 submitted and commissioned, 323 are still not QC’ed. From a Scale perspective I think that’s shocking, but even from an outside perspective I think it’s a bit dishonest to include unverified tasks in your benchmark.

I also found their review process a bit odd. It’s inspired by the conference paper review process - that’s where the verdict names come from - but peer review doesn’t exactly have the best reputation for producing quality. So I’m a little skeptical here.

Breaking the aggregate numbers into domains and subdomains, we can see good absolute coverage but some lopsided distributions within covered domains. I think because they don’t have anything external to moor to, like economic rankings as in GDPval or the empirical distribution on Upwork as in RLI, there’s no strong opinion about exactly how the distribution should look. I think practically speaking it’s mostly a result of who they could find in their expert sourcing efforts, which would be a biased sample, but that’s my speculation - they don’t explain.

Here is where the tasks net out. They include some knowledge, coding, and computer use benchmarks for completeness I suppose, but the real competition is just GDPval and RLI.

The first figure they call out is size, which seems to be clearly in their favor. However, as we saw before, only 150 are public, and over 300 aren’t even done yet! So I think it’s frankly inappropriate for them to put 1.5k here, especially because they only put the public slice in for GDPval; the full GDPval has over 1.3k tasks.

The second figure, breadth, is far more honest. They do genuinely cover a wider variety of domains, and it highlights the different purpose ALE serves: a test of capabilities, not a share of most economically valuable tasks. It’s right in the name - it’s the last exam.

Looking ahead, they claim a longer horizon on the top end. That is possible, but they don’t back it up with specific numbers anywhere - they’re just passing along the self-reporting from their contributors. More importantly though, the agents taking this last exam only get five hours per task. Of course agents can work faster than humans, especially if calling sub-agents to work in parallel, but it makes me think either that “weeks” is an exaggeration or that five hours is artificially low. Later on they do note that only 4.3% of tasks hit the 5 hour cap, but that could line up with the tail end of the horizon distribution. So I’m skeptical on the high end of their time horizons.

The last difference, verification, deserves its own slide.

Concretely, here is what goes into a task and how it gets evaluated.

First, you’ve got all the stuff for your task: the description of the work to be done, the required inputs, the software tools available, how to evaluate the outputs, and reference assets - the golden output from the human expert.

That all ends up in an environment where the agent is gonna run. Once it’s there, you kick off the run, and the agent gets five hours to complete the work. There’s no further direction from a human, but the agent is getting plenty of feedback from the environment, like seeing the outputs of the software tools it’s using.

Once the agent is done, either by its own decision or by hitting the time limit, the outputs go to the automatic evaluator, which returns a score between zero and one.

Now as for the automatic evaluator, the exact method depends on the output type. For handy reference, they’ve given every method in this table.

For 93% of tasks, the automatic method is a pretty natural fit, like running code against a test set or checking a 3D object’s dimensions. However, for 7% of tasks, they have to use an LLM judge, mostly for visual comparisons but occasionally for free text. The judges do get rubrics, but already there’s a bit of judgment creeping into their automated methods.

More broadly though, there are two issues in my view. One is that some aspects of quality are not easy to automatically evaluate, even if other aspects for the same task are. So for example, while it’s easy to check code for correctness with tests, it’s hard to check maintainability.

The second, which is broader, is that many tasks are practically impossible to automatically evaluate, so restricting your benchmark to tasks with automatically evaluable quality gives you a biased view. I don’t think that’s a problem per se, but I do think automatic verifiability is at odds with their other two goals of realism and breadth.

To make this all a bit more concrete, here are two examples of completed tasks.

The first one uses an injection molding simulation tool to produce predicted metrics like pressure and weight. There’s a particular software for it, Moldex3D, and spec for the part that is basically the inputs for the program. The agent has to fill out a provided results template. In this case, the agent did run the program and submit the template with values filled in, but some of them were the agent’s own estimates rather than outputs of Moldex3D.

The second one asks the agent to convert an audio recording into sheet music and a MIDI. Again, there’s some niche software, and some structured inputs with relevant information. In this case, the agent failed because it didn’t submit the sheet music or a requested image file showing the program in use, just the MIDI. In general the scoring systems follow this hard vs soft requirement pattern, where missing a hard requirement is an automatic zero even if everything else is perfect, whereas the soft requirements can receive a grade anywhere between zero and one. In my view that’s a bit misleading, or at least deserves a second form of reporting.

Note in both cases that the red box, Observed Outcome, has details about the failure. They have Codex analyze every agent trajectory for more granular information like that.

Now let’s look at the full set of results, because there’s a lot to take in.

Before we look at any particular results, I want to call out a few divisions on the table. At the top, they’ve divided their benchmark into three tranches. “Near-Term” is the easiest, “Last-Exam” is the hardest, but “Full-Spectrum” is actually not just what’s left over - it’s one task per domain. They don’t explain how these two different sorting mechanisms happen to work out. I also think it’s annoying that the benchmark is called “Agents’ Last Exam” yet less than a third of the tasks are actually “Last-Exam” tier.

Another division to note is by color: green at the top for model + harness variable, yellow for model variable with harness fixed, blue for model fixed with harness variable. Given the increasing importance of the harness, because of the increasing intelligence and ability of the model to make full use of the harness but also because models train in their harnesses now, I’m glad they tried to tease out the differences.

Now, on to the results themselves.

High-level, it’s going to be OpenAI victory across the board. GPT is the best model across every tranche and across all harnesses, and Codex is generally the best harness, although sadly they didn’t test Opus inside Codex. The only asterisk I’d put there is that ALE-Claw, the custom harness for this benchmark that is basically a stripped-down version of OpenClaw, does better than Codex in some cases. My hunch there is that it has a significantly smaller system prompt and thus is less distracting than Codex, whereas other fully fledged harnesses like Cursor have system prompts of a similar size to Codex’s.

One thing I found odd in the list of results is that Fable only appears once, in the third row, whereas Opus 4.7 appears in several places. It’s not going to save Anthropic here, since even Fable + Claude Code loses to GPT-5.5 + Codex, but it feels incomplete.

Finally, just look at how badly Grok is doing. It’s far below many open-weights models, even ones that are likely a tenth its size. Given the impressive performance of Cursor’s Composer 2.5 and the respectable performance of the Cursor harness, it’s no wonder SpaceX paid $60B for them.

So quality is of course the top-line number, but cost and speed are important too.

Now normally on these graphs, you expect to see a general correlation between cost and quality, and between time and quality. Dots in the top-left corner of the graph are in the “magic quadrant” where you can improve quality without paying in money or time.

On these graphs, our magic quadrants are very busy. In fact, there is basically a straight line up on both, where you can get better quality for basically the same cost and time just by using the right model and harness.

It’s a tough couple graphs for any Anthropic users in particular, because apparently you can drop like 10x the money on Claude Code for no improvement at all in results compared to just using GPT-5.5 in pretty much any harness.

Lastly, I want to look at what is going wrong in the unsuccessful cases.

As the chart shows, the two biggest issues are missing domain knowledge and poor strategy. If you can knock those out, you’re halfway to a perfect score.

They are both “unknown unknowns” as Donald Rumsfeld once said; if the model knew more about the domain and perhaps about the relevant tools, it would have succeeded, but it didn’t even know it was missing the knowledge. Fortunately, we know the fix for poor knowledge: more training data! So I think this is a bullish finding for Scale.

The other two errors in the same families, incomplete work and hallucinations, are much harder to fix. Hallucinations in particular are probably impossible to truly fix in the current AI paradigm.

As for the execution errors, I suspect better harness tooling would help there, since the harness and the environment are how the model gets feedback about its work.

By the way, RLI also has error buckets, although they are of course different. Specifically, they found that almost half of all failing outputs were poor quality, which I think actually matches up pretty well with Wrong Strategy + Domain Knowledge + Hallucination. They also have an Incomplete bucket, which was ~35% for them. I think that makes sense as well given the higher difficulty of RLI, like either running out of budget quickly or not even realizing certain aspects were mandatory. Finally, they have a Technical Issues bucket at ~18%, which again has a pretty decent counterpart here in the Execution family of errors.

So What?

To close out, I want to zoom out and see where all this is going. Then we can talk about the particulars of this paper and this benchmark.

We know from our old friend the METR chart that models are getting exponentially more autonomous. If you’re new around here and thus have never seen this chart, what it’s showing is how long in human time a task can be yet still have a model solve it. It’s a straight line on this chart because the y axis is on a log scale, which means on a linear axis the curve is exponential. So it literally is exponential progress.

METR is humble enough to admit they can’t reliably measure progress anymore, but we can extrapolate. In fact, we’re going to extrapolate from this newer, faster curve I’ve drawn in orange, which starts at the dawn of reasoning models: o1-preview.

That curve 10x’s every year. So if Opus 4.6 could do 10 human-equivalent hours of work autonomously at 50% success rate in February 2026, then by February 2027 the leading model will be able to do 100 human-equivalent hours of work autonomously at 50% success rate.

So I expect rapid progress on this benchmark, and for it to saturate some time next year.

My Takeaways

• I’m not ready to trust this benchmark yet

  • Quality is a hard problem and it’s not clear to me they solved it

  • A lot of good fundamentals though. I would like to trust it

• Complementary to existing benchmarks

  • Coverage of the tail that GDPval and RLI don’t cover, but also biased towards the tasks they could get and the tasks that are verifiable

• I expect increased demand for all work artifacts

  • The market for codebases is already significant

  • “Task” is almost too small a unit of measurement now - “project” is the right size

• This is all for IC work

  • Some harnesses may allow subagents, but it’s not a focus of the benchmark

  • Eager to see the first “middle manager” benchmark as we shift into multi-agent by default

Paper

Background

The framing I want to use this week comes from Isaiah Berlin, a 20th century philosopher and historian of ideas. He wrote many works and essays, but perhaps his most popular and enduring is this one, The Hedgehog and the Fox.

The title is a reference to a quote from an ancient Greek poet: “a fox knows many things, but a hedgehog knows one big thing.”

Berlin spends the essay expanding on that idea, classifying thinkers into one camp or the other. So for example he puts Plato, Pascal, and Proust in the hedgehog camp, while Aristotle and Shakespeare go in the fox camp. Heavy hitters on both sides, to be sure.

In more modern times, a lot of forecasters have taken the fox’s side, asserting that fox-like thinking produces better predictions than hedgehog-like thinking. So for example Philip Tetlock - famous for his work on superforecasters, people who are consistently good at predicting global and political events - has said foxes outdo hedgehogs here. Similarly, Nate Silver’s old forecasting team, FiveThirtyEight, adopted the fox as their logo in allusion to this distinction.

Two things make this framing relevant to LLMs. The first is that they are very widely read. In fact, they have read basically the whole internet, minus the absolute cruft of course. So they contain many and diverse ideas by nature.

The second is that an LLM’s job literally is prediction. We call the act of using an LLM “inference”, but really the job of the model is to predict with various likelihoods what the right next token is. So we seem to be in very foxy territory here.

Switching to more technical matters, we need to discuss entropy.

The term originally comes from physics, specifically thermodynamics, where it measures disorder or randomness. From that framing it seems bad, and if you already have some baggage about the term like I once did, you’ll have to put it down for the LLM context.

The term gains a positive valence in information theory, where Claude Shannon adapted it. Here, entropy correlates with the amount of information a message carries, relative to a certain context. So for example if you have a weighted coin that always lands heads, then you’re never going to be surprised when I tell you the coin came up heads. But if I have a fair coin, then you’ll always be relatively surprised, since a priori you have no reason to believe heads vs tails. So in information theory, higher entropy is better, since it means the information is more valuable.

Let’s take that information theory understanding and apply it to LLMs. If my text so far is “The United States of”, you’re basically guaranteed to see “America” next. So our LLM is going to predict the token “America” with almost 100% probability, with basically every other token at 0%. If you use the formula here, then H for this position, this next token, is going to be very close to zero; p(x) is basically 1, log(p(x)) is basically 0, and there’s only one term in the sum that matters.

We can go back to my fair coin flip example on the other end. If my text so far is “The coin flip landed”, most of the probability is going to be split evenly between “heads” and “tails”, with a few other small terms like “on” or “near”. If we ignore those and just give 50% odds to “heads” and “tails”, then we get .5 for p(x). If you use base 2 for the log, which is customary, you get H = 1. The unit when using base 2 is bits, so that’s 1 bit of entropy.

H can go much higher than that of course. Like for a fair six-sided die, H is about 2.6 bits. The more events you plausibly could have, the higher H is going to be.

Here’s a related paper from last year that I really enjoyed. It’s from the Qwen team, and it looks at the relationship between RLVR and entropy.

If we think back to the elicitation hypothesis from last week, which says that RLVR just optimizes and brings forth good solutions rather than teaching new solutions, that should have implications for entropy. Like if RLVR is improving the odds of getting a good solution out of your model, it must be changing the distribution of token likelihoods at each step, probably making some of them much more likely at the expense of all the others. That should reduce entropy.

But as we established earlier, entropy is actually a good thing, something we want to preserve. The framing you sometimes hear is exploration vs exploitation - you need to be able to explore different areas of solution space as well as exploit the most promising areas. Higher entropy means more exploration. There’s such a thing as too high of course, but the concern with RLVR is getting entropy too low, basically cutting off all the exploring.

You can actually see this difference in the word clouds they assembled on the right. The high-entropy tokens are at the start of ideas, often new ones or representing changes in direction. The low-entropy ones, by contrast, are often part of calculation or totally deterministic, like a closing parenthesis that you know has to be there because you had an open parenthesis earlier.

The finding in this paper is that if you focus on the small share of high-entropy tokens for training and updates, you end up better off than if you made updates based on all the tokens. That’s not as relevant for us, but it does show there is value in preserving a certain level of entropy.

The Paper

So now that we’re up to speed on the value of diversity and entropy, we can dive in.

The basic idea of this paper is simple: your model stays more fox-like if you collect and emphasize less common solutions during training.

They give an example here and diagram it out. Let’s say you have to compute this series, summing the cubes of the integers from 1 to 100. This is RLVR, so we have a golden final answer, in this case it’s 25,502,500. What we want to train on, to reward, is the model getting that right final answer.

So we take our model, we have it generate a bunch of solutions, then we group those solutions together, using a more powerful LLM. That lets you see past mere changes in wording or variable names etc and just focus on the different strategies represented.

Then you assign advantages to each group. Here an advantage is a reward times a weight. The reward is going to be either 1 or 0 depending on correctness, classic RLVR. The weight, however, will depend inversely on the number of solutions in the group.

So as the diagram shows here, any group of wrong solutions is going to get zero advantage. A group of correct and common solutions is going to get a positive advantage, but on the low side. A group of correct and rare solutions will receive a high advantage, with the highest of all going to any “group” that only has one example.

That’s pretty much it! All you need for GRPO, for the algorithm that adjusts the model’s weights, is these advantages.

Here are a few examples of solution diversity. The have three problems, each with a varying number of solutions, plotted as shapes.

Now to measure coverage, they happened to have a substantial number of human examples to cluster, from web and textbook sources. They imply that having many human solutions means they likely covered all the possible strategies, but they weren’t rigorous about it imo.

Anyway, the result they’re showing here is how the initial model expands its coverage. So like in the middle example, there are apparently four different strategies, only one of which they saw in the initial model. But after some training, they saw two additional strategies, although they still didn’t observe one that humans got.

So does it work?

To answer that, they train three models with their technique: Qwen2.5, Qwen3, and Olmo-3, all in the same size range.

What they’re measuring here is AUC, short for “area under curve”. It’s a way to capture performance of pass@k for a series of k values as a single number, instead of having to report pass@k for each value of k. The higher the AUC, the more question you got right at lower values of k. And remember, the pass@k curve can never go down, because increasing k just means giving the model more chances to get something right. One consequence is that lower AUC numbers actually show differences better, since if you’re already doing well on pass@1, there’s not much room for improvement. So here for instance they drop AIME and Medicine after Qwen2.5 and focus on the more discriminating benchmarks, HLE and Physics.

Anyway, as you’d expect from a published paper, their method wins out. The baseline model, which they label Instruct, does the worst most of the time, although not always.

The other comparison they do is against what they call “SimpleRL”, which is your standard GRPO + RLVR, the foundation this paper builds on.

For Qwen3 you’ll notice two other methods, DAPO and Forking Token. DAPO is like a tweaked version of the GRPO algorithm that we don’t need to worry about. Forking Token is actually from the paper in the background slides, the ones about high-entropy tokens. I was pleased to see they liked that one as much as I did. Still, the simpler method of rewarding diverse strategies does best.

Of course we have to cover entropy too, since we know how valuable it is. Here they show the classic trend for standard GRPO + RLVR, where the entropy trends down as the accuracy trends up, regardless of the model.

For their method, by contrast, you preserve or maybe even increase entropy. So then you’re doing better exploitation while maintaining good exploration, exactly as desired.

Oh, and if you noticed the relative smoothness of the latter two graphs, I think that’s just from a change in plotting rather than a genuine difference between Qwen2.5 and the other two models.

My Takeaways

• Entropy is a first-class metric

  • Any training method that only reports accuracy (or pass@k for low k) is cheating

• Maybe we can use this for reasoning data

  • Their coverage evaluation used human-generated strategies as the ground truth

  • We could verify all the clustered strategies AND try to come up with alternate strategies not represented in the clusters

• Could you be more purposeful in exploration?

  • Example: ask SOTA for diverse strategies and then only grant max advantage if you display all of them

Paper · Blog post · Model

Background

I want to start our review today with some history. Specifically, the history of NVIDIA in the language modeling space.

Now NVIDIA chips, and libraries for using those chips, have been part of machine learning for a long time. Most of the breakthrough moments in what people used to call “deep learning”, like AlexNet or GPT-1, have used the NVIDIA stack.

But it wasn’t until September 2019 - over a year after the release of GPT-1 - that NVIDIA started putting in some language modeling efforts of their own. Specifically, I’m referring to this paper on the left: Megatron-LM, which is all about training large language models across many GPUs.

That sounds pedestrian today, with data centers containing thousands of GPUs all running in parallel to train the next SOTA model, but in 2019 it was a real pain point. For context, GPT-1 was 117M parameters and trained across just 8 GPUs. GPT-2, which was training when this paper came out, went up to 1.5B parameters and trained across 32 TPUs. Those are Google’s special ML chips, so not actually GPUs.

By contrast, the Megatron-LM paper trained a model with 8.3B parameters, which is a model size many researchers still use today, across 512 GPUs. The model itself didn’t make much of a splash, but the engineering did; the ideas and tools from Megatron-LM live on today - including in the original GitHub repo, shown on the right, which is still in active development.

Things scale up rapidly from there. GPT-3 comes out in 2020, rocking 175B parameters, having trained on 10k NVIDIA V100 GPUs. Then ChatGPT comes out in November 2022 and kicks off the generative AI explosion.

While OpenAI by this point has become closed, enough knowledge has gotten out that other folks are able to train large language models. The big name in the open-weights space, LLaMA, comes out in February 2023, with sizes from 7B to 65B parameters. Llama 2 quickly follows in July 2023.

Among the many other companies releasing open-weights models is NVIDIA, with Nemotron 3 8B in November 2023.

Now I know what you’re thinking: is this Nemotron 3 connected to Nemotron 3 Ultra, the model we’re discussing today?

Strangely, the answer is no. Other than both being language models from NVIDIA, the Nemotron of 2023 is not related to the Nemotron of today. The original line extended to Nemotron 4 in 2024, but then they rebooted the franchise so to speak, and released a new Nemotron 2 in August 2025, which we covered in a previous Friendly Paper Review.

I include all this confusing nomenclature because I find it emblematic of NVIDIA’s model efforts on the whole, and of my experience reading the Nemotron 3 Ultra paper in particular. Quite the opposite of Microsoft AI’s approach, which we covered last week and which I found refreshingly clear and principled.

So we know NVIDIA has been working on models for some time now. The next question is, why?

The answer boils down to one business principle: commoditize your complement.

Let me explain with reference to Windows and the PC market, which is the classic tale of commoditizing your complement. Once upon a time, PCs were differentiated - different hardware, different drivers, different standards. That differentiation is how PC makers stood out with their customers, but it was hell for operating system makers like Microsoft. If you’re Microsoft, of course you want all PCs to be interchangeable so that you can easily run on all of them. But more importantly, you want all PCs to be interchangeable so that manufacturers have to compete primarily on price rather than on features - that’s the hallmark of commodities markets. That price competition on hardware side then drives down the price of PCs overall but leaves the price on the software side untouched. The lower the cost for a PC, the more people can buy them, and the bigger the market for Windows is, at the same price.

Now let’s apply that wisdom today, but with the roles reversed: NVIDIA is the hardware maker, and the big model makers like OpenAI and Anthropic are the software folks. They are battling over who will extract the value in their chain.

Obviously they all want the demand for inference to go up, and that tide is really lifting all boats right now. But no budget is unlimited, and for a fixed amount of money there has to be a split between hardware and software. Right now OpenAI and Anthropic in particular collect fat margins on their tokens, competing on quality or perhaps some differentiated product experiences, which means fewer tokens for a given budget. If NVIDIA can somehow commoditize the token, making providers compete on price, then demand for cheap tokens is gonna go up - there’s gonna be a lot more inference. And that new inference is mostly gonna run on new NVIDIA chips.

Microsoft did it by playing PC makers off each other. NVIDIA is doing it by introducing its own competitor in Nemotron - in addition to funding lots of other model makers.

Now transitioning from business to technology, one thing to point out that blends both is how AI workloads are evolving.

As this fun graphic from the recent Anthropic report on recursive self-improvement shows, the paradigm has shifted from chatbots to agents to swarms of agents. And because of how agents work - looping through thoughts, actions, and observations indefinitely until the task is complete - they are token-intensive. Back in the days of the original ChatGPT, inference was too expensive to do this kind of thing, not to mention the severe constraints on context window size and the generally insufficient intelligence.

So the dropping cost of tokens encourages more token use overall - a more general economic phenomenon known as the Jevons paradox - and unblocks the switch from chatbots to agents. It’s also very Bitter Lesson, throwing scaled-up resources at a problem instead of eking out wins with human effort; a chatbot assisting a human in a task is more efficient in tokens but ultimately unscalable compared to a swarm of agents working autonomously in parallel to complete that same task.

All that is to say: NVIDIA is naturally going to focus on agents, especially the always-on ones like OpenClaw, which are going to consume even more tokens because of all the work they do in the background - you don’t even need a human there to push up token demand!

Also, NVIDIA is going to satisfice on quality while optimizing for throughput and cost. They are not racing to achieve ASI; they want their good-enough models deployed as fast, cheap agents at every enterprise in the world.

Completing our transition into technical topics, I want to discuss a major shift in language model training to parallelize yet another aspect of the process.

The technique in question is “Multi-teacher On-Policy Distillation”, or MOPD. Like any technology, it has a long lineage, but the citations for it typically go to the tech report for MiMo-V2-Flash, an open-weights model by Xiaomi.

This graphic from that paper breaks it down: after establishing a shared starting point, you train many specialist models, then use them as teachers to distill into one shared student - typically the same model you used as a starting point. So instead of taking your one model and training it on search, and then code, and then math etc etc, you do all that training in parallel.

Training in parallel gets you two benefits. One, just in general, parallelizing work makes it take less wall clock time. Like it takes the same amount of computer cycles, but for you as the researcher it only takes as long as the longest individual teacher. So that could be significant.

And two, you decouple the training of each teacher, so you can train in the way that is best for each domain. So if one is very SFT-heavy but another uses mostly RL, or even just different hyperparameters - basically the settings during training - you have that freedom. There’s no compromise, no sequential dependence, all starting from the same clean slate in the form of the SFT model.

Now I don’t want to worry about the exact details of the method here, because the Nemotron folks made some adjustments of their own, but the broad idea and the specific MOPD term comes from here.

And in a final instance of parallelism, the MAI-Thinking-1 paper from last week also used multiple teachers, although it’s not on-policy distillation; they choose to make SFT data from the teachers and then train the student on that, rather than directly comparing next token predictions given the same inputs, which is how on-policy distillation works.

The Paper

A quick note: given the volume of content and the interests of my audience, I glossed over most of the infra and training details. There is a lot of good detail in the paper, I recommend reviewing it if those are your interests.

As usual for new model releases, we’re going to start off with architecture.

Nemotron 3 Ultra is a big model. It’s not as big as the biggest open-weights options, like DeepSeek V4 Pro at 1.6T parameters or Kimi K2.6 at 1T parameters, but it’s up there.

Notably, it has more active parameters than either of those models: 55B here, compared to 49B on DeepSeek and 32B on Kimi. That’s due primarily to the number of activated experts in the MoE layer, which is two or three times higher for Nemotron than for a typical model of this size. The more experts you activate, the more parameters you activate.

The other factor is the sheer number of layers, which is higher than any of the other big open-weights models. And here I want to split out the impact of layer count on training vs on inference.

On training, the number of layers is mostly a function of your available compute. Adding layers provides diminishing returns, but it still adds returns, so if you can afford the compute you might as well go for it. And NVIDIA can certainly afford the compute.

On inference though, each layer is going to increase latency - there’s just more stuff to do for each token - and each attention layer is going to add to the KV cache, the “state of mind” of the model. NVIDIA combats this by mostly using Mamba instead of attention. See, whereas attention’s demand on memory grows as the square of the input length, Mamba is a different architecture that uses a flat amount of memory, regardless of the input length. It’s kind of like having a fixed-length summary instead of just adding to your notes.

They do add the occasional attention layer in, which is similar to how many models nowadays interleave local and global attention, but mostly it’s Mamba.

Now let’s talk about how they pretrain that model they’ve laid out.

I want to prime you with a reminder of what the Microsoft AI team did with their model, which was to use no synthetic pretraining data. We’re going to see a very different approach here, which is not a surprise when you think about it; NVIDIA has the GPUs, so they can make all the synthetic data they want. Of course at Scale we’re going to bring a skeptical eye to that synthetic data, and we know the Microsoft AI folks would agree.

They have plenty of natural data too, from the web and GitHub and the like, but they spend much more time in the paper describing how they make synthetic data than how they find or clean natural data. Again, a stark contrast to Microsoft’s approach.

Because there is so much detail in fact, I can only provide a summary of their synthetic pretraining data efforts. They include:

• Generating question-answer pairs based on the training sets of many public benchmarks

• Using the domain and difficulty distributions from those benchmarks to inspire even more QA pairs

• Extracting facts from a Wikipedia dataset and turning them into QA pairs

• Making chains of thought about moral scenarios

• Pulling legal codes and case law to then summarize

• Synthesizing random character profiles from a specialized model, Nemotron Persona, and inserting them into legal cases

Most of that is available for download by the way - another service NVIDIA has done for the research community.

Anyway, they end up pretraining on 20T tokens, shifting from diversity to quality between the two phases pictured here. So for example, “finepdfs-unfiltered” is in phase 1, but only the “medium” and “high” filtered splits are in phase 2. And just in general for all LLM training, you always want to increase in quality as you progress in training. In their case, they do that first quality ratchet after about 15T tokens of pretraining.

Here’s where the base model nets out, before post-training.

Competition in the base model space is a bit thin actually; it’s become less and less common to release one, partly for competitive reasons I think but also because base models have little or no safety training. So at minimum, the NVIDIA folks have done the research community a service.

However, I do take issue with this slide, because they left off their stiffest competition: DeepSeek V4 Pro, which we’ll see later when they compare post-trained models. Not sure why they did it, but it makes the results here much less meaningful.

Now that we have our base model, we can focus on post-training.

Our old friends SFT and RLVR are in here of course, but the fancy new thing is that multi-teacher on-policy distillation we touched on at the top.

The other unusual thing I’ll note here now but not dwell on later is MTP Boosting. MTP stands for “multi-token prediction”, and it does what the name implies - predicts multiple tokens at once instead of just one. That’s baked in from the start in the architecture and is part of pretraining, but it needs a little post-training of its own close to the end apparently. Don’t worry if the jargon in the yellow box escapes you.

Calling back explicitly to the Microsoft AI paper, the setup here is more complicated and less principled. Like they’re clearly building upon previous experiences rather than starting with a clean slate like the Microsoft folks did. Not better or worse necessarily, just a very different flavor in reading this paper. And of course the proof is in the pudding, but sadly the Microsoft AI model is not publicly available yet.

Anyway, on to supervised finetuning. I’m putting up all these other AI company logos because the researchers really went ham on synthetic SFT data, in terms of the domains and the content and so on but also in terms of the models generating that data.

Here is the entire list:

• gpt-oss-120b

• DeepSeek V3

• DeepSeek V3.2

• DeepSeek V3.2-Speciale

• DeepSeek V4 Pro

• Qwen3-30B

• Qwen3-235B Instruct

• Qwen3-235B Thinking

• Qwen3-Coder-480B

• MiniMax M2.1

• MiniMax M2.5

• GLM 5

• GLM 5.1

That is a ton of models! And notably all open weights models, surely self-hosted. The implication is that diverse training data produces more robust models, which is accurate, but they never give an explicit rationale. I did think it was a little odd they purposely used some much older models though, like in practice there’s no reason to use MiniMax M2.1 if you have MiniMax M2.5 on hand, but the principle is right.

This is in addition to models for filtering or judging data by the way. They use some models here for those purposes and also some internal models built for purpose.

As for the data itself, there’s a lot of it, and the researchers provide a lot of detail - albeit with many gaps in information.

The vast majority of the data is either pre-existing, taken from the training split of benchmarks they don’t plan to eval on, or synthesized. It’s a pretty incredible variety of data too. Like they have data for CUDA, their software stack for controlling their GPUs, as well as RTL, which is a hardware design language. Unique to NVIDIA from what I’ve seen but it makes complete sense.

By complete contrast, there are only two short paragraphs about their RLVR stage. They list the domains they target, like terminal use and instruction following and white collar workflows, but no specifics about the data. They don’t even specify which harnesses they use, just that they use many. It’s a bit baffling given the copious detail of the SFT section, although we’ll see more detail on the teacher-specific RL down the road.

Anyway, with RLVR done, we now have a student ready to transform into all the teachers. Although actually we have two students apparently, that Agentic SFT/RL box towards the bottom of the Prep column, and again that’s not really explained either. At minimum it’s from the same base model though.

So with our chatbot student and our agentic student, we’re ready to train some teachers. Each one gets its own particular training recipe, which is one of the virtues of MOPD, but it also makes for a nightmarish amount of content. So in the interest of time let me just note a couple commonalities.

One is the general use of RL over SFT. In particular, many of the training recipes use a technique from another recent NVIDIA paper called PivotRL, which takes SFT data and finds the most consequential steps - the pivots - and does RL from those points forward. It’s a nice way to squeeze more juice out of SFT data, basically turning one example into multiple examples.

Two is, again, the prevalence of synthetic data. They even have some tool called NeMo Data Designer they used for at least the Usability teacher. However, they do mention for the Office Work teacher that they bought data from a vendor called AfterQuery to help them on GDPval.

There are basically no numbers on their teacher training data, except for two places: 3.5k samples for Coding teacher, and 40B tokens for STEM teacher.

Now the distillation from teacher to student isn’t perfect, partly because the student’s weights have to optimize all these different skills at once, which inevitably leads to compromise - just like how no human can be an expert in everything.

So to quantify that compromise, they measure the pre-MOPD student, the student after both rounds of MOPD, and the teacher. The recovery is the improvement of the student divided by the improvement of the teacher. So in the first row for example, MOPD2 minus RLVR is 9.5, and Teacher minus RLVR is 5.5, and 9.5/5.5 is about 173% - the student exceeded the teacher.

Most cases weren’t like that of course, but generally the student recovered most of the teacher’s performance, although in a couple cases recovery was quite poor.

They hypothesize that distillation works best when the student reasonably could have made the correct choice, and works poorly when the right choice was not at all a possibility. Like if the student just uses suboptimal terms for a web search tool call for example, it probably has the right search terms somewhere in its distribution of likely next tokens. But if the student is doing a hard math problem, there’s really no guarantee at all that the right next token will have any likelihood - like how I would have no shot of guessing the next step of an algebraic topology problem.

To be clear, that is speculation and maybe they just have a skill issue. But it seems plausible to me.

So after all that post-training, including a little polish we again had to cut for time, this is where we land across their chosen benchmarks.

I have a few comments to make. First is the key: I highlighted the winner of each benchmark, with a 1% band below the top score to allow for ties due to noise. Like there is no practical difference between an 82% and an 81.7% score on IFBench or whatever.

Second is the selection of models: it’s all the best open-weights models, no comparison at all to even the closed non-SOTA models, like Qwen3.7 or the latest Grok. Even here though, the range of total and active parameters is wide, with MiniMax M2.7 and DeepSeek V4 Flash on the small side on both. On the big side, for total parameters it’s Kimi K2.6 and DeepSeek V4 Pro, but on active parameters it’s actually Nemotron 3 Ultra at the top! As I mentioned on the architecture slide, it’s due to all those activated experts and all those layers.

Third is, unfortunately, the relative underperformance of Nemotron in my view. Like despite being the newest and also having the most active parameters, it only wins outright on one benchmark, and it’s one I’ve never heard of, with suspiciously bad performance for Qwen and both DeepSeeks. And one of the three it ties on, the Scale benchmark Multi-Challenge, is a tie between basically every model except MiniMax. Worse yet, Qwen is smaller on both dimensions, and is four months older, yet it places first in twice as many benchmarks.

Remember what I said about NVIDIA though: they’re satisficing on quality and spiking on throughput and cost.

That’s the story we see here. At the top in the key you’ll see two versions of Nemotron: the full-precision one, which stores its numbers in the format BF16; and the quantized one, which stores its numbers in the format NVFP4. The “NV” is short for NVIDIA, it’s a special format designed to work well with their GPUs, preserving most or all of the quality yet taking up only a fraction of the space.

NVIDIA actually trains the NVFP4 version natively, basically meaning that quantizing down from BF16 to NVFP4 remains high-fidelity. And they do so because it speeds up the model massively, particularly on decode, i.e. when you’re generating tokens. It helps with prefill too - that’s the part where you’re reading all the input - but for agents with these long trajectories, decode speed is crucial.

The relative comparisons on the right are a bit misleading, since all the models on the chart are in NVFP4, but it is true that NVIDIA engineered the hell outta their serving infrastructure to get that speedup.

My Takeaways

• NVIDIA is the American open-weights champ

  • Nemotron 3 Ultra is their best model yet

  • I don’t know why it took them so long to embrace this position and put serious resources into near-SOTA models

• NVIDIA is also a gift to the research community

  • Base models, open pre-training and post-training data

  • Not to mention the hardware, software, and work in other domains (e.g. robotics)

• They rely too much on synthetic data

  • MAI-Thinking-1 is the perfect counterpoint here

  • Some Scale post-training data could help here 😉

• MOPD is the new normal

  • Still evolving somewhat but the high-level idea is firm

Paper · Website

Background

I think we have to start with, what are world models?

We actually covered this in a paper last year, where I explained that world models are not necessarily models of The World, like the physical world around us, although in this paper we actually are talking about that kind.

At its most abstract, the concept of a world model is about whether an ML model can form a coherent and predictive view of a given environment based on limited information.

That may sound like a requirement for making good predictions, which we know models can do, but it’s actually not. Let me give you an example.

Let’s say you’re flipping a coin and you don’t already know that coin flips are inherently 50-50. One way you could discover they’re 50-50 is to flip the coin a bunch of times and extrapolate from that data to predict future data. You don’t need to understand the nature of coins or the laws of physics to do that extrapolation, you just need basic data analysis.

Of course the way a human would conclude coin flips are 50-50 is to look at the coin, maybe toss it a few times to make sure it’s not weighted, and then reason a bit or just intuit that the coin toss will be 50-50. That does require a world model, because it requires underlying assumptions about how objects behave in the physical world.

Another example where you hear both the statistical model and the world model-ish views of a thing is in the stock market. Some people just look at trends, with no concept or view of the underlying company, the ticker symbol could mean anything. A lot of quant trading is like this, where you just feed a bunch of factors into a mathematical model and get a “buy” or “sell” recommendation out. On the other end of the spectrum are the fundamentals investors, who look at the company’s financial statements and the market and the strategy etc and form a model of the company’s future performance.

So a world model is just about where the reasoning seems to be, not about one particular place or another.

Here’s another example of a world model, this time in the world of code.

So normal code data for us would be a prompt and a response, if it’s SFT, or a prompt and a set of unit tests that verify whether the model’s answer works, if it’s RLVR. But in either case we’re training the model directly for writing code.

By contrast, this coding world model has a different goal: to predict how a piece of code will work. If the model can do that, then it probably has a good world model for the world of code, its “laws of physics” so to speak.

The way they train that in is to provide a piece of code and an example of using that code, then ask the model to predict the state and action at each step of the program.

Here the state is in yellow and the action is in blue. As you go down the rows, or “frames” as the paper calls them, you can see what the model is keeping track of and how it changes after each action. Like for example in the second frame, we get a new variable n, with the starting value 0. Then in the third frame we see n being tracked in state, along with its current value, 0. The two dots in quotes is just a visual shorthand that means the value of the variable hasn’t changed, so like s is “strawberry” for the first frame and then just the two dots for the rest of the frames.

Of course in the end what we care about is whether it can produce good code, like with any coding mode. But this idea of training or testing the world model, separate from the end result, will be relevant later on.

Now let’s look at some video generation models, what laymen might think of when we say “world model”.

This is a clip from Sora, OpenAI’s video generation model, when it first came out back in February 2024. I remember being very impressed when it came out, and honestly it still holds up at least in this clip, but you can see some weirdness happening. Take a look at the manhole for instance and how it kinda morphs into pavement, that’s a classic AI video-ism. Consistency and memory are tough for video gen models.

One reason it’s generally so accurate though is because of what they trained on: synthetic videos, generated by engines like Unreal Engine and Unity, which have programmed in very detailed rules of physics. If you train on tons of footage from physics engines, you’re going to end up with a pretty good sense of physics. OpenAI actually views Sora as a “general-purpose simulator of the physical world”. So they’re going for a world model here.

The latest and greatest in this space is Genie 3, which generates good videos but also allows real-time exploration. Look at the legend in the bottom-left, showing arrow key presses. Genie 3 generates the world in real time, lets you navigate it, and it remembers what has been in the scene before but is out of the scene now, to maintain consistency. Again, you need a really keen idea of the world to do something like this. So the question is, how do we put it to use in the actual physical world?

Now getting more into the robotics side, let’s talk a brief bit about inverse dynamics models, or IDMs. It’s a complicated name, but really the idea is simple: instead of a regular dynamics model, where you infer outcomes given actions, you invert it, inferring actions given outcomes.

OpenAI actually pioneered this concept in the modern AI age. This graphic is from a 2022 paper called Video Pretraining: Learning to Act by Watching Unlabeled Online Videos. Their goal was to teach a model to play Minecraft. To do that, they figured the best way was to take advantage of the many many hours of Minecraft gameplay video available out on the internet - some 70k hours apparently, after significant filtering. But how to teach the model with videos yet no key presses or mouse clicks? The model wouldn’t learn how to actually play the game!

To make their 70k hour corpus useful, they synthetically added actions in. They collected 2k hours of labeled video, i.e. with the actions recorded, and then trained an IDM to predict actions based on video. Once they had that, they put these “pseudo-labels” as they’re sometimes called on all the initial videos, then trained their Minecraft agent on the pseudo-labeled corpus.

Ultimately though they’re learning actions from videos, without any sort of world model; the IDM doesn’t demonstrably know what the key presses and mouse clicks mean, just what they correlate to.

We should also discuss briefly how a modern robotics model looks. For that we’re going to turn to the Pi series of models from Physical Intelligence, which is roughly the OpenAI equivalent of robotics model makers.

The Pi models are VLAs, vision-language-action models. That’s an LLM at the center, with a vision encoder added on the front so it can see, and an action expert added on the end so the model can do stuff - the action expert outputs signals for the motors and such. In the case of Pi 0.5 the LLM is Gemma 2 2.6B, made by Google. The vision encoder is SigLIP, also made by Google, so the vision-language model so far is 3B parameters. Then you add on the action decoder for another 300M parameters, so 3.3B parameters total. It’s a pretty small model, but it really has to be in order to produce actions in real-time. Gotta keep that latency down, no way to do that with the parameter counts of SOTA LLMs, which are in the tens or hundreds of billions, maybe even trillions. By the way, this is the same reason that people speculate Sora and Genie are also in the single billions of parameters. Kinda crazy that all the knowledge to model a world can fit in that few parameters.

So one of the major contributions of Pi 0.5 was this concept of “co-training”, i.e. training the VLM part on a bunch of multimodal web data. If you’re not familiar with this work already it may be somewhat surprising that a robot could learn to manipulate the world better by just looking at images or watching videos, but of course we know humans can do the same - we can learn how to do things just by watching. Hands-on practice is often better but it’s not always required.

They also throw in training data from other robots, like other embodiments that this model doesn’t get used on.

Anyway, the main thing to know here for our purposes is that VLAs are the currently dominant paradigm in robotics models, and they start from LLMs.

The Paper

First let’s talk hardware. This is the star of our show, AgiBot G1. This model has grippers, although they make a version with dexterous hands. The grippers can rotate. Each arm has two “elbows” so to speak. The torso can move up and down on the base, and the base can wheel around. Finally, there’s one camera watching each gripper, and one behind the faceplate watching the whole scene.

This is the robot they use for gathering data and for doing most of the evals. They do use a couple other robots, specifically a Franka Emika Panda and a YAM, but if you’re going to imagine one robot for these results I would pick this one.

Now let’s cover the model.

The big difference between normal transformers like your typical LLM and robotics models of any sort is that robotics models have to predict things in parallel, because robots have different parts that can move simultaneously. You don’t have to do that with text, you can generate one word at a time. But you can’t trade off which limb moves at a time or something like that, at least if you want to produce smooth motion.

Image and video generation models, on the other hand, do predict things in parallel, although that’s more because of the inherently parallel nature of vision than necessarily having to produce many pixels at once. Still, you do have a body of image generation research to draw on.

So the architecture here combines the transformer, which predicts in serial and is good for language, with diffusion, which predicts in parallel and is good for image. That’s the diffusion transformer, DiT. And the way it works is by predicting small blocks of actions where stuff can happen in parallel, but predicting each block in series, so only one block at a time. Specifically, they predict 1.6 seconds of action per block. That’s how far ahead the model is “thinking” or “envisioning”. Predicting many actions in a block like that results in smoother motion compared to predicting just one action at a time.

But it doesn’t blindly act in 1.6s increments and wait until the next increment to make adjustments; just like how your body frequently adjusts balance or grip while carrying your dinner across the kitchen, the model makes updated predictions on a similarly short timescale. With this model, this hardware, and a suite of optimizations we don’t need to get into, they can make new predictions every 150 ms. So if the researchers swap out an object in the scene, or the robot’s grip slips, or the wind blows a cup over, the robot can react in a reasonable amount of time. So you never get to the end of that 1.6s block, you always have a new block ready to start acting on.

Now the other thing they mention here that I want to dig into is this word “joint”, as in “joint video-action flow matching” and such. What they’re saying there is that given some inputs, they want to predict video and action simultaneously, NOT predict video and then predict action based on that predicted video. There are a couple reasons for this.

The first is that with two simultaneous predictions, errors in one are unlikely to appear in the same way as errors in the other. So like if my video part incorrectly predicts the plate I’m carrying starts to tilt, but my action part has predicted no change in my hands, that disagreement gets picked up as loss and thus is targeted in future training steps. But if I predicted the video and then predicted the action on top of the video, then my action part is basically stuck predicting something wrong in order to agree with the video prediction and thus minimize loss.

The second is that joint prediction shares the world model knowledge with video and action prediction. That’s the main theoretical thrust of the paper really, that there’s this world model implicit in a video generation model that gets locked away when you try to learn actions from video instead of learning actions alongside video. Like you don’t want the video to be an intermediary between the implicit world model and the action predictions, you want to go straight to the source.

Here’s a quick example of some generated video. We don’t get a side-by-side of generated vs actual sadly, but you can see they line up generated video with where the real action happened. In the Generated row, the top-left square is the faceplate view, the top-right square is the right arm, and the bottom-left square is the left arm. You can see the robot uses the left arm to get a second view of the scene, which is kinda cheating from a human point of view.

OK, on to the data. They need this robotics data to transform their starting point, the open-weights video generation model Wan 14B from Alibaba, into a world action model, a WAM.

The main body of data they collect is here, ~500 hours of teleoperation data on their AgiBot G1 across 7.2k episodes and 22 different environments. Since one key advantage of WAMs is that they rely on their world models to guide actions, they don’t need tons of demonstrations per task to then imitate. That allowed the researchers to focus on diversity instead of repetition with their time budget.

The average episode is about 4.4 minutes and has over 40 steps, what they call “subtasks”, which is relatively long-horizon for this type of thing. So with this pretraining data, we can turn Wan 14B into DreamZero, which can predict video and actions.

So with their 500 hours of pretraining data in hand, they can train some models and start comparing results.

Let’s cover the models and terminology first. On the bottom you’ll see three models mentioned: GR00T, a VLA also made by NVIDIA; Pi 0.5, a VLA by Physical Intelligence that we’ve covered before and that we saw the architecture of earlier; and DreamZero.

They also give two different descriptors in parentheses: scratch and pretrained. Scratch means no robotics data other than the ~500 hours these researchers collected, to give a fair comparison between DreamZero and the VLAs. Pretrained means it does have other robotics data in it, so that’s like the full versions of GR00T and Pi 0.5 against the full version of DreamZero.

Finally, for the tasks, PnP means “pick and place”, things like putting fruit in a bowl; and Contact-Rich means folding clothes basically, which is super common in robot evals and also is one of the most common use cases for robots in the real world actually, like to a surprising degree.

In any case, DreamZero outperforms the other two on all tasks and both embodiments. It seems like that world model really does confer a lot of physical common sense and allow the robot to handle a wider variety of cases.

Now for the previous chart, those were all seen tasks, tasks present in the training data, although the evals were in new environments and with different objects.

In this chart, the tasks are all unseen, not present in the training data. Again we see the same general trends, although Pi 0.5 does come close to DreamZero in a couple cases. To be fair to the Physical Intelligence folks, 0.5 is not the latest version of their model, but it is the most recent open-weights one so it’s a fair base of comparison.

One fun note from the authors: apparently the VLAs often try to grab objects regardless of the prompt given, suggesting they’re overfit to those types of tasks. Makes sense given the most common tasks are versions of pick and place, I like the variety of actions they represented here.

So the results we’ve been looking at so far are for models that are pretrained only, i.e. no task-specific post-training on demonstration data. Here they change that.

The post-training data is for three tasks: shirt folding, collecting 33 hrs; fruit packing, collecting 12 hrs; and table bussing, collecting 40 hrs. In all cases they use a variety of objects, different counts and positions, etc.

Here we see near-parity between Pi 0.5 and DreamZero. Unfortunately we don’t have pretrained vs post-trained results to understand the impact of post-training per se, but my interpretation is that WAMs can learn from demonstrations just as well as VLAs can. Like sure, WAMs seem to do better right out of the box, but you could have objected and said we don’t care about out of the box performance, we care about absolute performance, and maybe VLAs have a higher ceiling because they learn so well from demonstrations, i.e. from post-training data. But that doesn’t appear to be the case; the ceiling for a WAM seems just as high as the ceiling for a VLA.

The bull case here might be that WAMs don’t even need post-training data or might only need a few human egocentric examples rather than teleop data, but again, they don’t show these evals before post-training.

Finally, to drive home the point about most work happening within the world model rather than within the robot-specific parts, they collect two small sets of data for a set of new tasks:

• 12 minutes of human egocentric video

• 20 minutes of video from YAM, another robot. No actions collected, just video

Then they train on them and run the evals on three versions of DreamZero: the initial version, a version trained with the human data, and a version trained with the YAM data. Then they see how well DreamZero learned to do these new tasks from the human video and the robot video.

What they found was that both transferred about equally well, meaning you don’t need expensive robot hardware to make good data. All you need is a human with head- and wrist-mounted cameras who’s willing to visit a lot of new environments.

Now running in the opposite direction, they took 30 minutes of video from YAM on a different set of tasks, unrelated to any evals, and trained their model on that. And suddenly the model was able to operate the YAM, even though its pretraining data was only on AgiBot. So in both directions the embodiment seems to matter only a little, and the bigger factor by far is whether anything in the pretraining corpus looks like the task at hand.

The authors give an explanation that I want to quote in full to round out our review:

“Learning an implicit IDM from predicted videos may be inherently more sample-efficient than direct policy learning - the model only needs to learn the mapping from visual features to actions, while leveraging the pretrained video model’s existing understanding of physical dynamics. Consistent with our AgiBot findings, failures primarily stem from video prediction errors rather than action extraction, suggesting that increasing task diversity during post-training could further improve performance.”

My Takeaways

• We might be able to ditch the robots, at least for pretraining data

  • For evals you always want to test in real conditions, i.e. with the robot

• Post-training for improving video generation will be valuable

  • RLHF

  • Rubrics? Seems challenging but potentially worth it

• This could be the start of a shift away from VLAs to WAMs

  • Even at similar quality, the data for WAMs is more scalable

Paper · Blog post

Background

So before this announcement, I think most people weren’t aware Microsoft worked on any models at all. Certainly they use models, like in their various Copilot products, but that’s pretty much all models from other providers.

But actually, one arm of Microsoft has been releasing models for some time now. That arm is Microsoft Research, and they have a good reputation for their Phi series of models in particular. The Phi Family as the image calls them is a wide array of research-size models, runnable and trainable on relatively modest hardware, and covering a lot of different areas. The Phi models aren’t great generalists, but they have historically done well on code and STEM work.

Microsoft Research has a lot of other models on their Hugging Face page, and they release a decent number of papers as well, but at least in my mind, I have historically considered “Microsoft model” and “Phi” synonymous.

But this paper and this release are from a different arm: Microsoft AI. And to talk about Microsoft AI, we have to talk about this man: Mustafa Suleyman.

Mustafa Suleyman is a giant of the field. In 2010 he and Demis Hassabis cofounded DeepMind, which Google acquired in 2014. DeepMind is the original seat of Google’s own generative AI efforts, so already in his bio we can credit him as the father, or perhaps the grandfather, of Gemini.

Fast forward to early 2022. Suleyman leaves Google, joins a VC firm, then quickly leaves to cofound a new lab with Reid Hoffman called Inflection AI. Keep in mind, this is before ChatGPT came out in late 2022, but well after GPT-3 came out, in mid 2020. So folks in the know are seeing the pace of AI progress is starting to really pick up - inflecting, if you will.

In 2023, Inflection launched its ChatGPT competitor, a chatbot named Pi, powered by the Inflection series of models. Inflection-2 came out in November 2023, Inflection-2.5 came out in March 2024, but by then it was already clear OpenAI was well ahead on both capabilities and chatbot market share.

So on March 19th, 2024, their lead investor bailed them out: Microsoft hired Suleyman and most of the 70-person team to become Microsoft AI. They also paid a huge “non-exclusive license” fee. So this was one of the first of what I like to call “fake acquisitions”: a bigger company scooping out their target talent from a smaller company, delivering a big payout, and leaving the remains of the smaller company alone. This is the same playbook Amazon used on Adept and that Google used on both Character AI and Windsurf.

It does share some characteristics with the investment Meta made in Scale in 2025, with Alex and a few other folks leaving to form Meta Superintelligence Labs, but the other transactions were “licensing fees” with no economic upside for the bigger company. Meta actually has a stake in Scale’s success, and Alex is on the board. Obviously the pattern-matching led to some rough vibes at the time of the transaction, but Scale is very much still a going concern, while the other companies I mentioned are shells of their former selves.

Anyway, now two years later Suleyman and his Inflection crew, with a lot of other talent on board, have finally released their first models.

The parallel thread here, and the subtext to a lot of this, is the relationship between Microsoft and OpenAI.

Microsoft first invested in OpenAI all the way back in 2019, before the release of GPT-3, as the exclusive cloud provider for OpenAI. At the time, Microsoft execs felt they had fallen behind Google on machine learning, given the latter’s high-profile acquisition of DeepMind. So as part of the investment, Microsoft got a license to anything OpenAI built. It also gave Microsoft 75% of OpenAI’s profits, up to the amount they originally invested. Pretty wild deal in retrospect huh?

So Microsoft continued deepening the partnership, with more big investments in 2021 and 2023, and deployment of GPT-4 across tons of Microsoft product surfaces. And with Microsoft as the exclusive provider of compute, any increasing demand for GPT meant increasing demand for Azure, and thus increasing profit for Microsoft.

Then in November 2023 came the near-acquisition moment: the OpenAI board fired Sam Altman on a Friday, over the weekend Microsoft offered to hire Sam and whatever staff wanted to move over from OpenAI, but by Monday Sam was back in the driver’s seat. As we know now, that left the door open for Suleyman at the future Microsoft AI.

Since then, much of the partnership has come undone. Microsoft is no longer the exclusive provider of compute. Their ownership stake is down, although owning roughly a quarter of a $1T company ain’t bad. And Microsoft’s IP and royalties arrangement with OpenAI is looser than before. Now with the release of the MAI models, it’s almost a frienemy relationship. And I’m sure Suleyman would love to best GPT one day.

Now briefly on the science side, I do want to cover one thing, which is loss.

Loss is the way you quantify how good your model is in many stages of training. It’s a single number that you set out to optimize, to get as low as possible via training.

For this paper, the form of loss we need to discuss is negative log likelihood, or NLL. NLL is what we use during pretraining and during supervised fine tuning (SFT).

Conceptually, NLL looks at the likelihood your model gave for the actual next token in your training data. If your model predicted the correct next token with 100% probability, the loss is 0. If it predicted the correct next token with 0% probability, the loss approaches infinity.

NLL is a nice way to measure performance because it is fast and objective compared to more intuitive forms of benchmarking, like human evals or leaderboards. It is also continuous between pretraining and SFT, so you can compare apples to apples. It also works for any text you think is trustworthy: just feed in some and see if the next predicted word is what your text says is next. That’s quite different from the training and eval data we make here, where we spend a lot of time ensuring quality and diversity and fit to spec and all that. Of course there are advantages to the other approaches and the other data like we make, and we’ll see MAI use them.

So for all these reasons, NLL is the consistent ruler we will see the MAI folks use to measure performance. Although occasionally we will see its cousin, “bits per byte” (BPB), which accounts for the differences in tokenizers.

The Paper

Before we start, just note that I had to skip a lot of things in the interest of time, particularly the serving and training infrastructure. If you want that detail, it’s all in the paper - they were quite thorough.

Let’s start with architecture.

MAI-Base-1 is a 1T parameter model, with 35B parameters active, across 78 layers and with 512 total experts, 8 of which are active at a time, for a sparsity ratio of 64. That is all very similar to many other modern models where we have architecture details. For example, Kimi K2.6 is 1T total parameters and 32B active parameters. DeepSeek V4 Pro is larger but in the same ballpark. So far so good.

Where it starts to go off the beaten path is in the details of the layers.

So for attention, which is where the model forms a holistic picture of the input, they switch between local and global attention, in a 5:1 ratio. So that means for 5 attention layers, the attention is only looking 512 tokens back, and then for the sixth layer it looks all the way back to the first token, assuming we’re within the model’s 256k context window. Switching between local and global is common, but 5:1 is on the more extreme end.

The weirder thing, which I have never seen before, is switching between dense and mixture-of-experts feed-forwards - the layer where all the processing or “thinking” happens. They’re switching it up every time, so in a 1:1 ratio. They claim that interleaving dense and super sparse MoE like this yields the same results as a less-sparse MoE throughout, but faster, which makes sense given how successful interleaving attention types has been.

Speaking of MoE, on the right side they diagram theirs, which is also a bit weird. So they’re showing their 512 experts, and we know they pick 8 of them each time, but they have this down and up projection thing and these yellow boxes around the experts.

Here’s what’s going on with that: they are compressing the inputs before going to the experts. That’s what the down projector does. But routing, which is where you select experts to activate, is super sensitive. So they split out those functions: the router uses the uncompressed input to pick which experts to use, and the dispatcher relays the compressed input to the selected experts. Then they combine all that info and uncompress it.

Here’s a bit about how they settled on their MoE setup.

What they’re showing here is how performance improves on a series of benchmarks given more training and more experts. So the x axis is FLOPs, which is the standard unit of training compute. The y axis is something they call “efficiency gain”, or EG for short, which is basically how much better your experiment is compared to your baseline. And the different colors show different numbers of total experts - they only activate 8 every time.

The upshot is that more total experts improves performance, and training them for longer sometimes improves performance. But more parameters means more cost and more latency, so they compromise on 512.

This one decision per se is not critical for understanding the whole model, but it’s emblematic of how the MAI team approached all of their decisions: empirically, with lots of rigorous optimization experiments. It’s a refreshing paper for its rigor, and also for its openness.

By the way, the weighted average in the bottom-right weights code at 50%, math and STEM at 17.5% each, general at 10%, and multilingual at 5%. So you can really see how much they’re optimizing for code and reasoning compared to everything else.

Worth noting these benchmarks include at least some human-generated data from external vendors — a sign that demand for human data in GenAI isn't going away.

Here’s another example of that optimization loop and rigor.

What they’re showing here is how different mixes of training data impact performance, as measured by NLL on two different validation sets - that’s the x and y axes. Each dot is a different data mix, and each color is a different size of model they trained in order to test the mixes.

They’re only showing a few sizes of model here, so there’s maybe a few dozen dots on here, but they mention that in total they trained several thousand models of between 760M and 4B active parameters.

One particular aspect they pay attention to is how different datasets interact, both within their domain but also across domains.

Like here for example, they’re measuring performance on a graduate-level physics benchmark, which a vendor created for them for this purpose. You’ll see our old friend NLL on the y axis, so lower is better here.

What each of the six graphs show is how adding the given data type in impacts performance. Like before, each dot is a model trained on one specific data mix.

In this example, math and STEM stuff helped, which includes “web PDFs” - those are PDFs of scientific papers. General web content didn’t matter, and code actually hurt. I personally was surprised by that, because generally code data helps with math performance, and vice versa, and physics is mostly math. But that’s why you run the tests! Anyway, hopefully you’re getting a sense of how a team ends up training several thousand models in their pursuit of one final model.

Relatedly, even though this much training takes a lot of infrastructure work, we just don’t have time to cover it. But if you want to know all the gnarly bits about managing GPUs and various forms of parallelism, it’s in there.

So after all that fiddling, here’s where they net out on pretraining mix.

Keep in mind this is all naturally generated - none of it is synthetically made for the purpose of training data. They even take great pains to identify and remove suspected AI-generated text from scraped sources. However, given the high share of code and the pervasiveness of coding agents, I suspect a decent share of the data is ultimately from models.

Speaking of code, it’s over half the data! And it gets one of the highest multiples there in the right corner, meaning they replay the data about two times. That’s kind of a proxy for usefulness. So apparently STEM data, books and journals, and especially math data were all very helpful.

30T tokens in pretraining is pretty good by the way. That’s about the same pretraining corpus as DeepSeek V4. To go much higher than that you’d need to start generating a lot of synthetic data expressly for pretraining purposes, which is a gamble and thus was not where MAI wanted to start.

Now after pretraining there’s midtraining, kind of smaller and more focused pretraining where they only take the cream of the crop. It’s also typically where people extend the context window.

Both cases are true here. They filter heavily for quality and rebalance domains a bit, then they extend context out to 256k tokens. That adds another 3.5T tokens or so.

Here’s the actual loss graph from their final training run, just to give you an idea of what researchers actually look at. We have NLL on the y axis, and we see how it pretty quickly flattens out, with very gradual gains for the vast majority of the training run.

Kind of wild to think how much quality improvement and polish comes from this seemingly small drop in loss. Also, if they had more tokens, they probably would have gotten even better results. But as we mentioned, new pretraining tokens are hard to come by these days.

So once pretraining and midtraining are done, you end up with your base model. In case you’re not familiar, base models are basically like autocompletes, they can’t do chat. Chat and other abilities come during post-training, which we’ll look at after this slide.

Base models are actually not that common to release anymore, so the comparison selection is limited, but they did their best here and picked reasonable competition - you’ll see the Kimi and DeepSeek models I mentioned before for example. They also included an earlier version of their model to show how they improved.

As for what they’re testing on, the top-right is code from their infrastructure, while the other three are benchmarks built by external vendors — again, a signal of ongoing demand for vendor-built eval data.

So with their capable pretrained and midtrained model in hand, MAI are going to use reinforcement learning (RL) to post-train three specialist teacher models: SWE & Agentic, STEM, and Helpfulness & Safety. Then those three teachers are going to use SFT to distill down into one single student, which will get another round of RL training to form their final model, MAI-Thinking-1.

We will tackle each aspect in turn, but one thing I want to say here is how common this sort of pipeline is becoming. For folks who joined the DeepSeek V4 paper review, you’ll have seen this before. The question is, why? Why not just keep training your one model?

The biggest reason is actually right there on the diagram: you can train the teachers in parallel. Assuming you have the compute for it, which Microsoft clearly does, it speeds up overall training time to do these separate skills in parallel and then come together at the end.

A couple fun notes about their RL scheme.

First of all, they’re using GRPO basically, kind of the standard RL algorithm, but with one neat trick they call “adaptive entropy control”, shown here. Basically, if they find their model isn’t exploring enough, if the entropy of its next-token probability distribution is getting too low, they flip on this switch that lets the model make bigger moves on each training step. The top graph is entropy, with symbol H, and they’re targeting an average of 0.3. The bottom graph is their switch, which they flip on when H is too low and generally flip off when H is too high.

Second, they control maximum response length based on how hard a problem is. So if it’s a very easy problem, maybe only 8k token budget. If it’s a really hard problem, they let the model go up to 128k tokens in its response. That’s like 100k words for one response! They also penalize length in the reward.

Now over the course of their RL, they lock in progress every so often by distilling the model onto itself. In other words, they take their somewhat improved model, make SFT training data out of it, and then train the base model on it. That’s what all the stars are, points of self-distillation. And the different colors are different models they distill onto. This lets them lock in progress and work off a clean slate, and also look for potentially bad behaviors in the SFT data that they can filter out - things like language switching or reward hacking.

It requires on the order of 1M SFT examples to effectively distill from the teacher to the student, but they get them “for free” by just keeping the correct rollouts from RL. Although weirdly, they report that even keeping the incorrect rollouts, like the model attempts that don’t get the right answer, still improves the student model in training.

Now let’s look at each of the three teachers quickly in turn.

This figure is their pipeline for producing QA pairs from natural sources for the STEM teacher’s mix. They also buy QA pairs from vendors, but of course we don’t get any details about the vendor processes.

Anyway, there’s a ton of work represented here, but I wanted to call out the scoring part in particular, since it applies to any QA pairs, even ones built by external vendors, so it could be a hint at their quality control process for purchased data.

I want to call out the Consensus Grading step. Basically, they’re doing a lot of attempts with a good model and seeing what the consensus answer is, then having a judge model compare that consensus answer against the ground truth from the QA pair. To me that sounds workable for most of the difficulty distribution, but for the hardest problems I suspect even a good judge will get it wrong, so I wonder if they’re leaving something on the table here. Or perhaps that tip of the difficulty distribution just isn’t present in their original sources.

This teacher also covers code, which they say they leaned on vendors for more, with 160k total coding problems.

The second teacher is for SWE and general agentic work.

Since they’re doing agents, they need environments, which each consist of a task, a sandbox for the agent to safely work in, an initial state, and rewards at the end. They’re going to decide those rewards with a combination of rules (i.e. verifiable states) and LLM judges (for e.g. task interpretation, helpfulness, and trajectory quality).

For the SWE tasks, they harvested from GitHub, filtering 102M PRs all the way down to just 266k, across 94k repos. Those are all the PRs that have fully fleshed-out issues, with all passing tests, and reproducible environments.

For the general tasks, their major goal was getting the model to pick the right tools. So they flooded the zone, offering 50+ API and MCP tools to the model, and even including some prompts that didn’t require any tool use at all. They did a lot of the data generation synthetically, 130k+ tasks across 150+ environments, with diversity across task, environment, personas etc.

The last teacher is Helpfulness & Safety, a real grab bag of more subjective stuff, which is in orange at the top.

I want to call out a couple things here. One is how they view human vs synthetic data. Human data has better complexity, like harder or many-step tasks that actually make sense and aren’t contrived, but synthetic data has better coverage - you can design and program in all the distributions you want your synthetic data generator to touch.

Another is the change in rewarding. As you move from left to right on this table, you get more and more subjective. Like Instruction Following for example is sometimes verifiable by rule even, for example on word count, whereas style is rarely that way. Even within a consistent method of rewarding, namely with LLM judges, the details change: rubrics for Instruction Following, yet a simple 0-2 scale for Style.

Lastly, one interesting trick not on this table: they estimate the distribution of appropriate response lengths based on the prompt, then penalize the model for anything outside of that. Yet another tool to combat overlong responses.

So once you have all three teachers, it’s time to distill onto the student, here labeled “Consolidated Model”.

As before, they’re going to reuse the RL rollouts as SFT, and they’re going to filter for relevant characteristics for each teacher, like correctness for STEM and style heuristics for Helpfulness & Safety.

Also as before, the mix of examples from each teacher is a matter of optimization. They find the share of examples is all that matters, not share of tokens, but they report both and it’s wild to see how disproportionate the mix can get while still being effective.

After distillation, there’s a final round of RL, mostly polish - a lot of the Helpfulness & Safety data, with a bit of the other two and a bit of long context, just to preserve those abilities.

Now we can get to results.

The MAI model doesn’t win on any benchmarks, but it’s competitive. Similar to how Meta’s Muse Spark was good but not amazing, just to prove they could do it. I wanted to compare benchmarks on the two but there’s basically no overlap, and Muse Spark isn’t available by API so even the benchmark maintainers can’t run it.

MAI picked Sonnet 4.6 as their comparison, and on the bottom table that seems to be true. Their model isn’t out yet though, so can’t compare vibes, which is increasingly crucial given how hard it is to measure some of these differences.

Finally, since they are good scientists, they do some human evals in addition to running programmatic benchmarks.

They actually call out their vendor, Surge AI, by name - a real sign of respect. Otherwise there’s not a ton new to report here; they use the standard RLHF task setup of several pointwise criteria on a 0-2 scale, then a pairwise scale for overall preference on a 7-point Likert scale, i.e. 1 = “Response A was much better” and 7 = “Response B was much better”.

Again, they’re on par or a bit better than Sonnet 4.6, generally a bit worse than Opus 4.6. A respectable first showing for MAI.

My Takeaways

• This is the MSL playbook: build a good-enough model to prove you can do it

  • Meta’s Muse Spark isn’t really available outside of Meta products, and it may never be - or at least there’s no rush

  • MAI is in the same position

• I predict MAI will primarily be for Microsoft product surfaces

  • This is what Amazon and Meta currently do

  • It’s also the fallback case for Google - they would keep building Gemini even if nobody used the API or subscribed to the chatbot

• All the big tech cos are invested in either OpenAI or Anthropic or both anyway

  • In a way, OpenAI vs Anthropic is a proxy fight for the big tech cos

• Microsoft’s demand for data should continue and increase

  • MAI has plenty of time to prove itself and all the financial runway it needs

Paper · Repo

Background

Let’s start with DeepSeek, the authors of this paper. They are one of the two top-tier labs in China, along with the Qwen team. Where Qwen shoots for quantity though, DeepSeek goes for quality.

Like look at the models DeepSeek has released here on their Hugging Face page. It’s been over a year since they released their last major model, DeepSeek V3, there at the bottom. It was apparently updated in March last year but it originally came out at the end of 2024. And you may remember DeepSeek R1, the model that briefly crashed the stock market in January 2025. They made two minor releases in 2025, and they released an OCR model for turning images of text into text, but that’s about it. If you compare that to the Qwen page, where they’ve released models across pretty much all modalities in both major and minor versions in the last year, and you’ll see what I mean.

Fundamentally it’s all because of resource constraints. For one, China overall is way behind the US in terms of available compute. They also don’t have access to the latest and greatest NVIDIA chips. So they’re working with fewer chips from a generation behind. That means every lab in China has to get clever.

Since Qwen is part of Alibaba, they still have relative abundance within the Chinese AI ecosystem. That’s why they’re able to produce models of many types and sizes and so on. But DeepSeek is different. They are an outgrowth of a hedge fund. They have always been more of a science experiment, and even though the hedge fund does well they don’t have the budget of an Alibaba. So DeepSeek is the cleverest of all these clever Chinese labs.

And because they’re so clever and so focused on R&D, they end up producing some pretty impressive papers. Probably the most impressive one is this Mixture of Experts paper, from all the way back in January 2024. That’s incredibly early in the history of the MoE architecture, which is totally dominant now in SOTA models. While they didn’t invent the idea, which actually dates to well before the modern Gen AI era, they laid a lot of the groundwork for how those SOTA models do MoE today.

First, they made lots of smaller experts and selected several per token, rather than having a few bigger experts and only picking one. If you remember Mixtral, which was 8 experts of 7B parameters each where only one was active per token, you know how big of a change DeepSeekMoE was. The smaller experts allowed more specialization and fine-grained knowledge.

Second, they introduced a shared expert, which is always active no matter which other experts are selected. The one shared expert tends to learn basic things like sentence structure or formatting, leaving room in all the other experts for more complex knowledge.

The graphic on the right shows both changes. In a traditional MoE, you might have N experts and pick 2. In this example, they bump it out to 2N experts and have 4 active: 1 shared expert and 3 selected experts.

Now in modern MoEs they’ve turned these knobs all the way up. Like in this paper they have 64 experts with 8 active, for a sparseness of 8, whereas in the most recent DeepSeek model they have 256 experts with 8 active, for a sparseness of 32.

And remember, this was all born of resource constraints, not pure performance gains. The virtue of MoE models is they trade expensive compute for cheap memory. So like in this paper, their model is 16B parameters but only has 2.8B active per token, swapping experts in and out of memory as they’re selected. Performance matched that of Llama 2 7B, which takes more than twice the compute to run. MoE is more complex and finicky, but if you’re really skilled engineers like the folks at DeepSeek, you can handle the complexity to unlock efficient performance.

Keep those themes of sparsity and complexity for performance in mind. There’s a reason they call their findings “a new axis of sparsity”.

Now let’s get into the topics of this paper. I want to start from a human, intuitive point of view and then transition into the technical.

First I want to highlight the different types of memory in humans. People talk about short-term vs long-term memory, but the popular notion of short-term memory is confused; it’s actually on a pretty short timescale as the graphic shows, and the better term is “working memory”. An example of working memory is if I give you a phone number to remember and repeat back to me. Most people have a working memory of 5-7 things. In the phone number example, a number without an area code is 7 digits, and if it’s an area code you’re familiar with you don’t need to keep all the digits in working memory, you have it in long-term memory.

Now within long-term memory, I want to distinguish between declarative and procedural memory. Declarative is stuff you can accurately describe and verbalize, like past events or facts. Procedural is stuff you can do but may not be able to explain well, “muscle memory” and the like. Declarative memory is much more amenable to written records and is easy to look up, while procedural memory is more diffuse. That split is going to be important in how the authors structured the Engram model.

Here’s a little illustration of working vs long-term memory. Since working memory is difficult to expand, we get smarter and more capable by putting things in long-term memory and then referencing it in working memory. So again, thinking back to my phone number example, the area codes in your home state are in long-term memory, so if someone gives you a phone number you can just point to that set of three digits rather than taking up three slots in working memory.

By analogy, working memory is like compute for a model, whereas long-term memory is like parameters. If you can look up the right parameters and swap them in, then you can avoid constructing the same ideas from scratch with your compute. That’s going to save compute for the things that require it, like handling new information and doing reasoning.

Thinking further about memory and meaning, let’s recall another slide from that agent memory paper in January. It shows how text turns into tokens, then into token IDs, then finally into embeddings. Embeddings are literally vectors, just a series of numbers, but more abstractly they represent the meaning of a token. And as we know from natural language, words and concepts and proper nouns etc have more than just their strict meaning: there’s also connotations, synonyms, personal associations, many many layers.

All LLMs have this single-token embeddings table. It’s how the model turns words into vectors it can then process into a next token prediction. But you could also do the same thing with groups of tokens instead of individual tokens. Like you could have one row in an embeddings table for “Alexander the Great” as one group, one n-gram as researchers call it, in this case a 3-gram since it has 3 words in it.

Of course you can’t have a row for every possible combination of three tokens, there are just too many combinations. But there could be ways around that with enough cleverness.

As for how LLMs build meaning from n-grams now, here’s an example from a related paper about how models piece together context as they process input layer by layer. The middle column, “Latent State Translation”, is a model’s own description of its “thoughts” so to speak at that point in its processing.

The input phrase here is going to be “Diana, Princess of Wales”. And at the start, after the first few layers of attention + feedforward, the model has really only picked up the very basic fact that this input relates to a country in the UK and in Europe. In the fourth layer it picks up on princess, in the fifth layer it connects princess and Wales, and then finally in layer 6 it recognizes that hey, we’re actually talking about a specific historical person with that title who has other facts associated with her, like her lifespan and her work.

That’s helpful and where we want to end up of course, but notice how much work went into putting together something the model already knew and was not context-dependent in any way. Like you could look up that exact information in a barebones encyclopedia. Maybe there’s a way to cut out that reconstituting process and just get the fully constructed information right away.

The Paper

Well that’s exactly what they did here with Engram. This figure is from later in the paper but it illustrates the idea pretty well: if you find that the last two or three tokens formed a known entity or phrase or something, just look up that meaning rather than reconstituting it from the knowledge in your parameters.

So in the first example, “Great” is bright red because the 3-gram “Alexander the Great” is well known. Same with most of the tokens in his horse’s name.

Then in the second example, the “way” in “By the way” is also quite red, because that’s a stock phrase and doesn’t need to be interpreted as individual words or with much attention to other context.

It even works in Chinese. In the first example, the first red block is the end of a famous phrase, The Four Great Inventions. You can see after the colon there are four list entries, and each one of them ends in red blocks too. For reference, the four inventions are papermaking, the compass, gunpowder, and printing.

This example is for 3-grams as I mentioned, but they also did 2-grams. They tried 4-grams but it barely helped, so apparently most proper nouns and stock phrases and the like are two or three tokens.

Okay, we’ve arrived at the architecture piece, which is going to take the most explaining.

First I want to situate us. At core we are still working with a transformer, still seeing the core operations:

1. Embedding, which turns tokens into meanings

2. Attention, which calculates how each token relates to every other token and establishes the meaning of the entire input

3. Feed forward, which processes that meaning into a prediction for the next token. In this model it happens to be a MoE setup

The big thing we’ve added is this Engram piece, also with a slight tweak to the embeddings which we can ignore. Let’s take each new step in order.

So the Engram starts with three inputs: the result of the previous transformer block, which is called “Input Hidden” here; then there’s the 2-gram; and the 3-gram.

Input Hidden will come into play later. For the 2-gram and 3-gram, we’re going to turn them into row IDs for our embedding tables using something called a hash. So the 2-gram and 3-gram are both going to become h different embedding rows.

Then we’re going to put all those rows together into one bigger row, specifically a row that is 2h bigger than each individual row. That’s called concatenation. That one big row stores all the meanings our embedding tables learned.

After that we’re going to basically copy the attention playbook. But whereas attention looks at how each part of the input relates to every other part, Engram looks at how each part of the input relates to these meanings we’ve fetched from the embedding tables. So maybe in this example the prior context is about Greek history, or about famous horses, or about literary devices in myth. That’s going to get mixed in with these context-free meanings we have about Alexander the Great, using that attention technique.

Now not all 2-grams and 3-grams are going to be meaningful of course. In fact most of them won’t have a static, context-free meaning the way proper nouns or common phrases do. If the fetched meanings are dissimilar from the constructed meanings, then the scaled dot product is going to be low, because the dot product measures similarity. So then you end up ignoring the fetched meanings. That’s fine because the Engram operation is pretty cheap, and the gain you get when the fetched meanings are similar to the constructed meanings outweighs the cost.

Anyway, you do a quick convolution at the end for stability and then add that Engram output to Input Hidden. Now your attention and feed-forward layers have this extra meaning and can skip the work they would have had to do to calculate that same meaning.

So that’s how it works for one transformer block with Engram. Not every transformer block needs Engram though, like in this example they show it at layers 2 and 4, although real models are dozens of layers deep so this is not totally indicative. In fact, the full model they train has 30 layers, and they put Engram blocks at layers 2 and 15, for reasons we’ll see in a bit. Earlier is generally better though since the memory lookup work is most helpful when the model is still piecing meanings together.

This diagram is also helpful because it reinforces the point that Engram memories do NOT live on the GPU, on the device as they say here. Instead, the Engram tables are either in RAM or the harddrive, both of which are far cheaper than GPU compute or memory. Again, there’s a throughline with their MoE paper; they’ve traded a little of an expensive resource, compute, for a lot of a cheap resource. If compute is your constraint then you want to make as many versions of that trade as you can.

Here’s another view of how having this Engram block to look up memories helps. On the left we have a graph that basically shows how close a representation is to the final one. So in general your representation gets more and more accurate as you keep processing, which makes sense. The interesting bit here is that the Engram lines, in yellow and red, are lower than the normal model’s line in blue. That means the Engram blocks give the model a head start sometimes, like a little boost, on its way to the final representation.

The sort of blocky diagrams on the rest of the slide tell a similar tale. We don’t need to get into what CKA means, but basically the lighter colors mean more similarity. So you can see that the early Engram layers are more similar to middle layers of the baseline model, with jumps at layers 2 and 15 where the Engram blocks are.

So it seems like Engram helps. The question is, how much? To measure that they did a few things.

One version of this question is, if I only have a certain number of parameters, how should I split them between Engram and the rest of the blocks? That’s what the graph on the left checks out. The allocation ratio, rho, is the % of parameters allocated to the rest of the blocks. So at 100%, you have no Engram, just the baseline model. At 0% you would have just Engram which doesn’t really make sense, so they only investigate down to 40%. So they train all these test models with different allocation ratios, and they measure the loss at two different amounts of training. At both levels of training, you see the same pattern, where something like 80% is optimal. That means a full 20% of parameters in the optimal model go to Engram.

Another version of this question is, how much can I scale Engram? Like keep the other parameters fixed, just allow for bigger and bigger embedding tables. That’s what the graph on the right addresses. What it finds is there is no limit, that the more space you allocate for memory, the better your model does. It might become impractical, but if you can support or afford it it seems to always help.

Of course the ultimate question to how much Engram helps is how much better the end model is.

Here we compare four models, all with the same number of active parameters and thus the same demands on inference compute: a dense 4B, a MoE 27B as we’ve been seeing for the baseline, Engram 27B to keep the same total parameters as the baseline, and Engram 40B where we keep the same active parameters and experts as Engram 27B but really bump out the Engram part, more than tripling those parameters.

Keep in mind that these are base models, not instruct models, so all they’ve done is pretraining, no post-training.

Anyway, as you can see Engram 27B beats the baseline model on every benchmark, sometimes by 4 or 5 points. As we saw with that U-curve on the previous slide, allocating ~20% of parameters to Engram is going to really help performance.

What’s interesting is that Engram 40B is not uniformly better. For several benchmarks it’s less than a point better, which I consider basically equal, and for a couple of them Engram 27B is better by more than a point. The authors attribute the somewhat mixed record to the shared limit on pretraining tokens, which is likely too low for a 40B model. Also, when you bump out the Engram parameters without bumping out the other parameters, you get away from that optimal allocation ratio.

One area of performance they pay special attention to is long-context, which typically means 32k or higher, even up to 1M in rare cases. They examine long context because it tests their theory that Engram allows the model to refocus effort from entity or phrase recognition in local contexts, which Engram now handles, to reasoning and global contexts.

Here they deploy two benchmarks, LongPPL and RULER, both on 32k token sequences. We don’t need to examine every metric, but the three groups are helpful to know: that’s perplexity, which is a measure of surprise; NIAH, short for “needle in a haystack”, where the model has to fetch one or more words or phrases from the long context; and then the other tasks, which are frankly less artificial than NIAH, for example question answering.

So compared to the baseline model, with 50k pretraining steps and 1.63 pretraining loss, some version of Engram pretty much always wins. The one with the most pretraining steps does the best unsurprisingly, but the one with fewer pretraining steps yet equal pretraining loss to the baseline model generally does second-best. So it seems like you can get more performance for the same amount of pretraining, or use less pretraining to get the same performance.

Finally, they do what’s called an ablation, basically taking away some of the stuff you added to see how important is actually was. In this graph they ablate the Engram component, so like they train the complete model with two Engram blocks and then just delete the Engram blocks.

Here’s the result, grouped by benchmark skill. The more reasoning-heavy tasks like reading comprehension and commonsense reasoning take minor hits, while factual recall tasks fall off a cliff. This validates two claims: one, Engram offloads a lot of factual memory but not process memory; two, that offloading lets the rest of the model focus more on other work.

My Takeaways

• I suspect this will follow the same path as DeepSeekMoE: appreciated for good engineering but not really “in the water” until DeepSeek releases a ~SOTA model with Engram

  • One of their researchers confirmed on Twitter that DeepSeek V4, whose release is immanent, will not have Engram

• Post-training an Engram model will present new challenges

  • Recall that the models in this paper are only base models

  • If Engram becomes the new standard, we will need to track such differences

• Could make the most sense on edge devices, where power management is an additional concern

  • 1B params is ~2 GB at full quality (i.e. in BF16 format, before quantization)

  • The Engram parameters (~20% of the total) can live in storage rather than RAM

Paper · Source papers

Background

To start off, I want to take us back. All the way back, to before the dawn of GenAI.

When I first started at Scale back in early 2021, the first project I was ever on used ML. Specifically, we used classifiers - models that take whatever input and spit out a class, like taking in a drawing of furniture and giving a confidence or probability for each class. For my project, we took as input a description of a grocery item, and the model predicted what category that item was: Canned Tomatoes, Frozen Fish, Packaged Cookies, Thank You Cards, and so on.

When you train a classifier, you give it the whole set of classes it could predict in advance. Like for our grocery classifier, we had to say in advance what all the classes were. You couldn’t just pass in a description and say “What item is this?” the way you can now with LLMs. That’s what generative AI means, it can generate new stuff on its own.

So you have this vocabulary of classes, and you make training data by saying, this description is Precut Melon, this description is Wet Cat Food, whatever.

The architecture, the way those classifiers worked, is broadly the same as the architecture of most LLMs now: a transformer, using attention and feed-forward layers. If you don’t know what those are, no problem - just being concrete for those who do.

Now let’s extrapolate. Let’s say that instead of these grocery items as classes, I had every word in the English dictionary as my vocabulary. And instead of grocery item descriptions, I had just any old text. Then my “classifier” could “classify” each input by what word came next.

Now you have a generative model. It’s still just text in, text out, but now the output text can be anything.

That oversimplifies somewhat, about how generative models are typically decoders rather than encoder-decoders and about how it’s tokens rather than words the model is predicting. But it really is the same core idea, that if you can just predict one thing at a time really well, you can eventually just keep predicting basically forever, on anything.

Your training data is just whatever text you can find, above some minimum quality bar anyway. You can take a bit of text, train your model to predict the next token, see if it was right, give it feedback, over and over and over again for every next token you can find. That is how you train - or more specifically, “pretrain” - an LLM.

After enough pretraining, like tens of trillions of tokens worth, you now have a base model, i.e. a model that can predict the next token for some input text. It can’t chat, can’t engage in dialog yet, but it can autocomplete like nobody’s business. And as with our classifier, each next token prediction comes with a probability.

This is what the first GPT did, back when it came out in 2018.

Now since then LLMs have gotten a lot better of course. For one, they’ve gotten a lot smarter; they know much more, they are better at reasoning, they are more perceptive and so on.

For another, they’re no longer just autocompletes! Through the magic of post-training, for which Scale provides data in our GenAI line of business, base models become instruct models. Instruct models converse, they don’t simply continue the train of thought you provided.

This is what OpenAI did with GPT-3 to turn it into InstructGPT - which Scale contributed training data to - and then GPT-3.5, the model that powered ChatGPT at launch. But it wasn’t just the model that changed - it was the model’s environment too. Whereas GPT-1, GPT-2, and GPT-3 were almost exclusively API services, ChatGPT was a consumer experience: on the web, with a UI, engaging in dialog.

Now remember, the model itself was not different between the API of GPT-3.5 and the website of ChatGPT. Ultimately it received text input and produced text output, like all its predecessors. But exactly what text it got was different. For example, a user on ChatGPT has a location and a timezone, which the model might use to adapt its slang or know to say “Good night” instead of “Good morning”. So where the model interacts is already starting to cause diverging behavior a bit.

As many of us will recall, the earliest version of ChatGPT was really just a chatbot: you put in a message, it sent a message back. No web search, no running code it wrote, no memory outside of the current conversation. And by the way, the conversation could only be 4096 tokens - about 3k words.

As time went on, the barebones chatbot experience changed in a few important ways.

For one, that context window got longer. GPT-4 doubled it to 8192, then quadrupled it to 32k. By late 2023, only a year after ChatGPT came out, GPT-4 Turbo bumped it all the way up to 128k. That ceiling generally stuck around for a while, although now 1M is becoming the standard.

Of course the intelligence improved dramatically too. A fun toy of a chatbot became a productive tool for many people, by API or by GUI.

Crucially for us, the software around the model improved too. Web search, code execution, and memory - what we broadly call “tool use” now - all enhanced the model’s performance without altering the model itself. And they provided a source of information and interaction aside from the user.

So the software grew alongside the model. And new software capabilities influenced model training.

For one, training models to use tools at all takes work, since that type of interaction didn’t exist before LLMs and so isn’t going to be in the pretraining data you scrape from the internet. But more specifically, the LLMs need new tools, tools with different affordances and abilities.

That’s the whole reason behind Model Context Protocol, or MCP as it’s more commonly known. Models have different strengths and weaknesses than either humans or regular old software, so your tools have to lean into those. So for instance, models like structured text, like regular software does, but it can’t handle a flood of output like regular software does. So tools should return a large-but-still-limited amount of structured text.

Or think about memory, which in most cases is just a set of well structured folders and Markdown files. A normal software program would use a database and would have clear rules for storing and fetching data. An LLM needs to work memory in naturally and quickly. It’s a different sort of tool, and models need training on when and how to best use it.

So we see that the initially clean split between model and software has started to heal, where the optimal model performance comes only within particular software. Kind of like a marriage, where two separate people come to share one mind in many ways.

So the marriage between model and software gets us some pretty slick chatbot capabilities. But the fundamental framing of chat is still around a conversation, not about getting work done “out there” so to speak.

One capability in ChatGPT and other chat portals that started to break that pattern was Deep Research, which came out for ChatGPT in February 2025. In a regular chat session, you could start out with one question but typically would need to keep directing the model on research steps, then ask it to compile its work, etc.

With Deep Research though, the model was autonomous: you gave it a question, it might request some initial clarification, then it would go off and work. It felt like the model was “out there” on the internet, not there in your conversation, and that it would “come back” when it was done, report in hand.

That was the first in-production peek at agentic workflows. The framing had shifted; instead of having an AI assistant, you now had an AI worker, returning you substantially complete work in one go.

But some problems require more than just a chat interface in the browser, or even a chat interface with a secret room on the server for the model to go Deep Researching in. You might need a different environment for some work, and really a different paradigm for working.

Anthropic realized that. And a few weeks after ChatGPT Deep Research came out, they released Claude Code.

If you’re not familiar, Claude Code is a program you install on your computer. That program lets Claude do things on your computer, basically anything that can be done with a command line, which is most things. You ask it to do stuff, it goes off and does it until it needs you again.

That software is called a scaffold or a harness, people use both terms but they mean the same thing: a program that turns a model into an agent. The harness provides prompts and tools and guardrails that the model follows and uses and obeys when working in the environment of your computer.

And while the form factor and capabilities seem radically different, it really is a gentle spectrum all the way from pure LLM endpoint to simple chatbot to modern chatbot to agent. Yes, the points further along the spectrum require model capabilities, like long context for those long agent trajectories and better tool use for the increasing array of tool options - that’s why ChatGPT didn’t start out as an agent - but the actual variable on that LLM-to-agent axis is the software. And that’s what today’s paper is really about.

The Paper

So this is a survey paper, which means they basically found and read all the papers they could on a certain topic and then composed their findings into something coherent, with a bit of original analysis on top. Surveys are by nature broad, whereas the papers we typically cover are narrow but deep.

This graphic from the start of the paper is our roadmap. After defining terms at the top, we’ll go through how the harness looks, what the harness does, how agents can work together using shared harness infra, and what the leading edge looks like.

Let’s isolate the top part first. An agent is a model plus a harness. The harness lets the model act in an environment, like on your computer in the case of Claude Code.

One of the main points of original analysis in this paper is how vital code is as a working medium for agents, why code is “agent harness infrastructure” in their jargon.

In fact it’s literally a medium in their graphic; code is in between the agent and the environment. It’s what the agent writes and executes in order to act on, to make changes to the environment. And then through code it receives feedback, signals like a printout or a confirmation or an error.

That’s quite different from chat, where the model interacts with us through natural language. If we think of ourselves as the environment that the agent is acting on during a chat, the agent uses words to act on us and we use words to send signals back to it.

And as the graphic points out, code has some nice properties that make it better as a working medium for agents:

• Code has hard rules and is basically machinery for reasoning. Natural language has soft rules and can work for reasoning, but provides no guarantees or checks

• Code provides feedback, like detailed errors and stack traces. Natural language does not

• Code can store state, basically it has memory. Again, natural language does not - it is not machinery

I would also add that model makers, the big labs like OpenAI and Anthropic, want agents to be good at code so they can achieve Recursive Self Improvement (RSI). The better the model gets at code, the more it can contribute to research to make itself more capable. That’s what Anthropic hired Andrej Karpathy to do, and OpenAI says they will have an intern-level autonomous ML researcher agent by September 2026.

So code composes the harness, yes, but code is also what a model does via the harness, when you combine the model and the harness to form an agent.

This is not a new realization, and basically since the dawn of the GenAI era people have been working on code from AIs.

The top timeline, Code for Reasoning, looks at code as machinery for logic. The very first one, Program of Thought (PoT), is one I remember reading. Back when LLMs couldn’t reliably do even simple math like two-digit multiplication, people realized you could get math right if you just got your model to write the formulas as code instead of just numbers. So if your model can write a quick Python block that just says the two numbers you want to multiply, some software can watch for that Python block, run it, and return the result.

LLMs have gotten a lot better at calculation now, but they still know to invoke a calculator like that instead of relying on mental math.

The middle timeline is basically for writing and then using scripts as tools. Like if your model wants to navigate a browser, what it often does is actually write little code snippets for different action and then stitch those together depending on the task. We’ll get to it later, but there are many ways code acts as a medium instead of a model visually perceiving a screen and just saying what pixel it wants to click.

The bottom timeline is for code as an end, not just as a means. Many of you will recognize SWE-Bench, which is the OG agentic coding benchmark. CWM, which stands for Code World Model, is an interesting one; they taught the model to predict how code will run in order to improve its coding skills.

Now in terms of how the harness operates, we’re gonna tick through five different skills: planning, memory, tool use, control, and optimization.

Planning has evolved significantly from the early days. Kind of inherited from the chatbot days, you start with something linear and direct. Then people tried getting clever by putting those lines into different shapes, different workflows. But that’s not very Bitter-pilled, and agent intelligence quickly advanced to not need so much prescriptive planning from humans, and more open-ended planning methods like search took over. Now the leading edge is in orchestrating between multiple agents, which we’ll come back to later on.

Memory has also gotten significantly more sophisticated, and has different forms as we know from human cognition. So now people work on different types, like storing the outcome of experiences or storing the meaning of data as metadata in addition to storing the data raw. And again, on the leading edge we see multi-agent cropping up.

Tool use they call out explicitly but is a bit of a catch-all, since technically any action a model takes other than simply returning text is tool use. But they do show the general trend from input-based (“do this”) to output-based (“achieve this”), and they again end on multi-agent.

Control is a less intuitive one if you’re not coming from a programming background, but it’s basically about how much autonomy you grant the agent while maintaining safety. The narrative here is that software infrastructure is starting to change to accommodate and fit agents, just as it changed to fit different types of users when computers got popular.

The “plan, execute, and verify loop” from the top is also a crucial point. One reason code is so great for agents is that the feedback is automatic for the most part. Like until the user needs to make a judgment call, the agent can just iterate freely. The more free iteration you can enable, the more autonomous the agent can be. Part of that is safety, part of that is verification.

Now harness optimization is where things get really interesting. If agents are so good at code, shouldn’t they be able to to improve their own harnesses? It’s just code after all.

In fact that’s starting to happen, and it’s a big part of the RSI story. I don’t want to minimize the raw model capabilities of course, which Scale’s training data is critical for, but harnesses have gone from nice add-ons to integral parts of the product. In fact, I wouldn’t be surprised the big model makers stopped generally offering API access to their models and only offered them as part of their products. Like you wouldn’t be able to use Claude via API anymore, only on claude.ai or in Claude Code or in Claude Cowork etc. Or maybe it would only be available to enterprises for use in their products on certain terms.

Note the timelines on each of these areas. Memory in particular is a longstanding and thorny problem, and I’m sure anyone who has used AI products with memory enabled has had recent terrible experiences. I think it will take some time before memory as fluid as ours is available.

I’ll also call out optimization as a recent hot area. Again, models have only recently become capable enough for harnesses to become critical, but now that harnesses are first-class players, expect a lot more effort to go into self-improvement.

Now as I’ve mentioned a few times, multi-agent is also a hot area. Multi-agent swarms are valuable because they parallelize work and they break up the context window amongst the different agents. So you can get more work done faster.

I personally think multi-agent is still in its infancy, and that it will take a long time to work out organizational principles and sociology of machine intelligences, but there has already been some progress.

One immediate split people have imported from human organizations is division of labor, specialization basically. Usually that’s just a harness change on the same model, but sometimes for cost and speed reasons you might use different models for different jobs.

Another import from human organizations is ways of working, like interaction and topologies as the top-right shows. The topologies one in particular is interesting as the bottom-right shows; instead of humans encoding a topology from the start, emergent or task-dependent reorganization has become prominent.

Of course code as the working medium recurs as a theme, as the green and red boxes towards the bottom show. Anyone who has used git to coordinate software projects knows how important it is to have everyone working off the same codebase rather than individual copies or going through one person who has control. For the non-technical folks, the equivalent would be using a shared Google Doc to work instead of everyone having their own local files.

As we continue looking to the future, there are areas besides code we want our agents to be good at, like graphically using the computer or doing science or steering robots.

In keeping with the theme, the authors recommend framing all these new capabilities in terms of code. The more code-like you can make it, the better agents are going to be.

Obviously coding agents are going to be good at coding. One non-obvious detail they do note though is how trajectories from the coding agents end up as training data. So the model now has training in its particular harness, not just on harness-agnostic capabilities. That means that Claude will always work better in Claude Code than in any other coding harness. Same with GPT in Codex. That’s yet another way the model and the harness are becoming inseparable, and why pretty much every model maker has their own harness now. There are still model-agnostic ones like OpenCode, but to me it seems they’re at a structural disadvantage.

Similarly, generalist agents like OpenClaw and Hermes are also seeing proprietary competition. Google and Microsoft both announced their own versions. Meta has an internal one I suspect they will release. A lot of generalist agent stuff like personalization is on a different axis than the raw performance you need with coding, so I do see a world where model-agnostic options are alive and well

So What?

Since we’ve been mentally drawing trendlines and now extrapolating, I wanted to make the trendline clear.

This is the famous METR chart, which shows how long of tasks models can accomplish, in terms of how long it would take a competent human to do them.

We’re at a point now where the best model on the planet, Claude Mythos, is beyond our current capabilities to measure accurately. I’m sure we’ll catch up in this case, but the gap will emerge again and one day we won’t be able to catch up.

In terms of the trend, depending on how you draw the line, it predicts a 10x annual increase in task duration. So if Opus 4.6 was at a 10-hour horizon in February 2026, then in February 2027 we’ll have a model that can do the equivalent of 100 hours of human work autonomously - at 50% success rate anyway. For 80% success rate the respective numbers are 1 and 10 hours.

My Takeaways

• RSI will be a system, not a model

  • How many more optimizations loops can we add?

    • Innermost = model

    • Outer = harness/agent

    • Outer outer = optimizer? swarm?

  • Will it require optimizing further-flung aspects, e.g. the CLI or the OS or the hardware? The org chart or society of agents?

  • Will we Goodhart intelligence to just be coding, or is intelligence a package that includes e.g. creative writing?

• The optimal arrangement of swarms will depend on the task, but will often look different from how humans self-organize

  • The more Bitter the better

Paper · Repo

Originally presented as a live talk on May 20, 2026

Background

To understand this paper, we have to understand how a model on a GPU turns your prompt into predicted tokens.

Let’s start with what we already know from our daily use of LLMs: you pass in your prompt all at once, you wait a bit, and then you start getting back tokens one by one.

Already we can start to relate to this diagram. The part where you pass in your prompt is called prefill. During prefill, you are filling up the working memory of the model, what we call the KV cache. The diagram here kinda breaks up “KV” and “cache”, but you can see that the prompt turns into KV vectors, and then those get cached, hence “KV cache”.

So now we have all the input in our working memory. That can take a long time depending on the hardware you’re using and how big your input is, but importantly, prefill happens for all input tokens in parallel. On my GPU at home, prefill runs at several hundred tokens per second.

So once prefill is done, the model is ready to start making predictions. And it makes those predictions one at a time, as the diagram shows: first “jumps”, then “over”, etc etc. This stage is called decode, because we’re decoding the mathematical representations that the model works with into words that humans work with.

Because decode runs one token at a time, it is much slower than prefill. On my GPU, decode runs at something like 30 tokens per second.

Note that when we decode a token, it gets cached too - that’s why the dotted line at the top spans prefill and decode. So the KV cache, that working memory, grows every time the model outputs a new token.

Now let’s look a level deeper, at the hardware. On a GPU, you have your chip and your memory, aka your VRAM. When you start up your model server, the server reads the model from your hard drive and puts it into your VRAM, right next to your chip.

When you actually use your model, like by sending it a prompt and getting back a response, your model server takes one layer of your model at a time from VRAM and sends it to your chip for computation. So if I’m at the very first attention layer, it’s gonna take my input and the matrices that actually make up the first attention layer and send ‘em to the chip for multiplication and so on. Then it takes that first attention layer back from the chip, along with the KV cache created from the computation, and it’s gonna send in the first feed-forward layer for computation.

That process of loading and computing and unloading happens over and over again until you’re at the final layer, where the chip can finally produce the predicted token. Then you gotta do the whole routine over again to predict the next token.

So this shuttling of weights to and from the chip is typically what slows you down - the constraint is your memory bandwidth, not the speed of your chip. If you can somehow compute multiple tokens at once in decode, like you do for prefill, then you can avoid repeating that shuttling. Sure, it costs you a tiny bit of extra time on the chip to deal with more tokens, but that’s peanuts compared to the time you saved in transit.

The question is, how do you predict multiple tokens? If predicting a token requires understanding all the tokens before it, how can you predict more than one?

The answer is in the name of the technique: “speculative decoding”. Instead of making a brand new prediction every time, you speculate about what the next few tokens will be - you take a guess beforehand and then check.

Speculative decoding works for the same reason prefill is faster than decode: inputs get processed in parallel. As we said before, once you load the weights onto the chip, it’s quick to do one or two or three or four calculations. As long as you have a good way to guess tokens, the model can check them all in parallel.

Of course, if the first token fails, then the other ones you guessed after it will likely be wrong and you’ll have to throw them away. But if your method for guessing tokens is cheap enough, and you’re not wrong too often, it can work out.

One common method is to have a version of the model itself make the guesses. Specifically, a much smaller version, ten or a hundred times smaller in fact, so it’s much faster and also can fit on the same GPU. This “draft model” as it’s called is not nearly as smart as the target model that is actually producing tokens, but it’s often smart enough. After all, most tokens are not incredibly complex or subtle; language is chock full of common and supporting words, and a lot of sentences are pretty mundane, meant to support the occasional novel or surprising sentence. It’s even more true for code, which demands predictable structure in a way natural language doesn’t.

As the graphic shows, the draft model quickly produces a few tokens, which all go into the target model in parallel rather than in serial. And thinking back to the last slide about where the bottleneck is, because you now have multiple tokens ready for computation, you save all that shuttling from VRAM to the chip on the second and third and fourth tokens.

In the example here, we have indeed generated four tokens, but the third one gets rejected, and the target model’s predicted token takes its place. The fourth draft token doesn’t get checked at all, because it depends on the third draft token being correct, which it wasn’t.

Let’s zoom in on this example a bit more. We have our four speculated tokens from the draft model, and we’re going to verify them with the target model.

Specifically, we’re going to check if the probability of the speculated token for the target model is at least as high as the probability of that token from the draft model. Like in our case, the target model thought “Brown” was 93% likely, and the draft model though it was 92% likely, and since the target model is smarter, we take the increased probability as a sign that the draft model was pointing us in the right direction. Similar story for “Fox”.

But for “Hopped”, the draft model was more confident in that token than the target model was. That’s a bad sign and means we should reject the draft model’s choice.

Incidentally, when the target model rejects the third token, it substitutes its own - in this case it’s the word “jumped”. That extra token you get from the target model when it rejects the token from the draft model is called a “bonus token”, because you get it “for free” in the process of verification. If you’re really lucky and all your speculated tokens get approved, you get a bonus token after that, directly from the target model. Like if all four tokens had been right in this example, we also would have gotten a fifth token as well, with virtually no extra effort.

Now as you might imagine, the draft model is going to be better at predicting some tokens than others. Like on a hard reasoning problem, the acceptance rate will be quite low, maybe like 25%. But on a more structured and straightforward task, like using a web search tool, it could be near 100%. So speculative decoding isn’t a complete across-the-board speedup, but for a lot of mundane LLM uses it’s helpful. It’s really an empirical question depending on your use cases, your hardware, what model you’re using, stuff like that. Like me personally, on my hardware and for my Hermes Agent, speculative decoding 4 tokens at a time has been helpful and fits on my GPUs.

I should add that there are other forms of speculative decoding, or of trying to predict multiple tokens in one go anyway. We’ll briefly see one called EAGLE-3 in the paper for example.

Another example, which we’re looking at here, is literally called “multi-token prediction” or MTP. The difference with MTP is that is has to be part of a model’s training from the get-go, it’s not an external enhancement.

You can see it right there along the top, at the boxes labeled “Cross-Entropy Loss”. The loss is the single number that tells you how well your model is doing. In a normal model, your pretraining loss is based on how likely you predicted the actual next token in the training data would be. That’s the first box along the top, it has an arrow pointing to L_Main - that’s the symbol for loss.

But here in MTP, there are multiple losses! As you continue along the top, you’ll see L_MTP^1 and L_MTP^2 - the losses for predicting the first and second of the multiple tokens. So now your loss is from the normal token, the first MTP token, and the second MTP token. And if you do training right and minimize the overall loss, you can get pretty good at predicting multiple tokens.

As with so many thing, MTP is an invention of the DeepSeek crew, and all their models since V3 have had it. More recently, Gemma 4 and Qwen3.5 got it, which has been great for local LLM users.

Lastly, one thing I want to remind folks of is what LLMs actually predict. It’s not just one token, it’s actually a distribution of tokens, each with its own likelihood.

Technically the model predicts a likelihood for every single token in its vocabulary, which is usually in the high tens or low hundreds of thousands. Of course nearly all of them will receive virtually 0% odds, so in practice you end up with something more like the distribution here, where a small number sum to almost 100% probability. The shape of the distribution, and how far it extends out, can vary a lot and will be important later on. For now though, just remember that the model usually has a few thoughts on what could or should come next.

The Paper

So now that we’re equipped with the knowledge of speculative decoding, and other techniques like it, we can finally discuss speculative speculative decoding.

For reference, we have SD on the left. Our draft model speculates a few tokens in blue, our target verifies the first one but rejects the second and third, and in the process produces a bonus token, in yellow. A line of connected dots means a sequence of tokens, so we see the end result is two new tokens. Then the cycle begins again, with the next speculated sequence in red.

You’ll notice the draft and target turns happen in serial - one waits while the other works. Surely we could make use of the draft model while the target model is working, right? Like the draft model is fast, it could produce lots of tokens in the time it takes the target model to verify.

There are two problems though. One is that the target model is using up all the memory bandwidth while it’s working. So at least on the same GPU, there is no way for the draft model to work in parallel. That’s fixable with another GPU of course.

The second, bigger problem is that the draft model depends on having all the prior tokens available. If the target model isn’t done verifying, then we don’t know all the prior tokens yet! We know the prior speculated tokens, but of course not all the speculated tokens will be right. Even worse, the bonus token could in theory be any token in the vocabulary. What are we supposed to do?

Well as you probably guessed, there is a solution, shown on the right. Let me walk through it.

First off, you’ll see there are now two parallel paths, separated by a little dotted line down the middle. That’s showing the target model on one GPU and the draft model on another GPU. So they can work simultaneously.

Now the sequence of events is this:

1. The target produces a token, shown in green

2. The draft model returns a few speculated tokens

3. The target model starts verifying. Relatively speaking, that’s gonna take a while

4. On the draft side, we start preparing contingencies, shown as this kinda branching thing with tones of red and yellow. Specifically, at each token position, we are guessing what bonus tokens the target will produce in case the next speculated token gets rejected. So even for the first token, the green one from the target, we could get the first speculated token rejected. So now we have to be prepared for whatever bonus token in yellow the target model will produce

5. Since we have our bonus tokens predicted, we might as well speculate on what comes after those. Again, that’s the chains in red. So as long at the actual bonus token is one we anticipated and speculated on, we’re ready to immediately return that new speculated sequence

6. In this example, the target model verified only the first speculated token, then it produced a bonus token, and that bonus token was one of the three we anticipated. So the draft model immediately returned a new speculated sequence, in that medium shade of red. It’s a little hard to see, but in the tree thing they bolded the line showing the sequence of events

That’s basically it! You now understand the core concept of SSD. Now we look at optimization and performance.

So the big catch on this method is that you have to guess the bonus token, or at least guess it enough of the time that your extra effort isn’t wasted.

On its face, that might seem tough, since the bonus token can in theory be any token in the vocabulary. But of course in practice, it’s not a random choice, and the whole idea of a draft model is it knows how to make reasonable guesses at what the target model will say.

As we know, LLMs actually produce a distribution of tokens, so we have a straightforward way to guess bonus tokens from the start: pick the most likely token in the distribution as the one to speculate, and save a few others for your guesses at the bonus token, in case the speculated token was wrong.

On the surface that seems fine. But because of how verification works, it’s actually a big problem if your other guesses are overconfident, which they tend to be for draft models. Intuitively, since draft models are much smaller than target models, draft model distributions tend to be smaller and thus put a lot of probability on the few tokens they think could be right. By contrast, the smarter target models have wider distributions, accounting for genuinely different thoughts but also things like richer vocab.

So one key to making this work is to fiddle with the raw distributions of the draft model to spread probabilities out more, take out that overconfidence and just hedge a bit. That makes your bonus tokens a better safety net.

The other optimization they work on is how much speculating to do, and where exactly to do it. Like if you had all the time in the world, you would pick tons of bonus tokens at every position in the speculation. But in reality we only have the time it takes for the target model to verify, so the draft model needs to make the best use of it. How exactly they figure out the optimal number of bonus tokens per position is too far in the weeds for this paper, but you can see in the graphic that different positions have different numbers.

Okay, now let’s look at performance. Our target model here is Llama 3.1 70B, and the draft model is Llama 3.2 1B.

Here they compare three methods of decoding, of producing tokens:

1. Autoregressive, which is the baseline of just one token at a time

2. Speculative decoding, which runs draft and target models in serial

3. Speculative speculative decoding, which runs draft and target models in parallel

As you’d expect, SSD wins out, at 4x the speed of AR, and almost twice the speed of SD when using vLLM, which is a very popular LLM server program. SGLang is a relative newcomer and apparently does better with SD than vLLM does, but still, SSD is about a third better.

It’s important to note that the efficacy of speculative decoding, and other techniques like MTP, vary by domain and difficulty.

Like a very easy programming problem is gonna be super predictable, because it has lots of structured language and the draft model’s guesses are gonna be pretty good, because the problem is easy and because programming is a very common use of LLMs.

By contrast, SD will probably not work that well on high-minded creative writing. The draft model is gonna have bad guesses, and there’s just not nearly as much of that stuff in the training data in the first place.

So to that end, it’s important to measure speed gains on different benchmarks. Here we have one code, two chat, and one math benchmark, respectively. As we would expect, the programming and math benchmarks show higher speed gains.

On a related note, temperature also impacts SSD performance. It’s the same intuition, that less predictable text means a lower hit rate for SSD. Higher temperature flattens out the probability distribution and make guessing harder. That’s also why people conflate high temp with creativity, because it makes unlikely tokens relatively more likely, thus creating surprises.

My Takeaways

• There is simultaneously a compute shortage and compute overhang

  • Lots of efforts to squeeze more performance out of existing hardware

• This is a “yes and” approach

  • Innovation will continue on the other constraints (e.g. memory bandwidth)

• ML infra is still in its infancy

  • People have only been serving LLMs at scale for a few years

  • Innovations in architecture (e.g. MoE) and inference (e.g. reasoning/TTC scaling) will present different avenues for optimization

• I am waiting for AI to discover something like this

  • Anyone working on RSI will hit this stuff at some point

  • Great test of creativity in the wild

Paper · Repo

Originally presented as a live talk on May 27, 2026

Background

So to really understand this paper, we need to know a lot about how LLMs predict the next token.

We spend a lot of time talking about the three main parts of the transformer:

1. Embeddings, which turn tokens into vectors

2. Attention, which forms a holistic understanding of the input

3. Feed-forward, which processes or “thinks about” that holistic understanding

What we typically neglect though is what happens at the end of the transformer, after all N transformer blocks. What is that last step that turns the output of the last feed-forward layer into a token? The way it’s shown here, it make it seem like the last feed-forward layer just outputs tokens.

In fact, that’s not the case. After all your transformer blocks, you need some way to turn that final transformer output into whatever it is your model is supposed to output. In the case of an LLM, that’s going to be language.

So we attach and train a language modeling head, or LM head for short. The end result of the LM head is not a single token, but a probability for all tokens in the model’s vocabulary, which usually contains around 100k tokens.

Now the vast majority of them are going to be zero or near-zero probability. But depending on the context, you could have kind of a long tail of possible outcomes. Of course in other cases it’s going to be pretty certain, like if the sentence says “The capital of France is”, the token “Paris” is gonna get 99.9% probability or something.

So that probability distribution is what the LLM produces. But how do we get from a probability distribution to a single, selected token?

There are some simple ways you could come up with, like taking the most likely one every time, or picking randomly based on their probabilities. And people do both.

But there’s a lot more to it than that. And how exactly you sample from this distribution can be a surprisingly big deal. So we need to take a closer look at the mechanics.

So broadly, the knobs to twiddle for sampling from your LLM’s probability distribution are called “inference parameters”. Some other settings are inference parameters too, like the maximum number of tokens per response, but a lot of them are about shaping or sampling from the distribution at each position, each time you need a new token.

The most familiar one is temperature. If you’ve heard of it, you probably heard what it impacts, like creativity or predictability. That’s true, but it will serve us better to understand the math directly.

Temperature is a way to fiddle with the distribution before you sample it. When temperature is 1, you get the unaltered distribution, shown in the middle here. 1 is actually considered a high temperature, which might be surprising given the mathematical impact is to do nothing - like you’d naively expect that to be the baseline or neutral temperature.

The lower the temperature goes, the more uneven things get. So the most likely token gets more likely, and everything else gets less likely, sometimes dramatically so. In the extreme case, T = 0, the most likely token from the original distribution is the only token in the new distribution, so you always pick it. A lot of evals run at T = 0 because it makes results reproducible. Like if you always pick the most likely token, then given the same input you should always get the same sequence of tokens as outputs.

On the flip side, raising T past 1 flattens things out. In the extreme case where T approaches infinity, all tokens are equally likely. Practically speaking you of course would not want that, but even a temp like 1.5 shown here is considered pretty high.

You can see where the conflation of temperature and creativity comes from - if creativity means making unusual or unexpected choices, raising the temperature means choices that started unlikely are now relatively more likely.

So once you have your temperature set, there are a few other ways you can alter your distribution.

One of the most common ways is to keep just the top few token choices, drop the rest, and redistribute that dropped probability equally amongst the token choices you kept. That’s called “top-k”. In this example, k=2, so we kept the top two and dropped the rest and gave their probability mass to the two we kept.

Another common choice is top-p, where you set a cumulative probability and drop everything below that. So let’s say I picked p=0.95, meaning I want to keep adding tokens to my retained set until I cross 95% total probability. As soon as I cross that threshold, I drop the remaining tokens and again redistribute the probability mass.

You can do both by the way. Like you take your original distribution, do top-k so you only keep the top 10 or whatever, redistribute probability mass, and then do top-p. Depending on the shape of your distribution, doing one, the other, or both can make a big difference.

Now once you have your final distribution, after your rebalancing with temperature and truncating with top-k and top-p, you may want to characterize it, to describe it in a few metrics.

You’re likely familiar with the basic stats properties like the mean, the median, the standard deviation and so on. Maybe you even know fancier terms like skewness or kurtosis.

But for LLMs, the key metric for these next token probability distributions is the entropy.

The term originally comes from physics, specifically thermodynamics, where it measures disorder or randomness. From that framing it seems bad, and if you already have some baggage about the term like I once did, you’ll have to put it down for the LLM context.

The term gains a positive valence in information theory, where Claude Shannon adapted it. Here, entropy correlates with the amount of information a message carries, relative to a certain context. So for example if you have a weighted coin that always lands heads, then you’re never going to be surprised when I tell you the coin came up heads. But if I have a fair coin, then you’ll always be relatively surprised, since a priori you have no reason to believe heads vs tails. So in information theory, higher entropy is better, since it means the information is more valuable.

Let’s take that information theory understanding and apply it to LLMs. If my text so far is “The coin flip landed”, most of the probability is going to be split evenly between “heads” and “tails”, with a few other small terms like “on” or “near”. If we ignore those and just give 50% odds to “heads” and “tails”, then we get .5 for p(x). If you use base 2 for the log, which is customary, you get H = 1. The unit when using base 2 is bits, so that’s 1 bit of entropy.

If we work out the math for my example in the previous charts, we see more entropy, about 1.44 bits, since it contains more possibilities, but there is a clear favorite. If all four had the same odds, H would increase to 2 bits. If we had eight options all with the same odds, H would increase to 3 bits. Entropy is rising because we’re adding information about what could realistically happen.

Conversely, if one token had 97% odds and the rest just had 1% each, H would be about 0.24 bits, way lower than before, because the distribution would barely mean anything - you’re almost guaranteed to get the same token every time.

So we can measure the entropy at each position in our response. Some positions will have low entropy, where it was really predictable and basically guaranteed what token we would get at that position. Other positions will have high entropy, where it seems the model had a genuine choice to make and considered many viable options.

Entropy is important not only for measuring, but for training. In fact, for pretraining and for SFT, the entropy is how score your model!

Let me break it down. As we saw before, your model produces a distribution for the next token - what it thinks are the realistic tokens, for the given context, based on all the training data it has seen so far. In the formula above, we’re going to call the model’s distribution q, and we’re going to call the current token position x.

In addition to our model’s distribution, which we can measure and look at, there is also the true distribution, the likelihood of every possible next token given basically all the information in the world: all language ever spoken, the complete physical state of the person speaking it, the time of day, the weather, and so on and so on. On here, we call that p.

Of course that true distribution doesn’t concretely exist, in the sense that it can never be found and calculated. It’s more like a Platonic ideal.

But what we can do is treat any piece of training data as a sample from that true distribution. Like if I write an essay, that’s some real text from some combination of factors from the true distribution. So when we train on that, the model gets one particular view of the true distribution. And if we get lots of different samples, from all sorts of people and places and circumstances, we can get more and more samples from the true distribution. We’ll never fully build it, but we can keep getting closer with more data.

So when we model the true distribution, what we’re really doing is stringing all these individual discrete samples together into one smooth curve, which hopefully is close to the true distribution.

The difference between the samples of the true distribution p and our model q is the loss. And the loss has two parts: the entropy of the true distribution p, since it is changing all the time; and the distance between p and q. Those parts together form the loss. We have no influence on p of course, so loss is never zero, but we can keep trying to shrink the gap between p and q.

The Paper

This is the entire method for the paper. Hopefully some of the math looks a bit familiar, but I’ll break it down into English.

First, they take a model and use it to produce one response per prompt, with certain values for temperature, top-k, and top-p.

Then, they use those prompt-response pairs to do SFT.

Then they test the model, using potentially different values for temperature, top-k, and top-p.

That’s it! They don’t check the responses at all, there is no QA. It’s just raw synthetic data from the same model they train, with certain temperature and top-k and top-p.

Incredibly, this works. We’re going to come back to this table of results once we understand more about why their method works, but for now I wanted to flash the results.

So the core insight of the paper in my view is this: some parts of a response are very predictable, while others are very uncertain.

The most illustrative domain for their insight is code. A lot of code is pretty standard, pretty formulaic, because code has lots of hard and fast rules. Like in Python here, when you define a function, you have to write “def” and then a function name with parenthesis around the arguments and then a colon. If it doesn’t look like that, you’re going to throw an error. There are lots of rules in natural language too, but almost none of them are hard and fast; natural language is famously flexible, it has artistic license and can even contain mistakes like misspellings without destroying the meaning.

Of course, since code is just a way to solve problems and one problem can have many solutions, some code is quite flexible. As the authors show here, there are many different ways to sort a list of things, and which one to pick can be a subtle matter, or may be only a matter of taste depending on the circumstances.

The authors coin two terms for the ends of this spectrum: “fork” for the uncertain cases, and “lock” for the certain ones.

As the charts illustrate, forks and locks have different needs. At a fork, the model needs to explore, to have lots of roughly equal next token probabilities so that it can learn better and so that minor changes to upstream context can tip it towards a different path easily. So there we would like a higher temperature.

By contrast, at a lock, the model needs to be sure. Locks have one right answer, and all other tokens in the distribution are distractors. At locks, we want low temperature.

The problem is that we can’t adjust temperature on the fly like this - it’s a parameter we provide once for the entire inference, for the entire response. In practice we end up compromising, but it’s not optimal.

So you might think we could just make a better rule, right? Like if the distribution looks like this, then change the temperature like that, maybe ignore the tail as defined by a certain probability, things like that. But that’s not very ML of us, right? That’s not very Bitter Lesson, putting human-engineered rules on a thing that learns. Instead, what if we could teach the model to change its distributions? After all, producing the right distribution is already what we train models to do!

What if there was some way to train models to dynamically adjust temperature, top-k, and top-p, so that we get distributions more like this - flatter at forks, more peaked at locks, and always ignoring distractors. Or if it’s not literally learning different T, k, and p values, at least learning to change distributions as if it did learn different T, k, and p.

In the interest of reducing variables, the researchers try out a much simpler model: a finite state machine (FSM). A FSM is not a transformer. It’s barely even a model. Very concretely, each node on here is a set of sixteen numbers, each representing the probability of going in one of sixteen possible directions. They call the directions “tokens” to match LLM terminology, which is why you see “tok” and then a number on all the arrows here.

The idea here is they can construct trajectories to only be made of forks and locks. Like starting from the root, tokens 0 and 1 are both viable, so there is a genuine choice to make. But then after the choice, from either Fork-A or Fork-B, there is only one right choice to make - there are three locks in a row.

The way the model operates is by selecting from its “tokens” in the same way an LLM selects from the tokens in its vocabulary: take the raw probabilities, adjust with temperature and top-k and top-p, then pick based on these final probabilities. So you’re isolating just the impacts of inference parameters and training from all the other LLM stuff, and you have this fake, tightly controlled scenario.

Importantly, you can train this FSM using cross-entropy loss on sequences of tokens, the same way you would train an LLM.

So what is their crazy method, what information exactly are they gonna train their toy model on?

You might think they can use the PASS or FAIL information. That would be perfectly reasonable, given we typically do SFT on good examples, or if you’re still in the RL mindset where we’re thinking in rewards. But that wouldn’t be teaching us anything new!

Instead, they’re just going to self-distill, to take a bunch of trajectories, whether PASS or FAIL, and train on them. The key variables to adjust are the inference parameters: temperature, top-k, and top-p.

What they show is that if you train on trajectories with the right inference parameters, regardless of the quality of the data, you can teach the model to shape its probabilities better.

This is what they actually observe, not just an illustrative example. For the lock nodes, the model learns to drop distractors and put more weight on the dominant token. For the fork node, the model learns to spread out more amongst the options left after truncation.

I want to give a quick intuition on why this should work before we get to the empirical results.

Let’s start with self-distillation at T=1, with no truncation. All we’re doing then is getting unaltered samples from the model and reinforcing those. But since we’re not changing the samples at all, or filtering them like we would if quality were a concern, then training on them should produce no change.

However, with SSD, we specifically avoid T=1, and we do use truncation, i.e. top-k and top-p. So now the synthetic samples are different from what the model normally would say, so there is something to learn from.

So really SSD is just reinforcing and sharpening instincts the model already has. If the distribution looks like a lock, truncating puts even more probability on the top token, and the imbalance is already large enough that temperature shouldn’t change much. If the distribution looks like a fork, temperature spreads out probability more, but truncation ensures you don’t explore too far down the tail.

It’s not gonna help if the model is trash in the first place. And it does bank on there being some untapped potential within the model to then tap. But on the flip side, it’s embarrassingly simple and practically free compared to producing high-quality training data.

Now let’s take a closer look at the empirical results.

For all models they tested in basically every segment on the two version of LiveCodeBench, SSD helps. If you’re not familiar with pass@1 and pass@5, it just means how many chances a model had to get the problem right. Pass@1 measures accuracy in the way an end user would expect, like of course we want the one response we get to be correct. Pass@5 is more a reflection of model potential, like is it able to get the right answer.

Two trends to call out. First, within the instruct models, the smarter models benefited more. The authors don’t explain why, but I think it boils down to the whole “self-distillation” thing. Like if we think SSD’s function is to bring out the potential of the model, then smarter models likely have more potential. It’s actually pretty similar to how RLVR works, if you believe in the elicitation hypothesis, which I do. I wish they had actually done something similar here, where they measured pass@256 or some other really high number in order to see whether SSD was doing the same sort of internal optimization. We know it can’t be learning from the data in the normal way, given how this SFT data is created and how poor the quality can be, at least by traditional measures.

Second, the models generally improved more on the harder problems. Again, the authors don’t provide a direct explanation, but if we think back to the forks vs locks thing, we’d expect harder problems to have more forks. Like on an easy problem, not everything is a lock of course, but the distribution of positions is gonna be more on the lock side. I imagine for harder problems there will be way more forks, so if SSD helps with forks and locks, you can see why it might help with harder problems more.

Of course this doesn’t scale indefinitely; at some point, a problem is just impossibly hard for a model, and no amount of training is gonna help. So that might be starting to bite on the hard problems for Qwen3-4B for example.

To drive home the point about how the training data isn’t operating in the conventional way, they really dial up the temperature on some of it. As you can see on the left, a temperature of 2 can lead to complete gibberish, let alone unrunnable code.

And yet, if you train on this data, where 62% of outputs contain no extractable code at all, you still improve.

To give you an idea of how much better SSD is compared to just picking the optimal temperature, they do a sweep over temperature values on different benchmarks and models. In every case, the best temperature for the untrained model is worse than the performance of the trained model.

I also think this graph is helpful for providing an idea of temperature’s impact. 0.5 to 1.4 is a pretty big range, but for most cases the difference isn’t very big, and the big differences themselves fall in unpredictable patterns. Like why is there a seven-point drop from 0.9 to 1 for the bottom-right graph? Very strange.

My Takeaways

• This is the elicitation hypothesis but with trash data

  • How well it works depends on how much potential the model has left to bring out

• Surely there is a cap to this work

• The result is more instructive than useful

• What are we losing?

  • Leans more into priors

  • So the model may be worse past a certain difficulty - just sharpens the boundary