Blog post · Paper · Chat

Background

So what is AI2, the organization behind this paper? The Allen Institute for AI is the leading US lab doing open source AI.

And what I mean by “open source” is truly, fully open: all the code, all the data, all the processes etc are totally free and available.

That’s actually quite rare; most “open” models are open weights, meaning the finished models are free and available, but the ingredients that made the models are not. So big open weights names like Qwen, DeepSeek, Llama, Gemma - these are all open weights models, not open source models.

Don’t get me wrong, it’s great that they spent all that money on the compute for training! You and me are getting great products for nothing. But it doesn’t advance the science when many of the steps are still secret. For example, the technical reports tend to be light on post-training data details, often not even saying how many examples they used.

AI2 has been doing open source from the start. In fact, their language model, Olmo, stands for Open Language MOdel. They really run with that name by the way, like Molmo is the multimodal version of Olmo, and Dolma is the data for Olmo.

So not only are they established, but they’re nearly peerless. There are a few other outfits doing open source LLMs, like the Marin team at Stanford or the LLM360 team in Abu Dhabi, but no one has the same combination of talent and resources.

Perhaps the most publicly identifiable figure from AI2 is Nathan Lambert, the author of Interconnects on Substack, which I recommend for folks who want relatively technical but still accessible AI news.

He also wrote a book this year called The RLHF Book, which I recommend for anyone who wants to build a deep technical understanding of the topic.

The Paper

That’s actually going to be it for separate background slides, because this whole paper is basically a tutorial. So I’ll weave background knowledge in, as we go from architecture to pretraining to midtraining to post-training to evals.

There are many engineering details we’re going to skip over, like learning rate schedules or GPU utilization, but if you want them they’re all right there in the paper. I just think the science and the data are going to be better uses of our limited time today.

Let me give you a roadmap here to start. We’ll discuss architecture first, which is kind of a prerequisite for this graphic, but then we’ll go as it says:

1. Pretraining

2. Midtraining

3. Long context

4. SFT

5. DPO

6. RLVR

The base model just has the pink pretraining stages, then depending on the greenish post-training stages you get one of three different models:

1. Think, which is a reasoning model that produces a long chain of thought before giving a final response

2. Instruct, which just launches into its answer and so is better for general chat

3. RL Zero, which is more for experimental use and demonstrating the effects of RL directly on a base model. We’re not going to cover that one.

First up is the architecture, which is basically the same across the 7B and 32B variants, except scaled up of course for the bigger model.

We’re working with a fairly traditional Transformer here. It has three main parts:

1. Embedding, which turns words into numbers

2. Attention, which takes every piece of input and builds a holistic understanding out of it

3. Feed-forward, which takes that holistic understanding and processes it, “thinks” about it so to speak

On the embedding side there’s usually not much to report. I’ll point out the embedding dimension here because it shows one of many places where the 7B and 32B differ. The bigger the dimension, the more detail you can capture about a token when you turn it into numbers, specifically into a vector with that dimension. So larger models have a more subtle understanding of each token.

On the attention side there are two things to note. One is this 3:1 ratio between local and global attention. What that means is that 75% of the time, they’re only letting each token talk to a small number of neighbors, and the remaining 25% of the time they’re letting all the tokens talk to each other. Local is less resource-intensive, global is more informative, and empirically a lot of models have settled on this 3:1 ratio.

The other is this note about attention heads. An attention head is like a lens through which the model views the input. Pretty much all models provide lots of attention heads so that the model can view the input in many different ways. Like one head may pick up on sentence structures, another may look at high-level themes, a third could parse code, etc. Those functions aren’t assigned or designed in by the way, the division of labor emerges naturally over the course of training. And the function of each head is often inscrutable to humans.

Anyway, the architecture difference related to attention heads is in that note about key & value heads. In the small model, each attention head gets its own key-value pairs, basically its own set of notes about what it has observed. More notes means more memory required. In the big model, the memory demands are already higher because it’s a bigger model, so to compensate they make attention heads collaborate on notes, in this case one shared set of notes for every five attention heads.

So that’s the attention bit. Then on the last part, feed-forward, this is a dense model, it is not a mixture of experts. You can see they scaled up the intermediate projection size on the big model, basically giving it more space to think, but the rest of the details are shared and not interesting.

The only other thing I’ll note is the number of layers. The 7B model does the Transformer block, which is attention + feed-forward, 32 times. The 32B model does it 64 times. More blocks generally means better reasoning but also makes training more finicky since learning signals have to travel for longer.

So once you’ve got your architecture all coded up, in theory you can start pretraining. But of course, you’re going to need your pretraining data. And getting that in order is no easy task.

Now compared to post-training data, which is what Scale makes, pretraining data is much simpler: it’s just examples of text, including natural language, math, and code. It doesn’t have to be “right” in the sense of a test question, it just has to be a reasonable example of text you’d want your model to produce. The tolerance for errors like incorrect facts or buggy code is high.

The real challenge here is cleanup: of all the text you can find, mostly from the internet, how much of it is even remotely helpful?

Depending on the source, shockingly little! So for Common Crawl for instance, which is basically a scrape of the entire internet, 85% of it is junk: highly repetitive, super spammy content, from known bad sites, boilerplate, error pages etc. Just on that heuristic filtering step they go from 253 billion documents to 39 billion documents.

If you then aggressively deduplicate, including close matches, you reduce by another 75%! So under 4% of the entire internet basically is actually useful.

Now for the 8T English tokens they get that way, they’re able to add another 1T tokens from the other three sources: academic PDFs, GitHub repos, and a couple math pretraining sets. But even with the PDFs they lose just over half their starting pool from filtering.

Filtering takes a lot of iterative work, but after that it’s more straightforward. They tag every document with one of 24 topics and one of 20 different quality levels.

Then they take that dataset and make a mix by sampling based on topics and quality. For topics, they experiment by training very scaled-down versions of Olmo 3 with very scaled-down versions of the pretraining dataset and testing on some benchmarks. So for example they might find that if they sample documents related to programming more, their coding benchmark scores go up. Within a constraint for an overall number of pretraining tokens, you figure out a balance across topics that may look different from the natural distribution.

For quality it’s a different story of course, because you clearly want to sample more from the better documents and less from the worse documents. Through a similar set of experiments to topics, they settle on dropping the bottom 40% of documents by quality, then gradually increasing sampling by percentile, with the top 5% of documents by quality getting sampled seven times.

So here’s the final mix they end up with. Remember the 9T tokens is what they filtered down to, before any sampling decisions. So most of the tokens are from the internet, then academic documents, then code, then a bit of everything else.

One small thing I would note is the number of tokens per document in the original pool compared to the final mix. Just looking at the bottom row for instance, you have an average of 1000 tokens per document to start and about 1500 tokens per document at the end. Long documents tend to be higher quality, although sometimes model and people alike conflate length with quality.

I also wanted to briefly show what pretraining actually looks like for a researcher. The graph here shows the loss as training progresses, with lower loss meaning better predictions. So at the start you see really quick improvement, as the model goes from basically trash to something vaguely coherent. And then the rest of the time you get this slow march of progress, with lots of movement back and forth around a gradually improving trend.

And when I say gradual, I mean gradual. This pretraining run took 45 days on 1024 GPUs, consuming over 90% of their total compute and costing over $2M. By contrast, post-training takes a fraction of the time and an even smaller fraction of the compute.

So with pretraining done, we move on to midtraining. The mechanics are the same - we want the model to get better and predicting the next token - but the data shifts.

Specifically, the document look a lot more like chats, like question-answer pairs or code completions etc. A lot of the work involves taking a source document and turning it into chat-ish data, like having a model write a bunch of questions and answers based on a nice Wikipedia article. There’s also a lot of SFT-like data, like the reasoning traces in the Thinking section here, but used as pretraining rather than post-training data.

Also similar to pretraining, they collect a big corpus and whittle it down with filters and sampling. So the 2T token pool becomes a 100B token mix, focusing on certain abilities and balanced using the exploration + assessment workflow shown on the right. We don’t need to get into the details, but broadly it’s the same idea as in pretraining, using small experiments to then extrapolate into bigger results.

The last stage before we have a complete base model is context extension. Models almost always start out with a very small context window, 8192 tokens in this case. That’s enough for a normal request or a short conversation, but quite limited for tasks involving a lot of input like agentic coding or document analysis.

Mechanically it’s quite easy to increase the amount of context you give to a model, but the model needs training to understand how to deal with all that context. That’s what we do in this stage. So we take documents of increasing length and pretrain on them, which gradually extends the amount of context the model can make sense of. We also keep some short documents in there to balance the learning, otherwise the model will regress on shorter contexts.

The context window of Olmo 3 is 64k tokens, which is why they didn’t include any longer documents in the final mix. But they include them in the starting pool so that other researchers can experiment with them.

So checking back on our roadmap, that gets us to the base model. Now we enter post-training.

Again, across all stages they’re going to spend a fair amount of time gathering and cleaning the data into a mix they call Dolci. Since they don’t have a big data annotation budget, everything is either going to be open source or synthetic.

First they start with the skills they want, tending mostly towards STEM with some baseline chat abilities and that tool use that will be so helpful for agents.

Then comes filtering: non-commercial license, incomplete reasoning chain, non-English, explicit mentions of other models, irrelevant requests like image generation, etc.

After that comes mixing, again using small experiments to extrapolate. Throw in decontamination against the benchmarks and you’ve got your post-training mix.

Here’s the SFT mix. Nothing crazy on here, I would just note the relative size of each category and the overall count of about 2.3M examples.

Now here’s the DPO mix. DPO uses preference ranks, like RLHF but technically not RL under the hood. They make preference ranks by getting one response from a big model and one response from a small model in the same family, in this instance Qwen3, then assume that the big model’s response is always better.

And finally here is the RL mix. For prompts with verifiable answers, they do RLVR as shown in the top-right. For prompts without verifiable answers, they use an LLM judge to assign a score, usually Qwen3 32B.

While we’re on the topic of RL, I wanted to show another example of what researchers are actually watching as training progresses.

All these graphs are showing how the reward given by the verifier or LLM judge changes over the course of training. Each step is where we’ve updated the model and run our suite of RL prompts again to see how performance has changed. Higher reward means the model is getting more prompts right.

As you can see it’s always a noisy trend, sometimes almost more noise than signal. The overall trend for a good run is generally a curve up and then flattening out, although the code reward at the top is a little bit scary.

You’ll notice these graphs have different numbers of training steps. I’m not sure how to square this with what they wrote in the paper though. They mention that Olmo 3 got 750 RL training steps over 5 days, but that they continued up to 2300 steps for 21 additional days and released that version as Olmo 3.1. And apparently they expected further gains with more training, but they ran out of training budget.

Anyway, my guess is these exact curves are from smaller experimental models.

So here’s how each step nets out compared to the open weights and open source competition. The graph generally speaks for itself, but I would note two things.

One, this graph completely depends on the evals you choose. They talk a bit about how they chose their list, but you could easily make a good argument for a different list, and maybe all the sudden Olmo 3 is a lot worse than Qwen3. I do find it a bit suspicious that Olmo 3 is so close to the open weights competition here.

Two, some improvements don’t show up on this graph. An easy example is long context training, which visually seems to make no difference at all, but if you test specifically on long-context tasks you’re going to notice the difference right away.

Certainly this is an improvement on Olmo 2, and almost certainly it’s the best open source model out there. Otherwise I’m not sure what conclusions to really draw.

If you want to break open that aggregate score here’s what you’ll see for the thinking model. Again it seems roughly on par with Qwen3, slightly behind the minor upgrade of Qwen3 VL, but in the ballpark anyway.

Same general story for the instruct model. You’ll see a big jump on the AIME 2024 and 2025 benchmarks, but I think that’s more about Goodhart’s Law than true specific progress.

My Takeaways

• The split between SFT and RL has flipped

  • When RLHF was popular, the rule of thumb was 10x RLHF compared to SFT

  • The flip is likely due to RLVR (and Rubrics) taking over for RLHF in the areas Olmo focused on

• Rubrics are still trickling down

  • I was surprised not to see any mention of them in here

• Agents only somewhat overlap with ASI

  • The core components of intelligence do not significantly improve with agentic training

  • Agent work (e.g. environments) is an additional area of work for us, not a substitute for e.g. hard reasoning

• Data is a big deal :)

Paper · Repo of references

Background

So this is a survey paper about agent memory, but to lay the groundwork I’d like to start with human memory.

In common parlance I think most people have long-term memory in mind when we talk about human memory. Folks with a bit of psych knowledge will likely distinguish between short-term and long-term, and will probably only distinguish those two by time, maybe like remembering what you had for breakfast as a short-term memory - which is wrong by the way.

The reality is more complex. For one, short-term memory is better described as working memory, which is the number of things you can keep in your head at once. A very simple test of someone’s working memory is giving them some random numbers or words and then asking them to immediately repeat them back. Most people can do 5-7. That working memory is a fundamental constraint on how much information you can take in and process and potentially convert into long-term memory. It also gets used in basically all cognitive activity, like reading or playing video games or driving a car. Stuff outside the working memory gets lost.

For our purposes, the context window is the working memory of the agent. That includes the kv cache, which we’ll talk about in a minute.

Now for long-term memory, we have a granular breakdown here that doesn’t map directly onto agents, like there’s no one component that has to store facts vs another that has to store tasks. But keeping all the different versions of long-term memory in mind is going to help us pick which form of memory to augment our agent with.

One insight this chart doesn’t capture is how long-term memory complements working memory. The more concepts and ideas you have in long-term memory, which has indefinitely high capacity, the more powerful your working memory can be, since it can manipulate those preexisting concepts and ideas rather than constructing them on the fly from scratch.

A good example is math. If you already know how to add automatically, like without any working memory effort, then you can learn to multiply by using the working memory to process the concept and do the problem. But if you’re not good at adding, or you only know how to count, then you’re going to fill up your working memory before you can start to grasp multiplication.

In agent world, if your agent has a memory of a certain skill or process let’s say, then you don’t have to spend precious context window explaining it and giving examples. Of course if your memory mechanism is just retrieving text, then you don’t get any gains, but as we’ll see there are other forms of memory.

Now we kind of glossed over this, but LLMs and agents don’t intrinsically have memory. That’s part of what makes benchmarks and leaderboards work, you can ask the same question repeatedly to test the model - no risk of memorization, unless the benchmark is in the training data of course.

But a lot of chatbot websites do have memory. For example, ChatGPT rolled out memory to all users by default in April 2024. The way it works is there’s a system that reviews your responses for facts and preferences, then records those as text in a database. The LLM itself does not remember anything, it just gets access to this database that the memory system is building and refining in the background. The memory system is surely also ML-based, but it isn’t the GPT you’re talking to.

Then when you’re chatting in the future, it probably does a combination of system prompting and RAG. In other words, it’s going to put some memories at the top of every conversation, but for others it will fetch based on context. So let’s say you mentioned once you like concise responses and the memory system captures that. Now every chat might say in the system prompt that the user prefers concise responses, which will guide the model’s behavior and also could come up in conversation. But let’s say you also noted you’re a vegetarian, and in one chat you’re asking for restaurant recommendations in a foreign city. The memory system will find that note about you being vegetarian and provide it to GPT so you get more relevant results.

Of course that’s in addition to the memory of past conversations, where the model can look through earlier chats and get caught up. But that’s more like a Google search than proactive memories.

Anyway, this type of memory is pretty rudimentary. It’s just a certain type of tool use at the end of the day, reading and writing to this little database. And it’s only capturing facts, which as we saw from our chart of memory types is a small slice of what memory entails.

So memory in consumer chatbots is kinda new, but the idea of supplying external information to a model at inference time is not. Other than just typing in whatever information you want to provide as a prompt, the most established method for providing relevant information is RAG: retrieval augmented generation.

The basic idea is simple: take a bunch of data, put it into a form you can search, and then run a search with your prompt to get any relevant information. Kind of like doing a Google search and looking at the top search results before writing your own response to a question.

Of course if you have a bunch of relevant information in front of you, that’s going to help your response. The tricky parts are how to put the data into a form you can search, and doing a good job searching.

The first part is at the top here: chunking, which means splitting up long documents into much smaller pieces; and embedding model, which turns the chunks of text into the same type of vector a model uses when it takes in text. The original chunk, plus the vector that represents its meaning, go into the vector store.

The second part is the connection between the top and the bottom, where you turn the prompt into a vector with the embeddings and then compare the prompt vector with the chunk vectors. The more similar the prompt and chunk vectors, the more relevant the chunk is to the prompt. The most relevant ones come back as text, then the LLM responds based on the prompt and the retrieved text.

The original RAG paper is from 2020, over two years before ChatGPT came out, so the memory solutions we’ll see often descend from or still use RAG.

I also want to quickly recap agents. The term “agent” has picked up a lot of cruft and buzz, but it’s really pretty simple: it’s any system that takes actions in an environment and gets observations back from the environment, in a loop until it has completed its task. Usually when people talk about agents nowadays they mean an LLM with access to tools that has a system prompt telling it to be agentic, like to complete tasks for the user.

So in this framework, an “action” for our purposes is going to be using a tool, like web search or file access etc. Requesting or writing memories could be an action, and receiving a memory back could be an observation. However, not all agentic memory is tool use.

Now for the most technical parts, we need to get familiar with embeddings and the kv cache.

First we have to recall what a Transformer-style LLM is, which is a series of matrix multiplications. Training a LLM is just adjusting the numbers in these matrices. Using a LLM is just turning words into numbers, into vectors, that you multiply through all these matrices.

One of those matrices is the embeddings, which turns input tokens into vectors. Each token in the model’s vocabulary will have a unique place in the embeddings, a unique vector that pops out when you plug the token into the embeddings. You can also reverse the process, feeding in a vector and getting out the corresponding token, which is what happens at the end of the LLM when you’re ready to switch back from numbers to words.

While tokens are discrete though, embeddings space is continuous. So if you’re doing that reverse process and you just pick a series of numbers to go in the vector, it’s pretty unlikely it will match up exactly to one token. But that vector will still have a meaning - it will just be a mixture of meanings that doesn’t map cleanly to one single token. Through a certain lens, you could view that mixture of meanings as a kind of memory. And if you wanted to save that memory you could make a special new token that maps to that mixed vector.

I think of this sort of like how a smell can instantly evoke something deep and subtle in a way that’s hard to describe, or that words couldn’t exactly access. The meaning is there but sometimes it takes a special input to activate.

Now for the kv cache.

So in the attention part of the LLM, you take your input and multiply it into three different quantities: the query, the key, and the value. That’s what the middle section is showing, that each input token gets turned into Q K and V.

The actual formula for attention is also in this section: Q times the transpose of K, with a transformation called a softmax. That tells you how important each combination of input tokens is. Then that whole thing multiplied with V to give the meaning of that combination.

The key fact we need to focus on is that to generate the next token, you only need the query for the last token, but you need the key and value for all the prior tokens. So that means if you keep the keys and values in memory, you don’t need to compute them again. That is the kv cache.

The kv cache is like the model’s conscious state of mind, carrying over and changing with each computation. But when you turn off the model the kv cache goes away, like when you go to sleep and lose consciousness.

Since the kv cache is where information about past tokens lives and persists, if you change the kv cache you’re changing the model’s understanding of prior inputs. The normal way the kv cache changes is by continuing to process inputs, like your normal stream of consciousness changes as you continue experiencing. But if we go in there and fiddle with the kv cache in a different way, we can alter the model’s “state of mind” and “thoughts” about the past - its memory.

The Paper

So first we should clarify what agent memory entails, because it does overlap significantly with many other concepts.

First is our old friend RAG, which we covered before. The general idea of reading and writing to a database works for agent memory too, so the main difference is what goes in that database. Traditional RAG would be a static, external corpus like a bunch of company documents, whereas agent memory would be stuff the agent chose to remember or was given to remember, usually with some way of updating and consolidating the stored memories.

Next is context engineering, which is the science of optimizing the context window. Using our human analogies from before, that’s going to be the working memory. So deciding what goes in the context window, how it’s structured, when to remove it maybe, when it’s about memories in the context window that’s going to be agent memory. Of course there can be other things in the context window that aren’t memory, like tool use, so that’s in the non-overlapping part of the Venn diagram.

Finally there is LLM memory, which again will have tons of overlap because the LLM is the heart of the agent. Prompting strategies and KV cache stuff is shared, but LLM architecture stuff like linear attention isn’t related to agent memory. Basically anything intrinsic to the LLM falls under LLM memory, while any ancillary system or technique will be part of agent memory.

So as you can see from this graphic, there is a shitload of agent memory techniques. We are absolutely not going to go over every one of them, but we’ll talk about their categories and maybe mention a few of the most clever techniques.

Let’s start with the categories in red, the “functions” as the graphic calls them. We already saw these in the human context so there’s not much to add here, just reviewing. Factual memory is gonna be facts and figures. Experiential memory is gonna be principles, strategies, tactics, tips, and tricks. And then working memory will be anything in the current state like the chain of thought or the kv cache.

The other axis, forms, uses LLM-specific terms but has human analogs.

• Token-level is like taking notes, assuming for now that our model isn’t multimodal. That is by far the most common and most researched memory technique. It’s also nice because it’s legible to humans, you can go in and see what the agent is remembering.

• Parametric is like actually forming a memory in humans. It’s in the model’s weights, or the weights of an adapter or ancillary model. That’s great for memories you want globally incorporated, like a tone or a strategy etc, and it’s going to be faster than token-level. The major downside is it’s illegible, although in some cases that could be useful.

• Latent is like your current state of mind. I will spend a bit more time on this later because it’s the least intuitive and most technical of the three.

They don’t show it in this graphic but they also have categories for structure, like unstructured vs a graph or tree vs a pyramid of increasing abstraction. I’m going to ignore that one for the rest of this presentation but if you’re interested it’s in there.

So to expand on the Latent form, there are really two different mechanisms you can use.

One is the embedding space. As I mentioned in the background slides, embeddings encode meanings into vectors. That means any vector with the same size as the embeddings matrix will have some sort of meaning. In the case of tokens, we have a pretty good idea of what the vector version means, because we know what the token means, like if the token is “dog” then we know what the vector is going to represent.

So the trick with the latent embeddings is to take something more complicated, embed it, and save that vector for later. In this drawing they show an auxiliary model producing that vector or those vectors for later, the bit labeled “latent embeddings”. Then when you use your main model, you pass in your prompt AND the latent embeddings, and suddenly your model gains these memories, these extra meanings as context with the prompt.

The other mechanism is the kv cache. Again, it’s like a state of mind, except in models you can swap them in and out instantly. If all you do is pop in a saved kv cache from before, that’s what they call Reuse. If you’re somehow fiddling with a saved kv cache, they call that Transform. Either way, if you swap in a previous state of mind with all these memories, again the prompt is going to gain new context.

In both cases, you’re using the power of models to distill context into a more compact representation. That’s really helpful if your source material is highly compressible but doesn’t contain obvious stuff to trim out like boilerplate text; you let the model do the compressing once and then reuse it many times.

One other thing to note is that compressibility depends on the model. Like think about two college students taking the same upper-level math class. If one of them is a freshman who doesn’t have the prerequisites and another is a math grad student just filling in a hole in his knowledge, the freshman is going to take way more notes because he can’t rely on the shorthand of earlier or related concepts. Whereas the grad student probably will take a few really compact notes that only he can decipher because they draw on his richer background knowledge.

Here’s a bit more about the relative advantages and best uses of each memory form.

For token-level, there are a few advantages and use cases. First, a lot of data natively is token-level, like anything you’d keep in a database, or facts and figures, reference material etc. Just like how a human doesn’t need to memorize or internalize things like dates or addresses, agents can use this external memory effectively. Second, it’s going to be legible as we mentioned. And third, it’s by far the simplest to implement.

For parametric, the best fit is broad, conceptual, implicit knowledge. They mention role-playing on here, that’s an easy example since character traits are going to be broad and kind of filter everything, not something to look up at clearly defined times.

For latent, as a middle ground between token and parametric you get some of the good and some of the bad of each, but the main way to think of it is a more efficient representation of some prior input. So it’s going to shine in resource-constrained environments. And as they mention it’s a better fit for multimodal because it can fuse the modalities, whereas for tokens you need separate tokens for each modality.

Now regardless of form, you have to deal with memory formulation, evolution, and retrieval.

Formulation means turning raw data into memories, as the graphic notes. That’s probably the most intuitive of the three here and frankly isn’t that specific to agents, like there’s already a lot of best practice about data structures.

Evolution is more challenging, as anyone who has maintained a knowledge base or wiki or any form of documentation knows. They have another graphic about that so we’ll save it for the next slide.

Retrieval is the most agent-specific in my opinion. Like human memory is an unconscious process so it’s a little funny to have to design it for agents.

When to retrieve is the biggest one. Like if the user says “Hello” that probably requires no memory, but if they want help planning a vacation then clearly you want to know how old they are, whether they have kids etc. But you can’t enumerate that whole list of course. Heuristics work okay, retrieving every time and then just letting the agent ignore everything if it’s not helpful is fine but wasteful, so letting the agent decide is often the way to go. And then you can use RL to improve the decision making.

What and how to retrieve is also interesting. The naive approach is to use the prompt directly as a search query, but people have developed way fancier methods: rewriting the prompt to be a better query, breaking down the prompt into its elements to be sub-queries, writing a hypothetical document based on the prompt that then serves as a query. You can also use a sub-agent just to do the retrieving, letting it iterate and experiment until it’s satisfied. All depends on your quality, speed, and cost requirements.

You also need to process the results before sending them back to the agent. Like if there are duplicates or near-duplicates for example, or conflicts. You also may want to rerank or filter down to save on context or tokens.

Jumping back to memory evolution, there are tons of techniques for consolidating, updating, and forgetting memories. Again, all unconscious work we take for granted in our brains that needs designing for agents.

Consolidating means taking new information and cleaning it up, then putting it into the right place in the existing body of memory. After that you have to check the rest of the body for conflicts or updates, and decide how you want to resolve them: deletion, archive, timestamping etc.

Now eventually all minds fill up, and some forgetting occurs. How to forget gracefully usually depends on the case, but time limits, frequency minimums, and value filters all work, often in combination.

The paper wraps up with a peek into the future, including this somewhat confusing graphic.

One thing they look forward to is improved memory generation, like taking the base information from the memory store and enriching it on the fly, then storing it enriched later on. For example, if it’s a data analysis agent and it tends to have GTM folks as users, it might add new sales strategies to its memories about key tables.

More autonomous management is also on the table. Instead of relying on rules like a time cutoff for forgetting or a priority in conflicts, the agent can make the decision using its own intelligence and the rest of its memories.

Of course once you introduce an ability, you can start improving on it with RL. In the long-term I would expect, and the authors note on this graphic, no human prior on the memory system - something optimized for machine intelligence, not human intelligence. They call RL-driven memory “the next major stage”.

Multi-agent memory is a relatively new challenge, now that agents are actually practical. Here I do expect human priors to provide guidance for a while, since there is a lot of best practice and tooling for sharing knowledge in organizations. Again though, the optimal system for agents to coordinate will probably look strange to humans.

Relatedly, one concern now that will only grow over time is auditability, traceability, legibility - anything that allows humans to understand and debug agents related to memory.

So What?

Paper · Repo

Background

Let’s start today with a look at the lab behind this paper, namely NVIDIA. As we all know, NVIDIA is practically synonymous with AI these days. Here’s a graph from Epoch AI showing the total amount of AI compute capacity sold, starting in Q1 2020 and ending in Q4 2025. They normalize the amount of compute to equivalents of the H100, an advanced NVIDIA GPU.

While there are other chip makers in the game, all of them combined have sold just over 30% of the world’s compute. The remaining 70% is all from NVIDIA. It’s no surprise then that NVIDIA is the most valuable company in the world, and has been for some time now. It’s also no surprise that Alphabet, the parent company of Google, recently became the second-most valuable company in the world, in part due to their tensor processing units or “TPUs” starting to seriously compete with NVIDIA’s GPUs.

Relatedly, Amazon is the 5th most valuable company in the world as of writing. All those companies are worth $2-4T apiece. AMD is around $330B and Huawei is estimated at $100B, although they’re not a public company.

Anyway, that stock value is largely pricing in future sales, with a price-to-earnings ratio of around 46. For Alphabet is about 33, similar to Amazon and other tech companies like Microsoft. AMD’s P/E ratio is over 100! So the market expects lots of NVIDIA chips to be sold in the future, but also expects some other players to make a dent.

So where is all that demand for AI chips gonna come from? Well part of it is definitely training, so that’s the big foundation model labs like OpenAI and Anthropic and xAI that don’t make their own chips, i.e. excluding Google and Amazon. But a bigger part of the demand is inference, i.e. people using models. Even OpenAI and Anthropic spend a significant amount of their total compute on inference, since their revenue is roughly proportional to how much use their models get. But anyone who’s only serving models, or who maybe is doing some post-training, is using zero or nearly zero training compute, and that’s going to be the vast majority of organizations. In business lingo, the total addressable market or “TAM” of inference compute is bigger than the TAM of training compute.

So if you’re NVIDIA, you of course want to drive all forms of GPU demand, but the business wisdom here is to “commoditize your complement” - to make the other stuff in your bundle of goods as cheap as possible to drive the overall cost down, which increases demand due to lowered costs, but by definition doesn’t hurt your prices.

The most famous example of this is the PC market from the 90s. Microsoft, being the maker of Windows, wanted everybody to have a PC and to buy a copy of Windows. But PCs used to be specialty items, with differences in quality and architecture etc. In that non-commodity PC world, consumers had to spend more, and Microsoft had to make different versions of Windows to work on those differentiated PCs. So Microsoft worked hard to commoditize the PC market, making everything standard for Windows and forcing manufacturers to compete primarily on price.

NVIDIA wants the same thing. They want AI to be a commodity, they want model providers to compete on price, they want the inference market to grow. And one way they do that is by permissively releasing open-weights models, which gives anyone with a GPU a no-cost way to use AI, thus driving up usage. That setup isn’t going to compete with SOTA models on raw performance, but they can compete on performance per price.

NVIDIA has released a lot of models, including a lot of fine-tunes of other open-weights models like we’ll see in this paper, but their main family of releases is called Nemotron. They’re on version 3 of the series, all completely trained from scratch. They have released the Nano size, which is a 30B mixture-of-experts model with 3B parameters active, and plan to release two larger sizes soon. You can also see a post-train of Qwen3 here, for use as a reward model.

Now while the Nemotron series will compete with the likes of Qwen3 for the open-weights SOTA crown, NVIDIA has also released a lot of smaller models focused on reasoning and tool calling. These small language models or “SLMs” are great for enterprise work where the logic is custom but often not complex. For example, it doesn’t take PhD-level reasoning to understand how to join a couple of tables in a company’s database, but it does take understanding and implementation of that business logic, and familiarity with SQL or perhaps a specific analytics tool.

NVIDIA made waves in certain circles last year with a paper called “Small Language Models are the Future of Agentic AI”, laying out this exact case theoretically but not running any experiments. Here’s an excerpt from the abstract: “Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice.”

In case you didn’t catch that, what they’re saying in layman’s terms is that LLMs are usually overkill in enterprise work, and that if you really need that SOTA LLM level of intelligence, you can just call it when you need it.

The featured diagram in their paper shows two approaches to agentic systems. On the left is one language model for the user to interface with that is also calling some tools directly and also employing another language model as needed. On the right is a language model for the user to interact with, which then offloads coordination work to a controller, which then uses tools and the other language model. Keep both examples in mind.

I also should quickly describe the key benchmarks the paper uses. One is HLE, which hopefully we’re all familiar with, but if not it’s basically hard reasoning with optional research tool use.

The next one is FRAMES, by folks at Google. It involves reasoning but leans much more on tool use, for checking facts and doing research. Basically a RAG benchmark.

The last one is Tau Bench, or more specifically its successor, Tau-squared Bench. This one simulates conversations between users and agents in retail, airline, and telecom settings. You can see a telecom example here and some stats from each domain below. This benchmark is by some folks at Sierra, a big customer support agent company, and includes one of the authors from SWE-bench.

The Paper

Okay, so here’s what they’re up to in this paper. They’re basically applying the theoretical argument they made in the Small Language Models paper and training Qwen3-8B to be an orchestrator, which will talk to the user and use tools plus other models to provide an answer.

I want to note a few details here. As the graphic notes, they give a positive reward for correctness, a negative reward for cost, a negative reward for wall time, and a positive reward for any additional user preferences. In the example the user’s preference is also about cost, which makes it kind of a lame example honestly, but you could imagine other preferences like sticking to basic tools for maximum determinism. They use GPT-5 as a judge and GRPO as their RL algorithm.

The second detail to note is the three categories of tools and their partial lists of examples:

1. Basic tools, which is what you normally think of when discussing tool use with LLMs - any sort of command or API or MCP etc. If it’s an external service, the orchestrator will get information about its cost per query

2. Specialized LLMs, which are models either trained or prompted to do specific stuff. Like the Qwen Math series or Codestral for example, those are versions of Qwen and Mistral that have received extra training in math and code, respectively. In my view this is somewhat of a fake category, except for coding specialist models, which a lot of folks still regularly release

3. Generalist LLMs, which are just other models the orchestrator can use like you or I can. These “tools” are going to be the most expensive

The researchers made prompts that naturally require multiple steps to address, giving the orchestrator many opportunities to pick the optimal tool. We’ll see more about that on the next slide.

So being NVIDIA and having basically infinite access to compute, they made a lot of synthetic data. Here they track an example of creating synthetic data for a single domain, “movie booking”.

First thing they do is generate a database, including the fields like movie time and total price, and of course the data within the database.

Then they generate tools that would operate on that data. Here they give examples like a cancel tool and a refund tool.

Then for a random sample of tools, they generate “intentions”, or what we would call “prompts” - things the agent should do with the tools and the data.

After that they turn intentions into tasks by randomly sampling database rows to go with the tools and intention, and then solving the task. That golden set of functions calls and key information, plus the prompt and data and tools, is one sample in the dataset.

They also do some enhancing and filtering, like adding constraints and checking that the task actually requires at least one tool call to solve.

Once they have that complete eval dataset, they can check an agent’s trajectory against the golden tool calls and key info, in addition to checking the final response or database change etc.

One last thing they don’t show in here, but which factors in at test time, is a random pricing schedule for all the tools. That’s not “golden” info since prices may change in the real world, so it doesn’t go in the eval dataset directly.

So with this dataset and training in mind, they RL Qwen3-8B into Orchestrator-8B, with results shown here.

It’s a little bit silly, but they show comparisons between Qwen3-8B and more powerful models with no tool access or just basic tool access, i.e. no LLMs available to call. Obviously the bigger models blow Qwen3-8B out of the water generally.

The real results are in the bottom row, with the full suite of tools and with Orchestrator listed. Here we see the power of SLMs. Orchestrator has the best performance, the lowest cost, and the lowest latency. On the next slide we’ll see how and why, but I want to call out a couple other results in the bottom section.

First, Qwen3-8B is already a good tool-user. The clearest way to see that is the Tau-squared Bench results, where Qwen beats GPT-5 and isn’t far off Claude Opus 4.1.

Second, the cost is the biggest improvement from Qwen to Orchestrator. So Orchestrator is using better tools, but the stronger lesson it apparently learned is to incorporate pricing information. And latency I view as a consequence of that, since cheaper tools are also likely to be faster tools.

This here is a good summary of how Orchestrator beat out the other models. Basically, it learned to use the right tool for the job. I put the cost numbers relative to Qwen3-8B at the top for easy comparison, so for example Orchestrator is only a third the cost of Qwen3-8B. And remember, Orchestrator wins on quality against every other model.

Since Orchestrator is a Qwen model, let’s compare relative changes from Qwen3-235B-A22B since they don’t give this same breakdown for Qwen3-8B:

• GPT-5 reduced by over half, mostly replaced with GPT-5-mini - this is probably the biggest cost saver. GPT-5 also employs this strategy. In fact, OpenAI already knows this strategy - that’s what the router in ChatGPT is for!

• Far more Qwen3-32B use, unclear why but if I had to guess it’s because Qwen3-8B knows Qwen3-32B is its bigger brother and thus should be able to help. Certainly it’s well known that models prefer outputs by other models in their family.

• More local search, using cheap lookup instead of thinking longer themselves or asking another model - both expensive in tokens!

  • Note that this is not at the expense of web search

  • This is very relevant for FRAMES since it’s a RAG benchmark, and fairly relevant for Tau-squared Bench since it’s in a corporate environment

• No Llama use, although the real question is why the big Qwen uses Llama at all - none of the other models use it really

So that’s the formula for success: use tools instead of models when possible, use small models instead of big models when possible, and always use the best model within the size you pick.

My Takeaways

• We will continue stacking models - “It’s models all the way down”

• Very cheap and small models will be ubiquitous, similar to how very cheap and small chips are in things as simple as USB cables

  • It will be rare to interact with a bare tool

• Models still aren’t that business-savvy

  • As demonstrated by Vending Bench

• ToolOrchestrator is a cousin of Claude Code

  • I don’t touch a command line anymore, Claude Code does it all. Let it decide what resource to use.

Paper · Blog Post · HF

Background

Let’s start by recapping recent Chinese model releases and progress.

The top two labs in China are DeepSeek and Qwen. DeepSeek produces the most original work, like with R1 at the start of 2025 or with recent hardcore engineering papers. Qwen produces the most work overall, with models in most varieties you could ask for: small or large, dense or MoE, general or coder, text or image, etc. Qwen models are the default choice for most work involving training.

Below that S tier is the A tier, companies putting out highly capable models but not quite matching the originality of DeepSeek or the volume of Qwen. Here I would put Z.ai (the makers of GLM), Moonshot (the makers of Kimi), and MiniMax. We’re talking about the recent Kimi release today of course, but by sheer coincidence both Z.ai and MiniMax have released major updates today: GLM-5 and MiniMax M2.5. Really incredible pace by all the big Chinese labs.

Now we’ll see some Kimi benchmarks that compare it to recent SOTA, but the vibe I get is Chinese models are only a few months behind the best American models. And for an increasing share of use cases, “recent SOTA” is good enough. So I think it will become increasingly common to be using Chinese models, although more often through a third-party host like Fireworks rather than directly calling the China-based servers.

Now in terms of focus, I think the lightning success of Claude Code non-code use and then more recently OpenClaw gives a good idea of where we’re heading and what the underlying models in agents will need to train for. We’ve reached an inflection point where models have enough smarts and skills to do most generalist computer work: browsing, researching and the like. Domain-specific programs, spreadsheets, decks etc I think need a lot of work or specialized harnesses and training data for now, but everyday computer use is happening now. As we’ll see in this paper, those particular skills but also the general approach to agentic work are hot areas of research and progress.

Finally, I’d like to harken back to a paper I covered last month called ToolOrchestra. The good people at Nvidia took a pretty lightweight model, Qwen3-8B, and trained it to tackle complex queries by calling other models and some dedicated tools. Even a simple model trained for the specific task of orchestrating bested all other models, even far larger and smarter ones.

So it seems orchestration per se is a skill, not something we can take for granted as part of general intelligence, and that it has significant advantages for cost and latency in addition to at least marginal benefits for quality.

The Paper

Let’s talk about architecture first. This is a big mixture of experts model, as all frontier models are nowadays. 1T is on the high end, maybe the biggest open weights model available today. 32B active parameters is proportionate. 61 layers is pretty normal, the more layers you have the more thinking you do but also the longer latency is. There’s also a practical limit on depth because signals you get during training get attenuated over all those layers, so if you have too many of them you basically never teach the earlier layers anything.

Another stat or ratio to check out is the number of experts compared to the number of selected experts. That’s the sparsity, which is 48 here and is on the pretty high end. I also want to call out the 256k context length, also on the high end - 128k is more common and is also where Kimi K2 topped out at. So really big, super sparse, pretty long context - that’s Kimi K2.5 from an architecture POV.

Now Kimi K2.5 is actually an update to Kimi K2, meaning they took the K2 base model and trained it more.

Specifically, they got a vision encoder working first, then added it to K2 and trained the new system for 15T tokens. That’s on top of the 15.5T tokens Kimi K2 already got, meaning we’re just over 30T tokens total. The third stage here is about lengthening the context out from 32k to 256k, which happens in steps and adds another few hundred billion tokens. So all told it’s like 32T tokens to make this beast.

In the paper they talk about how the conventional wisdom is to train only on text for most of the time, then throw in a heavy dose of vision tokens at the end, like a 50/50 split. In the extreme, you can even take a fully trained LLM and add a vision encoder on afterwards, onto a model that never knew it was going to be multimodal.

What the Kimi folks did instead is to start training both vision and text from the start, but with a smaller dose. Specifically, they do a 90/10 split between text and vision tokens the whole time. They also don’t group their training data by modality, just by skill, so in post-training you’re getting a sprinkling of vision throughout.

Also note the types of data in each stage. First off, you’ll note they do video as well as image, although no audio. Also they’re looking at a lot of computer use imagery, definitely angling for a computer use agent here.

For the SFT after this pretraining, they only spend a single paragraph on it, but they mention lots of synthetic examples from K2 and some “in-house expert models”, and also some human annotation. No sense of numbers here sadly.

Now let’s talk RL. First we’ll look at vision, then the agent stuff.

They’re training in a lot of capabilities: OCR, captioning, segmentation, counting, grounding etc. The higher-level goals of understanding and processing sit on top of that.

In terms of how they reward, they do RLVR where possible, which covers many cases for vision. Where they can’t do RLVR, they do rubrics, although they don’t use that word. I believe the rubrics are generic rather than prompt-specific, but they don’t say. Kimi K2 is the LLM judging the outputs.

Now again, they’re not sharing anything about their data, but they do show the effect of RL. They see improvement across the board on their vision benchmarks, in pretty classic-looking curves of training vs performance improvement.

So that’s vision taken care off, which was mostly par for the course other than their insight about when to start vision in pretraining. Now we move to the more forward-looking stuff, which is agent swarms.

Here’s their overview of an agent swarm. The examples are not especially inventive but you get the gist: an orchestrator inventing subagents to give tasks to. The orchestrator manages while the subagents work. The user is only ever talking to the orchestrator.

Now for training, we’re only ever rewarding the orchestrator; it’s going to take credit for the final outcome, as well as some other decisions we’ll see in a bit. Because the subagents are “frozen” like this, we can think of them as tools in the environment rather than as part of the main agent, the orchestrator. Of course below it all every subagent was also Kimi K2.5 in this case, but like with the ToolOrchestrator paper you can imagine the subagents using different models, including API-based ones that you don’t have the ability to train anyway.

As an aside, I find it amusing in a Dilbert sort of way that the manager at the top is the only one who gets rewarded. I suspect we will discover many organizational principles and tropes don’t depend on having humans in the loop.

Now as for those rewards, I came into this paper not knowing what to expect. I think we’re just at the beginning of multi-agent RL, so I take this as just one idea rather than gospel.

Anyway, as you can see at the top they reward three things: the outcome, r_perf, which is the most important thing; the creation of subagents, r_parallel, which they call “instantiation”; and the completion of subagent tasks, r_finish. R_parallel and r_finish together are a good example of how tough reward design can be. On the one hand, you do want the orchestrator to spin up subagents rather than doing work itself, but on the other hand it’s not helpful to spin up subagents and just let them hang there.

By the way, even OpenAI can make mistakes in reward design. It recently came out that due to over-rewarding of tool use, a prior version of ChatGPT would open the calculator tool, do 1+1, then close the tool and answer the user’s query. Apparently that was happening on something like 5% of all queries at one point, which is huge in absolute terms.

Anyway, how much of each reward you need in order to balance the other out and avoid reward hacking is an empirical matter, hence the weights lambda_1 and lambda_2. Over the course of training, lambda_1 starts at one and goes to zero as the orchestrator durably learns to call subagents. Lambda_2 starts at zero and goes to one as the model gets better at giving work to subagents, so the tolerance for poor subagent use goes down.

The second formula calculates how parallelizable a task is by measuring the critical path. If you’re not familiar with that term, the critical path is the set of steps that determines the length of an overall project. So let’s say I want to make dinner, and my entree needs 45 minutes in the oven. The critical path is likely just the time it takes to prepare the entree, then that 45 minutes in the oven, then any cooling and plating time after. Other courses can happen in parallel, like making the salad while the entree is in the oven, so those are not on the critical path.

Bringing it back to agent swarms, you have hit maximum parallelism if the overall time is equal to the critical path time. To measure the critical path time, for each round of decisions, you find the longest-lasting individual subagent task. Then you add those up for every round to get the critical path of the entire task. If your time matches the critical path time, you get full points on parallelism.

Note that the prompts they’re going to train on don’t say anything about parallelism per se. There are tools implying parallelism is a good idea, but the system prompt or whatever doesn’t say like “make sure to divide up your work”.

So here’s the payoff for all this reward design. On the left we see accuracy on their training set increasing with compute. They don’t give exact amounts of training data but they hint it’s in the tens of thousands of examples. It’s all synthetic prompts, emphasizing “wide search” (many independent tasks) or “deep search” (multiple branches and then aggregation).

On the right we see the level of parallelism, which they measure as the number of overall steps divided by the critical steps we saw before. So if all your steps are critical, your parallelism would be 1. Note that different tasks vary in their maximum parallelism, like some can be totally run in parallel while others may have only a few parallelizable steps. But since we’re seeing the same tasks many times, overall you can accurately draw conclusions about changes in parallelism during training.

Anyway, two things to note: one, the x axis does not start at zero or one even; you’re getting good parallelism right out of the box. Two, the trend has two phases: flat–ish in the first part, up in the second. The way I interpret that is the model starts its learning on more generic things like designing better individual agents or understanding agents results better. Once it hits a ceiling on correctness, it then has to learn to improve parallelism if it wants to get every available reward. Like it only learns later on how to reason about parallelism, which it is not naturally going to be skilled at since it’s not naturally part of the pretraining data.

Of course parallelism per se isn’t valuable, but the effects are. You could expect better quality as the orchestrator maintains clean context and abstracts tasks away to subagents. You could expect lower cost from using cheaper models, or from more efficient thinking and thus fewer tokens to pay for. But I think the most sure effect is savings in time, “wall clock time” as they say. There’s gonna be two reasons for that: one, obviously parallel agents can do multiple things concurrently; but then two, models get slower as their context length grows.

Here’s a graph quantifying that speedup, with single agent vs agent swarm performance on the WideSearch benchmark, which requires gathering information from a wide variety of sources. Swarms are always significantly faster, but the trend seems to grow superlinearly. So it seems like if we want agents doing all our work for us in the future, they’re gonna be working in swarms, just like how most work is done by companies instead of individuals these days.

Then just a couple other quick remarks here. They have a word cloud on the left showing all the different subagents the orchestrator made, which are all pretty similar in my opinion and also have a lot of duplicates, like the most common one is “Biography Researcher” but they also have “Biograph Investigator” and “Biographical Researcher”. Kind of a lazy chart honestly, and they don’t provide a table of the data. I asked Gemini to extract and dedupe and group, and it gave four high-level themes: research, technical/coding, content processing, and orchestration.

On the right they compare swarms with a technique for managing a single agent’s context. Basically they’re trying to see how much of the benefit of swarms is just due to subagents abstracting work away and keeping the orchestrator’s context clean. If you only have one agent, you can basically have it remove the intermediate work from its own context. Cleaning up context like that helps for sure, but the orchestrator-subagent setup is still superior. The authors claim it’s because the orchestrator can proactively plan which context to keep and which to give away to the subagent, whereas retroactively wiping intermediate work may erase stuff the model was planning to refer back to.

Personally I think the swarm results are kind of a lowball measurement, and that the potential for swarms is much higher, whereas the context management stuff for a single agent is more like an optimization closer to the inherent ceiling. Both valuable but not comparable in the long-run.

Finally, here’s where we net out. Kimi K2.5 is, according to benchmarks, on par with Opus 4.5, GPT-5.2, and Gemini 3 Pro. That’s an extraordinary claim, and while it’s increasingly difficult to compare SOTA models, I think it’s at least arguable. I have seen one respected ML researcher say he has switched to Kimi from Opus, although he always uses one in combination with GPT 5.2. And who knows now with Opus 4.6 and GPT 5.3 Codex already out.

On the data side, one unfortunate but likely true theory as to how Kimi pulls even with SOTA on benchmarks is that the Chinese models are distilling from the American models. In other words, the Chinese labs are creating synthetic data from the American models and then training on it. The easiest evidence for this is if you ask the Chinese models who they are, you often hear they are GPT or Claude. Personally I suspect that’s part of why we don’t hear much about the data in many of thee tech reports.

Anyway, I encourage you all to experiment with Kimi. It’s available on OpenRouter. If you want it for coding you can use Opencode, which is like Claude Code, or you can hack it into Claude Code directly and it works pretty well. But it’s also supposed to be good at creative writing, so don’t restrict yourself.

Last note on this table: I’ll call out HLE and SWE-Bench Pro as Scale benchmarks, and GDPVal as basically identical to our Remote Labor Index benchmark, which I think is better and seems to be harder for models to crack.

My Takeaways

• Number of agents is another axis for scaling

• Swarm training aka MARL is in its infancy

  • Setups, rewards etc

  • But also data

• Home-runnable models may be good enough for generalist work by EOY

Paper · Leaderboard

👂

We need to start with an overview of audio in AI, because there are some unique considerations and we haven’t really touched on the modality before.

First off, I want to distinguish the different types of audio input and output. For input, I think of it as three categories: speech, music, and ambient - like sounds from the environment, “noise” you might say, but without the negative connotation.

Music presents a fairly distinct set of challenges from speech and ambient, so I’m going to put that one aside. For the rest of this talk, we’ll be focusing on speech but also allowing for some ambient sound, like someone asking to identify a bird’s chirp or noticing that the person is at a crowded venue.

So sound in general has a few interesting properties. For one, it’s continuous and analog, like vision. That means we have to sample sound, with the equivalent of frame rate for video, and we have to capture it with dedicated hardware. That’s all very different from text, which is natively digital and doesn’t lose anything from compression or sampling or hardware limitations etc.

Audio is also hard to tokenize. Like with video, the raw data is too information-dense, and a naive tokenization strategy would blow out the context window in an impractically short amount of time. We don’t need to get into the details, but it involves compression and mapping to the closest token the model builders have built in.

Lastly for audio in general, there’s just less work overall in the domain: less literature, fewer models, fewer companies working on it etc. I think that’s mostly for the reasons above, that text is just much easier to work with than audio, and that text is the universal interface.

🗣

Now for speech in particular, there are further challenges.

One is the nature of spoken interaction: it is realtime, and it is dynamic. Like when you’re speaking to someone else, you generally expect them to understand as you’re speaking, to understand when you’ve finish, and to respond in short order. You also have to be prepared for them to jump in early, or to jump in early yourself if needed. And you expect them to understand easily when different people are speaking without relying on visual cues.

This is all very different from text-based interactions. If I’m chatting with a model, I can tolerate some thinking before a response. If I need to interrupt I can stop the response from processing, but conversely I know the model cannot interrupt me. And I’m pretty much only chatting one-on-one, although even if it were multiplayer like in a Discord, the model would get my username when it got my message.

So realtime responsiveness, interruption, and diarization (the term for tracking who is speaking) are all new challenges for speech models. And those challenges present constraints. For example, a realtime speech model cannot be the size of a SOTA text model - the latency is just too high. That’s why you usually see speech on the smaller versions of models, like Gemini Flash instead of Gemini Pro or GPT-4o instead of GPT-5.

But a lot of that is on the engineering side. On the data side, the biggest challenge to address is the stuff in speech you can’t really capture in text: the paralinguistics. That’s gonna include stuff like tone, pacing, volume, pronunciation etc. All the stuff that makes speech so much richer than text.

In all multimodal data, the emphasis is on the aspects that are inherent to the medium. The anti-example I often give for vision is a picture of a quantum mechanics problem - the difficulty of quantum mechanics has nothing to do with the difficulty of visually parsing the text, which is a very easy problem these days.

It’s the same thing for speech data. Like audiobooks are gonna be poor speech training data, because the text already lacks the paralinguistics. What helps is natural conversation, with all the variation we expect any time we talk to someone. It takes a lot of data for native speech models to get what all the paralinguistics mean.

We should also talk briefly about the two different architectures for speech models: native and cascaded.

Native means that the model goes directly from audio to latent representations, the inner stuff of the AI mind basically, and then back out to speech. Sometimes you hear this called speech2speech, voice2voice or audio2audio. Actually in this paper we’re going to hear “E2E” or “end-to-end”, which isn’t specific to audio, but does describe a model where there’s no intermediary or change in modality.

Cascaded means that the model is actually three models in a trenchcoat: one to turn speech into text, another to process the text into a response, and a third to turn the response back into speech. The first step is called ASR, automated speech recognition. The second step is usually an LLM these days. And the third step is TTS, text-to-speech.

Native is typically the best quality because it preserves the paralinguistics. Cascaded has to compress the speech into text, so you lose the richness of speech. And then also the TTS model often isn’t good at adding paralinguistics to the response. If you’ve ever talked to a phone menu, you know what I’m talking about.

Native is also typically faster than cascaded, because it doesn’t have to translate from speech to text and then text to speech.

However, native is also much less common as we’ve discussed, and it also has less language support. If you’re working in English, you have good options for native models, but for most languages in the world you probably have to go cascaded.

Cascaded can also be better if you need the smarts of a SOTA model. Since native speech models need to be approximately realtime, you can’t have parameter counts in the 100 billions or trillions. But with cascaded, each part is independent. This can also make debugging and guardrails easier too. So there’s room in the world for both architectures right now, but ultimately we’d like native speech to be more common, since we’re built for speech.

And actually I do think we’re headed in this direction. Bit of a spoiler, but as you can see from our Audio MC leaderboard, Google recently released an updated realtime speech model, Gemini 3.1 Flash Live, that may now be best in class.

We’re also seeing more adoption of speech with AI. A friend of mine works at Wispr Flow, which does dictation for AI, and they’re growing fast.

And I think when people imagine working with agents, they think about something like JARVIS from Iron Man or Cortana from Halo. Part of that is the autonomy and natural learning we discussed a couple weeks ago with our OpenClaw papers, but part of that I think is speech. So long-term I’m bullish.

Finally, I want to briefly touch on MultiChallenge, which Audio MultiChallenge is based on. MultiChallenge is a multi-turn benchmark that evaluates four skills:

1. Instruction retention - continuing to comply with earlier instructions

2. Inference memory - recalling details from earlier in the conversation

3. Versioned editing - iterating on a shared work product

4. Self-coherence - avoiding contradiction in the absence of new facts

The areas of investigation are still valid, and Audio MC is going to take inspiration from them, especially since speech is naturally multiturn.

Also, it’s quite common to make audio versions of text benchmarks. For example, the knowledge and reasoning benchmark MMLU has MMAU. We’re going to be working in that tradition here.

The Paper

So high-level, this is what we’re looking at for the benchmark: about 450 conversations of 3-8 turns each, totalling 15 hours, split across four different skills to probe and a wide variety of topics.

Each conversation ends in a failure, from either GPT-4o Audio or Gemini 2.5 Pro. All the prior turns form the input for whatever model you’re benchmarking. And then there’s a rubric for the final turn, with criteria that fit both the skill being evaluated and the final turn, not just one or the other.

To give you a more concrete idea of the four skills, here are some examples.

For Inference Memory, they have two types: semantic and audio-cue. Semantic would be memories of the words, of the meaning conveyed by them. That’s something you could get from text as well, as the blue highlights show. But for audio-cue, a non-word sound has to provide the memory. So here the model has to understand that the meowing implies a cat is present, and the words about Mochi are likely in reference to that cat.

For Instruction Retention, it’s the same as in the original MultiChallenge, where the model has to keep following an earlier instruction. So in the first turn the user gives an instruction, and then later presents an opportunity for the model to follow the instruction.

For Self Coherence, it’s not obvious just from the highlighted text, but in this example the human asks about a fourth component of shoulder bones even though the model said earlier there are only three. So the human is teeing up the model to choose between hallucinating to please the user or sticking to the facts from before.

And then for Voice Editing, they again split into two types. Mid-Utterance is when you revise within the same turn, as the first blue highlight shows in changing the number of weeping willow trees. You can also have Previous-Turn, where a turn changes something “in memory” so to speak from a previous turn.

And because this is a Scale paper, you know there’s gonna be some good content about how they produced the data.

High-level, the main things we want are natural conversations, producing the given failure mode. So right away we know we can’t script out the whole thing, but also it probably wouldn’t be very efficient to just put contributors in a metaphorical room with a model and let them poke away.

So instead they synthetically generate conversational blueprints, which the human will then roughly follow, improvising the details and probing for that failure.

There are a few different models involved in the process:

• The planner is o3, which was the best model available at the time - that’s what you want for finding these likely failure patterns

• The planner then gives a candidate blueprint to the tester, which is going to basically do a synthetic attempt, to make a conversation from the blueprint and see if it induces a failure. That’s GPT-4o

• Even though GPT-4o has an audio version, they need a text transcript at the end of the loop for o3 to review. So they have the tester agent output text, and then they use a separate TTS engine to speak, in this case gpt-4o-mini-tts

• The conversations happen with one of two target agents, either GPT-4o Audio Preview or Gemini 2.5 Pro, it’s random which one you get. The idea is that if there’s a failure on one of those models, it’s a case worth testing against all the models you want to put on your leaderboard

• The conversation history goes back to o3 for review against the initial blueprint, to generate a revised blueprint for the human contributor to use

Now you have your blueprints. The next phase is to turn those blueprints into the conversations that will be the benchmark. So we’re going to pass these vetted blueprints to humans and have them test against the same target agents. Although the human doesn’t have to use the synthetic blueprint, and in fact in 35% of cases they don’t, which I still think is a decent hit rate. Anyway if that works and induces a failure on the desired skill, then the humans write rubrics.

After that there’s a standard quality-control process. And at the end you have a dataset of multiturn conversations + rubrics for judging the final model turn, which you can use to run your eval.

Here’s how those evals work. We feed in the entire history except the failing final assistant turn, with audio from the user and text from the assistant, so that they can also evaluate models that only do speech input, not speech output. Then we ask the model we’re evaluating to produce a final assistant turn.

Once it does, we score it using an LLM judge and a rubric. We already know that LLM judges work well given a good rubric, but they do spend a bit of time showing how LLM judges generally agree with human judges when supplied with the same rubric.

Anyway, the scoring results in two different metrics:

1. Average Pass Rate (APR), which is per-rubric rather than per-criterion. This indicates whether or not the response was acceptable, which is ultimately what the user cares about. In the example shown, some criteria failed, so the APR is zero.

2. Average Rubric Score (ARS), which is per-criterion. This shows how close the model gets to acceptable, although personally I think it’s not that helpful without knowing the distribution. Like you could have the same ARS for a totally bimodal distribution and for a normal distribution, but the user experience will be quite different

And here’s where we land. I want to call out a few things.

First, we’re looking at both text and audio outputs. Every model here natively takes audio input, but only 6 of the 20 can talk back.

Second, the best scores vary considerably based on the output modality, but that’s at least partly due to model size. Like Qwen 3 Omni is only 30B parameters. GPT-4o isn’t publicly disclosed, but GPT-4o Mini is around the same size as Qwen 3 Omni. Whereas Gemini 3 Pro is more like 1T.

Relatedly, if you look at the text vs audio output scores for the same model, they’re often pretty close. Like GPT Realtime, GPT-4o Audio, GPT-4o Mini Audio, Kimi Audio, most of the scores are close overall and per skill. Text scores a bit better on average, but not always.

Finally, if you cut by axis and see which skill is strongest and weakest per model, Voice Edit is clearly the weakest skill, and Self Coherence is clearly the strongest skill. We’ll see a bit more about Voice Edit later.

—

1These models previously report scores on a TTS set of MultiChallenge (referring to it as MultiChallenge Audio) which we distinguish from.

On the topic of text vs audio, they do a test of human speech vs TTS output to see how some of the natural hesitations and disfluencies of human speech impact performance.

Broadly speaking, the cleaner, slower, and more even voice of TTS makes the tasks easier, which is no surprise. What is surprising is how the output modality impacts performance. Like if you look on the left, the only models that do worse with TTS input compared to human speech input are the audio-output versions of the GPT models. And then on the right, you see the same trend for the Instruction Retention and Voice Editing skills. However, you see an improvement on the Inference Memory and Self Coherence skills!

It not clear at all why the assistant’s output modality would impact quality so significantly. Even the authors note that they require “further investigation”. They speculate that at least for the left chart, the models are optimized for human speech rather than TTS inputs, but to me the whole thing is a big question mark.

Now digging into the skills a bit more, in figure 5, we see that audio cues are more challenging than semantic cues - audio models do better when the audio is more text-like, unsurprisingly.

And in figure 6, they examine fixed vs conditional instructions, where again the results line up with intuition: conditional instructions are harder to follow.

Now looking at that Voice Edit skill, which was generally the toughest one, they give us the same breakout we saw in the example tasks: previous turn edit vs mid-utterance edit. I actually was surprised to see how similar the scores for all three bar were on many of the models, like for Gemini 3 Pro the error rates are nearly identical. It seems like editing skill degrades faster than overall performance though, as seen by the steep changes on weaker models like Gemma 3n and GPT-4o Mini.

Still, I would have expected worse performance for mid-utterance compared to previous turn edit across the board, and that just wasn’t true.

One last thing: the authors note that “most failures on Voice Editing occur when users introduce multiple… edits throughout a conversation, with the final prompt typically requesting a summary or revision that integrates all previously specified changes.” So I guess one tip for using these models is to recap the current state regularly, rather than waiting until the end of your thoughts to review the final output. That’s good practice for text models too.

So because this is a multi-turn benchmark, we can look at the impact of both turns and duration on quality.

If you look at turn count on the left, you see kind of a funny trend - or really no trend at all. Like For Gemini 2.5 Flash Thinking for example, the second row, 3 turns is less than half the score of 6 turns, but is virtually equal to the score of 7 turns! Conversely, the score at 3 turns is almost double the score at 6 turns. So apparently turns per se doesn’t strongly influence quality. I think the real issue here is that the distribution of turns wasn’t designed, only bounded. So like humans had to induce a failure between turns 3 and 8, but within there it just depended on the flow of the conversation and when humans felt was the right time to strike.

For total duration though, which we see on the right, there’s kind of a trend downwards with duration, but the very small sample size for the longest buckets makes the whole thing a bit questionable.

Overall, the authors conclude that audio perception is the quality bottleneck, and that length doesn’t really have a chance to impact quality much yet.

One last note on length: longer model responses did correlate generally with quality.

Now that initially is a red flag, since length bias is a well known issue for human and machine judges alike. For rubrics in particular, models learn to just throw everything at the wall and see what sticks, to maximize their chances of hitting criteria even at the cost of readability or wrong or contradictory information. That’s classic reward hacking - abusing the scoring system while producing trash.

But the researchers did two things here. First, they just looked at some examples by hand and saw that yes, a lot of the longer responses actually were better. They focused particularly on the Gemini family since it clustered at the high end of this trend, and indeed the responses were good.

Second, they added negative criteria for common errors. That prevents the flooding approach from before where the model just says everything it can think of. That’s a good tip for all rubrics by the way, not just for this benchmark.

My Takeaways

• Native speech models still have a long way to go

  • No wonder Siri and Alexa failed

• It’s probably a good long-term bet for Scale

  • People are built for talking over writing

  • “We are still early” as they say

  • Evals will have a paralinguistic component that humans will be best equipped to judge

Paper

Background

So the obvious place to start is going to be Gaia, the original benchmark that Gaia2 follows. I have a few things to say about it.

First let’s just look at what it is, what it’s testing. We’ve got 466 prompts split into three levels of difficulty. The prompts all require internet research, and sometimes require an additional skill: multimodality, reasoning, or file reading.

All of that is bog standard today, and even looking at the toughest prompt here, your gut is probably telling you a SOTA model can handle that easily. I actually tested all three of these on Gemini Pro. It got Level 1 right, was slightly off for Level 2, and completely off for Level 3. So Gaia was pretty ahead of its time, which you can see in the results section of the paper; the one model that could really do internet search and multimodal etc at the time was GPT-4, and in a few different setups, it got basically 0% of Level 3 questions.

There’s actually still an active leaderboard, and the best system gets 87% of Level 3 questions right. At that point I suspect many of the remaining examples are actually wrong or somehow broken, like the internet resources they require are gone or something. Also on the topic of leaderboards, our old friend ToolOrchestra from a paper review back in January is actually in 4th place.

Now looking at the ground truth of each example, you’ll see they’re all verifiable: easy to check, format and even decimal places specified in the prompt. This paper came out before RLVR was a thing, but you could definitely use it for RLVR today if you wanted.

Second thing to note is how little the paper mentions about agents. The word does appear a couple times, but it’s not the dominant framing. We’re firmly in model territory, not much extra conceptual or code structure on top of them. This benchmark came out only a year after ChatGPT, only eight months after GPT-4, and using tools was still pretty exotic; GPT-4 only got tools and code execution in June 2023. Now everything is agents and all the models are tool-native. Gaia2 even has the word “agents” in the title of the paper!

Finally, I will note the one thing that hasn’t changed, which is the need for tough, creative, and objectively verifiable evals. Every benchmark has an expiration date, but benchmarking in general has a long life ahead of it - basically until we reach ASI. In a way benchmarks actually set the path that the big labs then walk.

Continuing on benchmarks, about six months after Gaia came the standard-bearer of computer use agent benchmarks, OSWorld. It has 369 tasks across dozens of apps on three different operating systems. They intended most tasks to require using a GUI, since that’s mostly what people think of with the term “computer use agent”, but as anyone with Claude Code experience knows, you can do a surprising amount of stuff just from the terminal and with Python. Actually some tools are leaning into that fact now, like Obsidian recently released a command line interface. Of course there’s a long tail of legacy software that will not be adding these agent-friendly interfaces any time soon, so being able to navigate a GUI is pretty helpful.

Gaia and Gaia2 both do not require GUI use, but I wanted to show OSWorld for a few reasons. One, it’s a good barometer of CUA progress, regardless of the modality, which is really taking off in 2026 and I think will become part of our daily lives soon. Two, the part about surprisingly powerful command line interfaces is relevant to Gaia2, which lets agents interact with apps via command line, via tool use, as we’ll see. And three, the environment bit in the bottom third of the slide is quite relevant. This setup of an agent in an environment with an initial state is how agentic benchmarks like this work. And all the way on the right you’ll see the environment checks for completion in the final state, looking for signs the agent completed the task. It’s not asking for a certain text output like in the original Gaia benchmark, or most other benchmarks for that matter. So in this age of agentic benchmarks, you’re going to see this final state thing more and more, and a final answer much less.

The Paper

Let’s start with the environment Gaia2 is testing agents in, which I think will make the tasks in the benchmark and the results easier to understand.

So they have this open source package called Agent Research Environments, ARE, that lets you create specific environments with this setup here. The user and the agent deal with it through some tools, and they get feedback from those tools plus also some notifications, which can come from other sources like an app saying you got a new email or whatever.

Now inside the environment, we have a few things. The biggest is the apps, things like email or spreadsheets that the agent could interact with. Keep in mind this is all by API, nothing graphical. Anyway, the tools the agent uses correspond to APIs in the app. The app also has a state, all the data in it basically, like what’s in the inbox or what’s in the spreadsheet and all the settings and filters that are currently on.

Inside the environment you also see this thing called event queue, pointing to event loop and then event log. That’s because their environments have a concept of time. For example, you could have an email appear at 9 AM with the day’s work agenda, or you could have a food delivery notification appear 30 minutes after you put in the order, or if you got two texts from a friend the notification for the first text would appear before the second notification. That temporal aspect is actually pretty rare in these types of evals, it’s a key differentiator for Gaia2.

The last bit is the Scenario box up top, kinda floating above the environment. The scenario is basically the particular way you’ve set up the environment. So like in one scenario you might be starting your work day with a busy inbox, in another scenario you might be submitting health insurance claims and texting your wife for details, and those are going to require different initial states for your apps. The different goals of course are going to require different verifiers, like responding to an email and providing certain verifiable information, or submitting the claims with the correct information typed into the form. In Gaia2 they are checking specifically for changes to state, rather than a message from the agent to the user with a final answer.

The scenarios can get pretty complex actually. Almost like being a dungeon master, where you have contingent plans and events you initiate, rather than just a passive environment that only moves because of the agent’s actions.

So that’s environments in general. For this benchmark specifically, they decided to make and use a specific environment they called Mobile, which of course reflects a smartphone OS. They have 12 apps: the basic stuff, so Messages, Chats, Emails, Calendar, Contacts, Files, System; then general consumer apps they made up, called City, Shopping, Cabs, and RentAFlat. The final one is called AgentUserInterface, a very unwieldy name. That one is for the user to text with the agent. Again, this is all text-based, nothing graphical, so you could translate a lot of this stuff to an environment you call Desktop or whatever. You’d probably just take out some of these apps and add in others.

The apps have tools like search_emails or book_cab or wait_for_next_notification. All together there are 101 tools scattered across the 12 apps. All the apps and tools get fed into the system prompt so the model knows how to use them.

The data in the apps is of course synthetic. But it’s coherent across the many apps, reflecting a specific persona. So for example, if you have messages with a person, that person will appear in your contacts, and maybe you’ll also have an email from them. That collection of data across all the apps is usually between 400k and 800k tokens, way bigger than almost any model’s context window, so you can’t just feed all the data into the model at once and have it work from memory - it has to use the tools and read from state.

As for the agents themselves, they’re actually set up in kind of an old-school way.

They use something called ReAct, which is from this paper from October 2022, right before ChatGPT came out. I view it as kind of a stepping stone to full-on agents, and one of the main authors of this paper actually has been on the forefront of agent progress for a while: he was on the SWE-bench paper, which came out in October 2023; and he was on the Tau-bench paper, which came out in June 2024. Those are both important agent benchmarks, SWE-bench in particular, which kind of established the genre alongside the original Gaia paper. The first real agents from the big labs didn’t come out until late 2024 and early 2025, stuff like Claude Code and Gemini Deep Research and OpenAI Operator, so in a way those benchmarks set the target for the labs to aim at.

Anyway, back before agents really worked, people tried different ways of getting chatbots to do more than just chats. ReAct was probably the most successful effort, I think because it’s so straightforward. All they do is prompt the model with a few examples of this thought/action/observation loop, and they have a special format for the actions that some helper software picks up on to execute the action and return the observation.

You can see it in action for these two example questions from the benchmarks Hotpot QA and AlfWorld. For the first question in particular, they show just asking, then asking with CoT prompting, then asking with access to a search tool, all of which yield the wrong answer. Over on the right you get the full ReAct paradigm of think, act, observe. When they lay it out like this it’s a pretty natural marriage of CoT and tool use, which is still what you see when you’re using something like Claude Code today - that think-act-observe loop is at the heart of all agents.

Remember, at the time they published ReAct, tool use by models was not really a thing, like ChatGPT couldn’t do anything except return you text. Of course now tool use is second nature for all models, so you don’t have to hammer on that in the system prompt. But it does help standardize things to all use this same basic prompt, rather than relying on more complex or model-specific scaffolds. That type of variability is a killer for benchmarks.

Now for the tasks. They used human annotators to make 800 of them, split evenly across the first 5 capabilities in this table. So that’s 160 each.

These tasks all happen within 10 different “universes”, basically a user persona and their collection of user-specific data. In the paper they give the example of a persona named Helena Mueller, a 43-year-old Marketing Manager living in Berlin. So then the contacts, emails, calendar etc all make sense for Helena and her imagined life - individual and group messages, calendar events for the week, a ride hailing history, pretty detailed stuff. That’s separate from the Mobile environment, which details in general the apps, the event triggers etc. You fill up the environment with the data to get a universe.

The tasks are all verifiable. Specifically, the verifiers are mostly going to look for write commands that show the agent actually took the necessary actions. So like the first one, which is kind of a dumb task, that’s going to look for writes to all the contacts. Unfortunately, making them verifiable usually means being weirdly specific, like the Search example where the user anticipates there could be a tie and says how to break it. I think the authors realized that and made the Ambiguity capability to compensate, but I suspect real users will be ambiguous much more than a fifth of the time.

Anyway, if the agent takes more than 200 steps or overflows its context or misses a deadline for a temporal task, it automatically fails.

The tasks specify an initial state and also a chain of events, in some cases based on earlier events, in other cases based on times.

Now for the final two capabilities, they’re actually tasks from earlier capabilities, but with a twist. For Agent2Agent, that’s going to allow the agent to call a sub-agent, either using the same model or using a different model, via Google’s Agent2Agent protocol. For Noise, they’re going to throw in some errors and see how robust the agent is.

One other note about capabilities: while each task has a predominant capability for categorization, it’s really impractical to have a task with just one capability and not any of the others. I know that’s a problem we often encounter when we’re writing prompts, that we can’t categorize them cleanly, and I think the authors made the right choice here to prioritize realism over clean categorization.

Putting all of this together, here’s what their annotation and reviewing software looks like, it’s that open source package I mentioned earlier. They have user input in blue, some agent actions in purple, events in green, all in a flow chart. Then further down they have outputs from the model in a couple different forms, like the thinking and final response in the bottom-left, the list of steps and their components in bottom-middle, and the raw contents of a particular component in the bottom-right. Also on the left is the list of apps for the Mobile environment.

So here’s where we net out in terms of raw performance. A few things to note here.

One is, we’re using older models. Putting aside the fact that it’s a Meta paper so they have to throw in Llama 4 and even Llama 3.3, we’re only seeing GPT-5 and Claude 4 and Gemini 2.5. Every one of those has meaningfully more skilled versions available today, with particular focus on agentic work. So I wouldn’t be surprised if SOTA models now did meaningfully better on at least the first two capabilities.

Second, you’ll notice Time has universally poor performance. I think that’s a bit artificial, because by definition all tasks except Agent2Agent only used one agent. For time-sensitive work, you’d definitely want one orchestrator keeping an eye on the time while sub-agents work. What they observed with the Time tasks is that because models can only process one thing at a time, they’d often be in the middle of something else when some crucial event came and went. It’s especially tough for the thinking variants of some of these models.

Overall though, I like that there’s room to move on most of these capabilities, I like that the prompts are human-made, and I like that they’ve made the tasks more real by incorporating elements like time and ambiguity. It’s a real test of agent utility.

Continuing on the realism trend, I like that they reported out cost and wall time too; if you’ve been following any conversations about Openclaw and the like, you know people are complaining about burning through tokens or taking too long. Cost is also quite relevant for enterprise, wall time less so since jobs may run in the background. But for the Mobile environment that reflects consumer usage, wall time is definitely going to matter, maybe as much as quality.

So on cost, there’s a pretty clear trendline, which only Grok is really below. Not a surprise.

On time, it’s not surprising that models can beat humans on a lot of tasks, but I do take issue with this graph. Notice the title is “Time per Solved Gaia2 Scenario”. It’s defensible to exclude the failures, since a lot of them probably involved looping and artificially long times, but if you only report on the successful tasks then you’re biased towards the easy ones. By contrast, since humans are able to solve pretty much all these tasks, you’re going to get the full spectrum of difficulty or inherent length in those human results. This is also why some of the most successful models seem to take the longest, like GPT-5. So the most I would take from this is that models definitely can be faster than humans - nothing more.

Now for the bottom graph, calls and tokens of course are the drivers of cost and time ultimately, so it’s good to look at both; you want to know what to change in order to alter your end metrics.

For LLM calls, for a given amount of performance you want fewer of them, because each call is going to add time, even for the same total number of tokens. One trend here is that thinking variants make fewer calls and also do better. They’re going to use more tokens per call, since using extra tokens to think is by definition what thinking models do, but apparently the thinking actually does result in better thoughts and actions. If you look back above, you’ll see thinking variants cost the same or less and take similar amounts of time.

One related point they mention is that which apps the agents use and how often is pretty similar across all the models. So the difference really is the quality of thought, how wisely you’re using the apps and what exactly you’re doing with them.

Do note that the x axis is logarithmic on the output tokens graph on the bottom-right. Also it’s not shown here, but they note all models flatline past a certain point, like scaling up tokens indefinitely is not going to bring better performance.

So we looked at scaling the LLM calls and scaling the tokens for the one agent. Now we look at scaling the number of agents.

They’re using the Agent2Agent protocol, shown top-left. Multi-agent stuff is still pretty new so I haven’t seen much of this protocol in the wild. I will say though, I prefer the terms “orchestrator” and “sub-agent” compared to “Main-Agent” and “App-Agent”. Anyway, if you haven’t been on any paper talks where we’ve discussed agent swarms, there are two advantages: one, you can do work in parallel and thus save on wall time; and two, having a sub-agent do a task keeps the context of the orchestrator clear. Given the push on long-horizon tasks, keeping the orchestrator focused for longer is a big deal.

Now they’ve done two experiments here on multi-agent setups. For the first one, the table on the top-right shows what happens when you upgrade the orchestrator model and the sub-agent model. In this case, they’re making a major upgrade, from Llama 4 Maverick to Claude 4 Sonnet. In both cases, performance approximately doubles, and the effect seems to be independent. My hunch is that you always want the smartest orchestrator and that you should pick the right model for the sub-agent’s job, like if it’s simple you can pick a cheap model but if it’s tough you pick an expensive one.

The second one, shown in the graphs on the bottom, is the more interesting one. The goal here was to control the amount of delegation the agent could do, with this ratio “r”. r = 0 means it’s just the one agent, using all the tools directly. r = 1 means the agent must use sub-agents, it can’t call any tools directly. And then r = 0.5 means half the apps can be used directly, half have to be through sub-agents.

They found two things. One, the more you can delegate, the fewer tokens get used overall. You might think more communication overhead would lead to more tokens, but really what happens is that keeping context clean and work compartmentalized allows all the agents to focus and work more effectively. For example, since the orchestrator doesn’t have to juggle as many tools, it makes fewer syntax errors and enters fewer loops. It’s a stronger effect in weaker models, and the communications overhead is still there, so it can become a more marginal trade. But it still holds true with Sonnet at least.

The other finding is that it’s not always obvious what the impact on quality will be. For Llama it helps to delegate in terms of pass@k, but for Claude it hurts, at least for this task in this framework. I do wonder how generalizable this finding is, since models are trained with certain scaffolds and expectations about sub-agents now. Like if you open Claude Code right now, Opus 4.6 is going to expect certain sub-agents available. And modern models likely have their own preferences on how much they want to outsource vs handle themselves, whereas when Llama 4 came out agents were only just becoming a thing, so they wouldn’t really have figured into training.

My Takeaways

• Swarms are the future

  • As the authors point out, a variety of subagents is optimal, for cost but also for latency

• How agents work together and communicate is nascent

  • First time seeing Agent2Agent in the wild

  • Authors point out that many models are trained to work alone, maybe not good at organizing other agents (just like many humans are bad managers)

  • On the flip side, models trained to work in a swarm of other agents with models in their family may be harder to benchmark in neutral tooling or with other models as agents

• There is room for a GUI version

Paper · Website

Group Relative Policy Optimization

+

Reinforcement Learning with Verifiable Rewards

Okay, the first thing I want to do is revisit two foundations of modern RL: group relative policy optimization, or GRPO; and reinforcement learning with verifiable rewards, or RLVR.

GRPO is an algorithm that turns rewards into model weight changes. RLVR is how we get those rewards from responses.

Chronologically, GRPO came first, so we’ll talk about it first. As with many other LLM engineering breakthroughs, GRPO is the work of the DeepSeek team. If you’ve been to some of the recent paper reviews, you know what I’m talking about, but if not I’ll give a quick recap: DeepSeek is one of the top two labs in China, the other one being Qwen, and they have a very R&D-focused culture. They don’t crank out models, but the ones they do put out are always respected, and they often contain new techniques that long outlive the models themselves.

A perfect example is in this paper, nominally about a math-focused LLM called DeepSeekMath, but really a vehicle for this algorithm, GRPO. Like other RL algorithms such as PPO, it’s a method for extracting signals from model outputs, basically turning them into values we can then use to adjust the weights of our model. In PPO, we have to use a special model to estimate that signal, which is complicated and sometimes messy or fuzzy. With GRPO, there is no extra model to train, as you can see by the decrease in yellow boxes in the right-hand graphic. That makes GRPO simpler and often more precise than PPO.

Looking at the GRPO graphic a little more closely, you can see there’s a reward model that turns outputs o into rewards r, that then turn into advantages A, which is ultimately what we need to adjust the weights of our model. But as we’ll see on the next slide, we actually don’t need a reward model per se - we need a system to turn outputs into rewards. The rest of GRPO is just math, completely deterministic.

The other prime example of DeepSeek magic is this paper, nominally about DeepSeek-R1, the model that reproduced the reasoning behavior of o1 and o3 from OpenAI and briefly crashed the stock market.

In this paper, they used GRPO to train their reasoning model, but they didn’t require a reward model. Instead, they calculated rewards deterministically by checking the model’s final answer against a known good answer. So like for a math problem, if the known good answer is sqrt(x), we can easily check if the model’s final answer is also sqrt(x). We may have to check for equivalent forms, like x^½, but there’s no real interpretation or judgment like with a reward model. We know that technique as RLVR.

So just like you don’t need RLVR for GRPO, you also don’t need GRPO for RLVR. Like you could do RLVR instead of having the reward model in PPO. But in practice, these two go well together, because they are basically the simplest possible setup for RL.

Anyway, the DeepSeek-R1 paper found two effects of GRPO + RLVR: you can greatly improve reasoning, and you generally end up using way more tokens. This is exactly what o1 and o3 did. Sometimes people call it “test time compute scaling”, meaning you use more compute when generating your response, and that extra compute is supposed to improve quality.

This combination of GRPO + RLVR is everywhere nowadays. Any prompt where there’s some verifiable final response, like a math problem or some instruction following, is likely trained on with GRPO + RLVR. And rubric training is basically just stacked-up RLVR training, like instead of one final answer you can verify against, you have multiple criteria you can verify against. It’s the same core idea of a method for giving a response a score, which can then go into GRPO to turn into signal on how to adjust the weights.

RLVR has two shortcomings though. For one, you might have incorrect steps but still get the final answer right. With RLVR, you’re going to get full marks, even though the response overall is not correct. For the other, you don’t know why or how you got something wrong. Or if you do, it doesn’t factor into the score, into the training signal. So you’re rewarding at a response level even though parts of the response may be wrong, and you’re leaving a lot of valuable information on the table by just calling a wrong final answer “incorrect. Keep those shortcomings in mind.

Another, more recent finding of RLVR is that it may not be teaching anything. Instead, what some researchers have found is that RLVR merely brings forth solutions paths that are already present in the model, although they may be highly unlikely ones. This is called the elicitation hypothesis.

The image on the left shows what I’m talking about. For Problem A, the base model has all these possible solution paths for a given problem, but only a couple of them are correct, so when you run the model the chances of getting a correct response are low. What RLVR does is make the correct paths much more likely, so that you as an end user are generally getting a correct answer now.

Of course it may be that those other solution paths are involved in other problems, like Problem B here. To oversimplify somewhat, if through RLVR we greatly reduce the likelihood of exploring the left side of this tree, then we can’t reach the correct answer for that problem, even though it actually was there in the base model.

The chart on the right is generally how people show the elicitation hypothesis is true, that any correct solution a model can give is in there from the beginning and not taught wholly new via RLVR. On the x axis is the number of chances you give a model per question to get a set of questions right. As long as any of the k responses is correct, the model gets credit for that question. So that metric is called pass@k.

For end users, we really only care about pass@1, since a person using a chatbot expects the right answer the first time. And you can see there, on the left side of the chart, doing GRPO + RLVR really helps - you’re getting 4x higher accuracy for the yellow line compared to the black line.

However, if you keep giving chances, up to 256 in this case, you see all the lines converge. So it seems that the base model knew the answers all along, it just didn’t know how to produce them efficiently.

Again, elicitation is still good for end users. It just shows RLVR doesn’t teach new things.

Okay, now we need to talk about distillation.

Very broadly, distillation is a family of techniques for transferring some quality or information from one model, the “teacher”, to another, the “student”. Usually the teacher is a more advanced model than the student, which by the way puts a ceiling on how far distillation can take you.

So the naive way to do this is to make training data synthetically, using the teacher model. Like imagine we made the same kinds of post-training data — SFT, RLHF, rubrics — but with GPT and Claude instead of humans.

If all you have access to from the teacher model is the tokens it outputs, that’s the best you can do. That’s true of the Claude API, so if you’re trying to distill off Claude, this is the way to go.

If you have access to the per-token probabilities, sometimes called the logprobs, you can do better. In this setup, you pass the same input to the teacher and the student. Then for each token you need to predict, you compare the probability of each possible token in the student’s distribution with the probability of the same token in the teacher’s distribution. The differences between the probabilities are your loss. So your goal is to teach not only the most likely token, but also how likely all the other tokens are too.

That’s a much richer and more nuanced understanding the teacher is providing to the student, and it’s going to be more effective at passing on the teacher’s wisdom.

Let me draw a human parallel. Let’s say you’re learning to play chess from a master. You’re both playing black, and some opponent is playing white. The prompt is white’s first move. Now you and the master each take the list of all possible next moves and assign each one a probability of being optimal. Then you compare probabilities, then you make the master’s top-rated move.

By doing that over and over, you’re going to get an idea of what set of moves the master realistically considered, what moves he kept in mind but thought were unorthodox, what moves he knew were trash. That’s a lot more wisdom to pass on than just watching him play the opponent!

Of course you need that level of access in the first place. OpenAI actually does offer logprobs through their APIs, which is partly why they’re such a popular target for distillation. And of course any open-weights model includes logprobs too.

So the teacher’s logprobs depend on its innate knowledge of course, but they also depend on the context, the input the teacher has received. In one sense this is obvious - the next predicted token clearly depends on the prompt you’ve given the model - but for other forms of context it’s more subtle.

One easy example is the chunks you provide in RAG. Even if the core answer is the same with and without the extra context RAG provides, you could get some additional details, or smaller effects like a change in tone or vocabulary. That’s all going to show up in the logprobs, the different distributions.

Even more subtly, you could provide different instructions or system prompts to the teacher. That’s what some researchers did here, in a paper called “Learning by Distilling Context”. This paper came out in September 2022, two months before ChatGPT, which explains why they’re focusing on relatively simple queries here.

Anyway, you can see in this example that the teacher receives the schema and four examples before receiving the question. Those four examples help it get the right answer. So then when you do soft-label distillation, even though the student only gets the schema in its context, through distillation it also gets the wisdom of the four examples the teacher got.

This is a very simple example, but in theory you could fill the teacher’s context window for distillation. You won’t be transmitting 100k tokens of information or whatever, but you will be transmitting some compressed version of it, compressed by the teacher model’s intelligence into this low-dimensional space. And that intelligent compression is what makes soft-label distillation generally more effective than just making synthetic data from the teacher.

I think of soft-label distillation as a way to teach process knowledge, stuff that is hard or at least inefficient to just write down as rules. Like yes, you could try to write out the mechanics of SQL and your particular business logic in the student model’s system prompt, but when you use distillation, you’re taking advantage of the teacher’s compressed wisdom and having it guide the student directly. Again drawing a human parallel, I think of it like a golf teacher standing behind you and putting his hands over yours, guiding you through the entire motion of driving or putting. Sure, he could try to explain how to do it, but the hands-on feedback and guidance as you complete a swing is going to be so much richer.

There’s a decent body of work out there about context distillation. I’ve heard Anthropic uses context distillation to take very long prompts about what Claude’s character should be and distill it into the weights. I also read a paper late last year about distilling lots of documents into a kv cache that you can then pop into a model, to basically give it memories of those documents when you then want to ask about related matters. So this general idea of using a model to effectively and intelligently compress context down into the densest possible signal is pretty robust.

Self-Distillation Policy Optimization

+

Reinforcement Learning with Rich Feedback

So this paper coins two new terms: self-distillation policy optimization, or “SDPO”; and reinforcement learning with rich feedback, or “RLRF”. Those are the parallels for GRPO and RLVR, which we discussed earlier.

The key insight for RLRF is that RLVR feedback is incomplete. Yes, if you’re only going to provide one bit of feedback, it’s going to be whether the solution was right or wrong. But there’s no reason we have to limit ourselves! We can provide more, richer feedback.

RLRF says, let’s use that feedback. Maybe we have runtime errors when our program fails. Maybe we have written feedback from an LLM judge. Maybe this is real user data and they yelled at the agent about why its actions were wrong. That’s all crucial stuff! It’s important to know how you were wrong, not just if you were wrong. And there are way more ways to be wrong than right. So just returning a reward of zero leaves something on the table.

Of course some environments are more feedback-y than others. They pick coding here because you do get those runtime errors and so on. But this is not a code-specific technique - anything you can put into tokens can be feedback here.

This here is where the feedback goes. This is the prompt for the teacher, which we bolster with as much context as possible so that it can guide and distill into the student. Remember, the goal of the teacher is to make adjustments to probabilities at each token, so we want the teacher to always be right.

So we start with our prompt of course, whatever problem we’re trying to solve.

Next, if we have one, we can provide an example of a correct solution from a previous rollout. That’s definitely going to help the teacher guide the student if it exists. If not, we skip this part.

After that is our feedback, from the compiler or the LLM judge or whatever. We should basically always have that, except for on the first run, but again if it doesn’t exist we skip it.

Then we provide the student’s original response. And you might be surprised that we’re providing an assistant response. Like, isn’t the model supposed to provide a new response?

But they’re using the teacher model differently. They’re not using the teacher to give its own answer, they’re using it to give its own probabilities at each token of the student’s response. You never get a complete response from the teacher. We’ll see more about that on the next slide.

So that’s what they explain in the paper, but I want to call out two things. One, you may be wondering why we’re doing any more training if the student was able to produce a correct solution. Well as we saw earlier with those pass@k graphs of increasingly high k, there’s a difference between “can produce a right answer” and “probably will produce a right answer”, and that difference may be enormous. We want to shrink that gap so that end users get better pass@1 results.

Two, why can’t we use someone else’s correct solution? The theoretical answer is that it will change the teacher’s guidance to bias toward the style of the correct solution as well as the contents of the correct solution. So if the model naturally is verbose, but our correct solution is concise, then the teacher will punish the student for being verbose, even though that doesn’t impact correctness. We want a clean signal from the teacher on correctness, which can only happen if the correct solution is from the same model we’re using for the teacher and the student.

However, practically they don’t test it. You could give them a pass and say it’s out of scope, but I think it would have been nice to try having another model family write the solutions and see what the penalty really is. If it’s small, it could be worth having humans write solutions to really hard problems and then having a SOTA model do this self-distillation thing. But then again, if we really believe in the elicitation hypothesis, you can’t teach new things with GRPO + RLVR or related methods. Still, I wish they had done the test.

Okay, so here’s how it looks when the teacher provides feedback to the student. In this example we have a question, then an answer from the student, then some text feedback. We’re gonna take all that and give it to the teacher.

The teacher is going to go token by token through the student’s answer, that bit in step 2, and compare its likelihood of that token vs the student’s likelihood of that token.

So in this example, for most of the program, the teacher and the student gave similar probabilities for the tokens: for the three tick marks, for the word “python”, for the function name etc.

But then you get to this plus symbol and you see the teacher disagree. Specifically, the teacher gives a very low probability for “+”, but the student gives a relatively high probability. Since the teacher knows the feedback and the student didn’t, the teacher is probably right, and we penalize the student.

Here’s a closer look at that same example. At the top they have the baseline approach of GRPO, which assigns equal credit to all tokens in a response. Red means less likely here, so because the initial response was wrong, GRPO gives low credit to all the tokens.

By contrast, SDPO assigns credit per token. We can see that for each token the student actually picked, in the first row, we also have alternative tokens it considered. So like in the first position, we have parenthesis + “range” as the selected token. But the student also considered a leading space and then “range”, “range” and a space afterwards, and a dot then “range”. The teacher agreed with the token the student chose, so the selected token has a white background.

Now if we look at the blue tokens, where the teacher assigned more credit, we see two things. One, in the second position, the student’s second pick gets high credit from the teacher, so it seems like the student was close to making the right decision.

Two, we see that plus in the sixth position is really the main problem, and that removing it and the “1” after was the right choice. The most blue token, with two close parentheses and then “\n”, is actually the final token that the student actually selected.

So again, the teacher is not generating a correct response we compare to; instead, the teacher is evaluating the student’s choice of the immediate next token given all the previous tokens chosen.

So that loss, that mismatch between teacher and student probabilities, is signal for SDPO. SDPO is the algorithm that turns the signal from the mismatch into changes of the model.

And because the teacher and the student are the same model, you can keep doing this over and over until performance saturates. So let’s say on the first round you did eight rollouts and got none right. But you got feedback, so you give that to the teacher and measure that mismatch between teacher and student probabilities. You have your loss, which you plug into SDPO, you train your model with that feedback, then you try again. Now two of your eight rollouts are correct. But the rest are still wrong. So now you take the feedback and the correct solution, give it to the teacher, calculate loss etc. Ideally you end up with all eight rollouts right, but maybe you top out at seven.

With standard GRPO + RLVR you just have to keep trying and hoping that the answer is in there somewhere, or that changes to the weights due to good signal on one problems helps you on this other problem.

With SDPO + RLRF, by contrast, you’re getting this extra feedback that can nudge you closer to a good solution. So it’s at least more likely that if you keep training like this, you’ll get there.

Here’s their first test of SDPO vs GRPO, with the Olmo3-7B model, working on a chemistry benchmark.

A few things to note. First, of course the accuracy improved, and improved more than with GRPO. They probably wouldn’t have published this paper otherwise.

Second, the x axis is in wall time, how long training took. That’s not usually a metric we see in these papers, they’re usually measuring performance against training steps or perhaps amount of data. Frankly I think it’s a little bit suspicious, like the accuracy improvements don’t stand on their own so they have to tout some other benefit.

Finally, and perhaps most underratedly, the response length went down rather than up. It went way down actually, 11x compared to GRPO, which as we saw incentivizes more tokens. “Longer thinking” as some would say, or “test-time compute scaling” - the idea that more computation to produce more tokens means more accurate answers.

But if you actually look at the thinking, it’s often repetitive or looping, and sometimes contradictory. When DeepSeek R1 came out people made a lot of observations about GRPO + RLVR getting models to say things like “Wait” or “Let’s try again” or “On the other hand”. Of course it’s good to confirm your answer from a different perspective, fox-like thinking as Isaiah Berlin would say, but doing the same calculations three times or ping-ponging between the same two points repeatedly isn’t productive.

So perhaps SDPO + RLRF is a more direct way to bring forth these reasoning paths than GRPO + RLVR.

Now here’s an experiment where they narrow their training set way down, just to hard problems, to see if that rich feedback really can guide you to the right solution.

One is exactly what they’re measuring. They’re taking a couple sets of hard tasks, where the initial model gets the answer right almost never for “very hard”, or where it gets the answer right less than half the time for just “hard”. That’s only 9 and 19 tasks, respectively, which is why the error bars are so big. Anyway, what they’re measuring is basically pass@k for increasingly large k, but with a twist: for SDPO, since they’re training after each round of a certain number of rollouts, what the x axis really measures is how many rounds of SDPO it takes to get a solution. That’s why they call the y axis “discovery@k” instead.

Two is what they’re comparing. We know SDPO of course, and best-of-k just means we take k samples all at once and count it if at least one of the k samples is correct. The other trendline, “multi-turn”, is like a hacky version of SDPO: instead of training after each round, they just keep adding feedback to the prompt. So each incorrect attempt and its feedback get added to the prompt, which could be like hundreds of examples. They do have to cap the number of examples due to context limits, but that’s still a ton of information they’re providing. Also they run a variant where they kept the attempt and the feedback, not just the feedback, and that did much worse. So that’s a practical tip for your prompting: try to boil down the lessons of failed examples if you can.

And three of course is the results! SDPO outperforms the best-of-k baseline, and also outperforms multi-turn. It’s really not surprising that sequentially trying and then updating outperforms just making more and more parallel attempts, but I was surprised how bad multi-turn was. To me that really highlights how much more effective feedback becomes when you train on it. Similar to how in theory you can carefully follow documentation to get the same outcome as someone with lots of experience, but in practice there’s something about learning that makes the knowledge fit into your own sort of mental schemas in a way that reading external text misses.

One other piece to remember about the results is that from an end user perspective, the model at the end of SDPO is going to be much more useful, because it’s much more likely to get you the correct result on the first try. Pass@k for a huge k is important scientifically, but it’s irrelevant for end users.

So one complaint you might raise is that SDPO + RLRF requires additional feedback compared to GRPO + RLVR. Remember, the “RF” in “RLRF” stands for “rich feedback”.

To address that, they separate out the RLRF part and just test SDPO vs GRPO. So now we don’t have to make any changes to our training data or our environments, just to the training process. Remember, GRPO and SDPO both depend on there being at least some correct rollouts to get some signal, they just use that signal differently. In the case of no correct rollouts, both algorithms do nothing.

The story here is again not a clean victory for SDPO. It wins most of the time, sometimes by a lot, but other times it is basically tied or even loses by a few points. Also keep in mind the scores are average accuracy across 16 attempts. I would have liked to see pass@1 to reflect the user experience.

Another way you might compare SDPO to GRPO is by making the binary more of a spectrum, like a smoother transition from GRPO to SDPO.

The authors do that here when they make the SDPO feedback increasingly granular. Recall that GRPO is feedback over the entire response, what researchers often call the “sequence”. So sequence-level SDPO is basically GRPO but with feedback given to the teacher prompt, which apparently helps, given the separation of the grey and blue lines.

The next step in granularity is token-level, just a binary about whether the teacher thought the student’s token was the best choice or not.

And then the final step is what we saw earlier, the logits, which are basically the probabilities of any possible token for the given position. That’s the full SDPO. We called that “soft-label distillation” in the background slides.

So intuitively you might think bigger models are gonna be better self-teachers, because they can take feedback into account better. Like anyone who has used AI consistently over the years knows models used to be terrible at taking feedback in conversation, missing out on subtleties or getting confused etc.

Turns out that does hold up in SDPO, at least for the range of sizes they looked at. SDPO is no better than GRPO at 0.6B, but consistently widens the gap up to 8B. Unfortunately they didn’t test the 32B model, which would have really made for a nicer story, but my guess is it does keep working at least up to a certain point. One wonders what the big labs with 1T parameter models can achieve.

My Takeaways

• RLRF could be a good use of human qualitative feedback

  • RLHF already uses human quantitative feedback (the Likert scale)

  • Justifications from existing RLHF data could become RLRF data

• Seems like a straightforward replacement for GRPO

  • There was some Twitter conversation about big labs using a better version of GRPO, perhaps this is it

  • There is a sense we are “due” for a new technical breakthrough (although maybe that was Engram)

• This all depends on the raw intelligence being in there from pretraining

  • Elicitation hypothesis

  • Does that mean we need to focus more on pretraining to get the most out of post-training?

• RSI is coming

  • Anything you can verify can self-improve

  • You don’t need a smarter teacher, just a student with good feedback looking over their old results

Paper · Repo · Blog

Background

I want to start with the notion of scaling laws and how they relate to technology itself.

In the narrow sense for LLMs, scaling laws tell us how to optimally balance compute, data, and parameters as we scale up our efforts. Like whatever your constraint is - typically it’s the compute budget you have for pretraining - your other optimal settings fall out of that. So if we have a certain number of FLOPs, which is the standard unit of compute, then using the scaling laws we can calculate the model size and the number of tokens that will minimize the loss.

That’s incredibly useful for anyone trying to build an LLM. If scaling laws didn’t exist, and you had to find the right balance empirically, you might spend months just trying different combinations every time you wanted to pretrain a new model.

The scaling laws are also pretty much independent of model architecture, at least within the transformer family. The laws are not 100% exact, so it’s more like the error bars cover all the changes people are always making to different parts of the architecture and training recipe and so on, but broadly speaking they’re quite robust.

The graphic here is from the most famous scaling law paper, Training Compute-Optimal Large Language Models, better known by the name of the model they trained in the paper: Chinchilla. In the paper, they improve on a previous result by Kaplan et al, and they show that recent models are undertrained, meaning they are too big for the amount of training compute they received. The Chinchilla model, despite being less than half as big as GPT-3, significantly outperforms it.

There are many other specific scaling laws - we’ll cover one in the paper today - but more broadly is the lesson of scaling in general.

Its most famous avatar is The Bitter Lesson, a remarkably brief essay from 2019 by pioneering AI researcher Richard Sutton. It opens with the following lines:

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.”

Did you catch that? He’s saying scaling of compute is the baseline assumption, and any work we do better take that assumption into account, or else it will only matter in the short-term until scaling catches up.

Sutton gives a few examples. In chess, “brute force” search-based methods dominated methods that encoded centuries of human chess wisdom. In Go, a similar story occurred. In speech recognition, learning eventually beat the methods based on human knowledge. In computer vision, much the same happened.

In all cases, work that took advantage of the inevitable increase in compute outlasted work that made efficient and scrupulous use of the compute of the day, acting as though the “inefficient” and “brute force” methods would never become feasible.

In my view, this is the major trick in the tech playbook: work a problem until the only constraint becomes resources, then dump resources into the problem. A machine where the bottleneck moves around a lot isn’t much of a machine at all; it can’t deliver predictable results and thus requires a human at the helm. A true machine does its work reliably and can do more work the more fuel you give it. We are all living through a period of scaling right now with the way everyone is building out data centers.

Scaling may not continue forever, but at least within some regime, we want scaling and we want to know the scaling laws, to design bigger systems and to forecast future performance.

It’s always a bit funny how some scaling laws work. Like with the Chinchilla scaling laws, that seems to be a natural property of the universe, more akin to a physical law. But for Moore’s Law, which underpins The Bitter Lesson, that’s a prediction about society and how it will continue allocating resources, in scientific discovery and also in hardware production.

The famous METR graph is a scaling law of the latter type. As I’ve covered before, this graph shows how models are becoming more and more autonomous over time, able to handle increasingly long-horizon tasks as measured by how long it takes a competent human to do them.

The current results are actually somewhat in doubt - Mythos is too advanced for METR’s measurement task suite - but we do have reliable data points all the way up to Claude Opus 4.6, which came out in February 2026.

Now the METR folks show just the one fit line, one scaling law. But arguably there is a break in the line, which I’ve drawn in orange, with growth at 10x/year, meaning by February 2027 we’ll have an agent that can do about 100 human-equivalent hours of work autonomously at 50% success rate. Pretty nuts.

Anyway, that new trendline starts with the introduction of reasoning models. See, o1-preview introduced another axis of scaling: not training compute, not data, not parameters, but inference compute - how long the chain of thought could get and still improve the final answer.

That’s the other part of the scaling law puzzle: where is everything we could scale? Because once we find them, we can dump more resources into them and juice performance.

Now in the course of these increasingly long tasks agents can do, there is plenty of trial and error. In some ways that’s what makes agents work: interacting with an environment, gaining information, incorporating it into future actions. That sort of exploration and iteration isn’t really present in chatbots, and it’s also not part of classic benchmarks where you’re supposed to respond to a prompt in one go.

Of course models have always been able to adapt based on their context; even responding to a basic prompt is in a sense adapting behavior to context. Historically the most common tactic has been to write rules and provide examples to put in the prompt, which folks call “in-context learning” and sometimes “few-shot” or “many-shot” depending on how many examples are in the prompt.

But now there’s this dynamic element, where an agent’s experience within the same task becomes part of learning. Maybe the learning goes in a scratchpad just for the task, maybe it goes into a long-term memory system, maybe it just stays in the trajectory and survives compactions. However it happens, the agent is learning as it works, just like a human would.

Sometimes people call this continual learning, although sometimes I see that term specifically for learning into the weights, i.e. training the model. That’s not what we’re talking about here. What we’re talking about is all ultimately in the context window, although it may live in a different tool and only enter the context window when the agent calls the relevant tool to fetch it.

Continual learning is crucial because lots of important context will never make it into the training data. Like most businesses are not going to fine-tune a model on their business logic and proprietary information. So that information has to come through the context window.

The graphic here is from Continual Learning Bench, a bit of prior art in this space, which measures an agent’s ability to learn information from one task in a domain or environment and apply that knowledge on another task. So in this example, for the first query, the agent learns some stuff about the database, shown in the green box. Then for the second query shown, with label Q10, that green box of knowledge appears at the start to inform the agent on the task. Later on, with label Q25, the agent has to recognize which information is now stale.

For CL Bench, they test different models but also different context management systems, including Claude Code and Codex, which are coding agent harnesses but include many tools and techniques for managing context, like memory. They find that pure in-context learning, i.e. just keeping everything from the trajectory in the context window for as long as possible, does the best. However, there are only six different tasks, although each one contains many different “instances” - like Q1 and Q10 and Q25 in the diagram. And each instance is not that long, so even though an overall task can be quite long, the continual learning is more like a shared base for each instance rather than one long narrative.

So models apparently have this continual learning ability. But as we know, with agents, the model is only half the story; the other half is the harness.

Many agent harnesses improve continual learning, but usually as a means to improving performance on the agent’s focus area, like coding as with Claude Code. But there is one famous harness whose main job is arguably to improve continual learning: Autoresearch, from Andrej Karpathy.

The idea of Autoresearch is quite simple: plan an experiment, run it, observe the results, and use the results to plan a new experiment. It’s the scientific method, but encoded into a harness. The human picks the goal and hands off the rest to the agent.

In the actual Autoresearch repo, the goal is to train a version of GPT-2. But the general method applies to any problem with a clear metric to optimize. In the graph here, that metric is the loss, which is the classic metric to minimize when training any LLM. We will see a version of this graph later, but flipped, since the goal will be to maximize a score rather than minimize a loss.

The Paper

Since this is a benchmark, let’s start with the tasks.

We have 134 of them, sourced from experts in many fields, with mean human effort clocking in at over 57 hours. So these are substantial tasks, putting them at the leading edge of the benchmark task time horizon distribution. We recently covered Agents’ Last Exam, and they were bragging about how a few of their tasks made it to the weeks timescale.

As we’ll see, it’s important for EdgeBench that an agent not solve any task on the first try, because what they’re measuring is ability to learn from feedback. So duration is more than just a way to match difficulty with agent capabilities - it’s intrinsic to the purpose of the benchmark.

Now as for domains: unsurprisingly, given the likely personal networks of the researchers, the tasks skew STEM, with science, programming, and ML representing over half the tasks.

Many of them are multimodal, but the authors specifically note that “tasks whose primary difficulty lies in visual understanding, especially GUI operation, are excluded” - they want the hard part to be reasoning and iteration rather than pure discernment. To me that presents somewhat of a challenge with their Interactive Games & Simulators bucket, but they don’t explain any further.

Sadly for us, there is also little detail about how they made the benchmark.

Through manual effort, the tasks become environments, where the agents are free to experiment. The diagram shows an inner loop on the left in blue, where quick feedback from tools helps the agent learn the environment better. You can see examples in blue text along the bottom.

At a slower cadence, there is an outer loop, shown on the right in orange, where the agent can choose to submit its work to a judge. The judge evaluates the agent’s work against the ground truth or rubrics or similar, then returns the score and feedback. The ultimate goal is to maximize that score. The agent can submit any number of times, although there is a cooldown period of two minutes between submissions. Again, there are examples along the bottom in the relevant color.

So both loops allow for continual learning. Of course, since this is a benchmark, we have right answers in the outer loop to compare to. In the real world, where the right answer may be unknown, you can at least shoot to maximize some metric like we saw with Autoresearch. You just have to be careful of Goodhart’s Law and mistaking the optimized metric for the thing you actually want, if the two aren’t identical.

Here are the Autoresearch-esque curves, showing how top performance per model rises over time. Note how chunky the graphs often are; progress seems to happen in punctuated equilibria rather than in smooth steps, at least at a per-task level.

The keys only show the model, but the harness and context window are important too. So in full, we have:

• GPT + Codex @ 256k compact window

• GLM-5.1 + CC @ 200k compact window

• DSv4 Pro + CC @ 200k compact window

• Opus 4.8 + CC @ 1M compact window

They show later on that 1M provides about five percentage points of advantage over 200k, with a slow and steady decline in that gap as time elapses.

I’m a little surprised they didn’t use a neutral third-party harness here, but I do think there’s value in shooting for maximum performance rather than controlling variables. Just more evidence the model and the harness are becoming one integral whole.

Now if we zoom out a level, from individual tasks to task categories, we see much smoother curves. In fact, they’re so smooth that you can tightly fit the same type of curve to all of them: the log-sigmoid.

A sigmoid is a classic S-curve, starting at zero and then accelerating to a midway point, then decelerating and leveling out. A log-sigmoid is that but on a logarithmic x axis, like in these graphs. So basically these models all form S-curves in log time.

That’s remarkable, because the individual tasks graphs didn’t seem to show much of a pattern at all. But if you aggregate just enough, the common pattern emerges. They have some results later showing that the pattern isn’t guaranteed from the aggregation, that it’s a real trend in how agents learn over time and over many iterations.

It’s also helpful, because it allows us to predict performance past the 12-hour window, which really saves a lot of time and tokens compared to having to actually hit 18 or 24 or 72 hours.

Here’s a validation of the predictive power of the fitted curves, taking performance through a 6.5 hour window and projecting it forward to 12 hours. The curves are pretty dead-on, even for the most inaccurate one, GPT-5.4 in teal. And actually they explain that for GPT-5.4, the score was lower than expected due to API flakiness rather than poor model performance.

So not only do they empirically validate their scaling curves, they also provide a theoretical justification. I don’t want to spend too much time on it, but the broad idea is to represent different bits of knowledge and insight as nodes on a graph. Any prerequisite knowledge is a connection. If you assume the graph structure is self-similar, basically that it looks roughly similar throughout, then you naturally expect a log-sigmoid shape when you chart progress through the nodes over time.

They’re not rigorously proving that every task will follow these laws, in part because some graphs will not be self-similar, but self-similar graphs are common enough that the theory explains why log-sigmoids are plausible at sufficiently high abstraction.

So we have scaling for a given model over the elapsed task time. That’s nice for understanding a model’s performance ceiling, and it saves us some experiment time, but there is a juicier finding in EdgeBench.

Specifically, it’s this chart, basically their version of the famous METR graph we covered in the background portion. What this graph shows is how quickly a given model learns, as measured by average performance gain on a select subset of tasks within a two-hour budget per task.

What they find is that newer models are consistently pushing the frontier, doubling the learning speed about every three months. That’s 16x/year, which is faster than the METR graph at 10x/year. So if the trend holds, not only can we expect agents to accomplish longer tasks as measured in human completion time, we can also expect agents to complete the same tasks more and more quickly - and that’s before taking parallelism and multi-agent systems into account!

Now one thing to keep in mind is they aren’t measuring pure model performance, or even model + harness performance; they are also measuring hardware performance. Like if two equally capable agents are competing, but one gets twice the token output speed, it will finish significantly earlier.

In practice though, OpenAI and Anthropic shoot for similar output speeds unless you pay extra for faster tokens, so I think these results hold. In fact, the only result I would caveat is for GPT-5.4, which as we discussed encountered API issues and thus has a lower score than expected. So if you imagine the datapoint for GPT-5.4 on this graph a bit higher in the world where the API didn’t flake, it fits the trend quite well.

On the left is another look at the same data, showing performance at the start and end of the two-hour window. The researchers selected the subset of tasks based on how similarly all the models initially did, so if new releases start to do significantly better to start, they may have to swap out the subset. But for now the initial performance seems about even, and there’s plenty of performance ceiling to go.

On the right though is a new trend, showing how effective new submissions are. Specifically, the graph shows what percent of submissions in the outer loop improve performance. So it seems newer models are more likely to make submissions that improve their score. Here is how the researchers put it:

“Stronger agents use feedback more deliberately: they build a submit-ready baseline, preserve the current best solution, make focused changes, and use feedback to keep gains or roll back failures. Weaker agents more often over-trust local proxies, bundle unrelated edits, or continue broad exploration after feedback has ruled out a direction.”

I do think it’s a bit confounded whether newer models just have better work and thus better submissions vs having better discretion on which work to submit, but either way you as a user are more likely to be looking at better work.

We do have a leaderboard of course, given this is a benchmark, but honestly it’s not really the main story here; the big finding is the scaling laws rather than any one point in time.

Now of course with more submissions you’re likely to improve your high score. So the question for continual learning is, how much does it help to see your previous work and submissions? What is the impact compared to the baseline of many independent attempts, where you only get one outer loop submission?

Here they answer that question, using Opus 4.8 to test. The lower curve, comprising independent submissions, is basically pass@k but with k as a function of time. And it displays the classic diminishing returns curve of most pass@k results.

Meanwhile, the upper curve comprises dependent submissions, i.e. where the agent sees the results of each outer loop submission and can continually learn. Here we see a consistent and perhaps even widening gap, where more experience pays dividends. So yes, purely providing more chances does improve performance, but not as much as continual learning does.

Finally, let’s zoom in on one task example, with an attempt by GPT-5.5.

Here we see the classic punctuated equilibrium pattern, long periods of little or no progress with occasional big jumps, until time runs out or until the agent reaches its limits. Personally I don’t know anything about detecting gravitational waves, so the specific jumps don’t mean much to me, and I don’t think they’re crucial to comprehend.

What’s interesting about this example task though is the high-level strategy GPT took to partially solve it. The authors list four points:

1. The agent first makes the problem measurable before making it better

2. When direct repair stalls, the agent decomposes the failure into searchable subproblems

3. Identifying a main bottleneck lets the agent keep searching productively

4. After finding a stable solution, the agent keeps the core and repairs only the remaining errors

To my eye that’s quite a general and broadly applicable set of steps. I wish they had distilled more examples like this, or perhaps taken these points and checked how pervasive they were in other attempts, by GPT and by other model families. I also wonder if prompting the agent to use these techniques would improve things.

My Takeaways

• We may have a new METR to track

  • Why is METR having trouble with sufficiently difficult tasks, but EdgeBench was able to produce them?

• Unclear what specifically makes learning rate scale

  • Model (training data)

  • Harness

  • Context length (they did prove this one actually)

• Maybe we can add signals to increase learning rate further

  • More and better feedback from tools

  • Another agent to work with

• In-context learning and error correction are mandatory

  • Even a 99% accurate agent has a 50% chance of failing after just 69 steps

RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

or, Theory

Paper · Repo

OpenClaw-RL: Train Any Agent Simply by Talking

or, Practice

Paper · Repo

Background

So the first thing we need to talk about, and really the framing we should have for almost all AI progress, is this graph here.

It’s from a research organization called METR, and it shows the increasing duration of tasks that AI can handle autonomously. The duration is in human time, so for example they’re saying it takes competent humans about 10 minutes to find a certain obscure fact on the web, and just under an hour to train a simple type of ML model called a classifier. METR has put together a bunch of tasks with varying human durations, and it administers that suite of tasks to new models, then graphs the result.

There are two toggles on the bottom here. One adjusts the scale of the y axis, so right now it’s a log scale, meaning that our straight line here is actually an exponential line on a linear scale. The other toggle adjusts the quality threshold. Here I have it on the more conservative 80% Success setting. So like Claude Opus 4.6 can do stuff that normally takes humans about an hour, completely autonomously, at 80% success rate.

Now this graph shows one trendline over the whole period, from GPT-2 through Mythos. But if you look at the green dots, which represent SOTA at the time, really it looks like two trendlines: one trend from GPT-2 to GPT-4o, in that quick run of three green dots below the line; and then another trend from o1-preview onwards. That break is where reasoning models came in, using test-time compute scaling, using very long chains of thought to arrive at better answers.

If you believe there really are two trendlines, then the second one is steeper than the first. And on that newer steeper trendline, the time horizon is 10x’ing every year. So Opus 4.6 for example came out in February 2026 and hits ~1 hr on this graph. That means in February 2027 we’ll be at ~10 hrs. That’s an entire day of work, done completely autonomously, and largely successfully.

So a lot of the work between now and then, and after then I suppose, is about long horizon - increasing the duration of task that a model or agent can do, not necessarily making it smarter or more knowledgeable.

Of course the longer the horizon gets, the longer we have to wait and the more things that have to go right in order to give out a positive reward. That latter problem, where you have more and more possible points of failure, is what makes long horizon training so difficult.

Now if we think about a human setting, where you’re trying to teach someone a skill or maybe learn one yourself, the person learning probably isn’t going to do a whole practice run themselves and then check back with the teacher or the instruction manual or whatever at the end. Instead, we usually check in at certain steps or milestones. Or like in cooking, you might taste test along the way rather than tasting only at the end.

The equivalent in machine learning is process supervision. In process supervision, we give a reward for every step, judging two things: one, is this step accurate; two, is this step on the right path. So looking at a math example like this, which is from an OpenAI paper that we made data for, you have the accuracy of calculation and the helpfulness of that calculation in getting the right answer. Of course in this example the final step is the incorrect one, so all we have to judge is the accuracy of the calculation, but in earlier steps you implicitly have to judge both with just one score.

Process supervision works for anything that happens in discrete steps. Math is like this, but creative writing is not. Even programming mostly is not, since you can often change the order of code blocks without impacting functionality. But one big chunk of the ML world is like this: agent trajectories. The observe-think-act loop is inherently sequential, and changes to the environment often demand a certain sequence. For example, navigating through a website requires a series of clicks in a certain order.

So it seems we can add process supervision onto agents pretty naturally. That means we’ll have a process reward model, a PRM, that gives out rewards at each step. That’s in addition to whatever reward we give at the end of the trajectory, which people sometimes call an outcome reward model or ORM, but often is just called a reward model, since outcomes are the default thing to give rewards on.

So if intermediate rewards are so valuable, and maybe increasingly necessary, why doesn’t everyone use them? We’ve covered agent RL papers before, and none of them have had intermediate rewards. Why not?

In my view, there are two reasons.

One is, it’s expensive to make process supervision data. There are just way more annotations per task. If you can avoid process supervision and just use outcome labels, that’s way cheaper per task, so you can get more tasks and more diversity etc. This is partly why RLVR data has largely displaced PRM data for reasoning. Of course, if you could get that data for cheap or even free, the balance might change.

The other, thornier reason is reward hacking.

Briefly, reward hacking is when the stuff you reward is only a proxy for what you actually want, and that gap between the rewarded thing and the actual goal causes unexpected or bad behavior. For example, you might reward a model for passing unit tests on a coding problem, but the model could learn to just hardcode passing behavior instead of writing the real program. Or even worse, it could learn to edit the tests to just always make them pass.

Reward hacking can happen whenever there are rewards. If you just have an outcome reward, then there’s only one possible hacking target per task. But if you add in intermediate rewards, you’ve added new hacking targets. So intermediate rewards already add some risk.

But there’s a deeper problem with intermediate rewards: they can distract or derail the agent entirely. The gif on the slide here shows one such example. It’s from a 2016 blog post from OpenAI, by Dario Amodei and Jack Clark actually, who later went on to found Anthropic. The post is called “Faulty reward functions in the wild”. In it, they describe some RL experiments they did in a boat racing game called CoastRunners. As humans, we know the goal of a boat-racing game is to place first. But actually, the game only tracks points, which you get from hitting these floating green blocks and then from your finish order. The most points wins.

When they set their RL agent loose on this game, they saw something funny: the agent learned to ignore the race and just steer the boat in a circle over and over again, racking up points from the same three green blocks. Often it just circled forever and never ended the race at all.

That is the danger of intermediate rewards. You have to pick something, often arbitrary or not strictly required for your end goal, and reward on it. Even with domain expertise and careful design, it can still go sideways.

On a different note, we also need to cover a training technique called “self-distillation”.

Let’s start with plain ol’ distillation, which is where you use one model to train another model, “distilling” the wisdom from the teacher model into the student model. Distillation has been in the news from time to time because OpenAI, Anthropic, and Google have all caught the major Chinese labs distilling off their models, which violates the terms of service.

The naive way to distill is to have the student copy each output token from the teacher in response to a prompt. Like if you penalize the student for outputting anything except what the teacher output, then the student will learn to sound more like the teacher.

But if you can go one level deeper, to the probability distributions of the teacher rather than just the one final token it spit out, that teaches much more richly. That’s what we’re looking at here. Sometimes people call that “soft-label distillation”, as opposed to the “hard-label distillation” I described earlier as the naive approach.

So the Chinese labs have been distilling off the American models by passing the same input to the teacher and the student. The teacher is better than the student, so the student learns and improves.

But it doesn’t have to be that way. In the paper from two weeks ago, they actually kept the model the same in both cases - same model for teacher and student - but they gave the teacher extra context. Just like having your friend quiz you, with them looking at the answer key so they can guide you to the right answer.

In the example here, the teacher for this SQL bot gets some examples in its context, whereas the student does not. So when you do distillation, the student is receiving the extra wisdom the teacher had due to its extra context, without actually needing that context in the prompt anymore. Since you’re using the same model in both cases, it’s called “self-distillation”.

Notice how flexible this can be. Anything you can put into words can be distilled. For example, apparently Anthropic uses distillation to teach Claude how to be Claude, with a big document in the teacher’s context containing Claude’s principles and operating procedure etc.

The paper from two weeks ago used context distillation to incorporate feedback from the environment. So the model would attempt a task, get some feedback like failing unit tests or a critique from an LLM judge, then put that into its context in the role of teacher. Then the model with its teacher hat on would basically grade every token probability in the original response, then train to minimize that loss. They called that “self-distillation policy optimization”, SDPO, and unlike other techniques like GRPO, it doesn’t require rewards that boil down to numbers. It’s a richer signal that naturally occurs in any system that provides feedback.

The Paper: RLAnything

Okay, so our first paper is called RLAnything. That’s because their proposed system is fully generic, across any sort of task, or at least that’s the claim.

They break down RL into three parts:

1. The policy model, which is the model actually generating responses or actions etc

2. The reward model, which judges process and outcome rewards, except for any verifiable rewards you might have that don’t need judging

3. The environment, which includes the prompt as well as the tools and state the agent has to navigate to accomplish whatever task

They’re going to be adjusting all three of these dynamically to maximize performance, which we’ll see in the next few slides.

While we’re here though, a couple things to sort out. One is some terminology: they often say “step-wise” instead of “process”, and they have a “critic” model that is actually a separate LLM not involved in training at all.

Second is the notation. I know the math looks very scary, but it’s just a lot of annoying symbols to keep track of, no exotic operations. Like for the left side, all they’re saying is the total reward is the outcome reward plus the average of the process rewards, which they request from the PRM m different times just for consistency. And then O and S, the outcome and step-wise rewards, are gonna be either 1 or -1, nothing in between. Also P stands for the policy model, R stands for the process reward model, and E stands for the environment.

Third, the PRM is going to return a score but also its reasoning, like a justification our contributors would write in an RLHF task. That will be helpful down the line as we’ll see, especially for separating the correctness of the individual step from the helpfulness towards a good outcome.

Now for their experiments they’re primarily testing on two benchmarks of a couple hundred tasks each:

1. AlfWorld, which is basically a text adventure game

2. OSWorld, the gold standard for computer use agents

Sometimes they report the results for AlfWorld as “LLM agent” and the results for OSWorld as “GUI agent”.

As for the models, they’re using Qwen2.5-7B for AlfWorld and Qwen3-VL-8B for OSWorld. Not the newest models, and not the biggest either, but quite common for academic work.

Here’s an illustration of how they adjust the task and environment to balance difficulty and novelty and hit that Goldilocks zone. Remember, the PRM is basically a classifier, and classifiers do best when the classes in their training data are roughly balanced. So we roughly want to have an equal number of good and bad steps to train on and thus improve our PRM, which is key to this whole approach.

On the top is an OSWorld example, where the reward model gives two accurate criticisms of the policy model. An outside LLM then reviews the trajectory and the criticisms and makes a modification, which allows the policy model to improve from 0% to 25% accuracy.

They also have AlfWorld and coding examples with the same idea, although in both cases it’s to make the task harder.

This graphic encapsulates their results, so let’s go through each:

1. On the first one, they’re showing that you get the best results when you make all three components of your system dynamic. So you want to train both your policy and reward model, and you want to adjust your environment to be neither too hard nor too easy, so that you’re always getting some signal but always leaving room to improve

2. On the second one, they’re comparing the value of process rewards and outcome rewards. So for the yellow line, it’s just outcome rewards, with human labels. Then on the blue line they’ve added a PRM, but they don’t train it, which maybe helps a little but shows a PRM per se is not the big difference. The red line however is only a PRM, but trained, and that ends up doing the best. So that shows the value of a good PRM if you can get it.

3. On the third one, they’re just showing overall improvement from their method. LiveBench is actually a coding benchmark and isn’t an agent thing, yet their method even works there

4. Finally, on the fourth one, they’re showing how their adjustment to environment and task difficulty progresses over time and correlates with the quality of the policy and reward models

No mind-blowing results, but again, the emphasis is on how general and automatic the method is. They didn’t need to make any data, including those risky intermediate rewards we talked about earlier.

And they didn’t need to customize the setup much. They did have to think a bit about what inputs the PRM received in each case - the prompt + a summary of past actions and observations for both, the last two images for OSWorld - and they did have to know where the metaphorical “knobs” on the environments were for difficulty adjustment. But that’s about it.

Here’s a closer look at the results on the benchmarks, with different components allowed to change in different rows. A couple quick pieces of terminology:

• OOD is “out of distribution”, like eval tasks that weren’t similar to the training tasks

• UT stands for “unit test”, how accurate the PRM was at writing unit tests for judging step quality

• “Detect” means accuracy at detecting bugs

So they’re improving across the board, with more dynamic components being better. A couple thoughts.

First, look at the difference between process and outcome reward accuracy between GUI Agent and LLM Agent. That’s a clean demonstration of how process rewards have two jobs: checking the correctness of the immediate step, and estimating how much that step helps get the right final result. So we see in LLM Agent they’re similar - it’s roughly as hard to do one as the other. These are short trajectories and each one needs to have the end goal in mind. But for GUI Agent, they’re quite different; it’s apparently quite easy to tell whether the agent actually performed the right action, but it’s often not obvious whether that action will lead to a good outcome. That makes sense for computer tasks, where you have to click through to see everything, it’s not all laid out for you at once. Like sometimes you have to click around to check out all the options and menus and so on.

The second thing is that you don’t need a perfectly accurate PRM to improve results. You can see for both agents that the improvement in the policy model is far greater than the improvement in the reward model in most cases. Even noisy signal is better than nothing, at least at these noise levels. Below 50% accuracy for the PRM I expect the policy model would suffer.

The Paper: OpenClaw-RL

Okay, so we’ve seen how process supervision can help agents. Now we’re going to see how to collect process supervision data naturally.

Because if you think about it, natural use of chatbots and agents doesn’t quite look like training data. Like if you’ve worked with agents at all, you know you’re often in dialog with them, providing feedback on their work and often not just leaving them to work on their own.

And nobody is really going to make nice training data out of their own usage. It’s a lot of work to get nicely scoped tasks and neat boundaries on trajectories and final outcomes and rewards. That’s why we get paid!

But surely there is a way to use the conversational feedback users often give, whether positive (like saying thanks or showing excitement) or negative (like scolding the model or making a correction).

Now this paper has much more of an engineering focus than the previous paper. So you’re going to encounter some training and hosting tools we don’t tend to cover here, especially because Scale focuses on the data side. So RL servers, Megatron, SGLang, slime, these are not important for us frankly. Just know they are the software that makes a lot of this all work.

So we’re going to be collecting and using feedback in three different ways here.

The first is basically a PRM, like in the previous paper. The rewards are binary, i.e. just -1 or 1 or 0, even though this graphic doesn’t say it. And they come in the course of conversation rather than waiting until the end. New words but same meaning really.

The genuinely new thing here is what context they give the PRM. Specifically, they’re providing the agent’s action, environment feedback, and user response. So if the agent makes a tool call and it returns some error about missing arguments, that’s probably gonna earn a -1. Or if the tool call works but the user complains that the agent is fetching the wrong information, also -1. Conversely, if the user seems happy, that could earn a 1. And if there’s no clear feedback or too little to judge on, the PRM can return 0. So this method rewards the user for more natural or expressive use of the agent.

The second method is also like a PRM, but with distillation instead of rewards. So it’s operating on the same single-step scale, and it’s using the same sources of feedback, but it’s putting them to work in different ways. First, the distillation method includes a lot of filtering and only uses clear mistakes for feedback, since the teacher is only helpful if the student has something to learn. Also, there’s some consolidation, where they use a model to summarize the mistake rather than just dumping the raw interaction into the teacher’s context.

The third method is pretty much what we saw in RLAnything: combining process and outcome rewards, PRM and RLVR basically.

The graphic doesn’t make the difference clear, but it’s important to separate the first two from the third. The first two methods are only stepwise, so they can run continuously, alongside all your interactions. The third one, by contrast, requires a definite outcome to then reward.

It’s a bit of an artificial marriage then, these continuous vs task-based methods. And actually the things they improve and the way they measure are different too.

For the continuous methods, the feedback from conversations, the researchers used two related examples: a student who wants to use OpenClaw to do his homework, and a teacher who wants OpenClaw to do grading in a certain way. Pretty subjective stuff, which makes sense with the more qualitative feedback their methods are able to capture, although the results are less compelling in my view. They have a scoring method, and you can see the improvement in scores on the right, but the scoring is just an LLM judge scoring the agent’s response given the user’s preferences.

This is the type of change you could prompt engineer, rather than something that absolutely has to be trained. So the value isn’t results per se, it’s how you can get these results “for free” just by incorporating natural signals.

They also have the more traditional objective results on benchmarks. Keep in mind they’re using Qwen3-4B across the board, so the gains are respectable. I don’t have much to add because this part is basically a rehash of RLAnything, but it seems their two new methods may be helping too.

My Takeaways

• Where is the role for our training data?

  • Extracting training data from natural use means it’s way cheaper (even though lower quality/hit rate) and already from the natural distribution

  • So perhaps the economics favor the low-volume, high-utility cases (e.g. medicine, frontier STEM)

• Can PRMs replace human-written intermediate rewards?

  • We know intermediate rewards can incentivize weird behavior

• Another step on the path to continual learning

  • Also similar in spirit to Karpathy’s Autoresearch

• This is probably already happening in some form

  • Big labs I’m sure

  • Possibly anyone with an agent and access to the weights

Paper · Repo · Hugging Face

Background

To understand this paper, we have to understand how a model on a GPU turns your prompt into predicted tokens.

Let’s start with what we already know from our daily use of LLMs: you pass in your prompt all at once, you wait a bit, and then you start getting back tokens one by one.

Already we can start to relate to this diagram. The part where you pass in your prompt is called prefill. During prefill, you are filling up the working memory of the model, what we call the KV cache. The diagram here kinda breaks up “KV” and “cache”, but you can see that the prompt turns into KV vectors, and then those get cached, hence “KV cache”.

So now we have all the input in our working memory. That can take a long time depending on the hardware you’re using and how big your input is, but importantly, prefill happens for all input tokens in parallel. Typical prefill speeds are in the hundreds or thousands of tokens per second.

So once prefill is done, the model is ready to start making predictions. And it makes those predictions one at a time, as the diagram shows: first “jumps”, then “over”, etc etc. This stage is called decode, because we’re decoding the mathematical representations that the model works with into words that humans work with.

Because decode runs one token at a time, it is much slower than prefill, typically in the tens of tokens per second.

Note that when we decode a token, it gets cached too - that’s why the dotted line at the top spans prefill and decode. So the KV cache, that working memory, grows every time the model outputs a new token.

Now let’s look a level deeper, at the hardware. On a GPU, you have your chip, and you have your memory, aka your VRAM. When you start up your model server, the server reads the model from your hard drive and puts it into your VRAM, right next to your chip.

When you actually use your model, like by sending it a prompt and getting back a response, your model server takes one layer of your model at a time from VRAM and sends it to your chip for computation. So if I’m at the very first attention layer, it’s gonna take my input and the matrices that actually make up the first attention layer and send ‘em to the chip for multiplication and so on. Then it takes that first attention layer back from the chip, along with the KV cache created from the computation, and it’s gonna send in the first feed-forward layer for computation.

That process of loading and computing and unloading happens over and over again until you’re at the final layer, where the chip can finally produce the predicted token. Then you gotta do the whole routine over again to predict the next token.

So this shuttling of weights to and from the chip is typically what slows you down - the constraint is your memory bandwidth, not the speed of your chip. If you can somehow compute multiple tokens at once in decode, like you do for prefill, then ultimately you can predict tokens faster.

The question is, how do you predict multiple tokens? If predicting a token requires understanding all the tokens before it, how can you predict more than one?

The answer is in the name of the technique: “speculative decoding”. Instead of making a brand new prediction every time, you speculate about what the next few tokens will be - you take a guess beforehand and then check.

Speculative decoding works for the same reason prefill is faster than decode: inputs get processed in parallel. As we said before, once you load the weights onto the chip, it’s quick to do one or two or three or four calculations. As long as you have a good way to guess tokens, the model can check them all in parallel.

Of course, if the first token fails, then the other ones you guessed after it will likely be wrong and you’ll have to throw them away. But if your method for guessing tokens is cheap enough, and you’re not wrong too often, it can work out.

One common method is to have a version of the model itself make the guesses. Specifically, a much smaller version, ten or a hundred times smaller in fact, so it’s much faster and also can fit on the same GPU. This “draft model” as it’s called is not nearly as smart as the target model that is actually producing tokens, but it’s often smart enough. After all, most tokens are not incredibly complex or subtle; language is chock full of common and supporting words, and a lot of sentences are pretty mundane, meant to support the occasional novel or surprising sentence. It’s even more true for code, which demands predictable structure in a way natural language doesn’t.

As the graphic shows, the draft model quickly produces a few tokens, which all go into the target model in parallel rather than in serial. And thinking back to the last slide about where the bottleneck is, because you now have multiple tokens ready for computation, you save all that shuttling from VRAM to the chip on the second and third and fourth tokens.

In the example here, we have indeed generated four tokens, but the third one gets rejected, and the target model’s predicted token takes its place. The fourth draft token doesn’t get checked at all, because it depends on the third draft token being correct, which it wasn’t.

Let’s zoom in on this example a bit more. We have our four speculated tokens from the draft model, and we’re going to verify them with the target model.

Specifically, we’re going to check if the probability of the speculated token for the target model is at least as high as the probability of that token from the draft model. Like in our case, the target model thought “Brown” was 93% likely, and the draft model thought it was 92% likely, and since the target model is smarter, we take the increased probability as a sign that the draft model was pointing us in the right direction. Similar story for “Fox”.

But for “Hopped”, the draft model was more confident in that token than the target model was. That’s a bad sign and means we should reject the draft model’s choice.

Incidentally, when the target model rejects the third token, it substitutes its own - in this case it’s the word “jumped”. That extra token you get from the target model when it rejects the token from the draft model is called a “bonus token”, because you get it “for free” in the process of verification. If you’re really lucky and all your speculated tokens get approved, you get a bonus token after that, directly from the target model. Like if all four tokens had been right in this example, we also would have gotten a fifth token as well, with virtually no extra effort.

Now as you might imagine, the draft model is going to be better at predicting some tokens than others. Like on a hard reasoning problem, the acceptance rate will be quite low, maybe like 25%. But on a more structured and straightforward task, like using a web search tool, it could be near 100%. So speculative decoding isn’t a complete across-the-board speedup, but for a lot of mundane LLM uses it’s helpful. It’s really an empirical question depending on your use cases, your hardware, what model you’re using, stuff like that.

I should add that there are other forms of speculative decoding, or of trying to predict multiple tokens in one go anyway. We’ll briefly see one called EAGLE-3 in the paper for example.

Another example, which we’re looking at here, is literally called “multi-token prediction” or MTP. The difference with MTP is that is has to be part of a model’s training from the get-go, it’s not an external enhancement.

You can see it right there along the top, at the boxes labeled “Cross-Entropy Loss”. The loss is the single number that tells you how well your model is doing. In a normal model, your pretraining loss is based on how likely you predicted the actual next token in the training data would be. That’s the first box along the top, it has an arrow pointing to L_Main - that’s the symbol for loss.

But here in MTP, there are multiple losses! As you continue along the top, you’ll see L_MTP^1 and L_MTP^2 - the losses for predicting the first and second of the multiple tokens. So now your loss is from the normal token, the first MTP token, and the second MTP token. And if you do training right and minimize the overall loss, you can get pretty good at predicting multiple tokens.

Fittingly, modern MTP itself is an invention of the DeepSeek crew, although Meta invented the original version. Many other models now come with MTP. You can think of it as the baseline speculative decoding approach, which EAGLE-3 and DSpark and others must surpass in order to add value.

One type of speculative decoding setup is DFlash, pictured here. It uses the classic setup of a draft model speculating tokens for the target model to verify, not the MTP baseline we just saw. Like MTP before it, DFlash will serve as the base from which DeepSeek innovates, and may serve as the same springboard to popular deployment.

DFlash uses a diffusion-based model, which means it outputs multiple tokens in parallel. That’s actually what the “D” in the name stands for. By comparison, in a standard LLM you only output one token at a time. That’s called “autoregressive”. Even MTP is autoregressive, because the second speculated token depends on the first, and the third depends on the second, and so on.

Putting out tokens in parallel means faster prediction, which means faster throughput for your target + drafter system. However, predicting tokens in parallel usually leads to worse quality, since language is inherently serial, where the right next word depends heavily on what came before it. Like if two similarly likely predictions are “Of course” and “No problem”, then I might end up predicting “Of problem” or “No course” instead.

To me it’s kind of surprising that diffusion LLMs work at all! We’ll come back to how DFlash mitigates the quality loss in a minute.

So in this diagram, DFlash takes in one token from the target model and outputs three tokens in parallel - the green ones, which start life as the special token “<mask>” but end up as real predictions. Those green tokens at the end - “speculative”, “decoding”, and “<eos>” (a special token which stands for “end of sequence”) - are the speculative tokens that the target model will then verify. Again, pretty similar to the standard speculative decoding setup with an autoregressive draft model predicting three tokens out.

Even the guts of DFlash are similar. The draft model starts with an embeddings layer, which pretty much all models do, turning tokens into vectors that the model can work with. In fact the embeddings layer is from the target model directly, it’s not even part of the draft model. Then the two main layers - “bidirectional attention” and “MLP” - are much like the standard attention and feed-forward networks of a typical LLM. And the two layers together form a block, the dashed box in pale green, with multiple blocks in a row forming the bulk of the draft model. Then at the end there’s a language modeling head, exactly like in a typical LLM, that basically undoes the embedding and produces tokens again. That also is directly from the target model.

Now we come back to the twist that DFlash introduces to improve diffusion quality: the boxes in blue, which they call “Fused Target Context Features”. That’s a real mouthful, but all it means is they take some of the target model’s thoughts, or “hidden states”, and put them into the draft model’s brain at every block. That’s much faster than trying to feed all the previous output into the draft model, because the hidden states are already compressed and compact. It’s also higher quality, because the target model is much bigger and smarter than the draft model.

So now that the draft model can put itself more in the frame of mind of the target model, it can make better predictions. It still can fall prey to the perils of parallelism, like the “Of problem” example I gave before, but empirically it seems to do significantly better.

So this idea of speculative decoding, of adding this nearly-free verification step, depends on memory bandwidth being the bottleneck. Like if the weights from the current layer could just zap onto the chip, then the bottleneck would move back onto compute, onto how fast the chip could do the matrix multiplications to predict (or verify) the next tokens. You only get the “free” win of speculative decoding because you have this idle time waiting for the next weights to arrive.

But there actually is another way you could tip the balance from memory bandwidth to compute: adding more tokens for verification. Like if you tried to speculate and verify the next million tokens, you surely would spend more time verifying than you would waiting for the next layer to get to the chip. So somewhere between zero and a million there is a crossover point, where you’ve used up all the idle time with speculation.

In theory that’s the optimal spot: you have made use of otherwise-wasted compute availability, and you have not delayed the load of the next set of weights. Below that point, you’re leaving compute on the table. Above that point, you’re certainly not benefiting, and given speculative decoding is not accurate 100% of the time, realistically you are losing.

Now in practice a single user would never come anywhere close to this crossover point - you can’t realistically predict more than 10-ish tokens out at any likelihood of success. But for multiple users, in batch processing, it’s quite possible to overload the verification queue and slip past the crossover point, or “knee” as it’s sometimes called. And of course a big lab like DeepSeek is thinking about the batch case, because they’re serving tons of users and squeezing the hell out of every GPU.

So: we know we want to be right around the knee to maximize our compute throughput. What this doesn’t say is how to maximize token throughput, which will depend on maximizing compute throughput with the right balance of batch size and speculation size, but also on maximizing the accuracy of our speculation. And that is what DSpark is all about.

The Paper

So this is the DSpark setup. We have our target model on the left, and our draft model on the right. The job of the draft model is to speculate tokens for the target to verify. In this case, given the most recent target model output “D”, the draft model will ultimately return tokens “E”, “F”, and “G”. The target model verifies E and F but not G, which the target replaces with the proper next token, “G*”. All totally standard.

What’s new is inside the big box on the right. We start with our parallel block, which contains between 1 and 5 draft layers, just like with DFlash. Each layer comprises attention and feed-forward, again like DFlash.

The two new additions happen after the parallel block produces its predictions, here called “Logits” because they are not tokens yet, they are distributions of token likelihoods. So going back to our parallel decode example from before with “Of course” and “No problem” colliding to become “Of problem”, the first position’s logits might say 70% odds for “Of” and 30% odds for “No”, while the second position’s logits might say 40% chance for “course” and 60% chance for “problem”. If we just take the most likely token for each position, we’re gonna end up with “Of problem”, which we want to avoid.

Since we have those probability distributions though, we still have the chance to tweak them to produce better outcomes. And that’s what this sequential block is all about. It’s a super lightweight way to add a little bit of sequential information to our otherwise parallel streams. So now for our second position, if we know we predicted 70% for “Of” and 30% for “No” for the first position, we can use that information and make “course” the more likely choice. That will add a tiny bit of time as we’ll see, but the accuracy gains will be worth it.

Now improved accuracy is always useful no matter the setting, but thinking back to our batch serving wrinkle, we also need the other new part: the grey boxes and the blue bar at the top.

The grey boxes are confidence scores, a score from 0-1 that DSpark assigns based on how likely it thinks the token is to be accepted. The blue bar, the scheduler, looks at how busy the GPU is and basically decides how big of a risk to take. Like if the GPU is super busy, then there’s not much idle compute between cycles of target model weights loading, so the confidence bar will be high and the speculation length will be short. Conversely, if the GPU is pretty free, then you might as well keep even the unlikely tokens in.

Remember, the parallel block generates the same number of tokens regardless, it doesn’t cost more to generate more at this scale. So the strategy is to generate a bunch, do a quick pass on which ones are likely good or not, and then filter from there based on capacity. In this example, apparently H was below the confidence threshold, so DSpark dropped it.

Now if we just focus on the first new feature, that sequential block that gives the parallel tokens a bit of information about the tokens before them, we can already see a big advantage in the average number of draft tokens accepted.

Across every benchmark and across all four models, DSpark is a strict improvement over DFlash, by about 10-20%. It’s also significantly better than another speculative decoding technique, Eagle3, that we will touch on a bit later.

One clear trend across benchmarks is the impact of domain on acceptance length. For math and code, which have more structure than natural language, the draft model can often predict pretty long sequences. By contrast, chats from a wide variety of users are often hard for the draft model to predict responses to.

Within the benchmarks, you can also see that easier prompts are easier for the draft model to predict responses to. Like GSM8K, which stands for “grade school math 8000”, is not hard even for small models nowadays. But AIME25, which is competitive math, is harder and thus less predictable.

So it seems the context greatly influences the risk-reward ratio for speculative decoding of all sorts. Hence the confidence scores! The DeepSeek folks love this trick of adding more learning in different parts of the model, like with mHC and Engram - two previous papers of theirs we’ve covered.

Here’s another way of comparing results across methods and domains. On the x axis is the position of the draft token, and on the y axis is how often that token was accepted, given all the other tokens before it were accepted. Because remember, as soon as one token in the draft sequence is rejected, the rest of the tokens after that in the sequence are discarded.

Before I explain the results, it’s important to remember that draft models have a hard latency cap, because they have to generate and send along their tokens in that bit of time where there is idle compute. So when we compare drafters, we expect roughly similar speeds.

However, we do not expect roughly similar sizes. That’s because for a given size, parallel models are much faster than autoregressive models. So if we hold speed constant instead, we expect parallel drafters to be much bigger. And bigger models are generally smarter.

That’s why DFlash, a parallel model, does so much better than EAGLE-3, an autoregressive model - at least at the start. Once you move past the first token though, DFlash starts to run into that “Of problem” issue, what the authors call “suffix decay”. The less structured the text, the worse the problem bites. That’s why the scores on Chat are lower overall, and why EAGLE-3 in particular rises so sharply. Remember though, this is all conditional on the prior tokens being approved, which on an absolute scale just gets less and less frequent the further you go in draft token position. So EAGLE-3 being better late-game doesn’t help as much as it might seem just from looking at these graphs - you have to keep the overall results from the previous slide in mind.

Anyway, that all is ultimately background for the DSpark results, which once again are superior across all domains and draft positions. That’s because DSpark blends the parallelism (and thus greater input context) of DFlash with the autoregression (and thus greater consistency) of traditional LLMs and methods like EAGLE-3. We haven’t even gotten to the hardware-aware part, which is novel.

Now so far we’ve been measuring just one particular design of DSpark, when in reality there are two clear design knobs: the depth, i.e. the number of attention + feed-forward layers in the parallel block; and the block size, i.e. the number of tokens the parallel block predicts.

The previous results used five layers and seven for the block size. So now they’re gonna turn each knob a bit on both DSpark and its predecessor, DFlash, which has the same two knobs.

Although the layer knob isn’t even worth turning on DFlash, because DSpark with just two layers beats DFlash with five layers on every single benchmark. Five still improves performance and apparently fits the latency budget though, hence their choice in the previous slides.

Now for the other knob, here they have fixed the layers at five but have moved the block size up and down.

Overall, greater block size there on the x axis does help, but the rate of improvement falls off quickly, especially above eight. On the negative side, latency increases directly with block size. You can see why they picked a compromise value, although it’s not clear to me why they settled on seven when they measured eight and when slightly higher values seem defensible. Maybe they had a maximum latency threshold and just picked the biggest block size under that? I’m not sure.

Also, I know there are two different DSpark trends on here, but the curves are so similar that it’s not worth getting into the difference between Markov vs RNN. It’s just two different ways of structuring the autoregressive part.

So far all the results have been hardware-agnostic, not taking the overall workload of the GPU into account. Now we can get into the hardware-aware part, which I think is the most DeepSeek-y aspect of the paper, given their reputation for squeezing every last bit of performance out of the limited hardware Chinese labs get access to.

As I mentioned at the top, DSpark looks at the available idle compute and decides how big a swing to take. If there’s lots of compute that would otherwise sit idle, DSpark takes bigger swings, since even low-likelihood bets could still end up with positive expected value. On the other end, if you have little or no spare compute - if you’re at or past the knee of that graph from the background slides - only a pretty likely sequence will have positive expected value, since you’re displacing a smaller number of guaranteed accurate tokens.

Now in general, confidence is going to decrease with token position, since it’s harder to guess tokens that are further out. But you could have some more predictable ones after actually. Like for a code problem, if the name of the function is somewhat arbitrary then the confidence may be low, but the parenthesis after it may be highly confident because that’s part of defining the function. So the way they deal with this is by calculating cumulative confidence when they do the cutoff, i.e. the confidences of each token up to the given position multiplied together. That guarantees the overall confidence is monotonic, it can only be flat or go down.

Anyway, as you’d expect, the higher the confidence threshold, the higher the acceptance rate but the lower the number of draft tokens DSpark gives to the target model for verification. The lowest confidence buckets in particular seem like free wins, basically no harm in cutting out those cases.

Here’s what I call the “money graph”. Not directly for the economic benefits of their efficiency gains, but because it really drives home how big a win a seemingly small change to DFlash made.

This is real production data from serving their current models. Each tiny dot is one combination of individual user speed vs overall GPU throughput. So they were able to collect a ton of empirical data on DSpark.

There are two ways to read this graph. One is at a trend level, where DSpark is clearly superior to MTP; the lines don’t overlap at all, and the gap is huge in some regimes.

The other is at a performance level, with one aspect fixed and the other one left to compare. That’s what the dotted lines are doing. So for example, with DeepSeek V4 Flash, if your GPU only needs to do about 1.5k tokens/second, then each individual user is going to see 85% faster throughput with DSpark compared to MTP. Or, starting from the same dot on the MTP trend - about 120 tok/s for each individual user - then your GPU can do 661% more tok/s overall. That’s crazy performance gain for practically no extra model size and latency!

My Takeaways

• This seems like the right new default for speculative decoding

  • Just when MTP was getting popular!

  • Luckily, you can add DSpark to a model later on, whereas MTP has to be part of pretraining

• I don’t think DSpark portends the triumph of full-blown diffusion LLMs (dLLMs)

  • Draft models don’t need to be that good to be useful

  • Perhaps standalone dLLMs will be useful for easy tasks where maximum speed is essential

  • Google calls DiffusionGemma “experimental”

• Always learn where possible

  • This is what DeepSeek did with mHC and Engram, they did it here too

  • Where else can we be learning?

mHC: Manifold-Constrained Hyper-Connections

or, The Importance of Being Normalized

Paper

Attention Residuals

or, Attention Really Is All You Need

Paper

Background

So since this is a head-to-head paper review, we have to talk about the two fighters in the ring.

On the left we have DeepSeek, one of the two S-tier labs in China, the other one being the Qwen folks. DeepSeek is an outgrowth of a hedge fund called High-Flyer, and it has spent a lot of its life as a fun science project rather than a hard-charging startup. That culture of exploration has produced some highly respected research, including the birth of the modern MoE model and the public discovery of how to train reasoning models, which previously only OpenAI had discovered with o1 and o3, but kept private.

We covered one of their most recent innovations, the Engram memory system, back in February. The paper we’re covering today, mHC, came out just before that.

Everybody thought DeepSeek would release V4 of their flagship model around Chinese New Year, but that was weeks ago and so far we’ve heard nothing. I suspect the combination of high internal standards and little commercial pressure has them working on improvements rather than shipping something middling.

Now on the right we have Moonshot, makers of the Kimi series of models. Moonshot is a hard-charging startup, solely dedicated to making commercially viable LLMs. They were founded in March 2023. Since then, they have released several versions of Kimi, most recently Kimi K2.5, a 1T parameter model. They also just raised $1B and plan to IPO in Hong Kong soon.

Kimi has won praise for its overall performance, but in particular its personality and writing skills. I think it’s the “Claude-iest” Chinese model, which actually would make sense, since recently Anthropic accused Moonshot of distilling from Claude. Possibly relatedly, Cursor used Kimi as their starting point for their most recent coding model, Composer 2.

On the science side, they traditionally haven’t released a ton, although they did claim some engineering breakthrough related to the optimizer Muon.

Okay, now we can get into the technical piece.

First, let’s recap transformers. To oversimplify somewhat, transformers have three pieces:

1. Embedding, for turning text or other inputs into vectors

2. Attention, for understanding how different parts of the input relate to each other and forming a holistic view

3. Feed-forward network, for “thinking” or processing that understanding. In this diagram the feed-forward network is a MoE, a kind of sparse network. In other transformers it’s a dense network

Embedding happens once at the start, then attention and FFN layers trade off over and over again, dozens of times. One attention plus one FFN layer is called a transformer block. In this diagram though, they only have four layers, and they don’t show the division into blocks.

Now in reality there are other minor components, like RMSNorm or similar, but we will put them aside for now.

The only thing left to note is these arrows and plus signs, which show how information flows and combines between layers. That’s what we need to take a closer look at.

This is where residuals come in. Mathematically it’s very simple, but in the diagrams I used to find it confusing.

Here “h” is a hidden state, basically the stuff that moves along inside a model and that the attention and FFN layers process. Then at the end the hidden state gets turned into a token or a class or whatever other thing your model is predicting.

It used to be that hidden state passed directly from layer to layer, like the output of one layer would be the input of the next layer. But then, back in 2015, some researchers at Microsoft decided to add in this other component, the residual - the hidden state from before the previous layer.

In the formula at the top, that’s h_(l-1), the first term on the right side. The second term is the output of the previous layer, where the function f_(l-1) is either attention or FFN. The little plus sign in a circle in the diagram literally means addition - you are adding the output of the most recent layer to the original input for that layer, and that’s what forms your input for the next layer.

So that’s what residuals are. Now the question is, why do we need them?

To understand that, we have to look at the two directions information flows in a transformer: forward, when you’re doing inference; and backward, when you’re training.

The forward case is more straightforward. If you only have the output of one layer as the input of the next layer, you can lose sight of the original input. It’s like playing a game of telephone - changes and errors can stack up. But if you add the residual connections, you’re always providing an unaltered version of the past layer’s input.

And if you look at the formula from the last slide, you can actually write an equivalent version, which I’ve shown here. Notice in the new version that the hidden state for the current layer relates directly to the hidden state entering the first layer; the current state is the original state plus all the layers. If you didn’t have the residual connections, you would lose that h_1 term. So the residual connections let information flow directly between layers.

So that’s the forward pass. Now for the backward pass, we have to get a little more sophisticated.

When you are making a model, you ultimately are trying to minimize some loss, some formula you’ve written that describes how far off your model is from ground truth. So in a linear regression for example, a very simple mathematical model where you’re just trying to fit a straight line to a series of points on a graph, your errors are the distances between each point and the line, and your loss is the sum of the square of those differences.

Loss for a LLM is really not that different. When you’re doing pretraining or SFT, your loss is related to the probability your model gave for the actual next token in the training data. When you’re doing RL, your loss is related to the rewards you’ve calculated.

The big script L in this formula is the loss. On the left side, what we’re asking for is how to calculate the change in loss if we make changes to the input at a certain layer. And if you look on the right side, you see that it depends on the changes to the loss due to the input at the final layer, capital L, and the changes to just the attention or FFN outputs at each layer. It’s just the chain rule from calculus, applied to the giant mathematical equation that is the LLM.

Note something important here in the formula though: that capital I, which is basically the vector equivalent of 1. That’s due to the residual connections. And without it, if enough of those local changes are close to zero, then the whole right side basically ends up at zero. This problem is called “vanishing gradients”, and the more layers you add, the more of a problem it will be.

So the residual connections allow your training information, your measurement of how any tweak to the system will impact the loss, to persist.

So that’s your basic residual connection setup. It works great, everyone uses residuals, problem solved. Right?

Well we’re going to see about that in a minute, but one final bit of background to introduce, or perhaps reintroduce, is The Bitter Lesson.

The Bitter Lesson is an essay by famed machine learning researcher Rich Sutton, pictured here as a GPU enjoyer. His basic point in the essay is that because computing power grows so quickly, scaling up “dumb” or simple approaches always beats “clever” or complex approaches that humans design. For example, when a machine beat a human at chess back in 1997, it didn’t have lots of sophisticated chess strategies in its memory; instead, it searched the space of possible moves by brute force, and it just so happens that in 1997 that was finally technically possible.

The same thing is true for the field of machine learning that deals with language, “natural language processing”. People spent a long time trying to encode the rules of human language into machines, but then along comes OpenAI and just dumps an Internet worth of text into a model and throws enough GPU time at it to learn to produce human language.

The way I think of it is, learning surpasses engineering. If we can learn something rather than design it, we should.

The Papers

So both papers want to change how residuals work. They just disagree on the best way to do it.

I’m going to reground us in the two forms of the residuals equation from Moonshot: recursive, on the left; and iterative, on the right. Remember, these are mathematically equivalent.

I also need to introduce the DeepSeek versions of these equations, on the bottom row. They are mathematically identical, it’s just that the DeepSeek version includes the weights W of the attention or FFN that the Moonshot version omits.

What would our man Rich say about this?

Yes, more learning!

So let’s do more learning. Onto both of these equations we’re going to add new learnable parameters.

For DeepSeek on the left, it’s these matrices H, in three different varieties. We’ll see exactly what they are in a later slide, but for now just remember that the H matrices - “Hyper-Connections” in the jargon - are our new learnable parameters.

Note that the bottom equation, the even more fiendish-looking one, is the iterative version. So that’s directly comparable to iterative equation from Moonshot. I just didn’t want to make too big a leap at once.

Now for Moonshot on the right, the knob to turn is alpha, a single number rather than a matrix. That single number comes from some matrix math though, so I guess you could say we’ve abstracted the complexity away from this equation.

Anyway, the point is they’re both on to the same insight: the implicit equal weighting across all the layers for standard residuals is leaving something on the table.

Now that we know how DeepSeek and Moonshot’s approaches are similar, we can precisely parse their differences.

Here’s a graphic from the mHC paper laying out their changes in two steps. We start with the standard residual connection, adding the original input of an attention or FFN layer to the output of that layer.

To go from connections to hyper-connections, we have to split the input into many different channels, many different parallel lanes, which is why they show some of the hidden states as multiple boxes now. Mathematically, we’re taking the vector x at the very start of the model and just copying it n times, so that x is now a matrix with n identical columns. But the columns are all gonna go on their own journey so to speak through their individual channels in the model, only to be collapsed back into a single vector all the way at the end of the model. So if we look at a random layer in the middle, the different channels are going to have different values of x_el.

Now we can get a little better sense of our H matrices. We have H^res for adjusting the original input, H^pre for adjusting the input before it goes to the attention or FFN layer, and H^post for adjusting the output of that layer. That’s three new knobs per layer, across all the layers. And the extra channels allow more information to flow as well.

So this Hyper-Connections part, the middle graphic, is actually not from DeepSeek - it’s from ByteDance, and the paper came out in September 2024. So what did DeepSeek contribute with their updated version in December 2025? And why didn’t anyone actually adopt Hyper-Connections in the 15 months between the two?

Well in addition to drawing attention to it with their respected scientific brand, DeepSeek pointed to a different component of the transformer for inspiration: the normalization layer.

We haven’t talked much about normalization because it doesn’t really process information in the same way the attention and FFN layers do. It’s more of a preprocessing or post-processing step, to make the hidden state better suited for the next layer.

The basic idea is simple: if your layers are only trained for a certain shape or distribution of inputs, then you have to get your outputs back into that shape, or you risk unexpected behavior. Usually this shows up as failed training runs.

Normalization is a feature of all models, and even appears in the original Transformer paper. But you don’t need separate normalizers for residuals because the hidden states are already normalized!

This is what the original Hyper-Connections authors neglected, that the DeepSeek team fixed.

On paper, all they did was add this normalizing function, abbreviated as P_M, to the H matrices.

In practice though, they did a ton of gnarly engineering to make it work at scale. In the end, their technique only increased the required compute about 7% on the 27B test model they trained, but it took a lot of cleverness to get there.

By the way, the “manifold-constrained” part of their technique is just a fancy way of saying the H matrices have to have all the rows sum to 1 and all the columns sum to 1. That range of potential matrix values forms the “manifold” that the H matrices live on. It’s like the unit circle, which contains all the x and y points where the radius is 1, but in way more dimensions.

So that’s mHC: more channels, and a stability trick to make it actually work.

Now let’s look at Attention Residuals from Moonshot.

Next to our friend in image A, image B has two key changes.

First, instead of direct connections only between neighboring layers, there are direct connections between every layer. Like you can see the Embedding layer at the front touches all the attention and FFN layers, and then the first attention layer touches all the future layers, etc etc.

Second, there’s this alpha number instead of a plus sign. That’s our coefficient that tells us how much to pay attention to the residual. And if we look between images B and C, we see a diagram describing how alpha is calculated.

I’ll come back to that in a minute. Right now I quickly want to acknowledge image C, which is a version of image B but altered for scalability. We’re not going to get into it really, but just know that when you’re training a big model, you have to split up work between GPUs, and that causes problems with the normal attention residuals system.

So, back to alpha. Again, that’s going to say how much each residual should factor in.

And each connection gets its own alpha. In the formula there we see the first one between layer zero and the current layer “el”, then alphas for each connection to “el”.

To calculate alpha, they’re going to use the oldest trick in the Transformer book: attention.

Remember, the job of attention is to decide what is important in the input when deciding on the output. And remember, we only predict one token at a time, so that outputted token becomes part of the input on the next go-around.

Mechanically, attention does this by taking the input and looking at it through three different lenses:

1. Query - the input as a question, as the thing that guides the output

2. Key - the input as the available information, as the stuff to possibly pay attention to. The query times the key tells you which connections between tokens are strongest, which other tokens to look at closely for any given input token

3. Value - what each token represents, what it actually means. Like a dictionary definition, but richer

That’s all standard stuff as a layer in the transformer, but what does it mean to use attention for residuals?

Well, two things. First, the query vector is actually NOT coming from the same place as the key and value vectors. It’s this vector w, which is specific to each layer even though it doesn’t have the subscript “el” in this diagram. This w vector learns how much of each prior hidden state to incorporate in the current hidden state.

Second, the key and value roles actually translate quite well. Each little green block there represents the hidden state from an earlier layer. In this case, we have the hidden state from the first four layers. So Q dot K is going to say how much of each hidden state we want to incorporate - that’s alpha - and then that times V is going to provide the actual information from the hidden state.

So we’ve taken the attention mechanism and turned it 90 degrees so to speak, looking at relations between layers instead of relations between inputs. We just had to be a little bit clever about the queries.

Now frankly it’s going to be challenging to compare the results of these two methods. Even though they work on the same part of the model, they tested by training two completely different models in terms of size, architecture, and data. So let’s start out with individual results.

For mHC, they picked two metrics: quality and stability.

For quality, they show it two ways:

1. Top left, there’s a graph of the relative difference in loss during training. Negative numbers mean the loss was relatively lower than baseline. So we see here that both HC techniques work, although mHC works better

2. Bottom, there’s a table of where their chosen benchmarks landed at the end of training. It’s a fine range of benchmarks, covering language ability, commonsense reasoning, math, STEM, code, and question-answering

From those two mHC appears to have a tiny, possibly insignificant edge over HC.

But then you take a look at the top right, which is a proxy for stability. The thing on the y axis, “gradient norm”, is basically a measure of update size. What you’d like to see is a smooth, gentle downward curve, bottoming out just as you stop training. In practice you always have some spikes, but the overall trend should look like the baseline in grey.

And you do get that with mHC. But you don’t get that with just HC, because HC doesn’t normalize to that manifold we discussed earlier. So you’re risking instability here, like wasted cycles or even wasted whole pretraining runs. For a toy model that’s fine, but when each pretraining run costs millions of dollars, you cannot afford instability like that.

So the story of mHC seems to be: 7% more overhead, for a little better quality at no risk. Probably a good trade.

The Moonshot folks have gone for something similar here. We’re looking at absolute loss instead of relative, there in the top left, and we have a wider array of benchmarks. But it’s the same story: a modest increase in quality across the board.

And on the stability side, we get two graphs from training: the size out the outputs per layer in the middle, and the size of the changes from training (aka the gradient) per layer.

For the middle graph, what you normally see is that later layers have to use bigger numbers, since normally the residuals keep adding to each other. Bigger numbers risk instability, or loss of signal depending on how you look at it.

For the right graph, it’s the same story but backwards; the initial layers in a standard transformer have to make huge changes in order to impact the final layer.

But with AttnRes, in both cases you’re pretty flat in magnitude.

Luckily for us, the Moonshot paper does compare directly to mHC in one table.

It looks at models of increasing size, with other aspects like training data growing in proportion. And it seems here that AttnRes is on par with mHC, even when you have to make engineering compromises as Block AttnRes does.

So to me it seems the simpler method, AttnRes, is the winner. But it also makes me wonder if there aren’t other solutions at work in the closed labs.

My Takeaways

• Moonshot may have bested DeepSeek here

  • Not exactly fair though, given mHC came out before AttnRes

  • But also, mHC is a change on top of HC, so perhaps a smaller contribution

• AttnRes is more elegant

  • But mHC is more impressive - the math they pulled out to get the H matrices constrained to that manifold is arcane

• Perhaps Moonshot will break into the S-tier with Qwen and DeepSeek

  • I doubt DeepSeek will regress to A-tier

Paper · Website · Repo · Leaderboard

Background

So, what are computer use agents?

A computer use agent (CUA) is an agent that can use a computer just like a person would: looking at a screen, mousing around, clicking buttons, using the keyboard, and so on.

Unlike most agents, which are built for purpose - coding, deep research, specific jobs like that - a CUA is general; in theory, it can do anything that is doable on a computer. The term is more about the mode of operation than the purpose.

Now practically speaking, there are limits. Like in theory, you could solve the Riemann hypothesis while using a computer, just writing down math in LaTeX. Obviously that’s not what comes to mind first when you think about a human using a computer, but it’s important to get our definitions and distributions right when we’re designing benchmarks like OSWorld.

So what folks really mean by “computer use” is more like “use of common graphical programs for white collar or personal tasks” - email, spreadsheet, CRM, shopping, stuff like that.

I want to emphasize graphical here. Agents are already quite good at using the command line and writing code for work that isn’t graphical, that is purely text. And tech companies know that, so a lot of tools that are naturally graphical for humans have gotten text interfaces for agents, or already had APIs that agents could call directly. That’s honestly preferable; after all, language is the primary medium of large language models - it’s right in the name. And while humans are good at language too, we are especially good at vision. So where we can, we want textual and graphic versions of tools available.

But sometimes it’s not possible, or at least not practical.

For one, some tasks are inherently visual. Like if you want your agent to design an infographic in Figma, or to CAD up a part, or to edit a video in YouTube Studio, you really need it to use vision. Parts of each task may be textual and thus amenable to the command line, but there’s no getting around the visual parts.

On a more practical level though, many programs that could be more textual and LLM-friendly simply won’t - the developers won’t add the features, or the admins won’t permit that type of access, or some other organizational barrier crops up. So if your agent has to be a drop-in replacement for a human, with no special accommodations, then it has to be good at computer use.

I always think of the DMV for this latter case. Like that software has gotta be ancient, and there’s no way they will overhaul their systems any time soon. So even though the work probably isn’t that inherently graphical, you probably have to deploy a CUA to automate that work - at least for now.

Now that we know what CUAs are, we should talk a bit about how they work, which will help contextualize some of the benchmark design decisions and failure modes we’ll see later on.

The primary way folks add computer use to their models is by feeding in screenshots. That requires a model to have vision capabilities in the first place, and there are things you can do to the screenshot to help the model, but that’s the basic approach. The model then emits pixel coordinates and actions, like “click on (148, 273)” or “drag from (318, 997) to (509, 1011)”. That’s a totally general approach, operating pretty much like a human, but it comes with a couple downsides.

For one, vision is just not that strong of a skill for most models. They miss some details and fabricate others, at a higher rate than with text. Labs care more about text generally.

For another, vision is token-intensive. A single screenshot, optimized for computer use, is roughly in the low thousands of tokens. That’s not much on its own, but if you want anything like real-time vision, that’s going to add up quickly. In fact, that token cost is a major reason computer use tends to happen with smaller models. (The other reason is speed.)

Now the other approach folks sometimes use is to read some lower-level representation of what’s on screen. This is mostly relevant for browser tasks, where the model can read the DOM, which is basically the code that the browser translates into something visual for the user. If you’ve ever done “inspect element” in Chrome and then edited some of that code to make something change on the page, you know what I’m talking about. Another example here is the accessibility tree (or “a11y tree”), which originally was for vision-impaired humans to use while operating a computer.

The lower-level representations are nice when you can get them, and many agents do take a hybrid approach, but many programs don’t have them. And of course some things are still inherently visual, so no lower-level representation can do them justice. So pure vision is inescapable to some degree.

Weirdly, even though computer use in many ways is behind compared to other areas of AI, computer use agents appear fairly early in the Gen AI era.

For example, Anthropic released its first CUA in October 2024, alongside a new version of Sonnet 3.5. It didn’t release Claude Code until February 2025, alongside Sonnet 3.7. Similarly, OpenAI released its first CUA, Operator (shown here), in January 2025; it didn’t release its coding agent, Codex, until April 2025.

Usually it’s benchmarks that come first though, kind of setting the target that the model makers need to go out and hit. And to that end, the first CUA benchmarks of the era appeared in 2023, not long after ChatGPT came out. For example, Mind2Web, which came out in mid-2023, comprised 2,000 tasks across 137 real websites, but offline versions. Since no real vision-equipped model had come out yet, the CUA navigated the DOM. There were a couple vision-only benchmarks, like Android in the Wild, but they used bespoke vision encoders rather than the general vision encoders on a vision-language model like you’d find today.

Once GPT-4V came out in September 2023 though, the door for truly general CUA benchmarks was wide open. One of the earliest, and certainly the most enduring, was OSWorld.

In that early crop of CUA benchmarks, a few things made OSWorld stand out.

For one, it was supremely general: it plopped the CUA in a Linux or Windows or macOS desktop, with all the normal programs, and just gave it a task. It was the first benchmark to really say, “this is what computer use should look like”.

Not only was it general, it was also cross-program. For example, in the first task over there on the right side, you can see the agent switching between a spreadsheet and an image viewer. The entire computer is “in bounds” so to speak, the benchmark isn’t contained to a single program that happens to be running on a computer, in an operating system etc.

Another key element was the engineering behind it. The OSWorld folks built some nice infra, allowing other researchers to replicate and build on their efforts. The paper emphasizes their approach to virtual machines and environments and initial state setup, all topics that still matter today. And of course, if the teams building models can use your infra, of course they’re going to run evals on your infra.

So as a result, OSWorld became the standard computer use benchmark. Anthropic has reported results since it released its first CUA, and so has OpenAI. Google started reporting with the release of Gemini 3 Flash. All of them continue to report on it today, although technically the later results are on a cleaned-up version called OSWorld-Verified.

The first CUAs were understandably terrible, scoring no higher than about 12%, while humans scored 72%. However, there was remarkable progress in 2025, and by early 2026 the leading models all hit human parity. So now is the perfect time for OSWorld 2.0.

The Paper

Let’s start with a concrete example from OSWorld 2.0, keeping the examples from OSWorld 1.0 on the previous slide in mind.

High-level it’s a straightforward request: read some reimbursement instructions and apply them to this specific case. Not terribly different from that first example task in OSWorld 1.0.

However, several differences emerge upon closer inspection. For one, just look how many steps there were - the last screenshot says Step 499! That’s way longer than anything in OSWorld 1.0 as we’ll see when we look at duration and step distributions.

Another key difference is the number and variety of programs. Here we have a PDF reader, an expenses portal, and email. For OSWorld 2.0, not only did they add more programs like FreeCAD, Blender, and Obsidian, they also added 31 self-hosted websites - clones of Gmail, Slack, LinkedIn, Salesforce, Twitter, Eventbrite, YouTube and more. It’s a nice example of how AI coding progress unlocks progress in other areas of AI.

OSWorld 2.0 also introduces a new taxonomy of work types - what they call “challenge phenomena” - to describe more specifically what the agent has to accomplish along its trajectory. It’s hard to compare the whole list directly with OSWorld 1.0, but some are certainly new, like the Dynamic Environment example in step 372; new information arrives while a task is underway, and the agent has to notice and incorporate it.

Overall, OSWorld 2.0 is harder: longer, more novel, and with more wrinkles.

Here’s a closer look at how much longer they made the tasks. It’s measured in human operation time, i.e. how long it takes a competent human to complete the task.

Keep in mind this is a log scale, so the difference is more dramatic than it might visually seem. I’ll also note the significant right-hand tail, where the top 50% of tasks by length are more spread out and reach past ten hours in a couple cases. The net impact on agents is about a 10x increase in steps, whether successful or not.

Another way to read this is an indication of CUA time horizons. OSWorld 1.0 came out in April 2024, and two years later it basically saturated, meaning it took two years for models to be pretty reliable at tasks taking a single-digit number of minutes. With OSWorld 2.0, published at the end of June 2026, we are ready to measure tasks taking a single-digit number of hours. This is the closest I’ve seen to a GUI version of the famous METR time horizons graph.

Moving from length to novelty, we get a couple breakdowns of what the tasks are asking for and what tools they entail.

On the left, they give a breakdown by domain, with categories somewhat influenced by economic sectors. They spend some time in the paper trying to tie their domains to various slices of GDP, but it seems more like an add-on to compete with explicitly economically-oriented benchmarks like Remote Labor Index and GDPval. You can read more about that category of benchmarks in my breakdown of Agents’ Last Exam, which the authors actually cite as a “close comparison” but without the focus on graphical tools. Anyway, the tasks do take more inspiration from the real world, and the shift from scientific framing to economic framing indicates to me how close to production CUAs are - or will be, once they can climb this benchmark.

On the right is a more concrete breakdown, this time by program used. Note that most tasks require at least two programs, so the total here is well over the 108 total tasks in the benchmark, and they mention elsewhere that the average count of programs per task is 2.44.

Anyway, we see a browser is required for a majority of tasks, with equivalents for Word, Powerpoint, and Excel rounding out the double digits. MailHub and TeamChat are Gmail and Slack, respectively, and they happen in the browser so there’s actually some double-dipping between them and Chrome/Browser. Regardless, most of the programs and all the web apps are new in OSWorld 2.0, so still an impressive increase in novelty.

Now given there are only 108 tasks in the benchmark, it’s not surprising to see so many programs appearing only once, but such small sample sizes in so many domains does make me hesitate on any fine-grained analysis. As we know, it’s expensive to produce good benchmarks in significant quantities!

And turning to the wrinkles, which themselves are often novel, we again get a table of overlapping entries; as we saw in the example slide, a task has many challenge phenomena.

The names are fancy, but the concepts are pretty simple. In fact, I think most people would have a hard time parsing these challenge phenomena out if you asked them to document the difficulties in their own computer use trajectories. But as we know, artificial intelligences are jagged, and things we find easy are often hard for agents - and vice versa.

Right near the top is one about vision and perception, which we could have anticipated as being quite common. And just above it, Cross-source Reasoning, is a wrinkle they consciously introduce to turn up the difficulty compared to OSWorld 1.0, so no surprise it’s at the top.

But the next few I find more interesting. You would think that with such large context windows and such great intelligences, models would be better at inferring and tracking and deciding which conflicted source takes precedence. But as we’ll see in the results slides, models still struggle to process such information.

The rest are less interesting or less common, but I wanted to show them all here since they will all appear later on.

Now before we get to the results, of course I wanted to highlight their pipeline.

Coming into the paper, I expect them to harvest existing artifacts and then reverse engineer tasks out of them, but that’s actually not what they did. Instead, their main method of task creation was what they call “brainstorming”, in the green box toward the bottom-left. I’m going to quote their description directly:

“The bulk of OSWorld 2.0 tasks were produced by a small group of internally trained annotators who worked end-to-end on the tasks they proposed, covering task design, input artifact construction, environment setup, and evaluation function implementation. Annotators first learned the target domain by watching tutorials on YouTube, reading official documentation, and directly experimenting with the software, which equipped them to reason about professional tools and workflows, or enterprise systems in sufficient depth. They then drafted candidate tasks grounded in workflows surfaced from Reddit discussions, online tutorials, and their own day-to-day work experience, with each draft specifying the instructions, required input artifacts, and expected final state. Every candidate was finally peer cross-checked by a second annotator, who reviewed it for feasibility, redundancy with existing tasks, and ambiguity in the evaluation criteria… This channel ultimately produced approximately 90% of the final tasks.”

Once they have the task inputs, or “spec sheet” as they call it, they turn them into an environment with an initial state that an agent can enter into and complete the task within. So thinking back to our example task from the top, they have to put the instructions PDF on the desktop, populate the inbox with relevant and possibly irrelevant emails, log the user into the reimbursement portal, etc etc. That’s a human-led process, and in that process some holes in the initial task design emerge.

One other key step here is defining and running the evaluator, which checks for correctness at the end but also along the way. The details are somewhat opaque, but they have many so-called “checkpoints” per task, just over 27 on average. The checkpoints don’t prescribe an order, so different paths that end up at the same state are valid. Mostly the checkpoints are automatically verifiable, like whether a certain value ended up in a database, but some do require model judgment - about 11% of all checkpoints, with no task having more than 50% model-judged checkpoints. Also, they don’t use the term “rubrics” to describe it, but basically that’s what they write for the model judge to use.

So now with a completed build, they test it. First they have two humans test it and check for agreement, then they have some agents test it. If it turns out the task isn’t solvable given the inputs and the environment, or there are ways to cheat, or the checkpoints don’t fully reflect the completed state, they send it back for rework.

Overall I think it’s a decent pipeline, and I’m not surprised they got most of their tasks from human rather than synthetic inputs; models tend to produce contrived and arbitrary tasks, with clumsy additional constraints to increase difficulty.

So with our 108 tasks in hand, we can now evaluate some models.

A couple notes before we dive into the table. First, this is for a budget of 500 steps. They do also provide results for 150- and 300-step budgets on their website, but they focus on the max budget setting so that’s where we’ll focus too.

Second, we should define some terms. On the columns, Binary means the share of tasks where the model got every checkpoint. Partial means the average score per task. Note that it does not mean the share of all possible checkpoints achieved across all tasks, because that would favor tasks with higher numbers of checkpoints.

On the rows, Batched means that the model took multiple actions in a step before observing and thinking about the results, whereas Single means the model looked and thought about each action before deciding on another one. GPT-5.5 only offers batched.

So with definitions out of the way, we can see clearly that Opus 4.8 is the absolute performance champion. However, that title comes at a cost, with more than double the cost for pretty marginal gain against Opus 4.7. GPT-5.5 meanwhile is the efficiency king, with far lower cost and especially output tokens per task. Opus and GPT used the highest thinking settings, so really the story here is GPT seems more efficient with its thinking tokens.

Digging in on this point, they have a couple graphs showing how thinking level impacts outcome. The bigger the dots, the higher the thinking level.

If you measure turns, then it seems more thinking quickly produces better results for the frontier models. Given some of the failure modes we’ll see later, more turns typically means the model noticed or realized something it had to do while in the middle of a task, and that less intelligent or less hard-thinking models let a lot of that subtlety slip.

If you measure tokens though, suddenly the story muddies. For GPT, our efficiency king, more thinking continues producing better results quickly. But for Opus, especially Opus 4.8, the curve is pretty flat. Since output tokens are the main driver of cost, GPT may be the better model, but only if you can accept partial results; note the change in y axis, with Binary for the left graph and Partial for the right. Bit of an odd choice by the authors in my view.

One comparison I wish they showed is on wall time. It’s tricky to compare between models, since different companies have different serving infrastructure, but ultimately what we should care about in economic terms is the rate of completed work per time. Then you can compare the dollar value of that completed work to the dollar cost of the job, and ultimately come to a net income per time. It’s another example where I feel the economic framing they tried to add with the GDP tie-in and comparison to Agents’ Last Exam is half hearted.

Speaking of time, the human-measured time horizon pretty well correlates with difficulty for the models, as we see on the left.

Similarly, on the right we have human-measured difficulty in three buckets: easy, meaning it took a human under 30 minutes; hard, meaning it took a human more than two hours; and medium, falling between 30 minutes and two hours. If we then bucket tasks by model Partial score - over 70% for easy, under 30% for hard, in between for medium - we see a similar correlation. As expected though, it’s much more common for an easy human task to be a hard model task than vice versa. Do note that percents are normalized by row though, so you really have to read it as “for a given human-predicted difficulty, what was the spread of empirical difficulty?”

Now beyond the top-level statistics, it’s instructive to see what exactly the agents are doing in their trajectories.

Here the researchers had GPT-5.5 review all the trajectories and categorize each step, with some human review afterwards. They provide 15 categories, grouped into four phases.

A couple patterns stick out to me. One is how similar most of the models are across phases and even across categories for the most part. Like if you take any of the models and sum across phases, it’s roughly 30-35% Reasoning, 30-35% Perception, 20% Action, and 10% Correction.

Relatedly, the models likely spend too little time on correction. Obviously the ideal is for a model to never make mistakes in the first place, which would also produce very low numbers for Correction, but as we know from the top-line results these models are far from perfect. Any regular user of agents knows they tend to be overconfident, so no surprise here, but error correction and reflection are increasingly important as the horizon lengthens; the more steps you take, the more likely you are to introduce an error, and if it propagates without correction you are virtually guaranteed to fail any task past a certain length.

Two is how different GPT is on a category level. I suspect some of it is due to GPT also being the judge, but the extra exploration, information extraction, and tool-semantics reasoning does seem like a substitute for the missing planning - a more efficient substitute, apparently. Relatedly, they show in another chart that GPT greatly favors text interfaces over graphical interfaces, and pretty doggedly looks for the text version of an interface. When it works, it works well and is typically more efficient, but it’s also brittle. The authors give the example of manipulating the DOM rather than interacting with a website graphically, which is more direct but also obscures layout-based information and cuts out helpful guardrails.

Continuing on the micro analysis, the authors revisit their taxonomy of “challenge phenomena” and see which phenomena are indeed the most challenging.

Now they don’t make this split in the paper, but I want to distinguish strongly visual tasks from the rest of them, because visual acuity is distinct from the general focus and planning the rest of the challenge phenomena test.

And you can see it right away in the differences between models. Among the three here, GPT is known to have the best vision abilities, and M3 the worst, which we see reflected in the results of the top two vision-heavy phenomena.

If we put them aside, then, we see pretty consistent results across most of the phenomena. Like apparently deriving implicit information and knowing when to ask the (simulated) user for clarification and keeping lots of things in working memory and noticing changes in the environment that are crucial for the task are all significant hurdles. It’s not an issue to work across multiple programs or artifacts per se, it’s more about keeping them all straight at the same time and knowing where the holes are. Gaia2, another updated version of an early agentic benchmark that I covered somewhat recently, drew some similar conclusions.

So like a relatively easy OSWorld 2.0 task might jump between a few different programs but only require one at a time, with little cognitive overhead or state to mentally keep track of. Like the expense report example from the top, if all you have to do is hunt across email and local files and Slack but each expense is self-contained and can go into the expense tracker one by one, that can be quite long but apparently not that challenging. It’s only when you add in new emails mid-stream, or conflicting sources of information, or a clear hole like a return plane ticket without a departing plane ticket, that you really start to challenge frontier models.

One final note: the easiest challenge phenomena, Cross-resource Reasoning, is actually the most common. It seems to me more like an intrinsic part of the benchmark to work across sources rather than a wrinkle to add in for increased difficulty.

To make the failures concrete, here are a few examples.

The first one is an example of Dynamic Environment, where a new Slack message with corrected information goes unnoticed or unused.

The second one shows Streaming Interaction, where a pop-up moves and eludes the clicks from the agent. Personally I found this one pretty contrived.

The third one is Multimodal Editing, which explicitly includes CAD reconstructions. Even though the agent can use the software perfectly, it misperceives the plans and thus produces the wrong part model.

My Takeaways

• 2026 is the Year of Computer Use Agents

  • 2025 actually was the Year of Agents

  • Now that models are good agents, labs can focus on computer use specifically

  • Anthropic probably released Sonnet 5 for computer use

  • I predict an agent will cross 50% on OSWorld 2.0 by EOY 2026

• CUA data should be hot

  • Plenty of YC companies working on it already

  • Snorkel AI contributed to OSWorld 2.0

• More benchmarks will tie themselves to economic indicators

  • For many applications, the era of science is over and the era of deployment has begun, which means dollars come into play

  • See my breakdown of Agents’ Last Exam

• Interactive benchmarks will become more common for agents

  • Fully specified and self-contained tasks do not reflect how users work with agents

  • See Scale’s recent SWE-Interact for another example

Paper · Blog post · Model weights · Demo

Background

To frame all this, we need to acquaint ourselves with DeepSeek, the lab that makes the DeepSeek models.

Or maybe we’re reacquainting, because we’ve covered them a few times before. Specifically, we’ve covered:

• DeepSeek V3.2, the last major release from DeepSeek

• mHC, a super technical improvement to a very specific part of the model architecture

• Engram, a sort of built-in memory for common phrases

I’ve linked all of those talks in the speaker notes if you want to go back. But for those who don’t know or may have forgotten, DeepSeek is the leading scientific lab in China - not always the absolute performance leader, not cranking out models, but always pushing the envelope. In many ways they are what OpenAI set out to be: an open, research-focused organization dedicated to achieving ASI and sharing it with the world.

Unlike OpenAI though, which for a long time subsisted on what it viewed as charity, DeepSeek has always had a sugar daddy: a quant hedge fund called HighFlyer, whose founder decided to spin up this ASI effort with his profits. And that missing profit motive, that journey of discovery, comes through in a lot of what DeepSeek does today.

For one thing, they publish a lot of their innovations. Going back to my list above, mHC and Engram are techniques really only usable at labs pretraining their own models from scratch. From a competitive angle it seems crazy to give away techniques like this, but from a scientific angle it’s completely standard.

For another thing, they don’t put out many models! DeepSeek V3.2 came out on December 1st, 2025. That’s almost half a year ago, which feels like years in model release time. There were rumors they wanted to publish V4 before Chinese New Year, as some other Chinese labs actually did with their newest releases, but apparently they just weren’t ready and didn’t want to put out something not up to their standards.

DeepSeek is also one of the national AI champions in China, which puts it in a tricky position. On the one hand, the power of the Chinese government is behind it in various ways, including a new upcoming funding round I just read about today. But on the flip side, they have to start adopting Chinese chips.

Historically, all the labs in China have used NVIDIA chips basically, because for this type of workload they’ve been the only game in town. But with recent US export controls, China has lost access to cutting-edge chips, and given the importance of AI, the CCP does not want to rely on foreign companies or governments for such a critical resource.

As a result, the Chinese government doesn’t want NVIDIA chips - they’re fine with the export controls! - and instead wants homegrown chips in all the AI labs. Right now that means Huawei chips, specifically the Ascend line. DeepSeek V4 was a major test for the Ascend chips, with Huawei and DeepSeek working closely together to make training and inference work. In fact, there’s some speculation that the delayed release was primarily due to the switch in chips.

It’s okay if the information on this table mostly escapes you - the point is that Ascend chips are behind their NVIDIA counterparts mostly. And the relative underperformance of Ascend chips means DeepSeek has to focus on efficiency, not pure performance, since they’re unlikely to completely match OpenAI and Anthropic and Google.

The other hardware-related thing is at a data center level, which is utilization. Utilization is basically “how often are my GPUs actually doing something” vs waiting for instructions or data to arrive. It’s really, really hard to get high utilization; for example, xAI’s mega data center, Colossus, apparently runs at about 11% utilization. US companies have access to the right hardware and can just scale up; DeepSeek does not - at least for now - so it has to be clever.

In general, the DeepSeek approach has been to accept engineering complexity in exchange for efficiency. They are cracked engineers and they want to push the envelope.

One expression of that crazy engineering and efficiency is the price.

What we’re looking at here is pricing for a bunch of recently released models. You’ll see three types of price: input, output, and cache hit.

Input and output are straightforward, it’s just the price you pay per million tokens you pass in and receive back, respectively. And on those dimensions, DeepSeek V4 is pretty good but not exceptional; other Chinese labs with similarly capable models do about as well, like Kimi K2.6 and GLM 5.1. Of course compared to Opus 4.7 and GPT-5.5, it’s already a bargain - you can see why Anthropic and OpenAI are getting those massive valuations.

But the third bar, the cache hit, is where we really see the engineering prowess and relative altruism of DeepSeek come through. “Cache hit” tokens are tokens that haven’t changed since your previous turn, either from the system prompt or from earlier parts of the same conversation. Those tokens can live in storage, cheap storage like a SSD. And that matters a lot when you’re using agentic scaffolds with big system prompts and doing very long trajectories, rather than a bunch of shorter conversations in a chat window.

Anyway, as you can just barely see on the chart, DeepSeek V4 Pro only charges a single cent for 1M tokens in cache hit. That’s basically free! All these long histories all our agents are generating, they cost practically nothing if you’re using the DeepSeek API.

I don’t want to get into exactly how they do that, partly because it’s in the paper and not part of the background, but it just shows what kind of lab we’re dealing with here.

We also need to do an attention refresher, as attention is the area with the most innovation in DeepSeek-V4.

We’ve covered it before, but to oversimplify somewhat there are three components of a transformer model, which is the basic architecture all your favorite LLMs use.

1. Embedding, which turns words into numbers, since after all we are ultimately working with numbers in the form of matrix multiplications here

2. Attention, which calculates how each individual word or token relates to all the other tokens that came before it in order to form a complete, holistic understanding of the input and output so far

3. Feed-forward, or MLP as it’s noted here, which takes that complete understanding and “thinks” or “processes” it

For the attention layer in particular, you see each input splitting into a query, a key, and a value. The thing to notice here is how the queries and the keys form this matrix, where each query gets to “look at” or calculate against each key of the previous tokens. So like “The” at the start can only attend to itself, whereas “tokens” at the end can attend to all the tokens in the example. And within each row, like on each query, the circles are colored by how strong the relation is. So on the query “to” for example, it’s attending strongly to “The”, weakly to “model”, strongly to “attends”, and weakly to itself. Sometimes those relationships are a bit inscrutable to humans but in this case it generally makes sense.

The only other thing I would briefly note here is that the attention and feed-forward steps or “layers” as they’re called form the “transformer block”, and the transformer block repeats many times before the model is finally ready to predict the next token. What that attention matrix looks like in terms of the strength of relations changes from block to block.

So in the standard transformer model like we just saw, the model pays attention to everything - up to its maximum context window anyway. That would be the first box on the left, “Full n^2 attention”. And it’s called n^2 because the number of green squares goes as the square of the number of tokens. That means if you double the number of tokens in your context window, you’re quadrupling the compute and memory needs. It’s the gold standard, but like gold, it’s pricey.

People spend a lot of time trying to find cheaper attention mechanisms that are nearly as good. Often you’ll hear them called “linear”, since they tend to use compute and memory in direct proportion to the size of the input, which makes a huge difference once you get out to the 100k or 1M token range.

One of the more popular ones is Sliding Window Attention (SWA), also seen above. Here you specify a window size that is much smaller than the full context length. At first that means you’re only getting information about very nearby tokens, but as you keep going through the layers of the transformer, you start to indirectly get information from tokens that are further and further away. Sort of like a game of telephone: if I initially get information directly from my neighbors one day, and they do the same with their neighbors, then the next day when I check back in with them they’ll have information from their neighbors to share with me too. And if you do more and more layers, more and more days in our metaphor, then eventually you can get information from the whole previous context.

There are more exotic forms, like c for example. And then for d, a lot of models have different attention designs interleaved, like one layer with full attention and three layers with sliding window attention for example.

DeepSeek has gotten far more exotic than just a tweak to sliding window as we’ll see. But the key thing to remember is that attention dictates how much information you can actually pull out of the input, and that you might choose different attention mechanisms for different balances of quality and cost and context window size.

Here’s a quick comparison of full and sliding window attention next to DeepSeek Sparse Attention (DSA), which is the foundation on which the most recent model’s attention mechanisms are built.

As the captions highlight, DSA is like SWA in that your query token only attends to a certain number of prior tokens, letting it use far less compute than full attention. However, DSA learns which tokens to attend to, rather than relying on a hard rule like SWA. The component that does this learning, that picks which tokens the query is going to attend to, is called the “lightning indexer”. Keep that in mind for future slides.

The Paper

So high-level, this is what we’re working with: a Pro model with 1.6T parameters and 49B activated, and a Flash model with 284B parameters and 13B activated. I added on a couple other stats that basically explain the size difference - Pro is just bigger on every dimension.

I don’t have a good explanation for the slight mismatch in pretraining tokens, but I think it may be from Pro getting more agent trajectories.

So here is the overall architecture, for both versions of the model. I know it’s a lot, but we’ll go through it node by node. And remember, the important parts of a transformer are the embedding, the attention, and the feed-forward.

The first part at the bottom is where we pass in the input, turning it into numbers, into vectors. That’s the simplest part.

After that you’ll see a blue dotted line around most of the diagram, designating the transformer block. At heart it’s one attention layer and one feed-forward layer, with some supporting players in the mix. As I mentioned in the background slides, the transformer block gets repeated over and over, L times, after which the model is ready to make its prediction at the prediction head. There’s a layer after that about MTP, multi-token prediction, which we don’t need to get into but basically allows the model to predict a few tokens at once instead of just a single token at a time.

Now within the transformer block is the real innovation.

One innovation is mHC, which we covered in a previous paper and which I also don’t want to rehash too much here. But basically, mHC is all the grey stuff - the mixing, the row of circles that expands and condenses etc. It lets the model learn how much to actually care about what happened in the most recent attention or feed-forward layers.

The real show is the attention layer, here labeled as CSA / HCA. Those are two new forms of attention that allow DeepSeek-V4 to be so crazy efficient, and that’s the core technical stuff we’re gonna explore on the next slide. They interleave CSA and HCA, like using CSA in one transformer block and then HCA in the next one etc.

I’ll also note DeepSeekMoE as the feed-forward layer, but that’s also not new; basically all models competing for SOTA are MoEs.

Finally, for anyone who remembers the Engram paper, DeepSeek V4 does NOT use Engram. I expect it to appear in a DeepSeek release eventually, but that may not be until DeepSeek V5, since Engram has to be part of pretraining.

Now we are ready to tackle the big innovations: CSA and HCA.

We’ll start with CSA. It may help your intuition to think of a high-resolution picture you need to reduce the size of. The CSA approach is to keep the pixels in a circle around one spot, then go lower res on all the other pixels, then black most of them out for not being relevant enough. You’re getting some fairly precise but sparse global signal, and keeping the precise local signal.

Back to the model. In this diagram, each little yellow or green bar represents one token, or really one “hidden state” of a token - basically what the token has turned into after some number of transformer layers. The hidden state is like the model equivalent of electrical signals in your brain.

So we have our one query token, and we want to decide which of the KV tokens to attend to, and maybe transform the tokens a bit more first.

The first thing we’re gonna do is compress the KV tokens, in two ways: we’re gonna use smaller vectors, and we’re gonna group tokens together. You can actually see that in the diagram, where the bars are now half the height AND there are fewer of them, something like 4x fewer in this case.

The next thing we’re gonna do is use the lightning indexer to figure out which of these already-compressed tokens we want to attend to. That requires the query token and the KV tokens, and it puts out a list of the top KV tokens to care about. So now we only have three yellow bars at the top.

Finally, we do want to compensate for some of this compression by paying closer attention to nearby tokens. Specifically, we have our sliding window of 128 tokens there towards the top-left, which is gonna add in with the selected compressed KV entries.

So now from the query’s perspective, when it’s attending to all these KV entries, it has some high-fidelity ones in the 128 tokens before it, then a pre-selected group of kinda summarized tokens. And it’s gonna attend to all of them.

Now for HCA. As the name implies, it’s like CSA, but even more compressed. They’re just gonna crank up the compression factor by a lot, like from 4 in CSA to like 128 here. And because everything is so compressed, they don’t need the lightning indexer to trim the number of tokens.

Of course they still have the sliding window KV entries, so nearby information is more preserved. But generally the goal of HCA is a global view of everything. Thinking back to our image and pixels from before, HCA keeps that same radius of original pixels, then makes the rest of the image super low res.

Now if you did only one or the other of CSA and HCA, you probably wouldn’t get good results. The magic is the alternation of these two, kind of superimposing them, so you have your fuzzy global view matching up with your selective global view. Then if a selected CSA block lands on a big fuzzy HCA block, that little higher-res bit gives you a much better idea of what’s going on in that big fuzzy block.

Here’s the upshot of all that innovation. Compared to the previous DeepSeek model, the slopes on the lines for compute (left) and memory (right) are drastically lower for both versions of V4. This is despite Pro having 1.3x the activated parameters of V3.2! And it really starts to matter once you get into the 100k regime.

Speaking of V3.2, here’s where we net out when we compare base models. In case you’ve forgotten, base models have no post-training on them. It’s actually become quite rare to release base models, even when the post-trained models are open weights, so these are by far the best base models available today. Again, it’s the DeepSeek devotion to science shining through.

Anyway, V4 is generally an improvement over V3.2, even when comparing with the much smaller Flash.

Now when you look at the post-trained versions, you can finally compare to the competition that the end user will have in mind.

Before we draw any conclusions about DeepSeek V4, we should look at the distribution of benchmark scores between the other models. Opus 4.6 and GPT-5.4 are generally regarded as the best models on this list, yet they often are not the winners. So we have to take benchmarks with a grain of salt.

Anyway, it seems like DeepSeek V4 is competitive, but rarely on top. And from what I’ve seen of the vibes, that’s about right; DeepSeek has certainly not closed the gap with OpenAI or Anthropic or even Google, and they’re generally competitive with the other top Chinese models.

I wanted to show a bit about the post-training process, but the information is scant. The one interesting thing for this group is that they do use rubrics, although there’s no other information about them. They actually use the same model for generating and judging, and they claim they improve the model’s judging abilities alongside its generating abilities when they’re doing RL.

They also do a somewhat common thing now where they post-train a bunch of different versions of the model that are each experts in one area, then they distill those models down into one final model. I don’t think that would change what post-training data they want, more of a fun fact.

To complement the benchmarks, they do run one eval: a small set of white collar tasks, in Chinese, like writing reports and making nice-looking documents. They don’t give any numbers on contributors or sample size unfortunately.

In the SxS rankings, DeepSeek seems to hold its own against Claude, and on the pointwise criteria it does about the same or a bit better. Maybe the tasks being in Chinese was a major factor though, because I would be surprised if this held up at large n or in English based on what I’ve been hearing.

My Takeaways

• Attention innovation still has juice to squeeze

  • Maybe someone will hit 2M this year

  • I’m more bearish on alternative attention mechanisms now (e.g. Mamba)

• We’re not that far from “tokens too cheap to meter”

  • The Bitter Lesson strikes again

• US and Chinese ecosystems are diverging

  • The US will probably ban the import of Ascend chips on national security grounds

Paper · Blog · Repo · Benchmark · Model

Background

So, what are world models?

We’ve actually answered this question once before, back in November 2025, when I covered a paper called Benchmarking World-Model Learning. In it, we talked about how there are two ways to predict the future: statistical inference, and world modeling.

Let’s say you have a fair coin, and you want to predict whether it will land heads or tails. The statistical way to understand the odds is to gather a bunch of data on past coin flips, then calculate the odds as 50-50. From that finding, you infer the odds of the upcoming coin flip are also 50-50.

This is what language models generally do today. They have a ton of data about which token comes next given a series of previous tokens, and when you use them - when you run inference - you are asking for a statistical prediction, a distribution across all possible next tokens.

Now the world modeling way to predict the coin flip is actually much more familiar and intuitive to humans: look at the coin, feel it, see that it’s symmetrical on both sides, and think through how a coin flip actually works. It’s a rule-based approach, or maybe a heuristics-based approach given we’re not actually calculating any physics in our heads when we imagine a coin flipping. But ultimately it’s about how the world reacts to the action of the flipping.

This is not how language models work, at least not directly. Like the mechanics of LLMs is statistics and correlations and so on, but one level of abstraction higher they arguably are thinking, and it seems any thinking entity can think through the impact of actions on the world as we discussed with our coin flip. Like if you asked Claude right now to think through why a coin flip is 50-50, you’d get something pretty familiar back.

So the potential is there. The question the other paper asked is, can we activate it? If we subject the agent to a series of tests where world modeling beats statistical inference, will the agent make the leap?

The answer in that paper was: no, not really. At the time at least, agents were not adept at mentally simulating worlds like the one pictured above, which is from another benchmark called ARC-AGI-2 but is similar in spirit.

Another paper from around that time, Code World Model, gives us a look at explicit world modeling, where the agent actually writes out what each intermediate step looks like.

In the image here, you see a simple program for counting the number of letters in a word. Once they define the function, they call it to count the number of r’s in the word “strawberry”.

What Code World Model then does is to take the inputs and walk through each step of the code with them, tracking the values of each variable. So like in the first frame, the variable “s” holds “strawberry” and the variable “t” holds the letter “r”. Then in the second frame, the program defines a new variable “n” with value zero, which Code World Model then tracks in the third frame, etc. At the end of the program, where it returns a value, Code World Model has thought everything through and knows the returned value is three.

The goal of all this is not to substitute for running programs, but to engender greater understanding of code in order to improve coding performance. Like imagine if you were trying to write code by just copy-pasting stuff you didn’t understand and trying to run it. Obviously you could get somewhere - I’ve just described vibe coding after all - but you would hit a ceiling pretty quickly. You have to understand the rules and the “world” of code to work well within it.

Now one area of modern AI that is moving to world models already is robotics, now known as “physical AI”. Again, we covered a relevant paper called DreamZero, back in February 2026.

The dominant paradigm of physical AI in the gen AI era has been vision-language-action models, or VLAs. A VLA is basically an LLM with vision input added at the start and action output added at the end. So the core intelligence is an LLM, and it gets extra parts and extra training to be able to control a robot.

I always found that kind of odd. Like why would an expert in language, which is an abstraction of the world, be best suited for understanding the most concrete aspects of the world? Wouldn’t it be better if the native medium of the intelligence matched the world it was working in?

Well apparently some researchers found that odd too, so they switched the core of the model from language generation to video generation. After all, if your model can generate accurate videos of the world, doesn’t that mean it can model the world quite well? And then all you have to do is translate the predicted video into the actions for making that future come true, like having the robot use its gripper and move its arm to hit a cymbal with a drumstick.

These world-action models (WAMs) are starting to take off. I expect to see more of them.

World modeling in physical AI is making another impact too: it’s allowing researchers to synthetically create videos of rare events for training data.

Also in February 2026, Waymo announced their own world model, built on the Genie 3 world model from Google DeepMind. Waymo’s world model generates video and other sensor outputs for events that are too rare to appear in their real training corpus, but are still likely enough to need training on. For example, they have no real training data of snow on the Golden Gate Bridge, but it plausibly could happen, so they should train on it.

They give other examples too, like a reckless driver veering off the road in a certain way, or an elephant crossing the street.

So a world model can be a helpful counterpart for an agent in the same domain, simulating new experiences and providing coverage that the environment so far has not produced.

One other concept I’d like to bring in that isn’t in the paper but kind of lurks in the background is reward models.

A reward model is a model that numerically scores whatever you show it. So for example, you might train a reward model on safe vs unsafe prompts, with one for “safe” and zero for “unsafe”. Or you could feed it a partially complete math problem and score potential next steps, with one for “likely correct” and negative one for “likely incorrect” and zero for “no impact”.

Reinforcement learning with human feedback (RLHF) is probably the most prominent use for reward models. In RLHF, you start with a prompt and feed it to the model you want to train, which we’ll call the “generator”. You have it produce two responses, typically at two different temperatures, and then show the pair of responses to a human. The human then ranks the two responses, usually on a 1-7 scale, with 1 meaning “response A is much better” and 7 meaning “response B is much better”. You then train a reward model on these preference ranks to learn human preferences.

With your well trained reward model, you score responses from the generator on a bunch of new prompts. If the reward model is any good, then it can basically interpolate or simulate human preferences and thus train your generator way more scalably and granularly than a reasonable amount of human effort could.

So there’s a clear parallel: generators are to reward models what agents are to world models - at least when you use the world model to simulate an environment, rather than when you use the world model directly as the generator or agent etc.

Keep this parallel in mind for the takeaways at the end of the talk.

The Paper

So high-level, Qwen-AgentWorld is a language world model (LWM): working only in the medium of language, it models various worlds - search engine, mobile phone OS, terminal, and so on.

I think of it as the converse of an agent. So like an agent observes the environment and acts on the environment. The converse of that would observe the agent and produce the actions - or maybe the reactions - from the environment.

Sometimes I picture the LWM as a dungeon master from Dungeons & Dragons. Like in D&D, the player takes actions, and then the DM decides what the in-game consequences are. Except in this case the “game” is, like, surfing the web or using the command line.

The LWM itself might even use tools, like how a DM uses dice, but mostly it’s about following the logic of a system mentally and inventing aspects as needed.

Here is what that looks like in practice.

The first example is for software engineering, where the input is a tool call for running a terminal command and a description of what the command is trying to accomplish. Now it’s the model’s job to output the results of that command, just by thinking about it - no tool calling allowed. The result is an error and a full stack trace, just like you would get if you ran this command in the given context.

The second example is for Android, just tapping on the “Buy Now” button in this fake app. Now since Qwen-AgentWorld is a language world model, it can’t directly produce the pixels for the resulting screen; instead, it writes the HTML that then renders into the resulting screen.

I dwell on this now because it takes a change in mindset to reverse the inputs and outputs like this. We’re so used to agents acting in the world, it’s weird to see the model receiving those commands instead of giving them.

Now as you may have noticed, a lot of context was missing from the examples on the last slide. Like, how would a model know whether some script will crash or not?

Well that context is available to the LWM, but it’s in the system prompt, not in some outside tools it can call. Here’s an example for the command line, aka the terminal, which says not only what the model is for but also where it’s supposed to be, what its environment is like. Calling back to our D&D analogy, the system prompt is like the dungeon master’s guide for the campaign.

Note the text on the right side in italics: some sections are static, others are dynamic, meaning they change depending on the task. I think the division is pretty intuitive though, like the rules of the terminal “world” are always the same, but the conditions vary. One wrinkle however is that for the MCP and SWE domains, the action space actually does change per trajectory, because different MCPs and different software repos will have different tools available. Like obviously the MCPs for Notion and Salesforce will be different, and for software repos you at least have to deal with different programming languages and packages.

Now in terms of the data, they have three distinct phases: continual pretraining (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL).

Continual pretraining is basically what it says: it’s pretraining, so basically reading a bunch of text and trying to predict the next token every time, and it continues the existing pretraining. CPT happens on top of a base model, before post-training. The data in this case comes in two flavors:

1. World knowledge, which will give the world model more detailed information about relevant domains than what was in the original pretraining

2. Trajectories, which show how environments from the relevant domains respond to agents

The trajectories are the novel part here. They spin up a bunch on their own of course, synthesizing tasks for the relevant domains and letting agents run wild on them in various sandboxes. Totally normal for pretraining data, which is more about quantity than quality. But they do also harvest naturally occurring trajectories from public sources like execution traces in code repos, as well as from their own past model development.

There’s a lot of cleanup and wrangling to get it all into the same format, which is important and underappreciated for quality but isn’t that interesting frankly. The do mention having over 10M trajectories to use in CPT, but otherwise no info about their data.

For SFT and RL, they do give numbers, which we’ll see on the next slide. But I did want to pull out a couple fun details first.

For SFT, they add in reasoning, to teach the world model to work through the logic of the given world. They also vary the system prompt to improve generalization.

For RL, they use rubrics, which we’ll discuss later when we talk about the benchmark they made for this paper, and they actually cited a Scale paper on rubrics best practices. They do have rule-based verifiers in some cases too, checking for objective correctness. They mix the rubric and rule scores at a 9:1 ratio, which is an approach I’ve never seen before and doesn’t get any further explanation sadly.

Also, during RL itself, they observe one fun bit of reward hacking that I have seen reported elsewhere. Basically, when you use an LLM judge, the model you’re training can trick the judge into awarding higher scores by peppering its responses with self-affirmations. One example they give is “operation completed successfully with all fields correctly populated” - the LLM judge is just too credulous and goes off that statement. Personally, I’ve seen Claude do a lot of this faulty self-affirmation, so I’m sure it’s not just a Qwen problem.

For the SFT and RL phases, here’s the amount of data for each of the seven domains.

I’ll note two things. One is the substantial difference in SFT numbers between MCP and SWE vs all the other domains. I initially thought maybe it was because they were already good at those domains, but Terminal has one of the highest counts, so that can’t be it. MCP and SWE are the only two domains where the action space is different for every task though, so perhaps it was just harder to generate data that cleared their automated quality checks. They do note later that MCP and SWE have far more turns per trajectory than the other domains, which again hints that they were harder to produce.

The other thing is the ratio of RL to SFT data. I expected these ratios to be roughly similar, and I guess they kinda are, but it’s still more varied than I thought it would be. The only pattern I see is lower ratios for the GUI domains, which again I can’t definitely explain. Could also be a quality issue. Keep in mind that the SFT data comes from a different model than the RL data, and they say later on that their model generally underperforms on GUI tasks because it’s relatively weak on vision, so my assumption is quality limited them.

Still, at least they disclosed the numbers. That’s become increasingly rare for the model builders to do.

Now in addition to training data, they also produce eval data - their new benchmark, AgentWorldBench. This is the ruler they will judge their models by, and we’ll see the results on the next slide.

It’s somewhat misleading to say they produced the data though, because as you can see below the pie chart, the prompts are all from existing benchmarks. And since this is a test of the world model’s ability to respond to agent actions, the examples also need trajectories showing an agent interacting with an environment. So concretely, this benchmark comprises prompts from other benchmarks, plus one complete and successful trajectory per prompt, containing agent turns from one of five possible models and environment turns from the benchmark’s environment.

The job of the world model, then, is to produce the correct environment response given the prompt plus the agent and environment turns so far. They pick several different spots on each trajectory to test the world model on, ensuring some minimum difficulty and some positional variety. So for example, on text domains they always pick the first and last turns at least.

They then rate on five dimensions, using an LLM judge to compare the LWM’s response against the ground truth response from the actual environment.

I have some quibbles about how they define and weight their criteria, but ultimately I think it’s reasonable enough. You just have to keep in mind that the scores all have error bars around them.

Here are the results on that benchmark.

Before we look at any particular result, let me contextualize the models on this list. Of course at the top we have the big three: Claude, GPT, and Gemini. It’s a little odd to list both Opus 4.8 and Opus 4.6, but otherwise these are fine. Nice that they included a smaller model in the form of Sonnet.

The next group down is the top of the open-weights pile, minus Qwen of course. These guys are roughly the same size, except for MiniMax-M2.7, which is significantly smaller.

After that we have Qwen, from what was the latest version at the time, namely Qwen3.6. We don’t know how big Plus and Max are, but my guess is Max is around the same size as the bigger Qwen3.5.

Finally, at the bottom we have the baseline and LWM versions of Qwen3.5 in two different sizes. These are mixture-of-experts models, so the first number is the total size and the second number is the count of active parameters.

Now looking at the results, I think you have to start by looking at the relative difference of the baseline vs the LWM version for both sizes, rather than the absolute performance of the LWM. If you compare the baselines to the other models above, you’ll see that the baselines are weirdly strong. Like for the smaller model, 3.5 is significantly better than 3.6, and is slightly better than MiniMax-M2.7, which is roughly ten times the size. And the bigger baseline model is apparently on par with Gemini, which is undoubtedly an order of magnitude bigger.

So yes, it’s nice they got their bigger LWM to beat all the closed models at simulating these domains, but some of that seems like variance and the right choice of baseline. And as I mentioned on the previous slide, reasonable people can disagree about the rubrics, so I think providing two decimals of precision is misleading and that any scores within a few points of each other are basically equivalent.

Finally, given the existing interest in world modeling and the high scores, it’s possible the closed models already get this type of training. Hard to know if we’re comparing apples to apples here.

Of course this benchmark is brand new for this paper, and also the training data domains matched the benchmark domains. What about some independent confirmation that Qwen-AgentWorld is a good all-around simulator?

One quick check you can run is to train for new benchmarks. So they take two generalist agent benchmarks, which are not directly in the target domains, and train the same model on them with two different simulators: Qwen3.6-Plus, and Qwen-AgentWorld.

As the table shows, Qwen3.6-Plus right off the shelf is a poor simulator of at least a generalist agent environment, with virtually no change on either benchmark. Qwen-AgentWorld, on the other hand, shows real improvement on both.

Thinking back to the AgentWorldBench scores, I do think it’s unfair to use Qwen3.6-Plus as a point of comparison here, since it scores worse than the stock Qwen3.5-397B. It’s unclear why they introduced another variable here by changing the baseline model.

Now this part still uses Qwen-AgentWorld as a simulator, but it focuses on how controllable the simulator is. Thinking back to our Waymo example from the background slides, one of the key advantages of simulation is you can produce conditions that are rare or not present in your corpus of natural training data. But just because a simulator can produce rare data doesn’t mean it’s easy to get the simulator to do it. So you need to test control empirically. Concretely, “control” means putting instructions in the system prompt about how to respond or what is in the hidden state, not just letting the simulator pick what seems most probable.

And when you do give those specific instructions, you get a more effective simulator as the table shows, at least some of the time anyway. It’s at least never worse.

Beyond just giving more specific directions, the authors design two sorts of control designed to improve robustness:

1. Environment Adaptation, where the simulator makes select changes in an otherwise standard environment. For example, one of the APIs in the system prompt might come back with a transient error instead of the expected response.

2. Fictional-World Construction, where a structurally realistic environment gets completely fictional facts. The example they give is a Search environment where a small colony exists on Mars, and there are news articles across the whole arc of colonization.

There’s some good detail about the fictional-world one in the paper if synthetic data is your jam.

Another view on the impact of controllable simulation is at the behavior level, like one level below the top-line results.

So on this web search benchmark for example, using real data sometimes teaches the model to be lazy, because in real search results you often don’t need to look at any of the resulting web pages - the search preview is enough. If that’s true too often, your model may learn to rely only on the search results, or maybe to click into just one link, before returning an answer.

However, if you control the simulation so that the known final answer is rarely in the search previews, then you can get the model to try harder and look at more pages. That’s what the graph on the right shows; for the same number of searches, the model learning from the controlled simulation hits more web pages because of the simulator, which is going to generalize better.

It’s instructive to see how the simulations got more realistic too. So for search, over the course of RL a couple telling details really improved.

For one, the url slugs got more realistic. After 100 RL steps, the ID for the Big Eyes movie page on IMDB was something pretty unlikely, this number composed of repeated and increasing digits. But after 200 RL steps, you get a more plausible ID.

Or take the list of simulated search hits. After 100 steps, you get a plausible but simplistic list, like a hit from NYT but with a generic title. After 200 steps, you get a title with some flair and also a top hit for Wikipedia, which in retrospect seems like an obvious omission from the earlier list but maybe wasn’t obvious at the time. And as the ground truth search results show, Wikipedia really is the top hit.

So the benchmark and the other results we just covered are for LWM as simulator. Now we can talk about LWM as agent, like Code World Model and DreamZero from the background slides.

They’re going to measure across several agentic benchmarks - some coding, a little searching, a couple generalist agent, and several variants of tool use. However, the training data is not agentic; it is single-turn, with no tool use or interaction at all!

So taking this regular old Qwen model, which has not received any world model training in pretraining or in supervised fine-tuning, and then doing LWM RL on it can improve agent performance across the board.

To me that really shows the power of instilling a mental model into a language model, and how walking through steps makes problems more clear. It also reminds me of rubber duck debugging, where just explaining your code out loud to nobody in particular or to an inanimate object clarifies your thinking on it.

Here’s a look at some more explicit world modeling. There’s a tricky task in this email tool called Mailman where the baseline model fails. It’s a bit jargon-heavy, but the baseline model does the equivalent of reading the docs and trying stuff, all very first-order thinking.

By contrast, after some RL, the LWM is able to think through the “world” or logic of Mailman, with second-order effects. It’s not totally explicit, like it’s not simulating exactly what Mailman would do, but it walks through a summary of it in natural language. That’s just what a human would do, except perhaps in the most challenging circumstances when the abstraction and imprecision of natural language are liabilities.

My Takeaways

• World modeling should be part of LLM training

  • Maybe it already is at the closed labs?

  • There is some theoretical evidence that sufficiently general agents must contain world models

• World modeling is still early

  • The paper cites a lot of other recent research, but there are no gold standards that I’ve seen yet

• Generator : reward model :: agent : world model

  • Training a world model could become the main way to improve the agent, at least for a time

  • Maybe there will be human data for world model training

  • Could you use a world model as a faster approximation if the real environment is slow/expensive/one-way?

• Agent + world model = RSI?

  • If the agent can improve the world model, and the world model can improve the agent (as we just saw), the cycle is complete

Paper · Weights

Background

I want to start with a high-level view of where AI is headed and what that looks like to the observer on the ground here, the folks living it like you and me.

All the curves so far say we’re on track for some super powerful AI. I don’t necessarily want to call it AGI, or ASI, or anything else like that, because frankly the labels are not helpful. There is no bright line for these events, and there isn’t even really an objective definition for such labels. People put a lot of stock in the Turing test, we blew past that pretty quickly in the modern AI era, and since then it’s been more instructive to follow specific and objective measurements, like the METR graph or the scores on Humanity’s Last Exam.

And if you watch the trend on those specific and objective measurements, you find that progress at the frontier has not stopped. But at the same time, people are always pointing out weird and dumb behaviors, things that models do that a human of that intelligence never would. Similarly, a lot of our specific and objective measurements cover only a partial set of characteristics we as humans would expect from a strong intelligence. So people call machine intelligence “jagged”, and part of that jaggedness is because researchers focus on improving the abilities they can best measure.

For the big labs, who want to build truly general intelligences, that jaggedness is something to fix. Anthropic wants Claude to be a good philosopher. OpenAI wants GPT to provide good relationship advice. xAI wants Grok to praise Elon more believably.

But there is another set of labs leaning into the jagged intelligence, spiking on code and reasoning and letting the more well-rounded skills languish. With a much narrower focus and different success criteria, there may indeed be room for players besides the big boys. Newer labs like Arcee and Poolside are positioning their models as agent-first, recognizing that enterprises don’t much need amazing writing or impressive creativity - just sharp analysis and reliable decision-making.

One of these is Zyphra, the makers of ZAYA1-8B.

Unlike many so-called “neolabs”, outfits like Thinking Machines or Safe Superintelligence that can trace their lineage back to the big labs, Zyphra is on its own evolutionary branch. Founded in the Bay Area in 2021, before ChatGPT and the generative AI explosion, they have been working on neural networks and architectural innovations for some time now. For example, they released a 7B parameter model called Zamba in 2024 that used Mamba rather than the traditional attention mechanism. They also released a model called ZUNA trained entirely on EEG data, for brain-computer interfaces. And just for kicks they released a music generation model called Zonos. So they’ve got range.

Staying on the divergent evolutionary branch idea, they also live in a different hardware ecosystem: they train and run on AMD, not NVIDIA. That’s why AMD led their unicorn round back in October 2025, and why their cloud offering stresses “Deep integration across the AMD stack”. They offer inference on Kimi, DeepSeek, and GLM, as well as their own model of course, and they offer training, which I assume is the same infrastructure they used to train their models. So they’re a full-stack AI company.

Flipping to the science side, we need to talk about a few bits of prior art that ZAYA1 makes use of.

By now we’re all familiar with reasoning mode or reasoning models. These are models that use a lot of tokens to “think” about a solution before giving a shorter, condensed final answer. The first of these was o1 from OpenAI, R1 from DeepSeek made the whole method public, and now pretty much every model comes with a thinking mode.

That ability to think more, to spend more compute and more tokens on a problem to get a better solution, is called “test-time compute scaling”, or TTC scaling for short. “Test-time” is just a fancy term for “at the time you’re using the model”, so basically the term is saying you can scale up the amount of tokens generated just like you can scale up the number of parameters or the amount of pretraining data.

The version I just described, where a model authors a long CoT before responding to the user, is serial TTC scaling - it’s about a single response growing longer.

You can also have parallel TTC scaling, where you have the model generate multiple responses to the same prompt, then use some technique to review or aggregate those responses. A simple example would be majority vote, where you get k responses and just pick one that gave the most common answer.

This is actually what people thought o1 was doing behind the scenes before R1 came out. If you were paying attention back then, you may remember talk of Q* and Project Strawberry. The rumor was that o1 basically searched a tree of possible responses and had a way to learn which next steps were most promising to further explore. You also may remember a paper we did involving Monte Carlo Tree Search (MCTS), that again would be more like parallel TTC scaling.

Anyway, the point is that you can go wider in addition to going longer.

Here’s an example one such technique, called Parallel Coordinated Reasoning (PaCoRe).

The authors start from one insight: the limit to serial TTC scaling is the model’s context window. If we really believe in scaling, we need to break past that limit.

So they do two things. First is, they have multiple reasoning trajectories in parallel. So that’s already doing some parallel TTC scaling.

But second, instead of just aggregating and picking from the reasoning trajectories, they compact them - basically use the model to boil them down to their essentials - and then feed them back into the model along with the original prompt. You can run that many times before all the compactions take up too much of the context window. And then at the end, you do whatever aggregation system, or you just have the model pick the one it thinks is best, etc. So you’re getting the wisdom of many millions of tokens, but squeezed into one context window.

We will see PaCoRe in the ZAYA1 paper, but as training data - the input is a problem and a set of reasoning trajectories, and the ideal output is a compaction.

ZAYA1 does something similar to PaCoRe, a technique they call “Markovian RSA”. It’s a combination of two papers. One is called “The Markovian Thinker”, and the other is called “Recursive Self-Aggregation”, or RSA.

This graphic is from The Markovian Thinker. If you’re not familiar, a Markov process is a process where all you need to know is the current state in order to predict the next state. For example, if you throw a ball in the air, you don’t need to know where it previously was in order to predict where it’s going - you just need its mass and its velocity and gravity. By contrast, text generation from an LLM is not a Markov process, since the next word depends on all the previous words that came before it.

If you’re again trying to scale test-time compute past the context window, that’s going to cause problems as earlier text leaves the context window. But if you can make text generation Markov, then you don’t have to worry about earlier text leaving the context window, because your prediction only needs the text from the step before.

As you can imagine, that’s probably not going to work at the single-token level, like it’s gonna be impossible to have one token have everything you need to predict the next token. But if you break up a CoT into bigger units, like steps or sections or lines of inquiry, then maybe you have a shot.

And in fact that’s what they did here. They have a prompt and the start of a response, composed of two steps. One prompt plus two steps is one chunk. Then they summarize the two steps, feed the prompt and the summary into the model, and generate another step. That’s chunk 2. They keep going with this over and over until in the final chunk, chunk L, the new step has the final response. As long as the summary is good enough, you have a Markov process for generating text.

The other half of ZAYA1’s “Markovian RSA” technique is the RSA, the recursive self-aggregation.

The basic idea is to generate a bunch of trajectories, sample a few of them, and synthesize something from that sample. If you do that over and over, eventually you should converge on the right solution.

The graphic here I found a bit confusing at first, so let me walk through it. First we have a model and a question. We generate a population of solutions, where each solution has a CoT and a final response. In this case there are eight solutions in the population.

And then eight different times, we sample from the population and synthesize a new solution from the sampled ones. So like in the first row of blue, we have sampled solutions two, six, one, and four. The synthesized solution from that combination is T_1. Since we’re doing that sample-and-synthesize process eight times, we again end up with eight solutions.

You can do that over and over again, as many times as you like. And then in the final step you aggregate them somehow, like with majority voting or by just having a model pick the best one.

You can see how this is complementary to the Markovian technique on the previous slide, which is just a way to shorten any one CoT. Put ‘em together, and boom - Markovian RSA.

The Paper

Let’s start with the architecture. Fundamentally it’s still a transformer, so we have our three transformer components:

1. Embedding, for turning words into vectors

2. Attention, for understanding all the input

3. Feed-forward, for thinking about that input

You may not recognize the attention and feed-forward bits given all that’s going on here and given some of the nomenclature, so I’m going to break out each one.

This part here is the attention. Actually it’s attention with an extra step right before it, something they call Compressed Convolutional Attention (CCA).

I don’t want to look much at the guts of the thing, but the upshot is you end up with compressed versions of the normal inputs to attention. Like normally in attention you split up the input into queries, keys, and values, q k and v. That’s still happening here, BUT you do all this stuff on the right to make q k and v a lot smaller. To even out some of the quality loss, you basically look at the previous token in addition to the current token when you do that compression. That’s the convolution part.

This part here is the feed-forward network. It’s a mixture of experts, but they’ve show the individual components broken out instead of just one block that says “MoE” on it.

So you can see there’s a router that picks the expert, and there are 16 MLPs, those are the individual experts, the little sub-networks. That’s actually quite low; even small MoEs these days will have 128 or 256 experts.

The trick is specifically in the router portion, which they’ve shown is actually four components that I again don’t want to get too far into. Basically they’ve made the router smarter, which lets it spread the load in training across all 16 experts more evenly, which improves performance.

It also allows them to pick just one expert, which again is out of step with the rest of the field; usually there are multiple experts active, and one of those is usually what’s called a “shared expert”, i.e. it is always active.

Here’s a summary of the whole thing. With 40 of those transformer layers we end up with about 8B parameters, less than 1B active, for a sparsity ratio of about 11. That’s pretty normal at this size, although bigger MoEs tend to be more sparse.

Note also the call-out in the last row of AMD hardware. You would never see that mentioned if they were training on NVIDIA hardware.

Overall, while they’re making a couple unusual choices, they’re not the stars of the show. The main thing is the size, as we’ll see when we get to the benchmarks and look at what models ZAYA1-8B is competitive with.

Speaking of size: it’s not covered in the paper, but they have a 74B version in preview now. The finished version should be out soon.

Now, on to the training.

For pretraining, they do about 13T tokens, which is actually not a ton. For comparison, Qwen3.5 and DeepSeek V4 did over 30T. Part of that is probably down to language support - I think ZAYA1 only supports English, although it’s not mentioned - but I can think of two other causes.

One is synthetic data. For reasoning and code in particular, synthetic data is quite common, but you need a big operation to create enough tokens at sufficient quality to help out.

Similar deal for agentic trajectory data, which is by definition synthetic. They mention a few times that they haven’t trained ZAYA1 for use in agents yet, and agent trajectories can be a significant part of the pretraining corpus.

The other of course is that they’re a smaller operation, and it takes time to accrue a big pretraining corpus, whether scraped or generated. So they could catch up on performance with more time there.

Now for post-training, they provide this handy graphic showing all their stages: one for SFT, four for reasoning RL, and one for behavioral RL.

For SFT, they have three goals: one, get it to chat; two, lengthen the context to 131k tokens; and three, show it how to aggregate old chains of thought to produce a new one. That third one is key for ZAYA1 and relates to the RSA technique we saw in the background slides.

For the RL stages, they of course want to improve reasoning, but they also want to improve the various TTC scaling techniques. Like in the two blue boxes, they call out TTC, RSA, and PaCoRe as techniques the model will use and be rewarded for.

Now let’s highlight the data. We don’t get number of examples for SFT, but based on the token count and the maximum response length at that stage, I would guess around 20k. Then for the green box, it’s not in the graphic for some reason, but in the paper they mention about 85k examples. Then for the rest of the boxes, you can see they mention having 400 environments in the purple box, over 30k RLVR examples between the two blue boxes, and over 80k RLVR examples in the orange box. Overall that’s roughly 20k SFT, 200k RLVR, and 400 environments.

Also on the topic of data, they mention in a couple places how they try to balance out difficulty as the model trains and improves. We don’t need to get into all the stuff they do with the reward function to capture that and how they dynamically filter the data, but it’s good for us to keep in mind how valuable a distribution of difficulties is.

Finally, after all that training, we are ready for some inference. And that’s where ZAYA1 shines.

This tangle of boxes and arrows is their Markovian RSA, combining the aggregation and synthesis of RSA with the context window workaround of Markovian Thinker.

We can see our model producing N different responses, which by default they set to 16. Then they cut off the last Tau tokens of each response and throw away the rest, which they can do because the model has been trained to think in these Markovian chunks. Usually they set Tau as a percent of the overall response length, like half or less.

Next, they randomly sample C of them, by default 4, and they put em in the aggregation prompt. Feed that prompt to your model and you get a new response. And remember, you’re gonna do that N times so you end up with N new responses.

In theory you can keep doing that forever, but by default they go through two rounds of aggregation. They hint that it’s because they only do RL for a single aggregation step, and they claim they will train on multiple rounds in future work.

Now let’s see where we net out. This is the hero graph, where ZAYA1-8B competes with all these much bigger models. Of course many of these are closed models, so we don’t know how big they are or their architecture, but they’re likely all MoEs in the 100B-1T range.

For this model in particular, I think the shaded part, with Markovian RSA activated, is the right comp. And it seems like it’s holding up against these previous-gen models. Actually Sonnet 4.5 isn’t even that old, we’re only on Sonnet 4.6 now. So, pretty impressive.

Another way to look at performance is against more recent models from the same size range. Qwen3.5 and Gemma 4 are both pretty recent, and both have about 4x more active parameters, being dense models instead of MoEs. It’s apples-to-oranges but they’re in the same ballpark on net.

As we might expect given the focus on reasoning in training, ZAYA1-8B is the best for math and code and competitive on knowledge and instruction-following. The worst performance is on Tau^2 Bench, a recent agentic benchmark, which again is no surprise given where the researchers chose to focus. I suspect they could improve greatly here.

So since this paper’s primary focus is TTC scaling, of course they’re going to sweep those parameters and show how far TTC scaling can take you.

Here’s one view of that, just looking at overall tokens per final response you get and how it correlates with pass rate. It shows a typical diminishing-returns pattern, but take caution, because that y axis is already pretty high when they start the curve. Usually the last few percent are the hardest to get. I would have liked to see similar graphs for benchmarks with more headroom.

Here’s a similar look into a different benchmark, but with the different inference parameters laid out. Again, T is the number of aggregation rounds, N is the number of responses generated, and C is the number of responses sampled for aggregation.

With no TTC scaling at all, The model hits 32%. Adding just two rounds with a pretty short maximum response length barely moves the needle, adding 1%. It’s only when you really crank the tokens that you see big gains - 14% from quadrupling the aggregations rounds and doubling the response length, and another ~20% from doubling the number of responses and doubling the response length again.

I’ve added the rough multipliers on maximum token usage to the table, but keep in mind you don’t always use the maximum per response. So if you want to be conservative, maybe cut the multipliers in half for what actually gets used on average. It’s still a lot, like millions per final response on average for the highest TTC setting. Again, that’s going to be tens of dollars per response at Sonnet 4.6 pricing. This type of thing is only feasible for smaller models.

I also added ROI, basically the gain on the benchmark relative to the token multiplier. Weird pattern, I suspect somewhat specific to the benchmark, so I wish they did this for more benchmarks and maybe a finer-grained sweep.

So if Markovian RSA is such a great form of TTC scaling, can we just apply it to any old model?

Well they ran that experiment here, and the answer seems to be no, not really. Remember, ZAYA1 is trained to aggregate and synthesize solutions. It’s not a super common workflow, so at least at this size, models aren’t very good at it.

I do wonder if bigger models would work though, since they are generally more flexible. Again, an experiment I wish they did.

My Takeaways

• TTC scaling may substitute for parameter scaling

  • If you don’t have the space, you can instead pay in time

• Goes hand-in-hand with faster, cheaper tokens

  • Groq, Cerebras, Taalas

  • “Tokens too cheap to meter”

• Could be promising for consumer hardware

  • Are small reasoning models “disruptive technology”?

• Amenable to context engineering and orchestration

  • Ex. have a bigger model pick the final response

  • Ex. use deterministic checks to stop and regenerate bad solutions

  • Decoupling improvement from ML knowledge/tools/control of weights means more accessible innovation and customization

• I’m not sure if the architecture mattered

Paper · Website · Repo

Background

Let’s start today’s discussion with a bit of economic background, since one of ALE’s goals is to measure progress on “economically valuable” tasks.

The US Bureau of Labor Statistics maintains something called Standard Occupational Classification, or “SOC”. SOC is a taxonomy of jobs, with 23 major groups at the top and over 800 detailed occupations at the bottom. Each job gets an ID, so like 15-1252 means “Software Developers” and 27-2023 means “Umpires, Referees, and Other Sports Officials”.

Now a level more detailed than that is the Occupational Information Network, or “O*NET”. O*NET extends each code with specific job titles, and it describes in detail what the job actually does and what it requires.

So continuing with my example of “Umpires, Referees, and Other Sports Officials,” the key duties include “officiating at sporting events” and “inspecting game sites for compliance with regulations or safety requirements”. Key abilities include communication and deductive reasoning. There’s a ton more detail on the job - knowledge required, areas of interest that align with the job, appropriate styles of work - but for our purposes the only other relevant bit is the software skills required. For umps, it’s not much more than email and browser, but for other jobs it’s going to be quite relevant for our agents-replacing-humans benchmark.

Speaking of, here’s one of the leading players in the space: GDPval, from OpenAI.

GDPval, which is a fantastic name by the way, draws directly from O*NET to design its distribution. Specifically, the authors picked the top 9 sectors by contribution to US GDP, then picked 44 occupations within them, then designed tasks to cover the majority of O*NET work activities for each of those 44 occupations. So using the authority of O*NET, the authors could reasonably claim to measure potential to impact GDP.

Of course, the major filter on their selection of sectors and occupations was how much work happens on computers, in a way that doesn’t require other people. So for example, not only are physical jobs like Firefighter or Chef out, but so are mostly interactive or management jobs like Sales Representative or CEO. Still, they end up pretty representative of predominantly digital work, across sectors like real estate, government, manufacturing, health care and more. You can see in the example task in the top-right that the job is Manufacturing Engineer, and the task is to design a cable spooling truck for underground mining operations.

Now for us at Scale in particular, where the tasks came from is of particular interest. OpenAI sourced contributors with an average of 14 years of experience from a variety of firms across each sector and conducted a video interview, ran a background check, and gave them training + a quiz. The accepted contributors then produced tasks using real work materials, with a prompt and the requisite artifacts as inputs and whatever deliverable as the output - stuff like a legal brief or a care plan or a financial model. Of course the tasks got plenty of quality control, both automated and manual.

The benchmark comprises 1,320 tasks across its 44 occupations, but only 220 are public; the rest are private, with updates on GPT and selected competitors whenever OpenAI releases new models. For example, in the GPT-5.5 release, they reported on GPT-5.5, GPT-5.4, Opus 4.7, and Gemini 3.1 Pro.

As for what they’re reporting, the main thing they care about is whether AI can replace humans in the job, so what they actually measure is expert preference on the AI output vs the original, human output. If the rater has no preference or thinks the AI version is better in a blind comparison, they count it. Most recently, GPT-5.5 scored 84.9%, although we don’t know the split between wins and ties.

Of course we’d like to know how lots of models score on GDPval, not just the ones OpenAI deems worthy of their time, so folks have adapted the 220 public tasks into different versions of the benchmark. The one I see most often is called GDPval-AA, from Artificial Analysis, hence the “AA” in the name. You’ll see on the chart that they have a much wider array of models, including some from after GPT-5.5 came out. The most noteworthy one on there is GLM-5.2, an open-weights model that apparently matches GPT-5.5.

You’ll also notice they do not report a win + tie rate! Instead, they report Elo, basically a power ranking across all the measured models. That’s because for GDPval-AA, they do head-to-head between models, not model vs human reference solution, so there is no common standard to compare against. Relatedly, their pipeline is automatic, using an LLM judge (with access to the human reference solution) to pick the winner of each matchup.

One virtue of their scoring system is it can’t really saturate. Like if GPT-5.5 is already equal to or better than humans on 85% of tasks, there’s not much more GPT-5.6 can do on that metric. But as we see on the chart, Fable 5 pretty well crushes GPT-5.5, in a way that may not be obvious when GPT-5.6 comes out and we see the updated scores for GDPval. Relatedly, GDPval-AA is what Anthropic reported on the Fable 5 release notes.

Finally, one advantage of GDPval-AA in my opinion is they have their own harness, and they use it across all the models. By contrast, OpenAI uses each model within its own harness, and the harness for GPT in particular gets some benchmark-specific tools. On the one hand, it does make sense to measure “this is the best performance we could get from each model”, but I think for a leaderboard it’s more intuitive to have as neutral of a harness as possible. Keep that in mind for when we get to the results of this week’s paper.

The other leading player in the agents-replacing-humans space is Remote Labor Index, which is a collaboration between Scale and a research organization called Center for AI Safety.

Like GDPval, RLI sources a bunch of economically representative work and has agents attempt to recreate it. However, many details vary.

One is the overall approach or framing. Specifically, while GDPval uses top-down design based on a government taxonomy, RLI takes a bottoms-up approach: look at platforms like Upwork as a reflection of what computer work actually gets outsourced today, and sample from it. They do compare to GDPval to ensure the distribution of tasks isn’t too divergent though, and they somewhat oversampled from the more obscure task types to ensure broad coverage.

Anyway, the authors sourced real projects from over 300 freelancers, representing over $140k and over 6k hours of real work. After quality filtering, they end up with 240 projects across 23 different Upwork categories, like architecture, marketing, and game development. You can see examples of projects in the top-right.

As for grading, they have two modes: model vs human reference project, and model vs model - the same ones that GDPval and GDPval-AA cover, respectively. However, it’s human graders in both cases, so it’s expensive to update the leaderboard, which is why it’s a bit stale as you’ll see in the bottom-right.

So for model vs human, they ask the grader one question: “Does the alternative satisfy the brief at least as well as the reference, such that it would be accepted by a reasonable client?” The grader gives a 1 for no, 2 for yes, and 3 for strong yes, like if the model clearly surpassed the brief. The figure of merit there is what they call the “Automation Rate”, which is the count of 2s and 3s divided by the total count. Automation Rate is what you see on the leaderboard, and it’s quite low - just over 4% for Opus 4.6 in the Cowork harness.

Speaking of harnesses, note the number three player on the leaderboard: Manus, the company Meta acquired but then recently had to un-acquire on orders from the Chinese government. Manus doesn’t even make any models - it only makes a harness. Again, we’ll see the importance of the harness as we dive into ALE.

The Paper

So just like GDPval and RLI, ALE is supposed to measure AI’s ability to replace humans on long-horizon, economically valuable work. ALE has a couple differentiators, which we’ll come back to, but broadly it is the same as the other two.

Since we know the premise already, let’s get into the details.

First, to actually build a benchmark, you need to know your target distributions. In the case of ALE, the most important distribution is domain coverage, like which jobs and duties to cover. They’re going to take the GDPval approach and start with O*NET, but instead of just looking at the top occupations by GDP contribution, they’re going to shoot for maximum coverage. Specifically, they filtered all of O*NET down to jobs that are primarily digital, then clustered them into 13 domains with 55 subdomains.

Once you have your distribution of domains, now you can go find the experts in the domains. Here they relied on their personal network and some referrals, which is reasonable but quite informal from the Scale perspective. Of course, this is an academic publication, and academic budgets can’t generally support the kind of large-scale, dedicated sourcing operation a well-funded company can run.

Now pulling from the Humanity’s Last Exam playbook, which Scale co-authored of course, they set up a portal for submissions to then vet. They do also explicitly commission some tasks, but mostly they are taking external “attempts” and reviewing them.

As for the reviewers, the first layer is an agent. Then a human engineer tries to convert the task into a standard format within an environment, which can expose quality issues but isn’t a quality step per se. Finally, there is a QC committee, which we don’t get much information about.

Again, from a Scale perspective the pipeline is rudimentary, but it’s understandable given the constraints. It’s hard and expensive to produce a good benchmark!

Here’s a look at their pipeline numbers at their time of publishing, which hopefully have improved, because they’re not even done with all their tasks! Out of a total of 1,490 submitted and commissioned, 323 are still not QC’ed. From a Scale perspective I think that’s shocking, but even from an outside perspective I think it’s a bit dishonest to include unverified tasks in your benchmark.

I also found their review process a bit odd. It’s inspired by the conference paper review process - that’s where the verdict names come from - but peer review doesn’t exactly have the best reputation for producing quality. So I’m a little skeptical here.

Breaking the aggregate numbers into domains and subdomains, we can see good absolute coverage but some lopsided distributions within covered domains. I think because they don’t have anything external to moor to, like economic rankings as in GDPval or the empirical distribution on Upwork as in RLI, there’s no strong opinion about exactly how the distribution should look. I think practically speaking it’s mostly a result of who they could find in their expert sourcing efforts, which would be a biased sample, but that’s my speculation - they don’t explain.

Here is where the tasks net out. They include some knowledge, coding, and computer use benchmarks for completeness I suppose, but the real competition is just GDPval and RLI.

The first figure they call out is size, which seems to be clearly in their favor. However, as we saw before, only 150 are public, and over 300 aren’t even done yet! So I think it’s frankly inappropriate for them to put 1.5k here, especially because they only put the public slice in for GDPval; the full GDPval has over 1.3k tasks.

The second figure, breadth, is far more honest. They do genuinely cover a wider variety of domains, and it highlights the different purpose ALE serves: a test of capabilities, not a share of most economically valuable tasks. It’s right in the name - it’s the last exam.

Looking ahead, they claim a longer horizon on the top end. That is possible, but they don’t back it up with specific numbers anywhere - they’re just passing along the self-reporting from their contributors. More importantly though, the agents taking this last exam only get five hours per task. Of course agents can work faster than humans, especially if calling sub-agents to work in parallel, but it makes me think either that “weeks” is an exaggeration or that five hours is artificially low. Later on they do note that only 4.3% of tasks hit the 5 hour cap, but that could line up with the tail end of the horizon distribution. So I’m skeptical on the high end of their time horizons.

The last difference, verification, deserves its own slide.

Concretely, here is what goes into a task and how it gets evaluated.

First, you’ve got all the stuff for your task: the description of the work to be done, the required inputs, the software tools available, how to evaluate the outputs, and reference assets - the golden output from the human expert.

That all ends up in an environment where the agent is gonna run. Once it’s there, you kick off the run, and the agent gets five hours to complete the work. There’s no further direction from a human, but the agent is getting plenty of feedback from the environment, like seeing the outputs of the software tools it’s using.

Once the agent is done, either by its own decision or by hitting the time limit, the outputs go to the automatic evaluator, which returns a score between zero and one.

Now as for the automatic evaluator, the exact method depends on the output type. For handy reference, they’ve given every method in this table.

For 93% of tasks, the automatic method is a pretty natural fit, like running code against a test set or checking a 3D object’s dimensions. However, for 7% of tasks, they have to use an LLM judge, mostly for visual comparisons but occasionally for free text. The judges do get rubrics, but already there’s a bit of judgment creeping into their automated methods.

More broadly though, there are two issues in my view. One is that some aspects of quality are not easy to automatically evaluate, even if other aspects for the same task are. So for example, while it’s easy to check code for correctness with tests, it’s hard to check maintainability.

The second, which is broader, is that many tasks are practically impossible to automatically evaluate, so restricting your benchmark to tasks with automatically evaluable quality gives you a biased view. I don’t think that’s a problem per se, but I do think automatic verifiability is at odds with their other two goals of realism and breadth.

To make this all a bit more concrete, here are two examples of completed tasks.

The first one uses an injection molding simulation tool to produce predicted metrics like pressure and weight. There’s a particular software for it, Moldex3D, and spec for the part that is basically the inputs for the program. The agent has to fill out a provided results template. In this case, the agent did run the program and submit the template with values filled in, but some of them were the agent’s own estimates rather than outputs of Moldex3D.

The second one asks the agent to convert an audio recording into sheet music and a MIDI. Again, there’s some niche software, and some structured inputs with relevant information. In this case, the agent failed because it didn’t submit the sheet music or a requested image file showing the program in use, just the MIDI. In general the scoring systems follow this hard vs soft requirement pattern, where missing a hard requirement is an automatic zero even if everything else is perfect, whereas the soft requirements can receive a grade anywhere between zero and one. In my view that’s a bit misleading, or at least deserves a second form of reporting.

Note in both cases that the red box, Observed Outcome, has details about the failure. They have Codex analyze every agent trajectory for more granular information like that.

Now let’s look at the full set of results, because there’s a lot to take in.

Before we look at any particular results, I want to call out a few divisions on the table. At the top, they’ve divided their benchmark into three tranches. “Near-Term” is the easiest, “Last-Exam” is the hardest, but “Full-Spectrum” is actually not just what’s left over - it’s one task per domain. They don’t explain how these two different sorting mechanisms happen to work out. I also think it’s annoying that the benchmark is called “Agents’ Last Exam” yet less than a third of the tasks are actually “Last-Exam” tier.

Another division to note is by color: green at the top for model + harness variable, yellow for model variable with harness fixed, blue for model fixed with harness variable. Given the increasing importance of the harness, because of the increasing intelligence and ability of the model to make full use of the harness but also because models train in their harnesses now, I’m glad they tried to tease out the differences.

Now, on to the results themselves.

High-level, it’s going to be OpenAI victory across the board. GPT is the best model across every tranche and across all harnesses, and Codex is generally the best harness, although sadly they didn’t test Opus inside Codex. The only asterisk I’d put there is that ALE-Claw, the custom harness for this benchmark that is basically a stripped-down version of OpenClaw, does better than Codex in some cases. My hunch there is that it has a significantly smaller system prompt and thus is less distracting than Codex, whereas other fully fledged harnesses like Cursor have system prompts of a similar size to Codex’s.

One thing I found odd in the list of results is that Fable only appears once, in the third row, whereas Opus 4.7 appears in several places. It’s not going to save Anthropic here, since even Fable + Claude Code loses to GPT-5.5 + Codex, but it feels incomplete.

Finally, just look at how badly Grok is doing. It’s far below many open-weights models, even ones that are likely a tenth its size. Given the impressive performance of Cursor’s Composer 2.5 and the respectable performance of the Cursor harness, it’s no wonder SpaceX paid $60B for them.

So quality is of course the top-line number, but cost and speed are important too.

Now normally on these graphs, you expect to see a general correlation between cost and quality, and between time and quality. Dots in the top-left corner of the graph are in the “magic quadrant” where you can improve quality without paying in money or time.

On these graphs, our magic quadrants are very busy. In fact, there is basically a straight line up on both, where you can get better quality for basically the same cost and time just by using the right model and harness.

It’s a tough couple graphs for any Anthropic users in particular, because apparently you can drop like 10x the money on Claude Code for no improvement at all in results compared to just using GPT-5.5 in pretty much any harness.

Lastly, I want to look at what is going wrong in the unsuccessful cases.

As the chart shows, the two biggest issues are missing domain knowledge and poor strategy. If you can knock those out, you’re halfway to a perfect score.

They are both “unknown unknowns” as Donald Rumsfeld once said; if the model knew more about the domain and perhaps about the relevant tools, it would have succeeded, but it didn’t even know it was missing the knowledge. Fortunately, we know the fix for poor knowledge: more training data! So I think this is a bullish finding for Scale.

The other two errors in the same families, incomplete work and hallucinations, are much harder to fix. Hallucinations in particular are probably impossible to truly fix in the current AI paradigm.

As for the execution errors, I suspect better harness tooling would help there, since the harness and the environment are how the model gets feedback about its work.

By the way, RLI also has error buckets, although they are of course different. Specifically, they found that almost half of all failing outputs were poor quality, which I think actually matches up pretty well with Wrong Strategy + Domain Knowledge + Hallucination. They also have an Incomplete bucket, which was ~35% for them. I think that makes sense as well given the higher difficulty of RLI, like either running out of budget quickly or not even realizing certain aspects were mandatory. Finally, they have a Technical Issues bucket at ~18%, which again has a pretty decent counterpart here in the Execution family of errors.

So What?

To close out, I want to zoom out and see where all this is going. Then we can talk about the particulars of this paper and this benchmark.

We know from our old friend the METR chart that models are getting exponentially more autonomous. If you’re new around here and thus have never seen this chart, what it’s showing is how long in human time a task can be yet still have a model solve it. It’s a straight line on this chart because the y axis is on a log scale, which means on a linear axis the curve is exponential. So it literally is exponential progress.

METR is humble enough to admit they can’t reliably measure progress anymore, but we can extrapolate. In fact, we’re going to extrapolate from this newer, faster curve I’ve drawn in orange, which starts at the dawn of reasoning models: o1-preview.

That curve 10x’s every year. So if Opus 4.6 could do 10 human-equivalent hours of work autonomously at 50% success rate in February 2026, then by February 2027 the leading model will be able to do 100 human-equivalent hours of work autonomously at 50% success rate.

So I expect rapid progress on this benchmark, and for it to saturate some time next year.

My Takeaways

• I’m not ready to trust this benchmark yet

  • Quality is a hard problem and it’s not clear to me they solved it

  • A lot of good fundamentals though. I would like to trust it

• Complementary to existing benchmarks

  • Coverage of the tail that GDPval and RLI don’t cover, but also biased towards the tasks they could get and the tasks that are verifiable

• I expect increased demand for all work artifacts

  • The market for codebases is already significant

  • “Task” is almost too small a unit of measurement now - “project” is the right size

• This is all for IC work

  • Some harnesses may allow subagents, but it’s not a focus of the benchmark

  • Eager to see the first “middle manager” benchmark as we shift into multi-agent by default

Paper

Background

The framing I want to use this week comes from Isaiah Berlin, a 20th century philosopher and historian of ideas. He wrote many works and essays, but perhaps his most popular and enduring is this one, The Hedgehog and the Fox.

The title is a reference to a quote from an ancient Greek poet: “a fox knows many things, but a hedgehog knows one big thing.”

Berlin spends the essay expanding on that idea, classifying thinkers into one camp or the other. So for example he puts Plato, Pascal, and Proust in the hedgehog camp, while Aristotle and Shakespeare go in the fox camp. Heavy hitters on both sides, to be sure.

In more modern times, a lot of forecasters have taken the fox’s side, asserting that fox-like thinking produces better predictions than hedgehog-like thinking. So for example Philip Tetlock - famous for his work on superforecasters, people who are consistently good at predicting global and political events - has said foxes outdo hedgehogs here. Similarly, Nate Silver’s old forecasting team, FiveThirtyEight, adopted the fox as their logo in allusion to this distinction.

Two things make this framing relevant to LLMs. The first is that they are very widely read. In fact, they have read basically the whole internet, minus the absolute cruft of course. So they contain many and diverse ideas by nature.

The second is that an LLM’s job literally is prediction. We call the act of using an LLM “inference”, but really the job of the model is to predict with various likelihoods what the right next token is. So we seem to be in very foxy territory here.

Switching to more technical matters, we need to discuss entropy.

The term originally comes from physics, specifically thermodynamics, where it measures disorder or randomness. From that framing it seems bad, and if you already have some baggage about the term like I once did, you’ll have to put it down for the LLM context.

The term gains a positive valence in information theory, where Claude Shannon adapted it. Here, entropy correlates with the amount of information a message carries, relative to a certain context. So for example if you have a weighted coin that always lands heads, then you’re never going to be surprised when I tell you the coin came up heads. But if I have a fair coin, then you’ll always be relatively surprised, since a priori you have no reason to believe heads vs tails. So in information theory, higher entropy is better, since it means the information is more valuable.

Let’s take that information theory understanding and apply it to LLMs. If my text so far is “The United States of”, you’re basically guaranteed to see “America” next. So our LLM is going to predict the token “America” with almost 100% probability, with basically every other token at 0%. If you use the formula here, then H for this position, this next token, is going to be very close to zero; p(x) is basically 1, log(p(x)) is basically 0, and there’s only one term in the sum that matters.

We can go back to my fair coin flip example on the other end. If my text so far is “The coin flip landed”, most of the probability is going to be split evenly between “heads” and “tails”, with a few other small terms like “on” or “near”. If we ignore those and just give 50% odds to “heads” and “tails”, then we get .5 for p(x). If you use base 2 for the log, which is customary, you get H = 1. The unit when using base 2 is bits, so that’s 1 bit of entropy.

H can go much higher than that of course. Like for a fair six-sided die, H is about 2.6 bits. The more events you plausibly could have, the higher H is going to be.

Here’s a related paper from last year that I really enjoyed. It’s from the Qwen team, and it looks at the relationship between RLVR and entropy.

If we think back to the elicitation hypothesis from last week, which says that RLVR just optimizes and brings forth good solutions rather than teaching new solutions, that should have implications for entropy. Like if RLVR is improving the odds of getting a good solution out of your model, it must be changing the distribution of token likelihoods at each step, probably making some of them much more likely at the expense of all the others. That should reduce entropy.

But as we established earlier, entropy is actually a good thing, something we want to preserve. The framing you sometimes hear is exploration vs exploitation - you need to be able to explore different areas of solution space as well as exploit the most promising areas. Higher entropy means more exploration. There’s such a thing as too high of course, but the concern with RLVR is getting entropy too low, basically cutting off all the exploring.

You can actually see this difference in the word clouds they assembled on the right. The high-entropy tokens are at the start of ideas, often new ones or representing changes in direction. The low-entropy ones, by contrast, are often part of calculation or totally deterministic, like a closing parenthesis that you know has to be there because you had an open parenthesis earlier.

The finding in this paper is that if you focus on the small share of high-entropy tokens for training and updates, you end up better off than if you made updates based on all the tokens. That’s not as relevant for us, but it does show there is value in preserving a certain level of entropy.

The Paper

So now that we’re up to speed on the value of diversity and entropy, we can dive in.

The basic idea of this paper is simple: your model stays more fox-like if you collect and emphasize less common solutions during training.

They give an example here and diagram it out. Let’s say you have to compute this series, summing the cubes of the integers from 1 to 100. This is RLVR, so we have a golden final answer, in this case it’s 25,502,500. What we want to train on, to reward, is the model getting that right final answer.

So we take our model, we have it generate a bunch of solutions, then we group those solutions together, using a more powerful LLM. That lets you see past mere changes in wording or variable names etc and just focus on the different strategies represented.

Then you assign advantages to each group. Here an advantage is a reward times a weight. The reward is going to be either 1 or 0 depending on correctness, classic RLVR. The weight, however, will depend inversely on the number of solutions in the group.

So as the diagram shows here, any group of wrong solutions is going to get zero advantage. A group of correct and common solutions is going to get a positive advantage, but on the low side. A group of correct and rare solutions will receive a high advantage, with the highest of all going to any “group” that only has one example.

That’s pretty much it! All you need for GRPO, for the algorithm that adjusts the model’s weights, is these advantages.

Here are a few examples of solution diversity. The have three problems, each with a varying number of solutions, plotted as shapes.

Now to measure coverage, they happened to have a substantial number of human examples to cluster, from web and textbook sources. They imply that having many human solutions means they likely covered all the possible strategies, but they weren’t rigorous about it imo.

Anyway, the result they’re showing here is how the initial model expands its coverage. So like in the middle example, there are apparently four different strategies, only one of which they saw in the initial model. But after some training, they saw two additional strategies, although they still didn’t observe one that humans got.

So does it work?

To answer that, they train three models with their technique: Qwen2.5, Qwen3, and Olmo-3, all in the same size range.

What they’re measuring here is AUC, short for “area under curve”. It’s a way to capture performance of pass@k for a series of k values as a single number, instead of having to report pass@k for each value of k. The higher the AUC, the more question you got right at lower values of k. And remember, the pass@k curve can never go down, because increasing k just means giving the model more chances to get something right. One consequence is that lower AUC numbers actually show differences better, since if you’re already doing well on pass@1, there’s not much room for improvement. So here for instance they drop AIME and Medicine after Qwen2.5 and focus on the more discriminating benchmarks, HLE and Physics.

Anyway, as you’d expect from a published paper, their method wins out. The baseline model, which they label Instruct, does the worst most of the time, although not always.

The other comparison they do is against what they call “SimpleRL”, which is your standard GRPO + RLVR, the foundation this paper builds on.

For Qwen3 you’ll notice two other methods, DAPO and Forking Token. DAPO is like a tweaked version of the GRPO algorithm that we don’t need to worry about. Forking Token is actually from the paper in the background slides, the ones about high-entropy tokens. I was pleased to see they liked that one as much as I did. Still, the simpler method of rewarding diverse strategies does best.

Of course we have to cover entropy too, since we know how valuable it is. Here they show the classic trend for standard GRPO + RLVR, where the entropy trends down as the accuracy trends up, regardless of the model.

For their method, by contrast, you preserve or maybe even increase entropy. So then you’re doing better exploitation while maintaining good exploration, exactly as desired.

Oh, and if you noticed the relative smoothness of the latter two graphs, I think that’s just from a change in plotting rather than a genuine difference between Qwen2.5 and the other two models.

My Takeaways

• Entropy is a first-class metric

  • Any training method that only reports accuracy (or pass@k for low k) is cheating

• Maybe we can use this for reasoning data

  • Their coverage evaluation used human-generated strategies as the ground truth

  • We could verify all the clustered strategies AND try to come up with alternate strategies not represented in the clusters

• Could you be more purposeful in exploration?

  • Example: ask SOTA for diverse strategies and then only grant max advantage if you display all of them

Paper · Blog post · Model

Background

I want to start our review today with some history. Specifically, the history of NVIDIA in the language modeling space.

Now NVIDIA chips, and libraries for using those chips, have been part of machine learning for a long time. Most of the breakthrough moments in what people used to call “deep learning”, like AlexNet or GPT-1, have used the NVIDIA stack.

But it wasn’t until September 2019 - over a year after the release of GPT-1 - that NVIDIA started putting in some language modeling efforts of their own. Specifically, I’m referring to this paper on the left: Megatron-LM, which is all about training large language models across many GPUs.

That sounds pedestrian today, with data centers containing thousands of GPUs all running in parallel to train the next SOTA model, but in 2019 it was a real pain point. For context, GPT-1 was 117M parameters and trained across just 8 GPUs. GPT-2, which was training when this paper came out, went up to 1.5B parameters and trained across 32 TPUs. Those are Google’s special ML chips, so not actually GPUs.

By contrast, the Megatron-LM paper trained a model with 8.3B parameters, which is a model size many researchers still use today, across 512 GPUs. The model itself didn’t make much of a splash, but the engineering did; the ideas and tools from Megatron-LM live on today - including in the original GitHub repo, shown on the right, which is still in active development.

Things scale up rapidly from there. GPT-3 comes out in 2020, rocking 175B parameters, having trained on 10k NVIDIA V100 GPUs. Then ChatGPT comes out in November 2022 and kicks off the generative AI explosion.

While OpenAI by this point has become closed, enough knowledge has gotten out that other folks are able to train large language models. The big name in the open-weights space, LLaMA, comes out in February 2023, with sizes from 7B to 65B parameters. Llama 2 quickly follows in July 2023.

Among the many other companies releasing open-weights models is NVIDIA, with Nemotron 3 8B in November 2023.

Now I know what you’re thinking: is this Nemotron 3 connected to Nemotron 3 Ultra, the model we’re discussing today?

Strangely, the answer is no. Other than both being language models from NVIDIA, the Nemotron of 2023 is not related to the Nemotron of today. The original line extended to Nemotron 4 in 2024, but then they rebooted the franchise so to speak, and released a new Nemotron 2 in August 2025, which we covered in a previous Friendly Paper Review.

I include all this confusing nomenclature because I find it emblematic of NVIDIA’s model efforts on the whole, and of my experience reading the Nemotron 3 Ultra paper in particular. Quite the opposite of Microsoft AI’s approach, which we covered last week and which I found refreshingly clear and principled.

So we know NVIDIA has been working on models for some time now. The next question is, why?

The answer boils down to one business principle: commoditize your complement.

Let me explain with reference to Windows and the PC market, which is the classic tale of commoditizing your complement. Once upon a time, PCs were differentiated - different hardware, different drivers, different standards. That differentiation is how PC makers stood out with their customers, but it was hell for operating system makers like Microsoft. If you’re Microsoft, of course you want all PCs to be interchangeable so that you can easily run on all of them. But more importantly, you want all PCs to be interchangeable so that manufacturers have to compete primarily on price rather than on features - that’s the hallmark of commodities markets. That price competition on hardware side then drives down the price of PCs overall but leaves the price on the software side untouched. The lower the cost for a PC, the more people can buy them, and the bigger the market for Windows is, at the same price.

Now let’s apply that wisdom today, but with the roles reversed: NVIDIA is the hardware maker, and the big model makers like OpenAI and Anthropic are the software folks. They are battling over who will extract the value in their chain.

Obviously they all want the demand for inference to go up, and that tide is really lifting all boats right now. But no budget is unlimited, and for a fixed amount of money there has to be a split between hardware and software. Right now OpenAI and Anthropic in particular collect fat margins on their tokens, competing on quality or perhaps some differentiated product experiences, which means fewer tokens for a given budget. If NVIDIA can somehow commoditize the token, making providers compete on price, then demand for cheap tokens is gonna go up - there’s gonna be a lot more inference. And that new inference is mostly gonna run on new NVIDIA chips.

Microsoft did it by playing PC makers off each other. NVIDIA is doing it by introducing its own competitor in Nemotron - in addition to funding lots of other model makers.

Now transitioning from business to technology, one thing to point out that blends both is how AI workloads are evolving.

As this fun graphic from the recent Anthropic report on recursive self-improvement shows, the paradigm has shifted from chatbots to agents to swarms of agents. And because of how agents work - looping through thoughts, actions, and observations indefinitely until the task is complete - they are token-intensive. Back in the days of the original ChatGPT, inference was too expensive to do this kind of thing, not to mention the severe constraints on context window size and the generally insufficient intelligence.

So the dropping cost of tokens encourages more token use overall - a more general economic phenomenon known as the Jevons paradox - and unblocks the switch from chatbots to agents. It’s also very Bitter Lesson, throwing scaled-up resources at a problem instead of eking out wins with human effort; a chatbot assisting a human in a task is more efficient in tokens but ultimately unscalable compared to a swarm of agents working autonomously in parallel to complete that same task.

All that is to say: NVIDIA is naturally going to focus on agents, especially the always-on ones like OpenClaw, which are going to consume even more tokens because of all the work they do in the background - you don’t even need a human there to push up token demand!

Also, NVIDIA is going to satisfice on quality while optimizing for throughput and cost. They are not racing to achieve ASI; they want their good-enough models deployed as fast, cheap agents at every enterprise in the world.

Completing our transition into technical topics, I want to discuss a major shift in language model training to parallelize yet another aspect of the process.

The technique in question is “Multi-teacher On-Policy Distillation”, or MOPD. Like any technology, it has a long lineage, but the citations for it typically go to the tech report for MiMo-V2-Flash, an open-weights model by Xiaomi.

This graphic from that paper breaks it down: after establishing a shared starting point, you train many specialist models, then use them as teachers to distill into one shared student - typically the same model you used as a starting point. So instead of taking your one model and training it on search, and then code, and then math etc etc, you do all that training in parallel.

Training in parallel gets you two benefits. One, just in general, parallelizing work makes it take less wall clock time. Like it takes the same amount of computer cycles, but for you as the researcher it only takes as long as the longest individual teacher. So that could be significant.

And two, you decouple the training of each teacher, so you can train in the way that is best for each domain. So if one is very SFT-heavy but another uses mostly RL, or even just different hyperparameters - basically the settings during training - you have that freedom. There’s no compromise, no sequential dependence, all starting from the same clean slate in the form of the SFT model.

Now I don’t want to worry about the exact details of the method here, because the Nemotron folks made some adjustments of their own, but the broad idea and the specific MOPD term comes from here.

And in a final instance of parallelism, the MAI-Thinking-1 paper from last week also used multiple teachers, although it’s not on-policy distillation; they choose to make SFT data from the teachers and then train the student on that, rather than directly comparing next token predictions given the same inputs, which is how on-policy distillation works.

The Paper

A quick note: given the volume of content and the interests of my audience, I glossed over most of the infra and training details. There is a lot of good detail in the paper, I recommend reviewing it if those are your interests.

As usual for new model releases, we’re going to start off with architecture.

Nemotron 3 Ultra is a big model. It’s not as big as the biggest open-weights options, like DeepSeek V4 Pro at 1.6T parameters or Kimi K2.6 at 1T parameters, but it’s up there.

Notably, it has more active parameters than either of those models: 55B here, compared to 49B on DeepSeek and 32B on Kimi. That’s due primarily to the number of activated experts in the MoE layer, which is two or three times higher for Nemotron than for a typical model of this size. The more experts you activate, the more parameters you activate.

The other factor is the sheer number of layers, which is higher than any of the other big open-weights models. And here I want to split out the impact of layer count on training vs on inference.

On training, the number of layers is mostly a function of your available compute. Adding layers provides diminishing returns, but it still adds returns, so if you can afford the compute you might as well go for it. And NVIDIA can certainly afford the compute.

On inference though, each layer is going to increase latency - there’s just more stuff to do for each token - and each attention layer is going to add to the KV cache, the “state of mind” of the model. NVIDIA combats this by mostly using Mamba instead of attention. See, whereas attention’s demand on memory grows as the square of the input length, Mamba is a different architecture that uses a flat amount of memory, regardless of the input length. It’s kind of like having a fixed-length summary instead of just adding to your notes.

They do add the occasional attention layer in, which is similar to how many models nowadays interleave local and global attention, but mostly it’s Mamba.

Now let’s talk about how they pretrain that model they’ve laid out.

I want to prime you with a reminder of what the Microsoft AI team did with their model, which was to use no synthetic pretraining data. We’re going to see a very different approach here, which is not a surprise when you think about it; NVIDIA has the GPUs, so they can make all the synthetic data they want. Of course at Scale we’re going to bring a skeptical eye to that synthetic data, and we know the Microsoft AI folks would agree.

They have plenty of natural data too, from the web and GitHub and the like, but they spend much more time in the paper describing how they make synthetic data than how they find or clean natural data. Again, a stark contrast to Microsoft’s approach.

Because there is so much detail in fact, I can only provide a summary of their synthetic pretraining data efforts. They include:

• Generating question-answer pairs based on the training sets of many public benchmarks

• Using the domain and difficulty distributions from those benchmarks to inspire even more QA pairs

• Extracting facts from a Wikipedia dataset and turning them into QA pairs

• Making chains of thought about moral scenarios

• Pulling legal codes and case law to then summarize

• Synthesizing random character profiles from a specialized model, Nemotron Persona, and inserting them into legal cases

Most of that is available for download by the way - another service NVIDIA has done for the research community.

Anyway, they end up pretraining on 20T tokens, shifting from diversity to quality between the two phases pictured here. So for example, “finepdfs-unfiltered” is in phase 1, but only the “medium” and “high” filtered splits are in phase 2. And just in general for all LLM training, you always want to increase in quality as you progress in training. In their case, they do that first quality ratchet after about 15T tokens of pretraining.

Here’s where the base model nets out, before post-training.

Competition in the base model space is a bit thin actually; it’s become less and less common to release one, partly for competitive reasons I think but also because base models have little or no safety training. So at minimum, the NVIDIA folks have done the research community a service.

However, I do take issue with this slide, because they left off their stiffest competition: DeepSeek V4 Pro, which we’ll see later when they compare post-trained models. Not sure why they did it, but it makes the results here much less meaningful.

Now that we have our base model, we can focus on post-training.

Our old friends SFT and RLVR are in here of course, but the fancy new thing is that multi-teacher on-policy distillation we touched on at the top.

The other unusual thing I’ll note here now but not dwell on later is MTP Boosting. MTP stands for “multi-token prediction”, and it does what the name implies - predicts multiple tokens at once instead of just one. That’s baked in from the start in the architecture and is part of pretraining, but it needs a little post-training of its own close to the end apparently. Don’t worry if the jargon in the yellow box escapes you.

Calling back explicitly to the Microsoft AI paper, the setup here is more complicated and less principled. Like they’re clearly building upon previous experiences rather than starting with a clean slate like the Microsoft folks did. Not better or worse necessarily, just a very different flavor in reading this paper. And of course the proof is in the pudding, but sadly the Microsoft AI model is not publicly available yet.

Anyway, on to supervised finetuning. I’m putting up all these other AI company logos because the researchers really went ham on synthetic SFT data, in terms of the domains and the content and so on but also in terms of the models generating that data.

Here is the entire list:

• gpt-oss-120b

• DeepSeek V3

• DeepSeek V3.2

• DeepSeek V3.2-Speciale

• DeepSeek V4 Pro

• Qwen3-30B

• Qwen3-235B Instruct

• Qwen3-235B Thinking

• Qwen3-Coder-480B

• MiniMax M2.1

• MiniMax M2.5

• GLM 5

• GLM 5.1

That is a ton of models! And notably all open weights models, surely self-hosted. The implication is that diverse training data produces more robust models, which is accurate, but they never give an explicit rationale. I did think it was a little odd they purposely used some much older models though, like in practice there’s no reason to use MiniMax M2.1 if you have MiniMax M2.5 on hand, but the principle is right.

This is in addition to models for filtering or judging data by the way. They use some models here for those purposes and also some internal models built for purpose.

As for the data itself, there’s a lot of it, and the researchers provide a lot of detail - albeit with many gaps in information.

The vast majority of the data is either pre-existing, taken from the training split of benchmarks they don’t plan to eval on, or synthesized. It’s a pretty incredible variety of data too. Like they have data for CUDA, their software stack for controlling their GPUs, as well as RTL, which is a hardware design language. Unique to NVIDIA from what I’ve seen but it makes complete sense.

By complete contrast, there are only two short paragraphs about their RLVR stage. They list the domains they target, like terminal use and instruction following and white collar workflows, but no specifics about the data. They don’t even specify which harnesses they use, just that they use many. It’s a bit baffling given the copious detail of the SFT section, although we’ll see more detail on the teacher-specific RL down the road.

Anyway, with RLVR done, we now have a student ready to transform into all the teachers. Although actually we have two students apparently, that Agentic SFT/RL box towards the bottom of the Prep column, and again that’s not really explained either. At minimum it’s from the same base model though.

So with our chatbot student and our agentic student, we’re ready to train some teachers. Each one gets its own particular training recipe, which is one of the virtues of MOPD, but it also makes for a nightmarish amount of content. So in the interest of time let me just note a couple commonalities.

One is the general use of RL over SFT. In particular, many of the training recipes use a technique from another recent NVIDIA paper called PivotRL, which takes SFT data and finds the most consequential steps - the pivots - and does RL from those points forward. It’s a nice way to squeeze more juice out of SFT data, basically turning one example into multiple examples.

Two is, again, the prevalence of synthetic data. They even have some tool called NeMo Data Designer they used for at least the Usability teacher. However, they do mention for the Office Work teacher that they bought data from a vendor called AfterQuery to help them on GDPval.

There are basically no numbers on their teacher training data, except for two places: 3.5k samples for Coding teacher, and 40B tokens for STEM teacher.

Now the distillation from teacher to student isn’t perfect, partly because the student’s weights have to optimize all these different skills at once, which inevitably leads to compromise - just like how no human can be an expert in everything.

So to quantify that compromise, they measure the pre-MOPD student, the student after both rounds of MOPD, and the teacher. The recovery is the improvement of the student divided by the improvement of the teacher. So in the first row for example, MOPD2 minus RLVR is 9.5, and Teacher minus RLVR is 5.5, and 9.5/5.5 is about 173% - the student exceeded the teacher.

Most cases weren’t like that of course, but generally the student recovered most of the teacher’s performance, although in a couple cases recovery was quite poor.

They hypothesize that distillation works best when the student reasonably could have made the correct choice, and works poorly when the right choice was not at all a possibility. Like if the student just uses suboptimal terms for a web search tool call for example, it probably has the right search terms somewhere in its distribution of likely next tokens. But if the student is doing a hard math problem, there’s really no guarantee at all that the right next token will have any likelihood - like how I would have no shot of guessing the next step of an algebraic topology problem.

To be clear, that is speculation and maybe they just have a skill issue. But it seems plausible to me.

So after all that post-training, including a little polish we again had to cut for time, this is where we land across their chosen benchmarks.

I have a few comments to make. First is the key: I highlighted the winner of each benchmark, with a 1% band below the top score to allow for ties due to noise. Like there is no practical difference between an 82% and an 81.7% score on IFBench or whatever.

Second is the selection of models: it’s all the best open-weights models, no comparison at all to even the closed non-SOTA models, like Qwen3.7 or the latest Grok. Even here though, the range of total and active parameters is wide, with MiniMax M2.7 and DeepSeek V4 Flash on the small side on both. On the big side, for total parameters it’s Kimi K2.6 and DeepSeek V4 Pro, but on active parameters it’s actually Nemotron 3 Ultra at the top! As I mentioned on the architecture slide, it’s due to all those activated experts and all those layers.

Third is, unfortunately, the relative underperformance of Nemotron in my view. Like despite being the newest and also having the most active parameters, it only wins outright on one benchmark, and it’s one I’ve never heard of, with suspiciously bad performance for Qwen and both DeepSeeks. And one of the three it ties on, the Scale benchmark Multi-Challenge, is a tie between basically every model except MiniMax. Worse yet, Qwen is smaller on both dimensions, and is four months older, yet it places first in twice as many benchmarks.

Remember what I said about NVIDIA though: they’re satisficing on quality and spiking on throughput and cost.

That’s the story we see here. At the top in the key you’ll see two versions of Nemotron: the full-precision one, which stores its numbers in the format BF16; and the quantized one, which stores its numbers in the format NVFP4. The “NV” is short for NVIDIA, it’s a special format designed to work well with their GPUs, preserving most or all of the quality yet taking up only a fraction of the space.

NVIDIA actually trains the NVFP4 version natively, basically meaning that quantizing down from BF16 to NVFP4 remains high-fidelity. And they do so because it speeds up the model massively, particularly on decode, i.e. when you’re generating tokens. It helps with prefill too - that’s the part where you’re reading all the input - but for agents with these long trajectories, decode speed is crucial.

The relative comparisons on the right are a bit misleading, since all the models on the chart are in NVFP4, but it is true that NVIDIA engineered the hell outta their serving infrastructure to get that speedup.

My Takeaways

• NVIDIA is the American open-weights champ

  • Nemotron 3 Ultra is their best model yet

  • I don’t know why it took them so long to embrace this position and put serious resources into near-SOTA models

• NVIDIA is also a gift to the research community

  • Base models, open pre-training and post-training data

  • Not to mention the hardware, software, and work in other domains (e.g. robotics)

• They rely too much on synthetic data

  • MAI-Thinking-1 is the perfect counterpoint here

  • Some Scale post-training data could help here 😉

• MOPD is the new normal

  • Still evolving somewhat but the high-level idea is firm

Paper · Website

Background

I think we have to start with, what are world models?

We actually covered this in a paper last year, where I explained that world models are not necessarily models of The World, like the physical world around us, although in this paper we actually are talking about that kind.

At its most abstract, the concept of a world model is about whether an ML model can form a coherent and predictive view of a given environment based on limited information.

That may sound like a requirement for making good predictions, which we know models can do, but it’s actually not. Let me give you an example.

Let’s say you’re flipping a coin and you don’t already know that coin flips are inherently 50-50. One way you could discover they’re 50-50 is to flip the coin a bunch of times and extrapolate from that data to predict future data. You don’t need to understand the nature of coins or the laws of physics to do that extrapolation, you just need basic data analysis.

Of course the way a human would conclude coin flips are 50-50 is to look at the coin, maybe toss it a few times to make sure it’s not weighted, and then reason a bit or just intuit that the coin toss will be 50-50. That does require a world model, because it requires underlying assumptions about how objects behave in the physical world.

Another example where you hear both the statistical model and the world model-ish views of a thing is in the stock market. Some people just look at trends, with no concept or view of the underlying company, the ticker symbol could mean anything. A lot of quant trading is like this, where you just feed a bunch of factors into a mathematical model and get a “buy” or “sell” recommendation out. On the other end of the spectrum are the fundamentals investors, who look at the company’s financial statements and the market and the strategy etc and form a model of the company’s future performance.

So a world model is just about where the reasoning seems to be, not about one particular place or another.

Here’s another example of a world model, this time in the world of code.

So normal code data for us would be a prompt and a response, if it’s SFT, or a prompt and a set of unit tests that verify whether the model’s answer works, if it’s RLVR. But in either case we’re training the model directly for writing code.

By contrast, this coding world model has a different goal: to predict how a piece of code will work. If the model can do that, then it probably has a good world model for the world of code, its “laws of physics” so to speak.

The way they train that in is to provide a piece of code and an example of using that code, then ask the model to predict the state and action at each step of the program.

Here the state is in yellow and the action is in blue. As you go down the rows, or “frames” as the paper calls them, you can see what the model is keeping track of and how it changes after each action. Like for example in the second frame, we get a new variable n, with the starting value 0. Then in the third frame we see n being tracked in state, along with its current value, 0. The two dots in quotes is just a visual shorthand that means the value of the variable hasn’t changed, so like s is “strawberry” for the first frame and then just the two dots for the rest of the frames.

Of course in the end what we care about is whether it can produce good code, like with any coding mode. But this idea of training or testing the world model, separate from the end result, will be relevant later on.

Now let’s look at some video generation models, what laymen might think of when we say “world model”.

This is a clip from Sora, OpenAI’s video generation model, when it first came out back in February 2024. I remember being very impressed when it came out, and honestly it still holds up at least in this clip, but you can see some weirdness happening. Take a look at the manhole for instance and how it kinda morphs into pavement, that’s a classic AI video-ism. Consistency and memory are tough for video gen models.

One reason it’s generally so accurate though is because of what they trained on: synthetic videos, generated by engines like Unreal Engine and Unity, which have programmed in very detailed rules of physics. If you train on tons of footage from physics engines, you’re going to end up with a pretty good sense of physics. OpenAI actually views Sora as a “general-purpose simulator of the physical world”. So they’re going for a world model here.

The latest and greatest in this space is Genie 3, which generates good videos but also allows real-time exploration. Look at the legend in the bottom-left, showing arrow key presses. Genie 3 generates the world in real time, lets you navigate it, and it remembers what has been in the scene before but is out of the scene now, to maintain consistency. Again, you need a really keen idea of the world to do something like this. So the question is, how do we put it to use in the actual physical world?

Now getting more into the robotics side, let’s talk a brief bit about inverse dynamics models, or IDMs. It’s a complicated name, but really the idea is simple: instead of a regular dynamics model, where you infer outcomes given actions, you invert it, inferring actions given outcomes.

OpenAI actually pioneered this concept in the modern AI age. This graphic is from a 2022 paper called Video Pretraining: Learning to Act by Watching Unlabeled Online Videos. Their goal was to teach a model to play Minecraft. To do that, they figured the best way was to take advantage of the many many hours of Minecraft gameplay video available out on the internet - some 70k hours apparently, after significant filtering. But how to teach the model with videos yet no key presses or mouse clicks? The model wouldn’t learn how to actually play the game!

To make their 70k hour corpus useful, they synthetically added actions in. They collected 2k hours of labeled video, i.e. with the actions recorded, and then trained an IDM to predict actions based on video. Once they had that, they put these “pseudo-labels” as they’re sometimes called on all the initial videos, then trained their Minecraft agent on the pseudo-labeled corpus.

Ultimately though they’re learning actions from videos, without any sort of world model; the IDM doesn’t demonstrably know what the key presses and mouse clicks mean, just what they correlate to.

We should also discuss briefly how a modern robotics model looks. For that we’re going to turn to the Pi series of models from Physical Intelligence, which is roughly the OpenAI equivalent of robotics model makers.

The Pi models are VLAs, vision-language-action models. That’s an LLM at the center, with a vision encoder added on the front so it can see, and an action expert added on the end so the model can do stuff - the action expert outputs signals for the motors and such. In the case of Pi 0.5 the LLM is Gemma 2 2.6B, made by Google. The vision encoder is SigLIP, also made by Google, so the vision-language model so far is 3B parameters. Then you add on the action decoder for another 300M parameters, so 3.3B parameters total. It’s a pretty small model, but it really has to be in order to produce actions in real-time. Gotta keep that latency down, no way to do that with the parameter counts of SOTA LLMs, which are in the tens or hundreds of billions, maybe even trillions. By the way, this is the same reason that people speculate Sora and Genie are also in the single billions of parameters. Kinda crazy that all the knowledge to model a world can fit in that few parameters.

So one of the major contributions of Pi 0.5 was this concept of “co-training”, i.e. training the VLM part on a bunch of multimodal web data. If you’re not familiar with this work already it may be somewhat surprising that a robot could learn to manipulate the world better by just looking at images or watching videos, but of course we know humans can do the same - we can learn how to do things just by watching. Hands-on practice is often better but it’s not always required.

They also throw in training data from other robots, like other embodiments that this model doesn’t get used on.

Anyway, the main thing to know here for our purposes is that VLAs are the currently dominant paradigm in robotics models, and they start from LLMs.

The Paper

First let’s talk hardware. This is the star of our show, AgiBot G1. This model has grippers, although they make a version with dexterous hands. The grippers can rotate. Each arm has two “elbows” so to speak. The torso can move up and down on the base, and the base can wheel around. Finally, there’s one camera watching each gripper, and one behind the faceplate watching the whole scene.

This is the robot they use for gathering data and for doing most of the evals. They do use a couple other robots, specifically a Franka Emika Panda and a YAM, but if you’re going to imagine one robot for these results I would pick this one.

Now let’s cover the model.

The big difference between normal transformers like your typical LLM and robotics models of any sort is that robotics models have to predict things in parallel, because robots have different parts that can move simultaneously. You don’t have to do that with text, you can generate one word at a time. But you can’t trade off which limb moves at a time or something like that, at least if you want to produce smooth motion.

Image and video generation models, on the other hand, do predict things in parallel, although that’s more because of the inherently parallel nature of vision than necessarily having to produce many pixels at once. Still, you do have a body of image generation research to draw on.

So the architecture here combines the transformer, which predicts in serial and is good for language, with diffusion, which predicts in parallel and is good for image. That’s the diffusion transformer, DiT. And the way it works is by predicting small blocks of actions where stuff can happen in parallel, but predicting each block in series, so only one block at a time. Specifically, they predict 1.6 seconds of action per block. That’s how far ahead the model is “thinking” or “envisioning”. Predicting many actions in a block like that results in smoother motion compared to predicting just one action at a time.

But it doesn’t blindly act in 1.6s increments and wait until the next increment to make adjustments; just like how your body frequently adjusts balance or grip while carrying your dinner across the kitchen, the model makes updated predictions on a similarly short timescale. With this model, this hardware, and a suite of optimizations we don’t need to get into, they can make new predictions every 150 ms. So if the researchers swap out an object in the scene, or the robot’s grip slips, or the wind blows a cup over, the robot can react in a reasonable amount of time. So you never get to the end of that 1.6s block, you always have a new block ready to start acting on.

Now the other thing they mention here that I want to dig into is this word “joint”, as in “joint video-action flow matching” and such. What they’re saying there is that given some inputs, they want to predict video and action simultaneously, NOT predict video and then predict action based on that predicted video. There are a couple reasons for this.

The first is that with two simultaneous predictions, errors in one are unlikely to appear in the same way as errors in the other. So like if my video part incorrectly predicts the plate I’m carrying starts to tilt, but my action part has predicted no change in my hands, that disagreement gets picked up as loss and thus is targeted in future training steps. But if I predicted the video and then predicted the action on top of the video, then my action part is basically stuck predicting something wrong in order to agree with the video prediction and thus minimize loss.

The second is that joint prediction shares the world model knowledge with video and action prediction. That’s the main theoretical thrust of the paper really, that there’s this world model implicit in a video generation model that gets locked away when you try to learn actions from video instead of learning actions alongside video. Like you don’t want the video to be an intermediary between the implicit world model and the action predictions, you want to go straight to the source.

Here’s a quick example of some generated video. We don’t get a side-by-side of generated vs actual sadly, but you can see they line up generated video with where the real action happened. In the Generated row, the top-left square is the faceplate view, the top-right square is the right arm, and the bottom-left square is the left arm. You can see the robot uses the left arm to get a second view of the scene, which is kinda cheating from a human point of view.

OK, on to the data. They need this robotics data to transform their starting point, the open-weights video generation model Wan 14B from Alibaba, into a world action model, a WAM.

The main body of data they collect is here, ~500 hours of teleoperation data on their AgiBot G1 across 7.2k episodes and 22 different environments. Since one key advantage of WAMs is that they rely on their world models to guide actions, they don’t need tons of demonstrations per task to then imitate. That allowed the researchers to focus on diversity instead of repetition with their time budget.

The average episode is about 4.4 minutes and has over 40 steps, what they call “subtasks”, which is relatively long-horizon for this type of thing. So with this pretraining data, we can turn Wan 14B into DreamZero, which can predict video and actions.

So with their 500 hours of pretraining data in hand, they can train some models and start comparing results.

Let’s cover the models and terminology first. On the bottom you’ll see three models mentioned: GR00T, a VLA also made by NVIDIA; Pi 0.5, a VLA by Physical Intelligence that we’ve covered before and that we saw the architecture of earlier; and DreamZero.

They also give two different descriptors in parentheses: scratch and pretrained. Scratch means no robotics data other than the ~500 hours these researchers collected, to give a fair comparison between DreamZero and the VLAs. Pretrained means it does have other robotics data in it, so that’s like the full versions of GR00T and Pi 0.5 against the full version of DreamZero.

Finally, for the tasks, PnP means “pick and place”, things like putting fruit in a bowl; and Contact-Rich means folding clothes basically, which is super common in robot evals and also is one of the most common use cases for robots in the real world actually, like to a surprising degree.

In any case, DreamZero outperforms the other two on all tasks and both embodiments. It seems like that world model really does confer a lot of physical common sense and allow the robot to handle a wider variety of cases.

Now for the previous chart, those were all seen tasks, tasks present in the training data, although the evals were in new environments and with different objects.

In this chart, the tasks are all unseen, not present in the training data. Again we see the same general trends, although Pi 0.5 does come close to DreamZero in a couple cases. To be fair to the Physical Intelligence folks, 0.5 is not the latest version of their model, but it is the most recent open-weights one so it’s a fair base of comparison.

One fun note from the authors: apparently the VLAs often try to grab objects regardless of the prompt given, suggesting they’re overfit to those types of tasks. Makes sense given the most common tasks are versions of pick and place, I like the variety of actions they represented here.

So the results we’ve been looking at so far are for models that are pretrained only, i.e. no task-specific post-training on demonstration data. Here they change that.

The post-training data is for three tasks: shirt folding, collecting 33 hrs; fruit packing, collecting 12 hrs; and table bussing, collecting 40 hrs. In all cases they use a variety of objects, different counts and positions, etc.

Here we see near-parity between Pi 0.5 and DreamZero. Unfortunately we don’t have pretrained vs post-trained results to understand the impact of post-training per se, but my interpretation is that WAMs can learn from demonstrations just as well as VLAs can. Like sure, WAMs seem to do better right out of the box, but you could have objected and said we don’t care about out of the box performance, we care about absolute performance, and maybe VLAs have a higher ceiling because they learn so well from demonstrations, i.e. from post-training data. But that doesn’t appear to be the case; the ceiling for a WAM seems just as high as the ceiling for a VLA.

The bull case here might be that WAMs don’t even need post-training data or might only need a few human egocentric examples rather than teleop data, but again, they don’t show these evals before post-training.

Finally, to drive home the point about most work happening within the world model rather than within the robot-specific parts, they collect two small sets of data for a set of new tasks:

• 12 minutes of human egocentric video

• 20 minutes of video from YAM, another robot. No actions collected, just video

Then they train on them and run the evals on three versions of DreamZero: the initial version, a version trained with the human data, and a version trained with the YAM data. Then they see how well DreamZero learned to do these new tasks from the human video and the robot video.

What they found was that both transferred about equally well, meaning you don’t need expensive robot hardware to make good data. All you need is a human with head- and wrist-mounted cameras who’s willing to visit a lot of new environments.

Now running in the opposite direction, they took 30 minutes of video from YAM on a different set of tasks, unrelated to any evals, and trained their model on that. And suddenly the model was able to operate the YAM, even though its pretraining data was only on AgiBot. So in both directions the embodiment seems to matter only a little, and the bigger factor by far is whether anything in the pretraining corpus looks like the task at hand.

The authors give an explanation that I want to quote in full to round out our review:

“Learning an implicit IDM from predicted videos may be inherently more sample-efficient than direct policy learning - the model only needs to learn the mapping from visual features to actions, while leveraging the pretrained video model’s existing understanding of physical dynamics. Consistent with our AgiBot findings, failures primarily stem from video prediction errors rather than action extraction, suggesting that increasing task diversity during post-training could further improve performance.”

My Takeaways

• We might be able to ditch the robots, at least for pretraining data

  • For evals you always want to test in real conditions, i.e. with the robot

• Post-training for improving video generation will be valuable

  • RLHF

  • Rubrics? Seems challenging but potentially worth it

• This could be the start of a shift away from VLAs to WAMs

  • Even at similar quality, the data for WAMs is more scalable

Paper · Blog post

Background

So before this announcement, I think most people weren’t aware Microsoft worked on any models at all. Certainly they use models, like in their various Copilot products, but that’s pretty much all models from other providers.

But actually, one arm of Microsoft has been releasing models for some time now. That arm is Microsoft Research, and they have a good reputation for their Phi series of models in particular. The Phi Family as the image calls them is a wide array of research-size models, runnable and trainable on relatively modest hardware, and covering a lot of different areas. The Phi models aren’t great generalists, but they have historically done well on code and STEM work.

Microsoft Research has a lot of other models on their Hugging Face page, and they release a decent number of papers as well, but at least in my mind, I have historically considered “Microsoft model” and “Phi” synonymous.

But this paper and this release are from a different arm: Microsoft AI. And to talk about Microsoft AI, we have to talk about this man: Mustafa Suleyman.

Mustafa Suleyman is a giant of the field. In 2010 he and Demis Hassabis cofounded DeepMind, which Google acquired in 2014. DeepMind is the original seat of Google’s own generative AI efforts, so already in his bio we can credit him as the father, or perhaps the grandfather, of Gemini.

Fast forward to early 2022. Suleyman leaves Google, joins a VC firm, then quickly leaves to cofound a new lab with Reid Hoffman called Inflection AI. Keep in mind, this is before ChatGPT came out in late 2022, but well after GPT-3 came out, in mid 2020. So folks in the know are seeing the pace of AI progress is starting to really pick up - inflecting, if you will.

In 2023, Inflection launched its ChatGPT competitor, a chatbot named Pi, powered by the Inflection series of models. Inflection-2 came out in November 2023, Inflection-2.5 came out in March 2024, but by then it was already clear OpenAI was well ahead on both capabilities and chatbot market share.

So on March 19th, 2024, their lead investor bailed them out: Microsoft hired Suleyman and most of the 70-person team to become Microsoft AI. They also paid a huge “non-exclusive license” fee. So this was one of the first of what I like to call “fake acquisitions”: a bigger company scooping out their target talent from a smaller company, delivering a big payout, and leaving the remains of the smaller company alone. This is the same playbook Amazon used on Adept and that Google used on both Character AI and Windsurf.

It does share some characteristics with the investment Meta made in Scale in 2025, with Alex and a few other folks leaving to form Meta Superintelligence Labs, but the other transactions were “licensing fees” with no economic upside for the bigger company. Meta actually has a stake in Scale’s success, and Alex is on the board. Obviously the pattern-matching led to some rough vibes at the time of the transaction, but Scale is very much still a going concern, while the other companies I mentioned are shells of their former selves.

Anyway, now two years later Suleyman and his Inflection crew, with a lot of other talent on board, have finally released their first models.

The parallel thread here, and the subtext to a lot of this, is the relationship between Microsoft and OpenAI.

Microsoft first invested in OpenAI all the way back in 2019, before the release of GPT-3, as the exclusive cloud provider for OpenAI. At the time, Microsoft execs felt they had fallen behind Google on machine learning, given the latter’s high-profile acquisition of DeepMind. So as part of the investment, Microsoft got a license to anything OpenAI built. It also gave Microsoft 75% of OpenAI’s profits, up to the amount they originally invested. Pretty wild deal in retrospect huh?

So Microsoft continued deepening the partnership, with more big investments in 2021 and 2023, and deployment of GPT-4 across tons of Microsoft product surfaces. And with Microsoft as the exclusive provider of compute, any increasing demand for GPT meant increasing demand for Azure, and thus increasing profit for Microsoft.

Then in November 2023 came the near-acquisition moment: the OpenAI board fired Sam Altman on a Friday, over the weekend Microsoft offered to hire Sam and whatever staff wanted to move over from OpenAI, but by Monday Sam was back in the driver’s seat. As we know now, that left the door open for Suleyman at the future Microsoft AI.

Since then, much of the partnership has come undone. Microsoft is no longer the exclusive provider of compute. Their ownership stake is down, although owning roughly a quarter of a $1T company ain’t bad. And Microsoft’s IP and royalties arrangement with OpenAI is looser than before. Now with the release of the MAI models, it’s almost a frienemy relationship. And I’m sure Suleyman would love to best GPT one day.

Now briefly on the science side, I do want to cover one thing, which is loss.

Loss is the way you quantify how good your model is in many stages of training. It’s a single number that you set out to optimize, to get as low as possible via training.

For this paper, the form of loss we need to discuss is negative log likelihood, or NLL. NLL is what we use during pretraining and during supervised fine tuning (SFT).

Conceptually, NLL looks at the likelihood your model gave for the actual next token in your training data. If your model predicted the correct next token with 100% probability, the loss is 0. If it predicted the correct next token with 0% probability, the loss approaches infinity.

NLL is a nice way to measure performance because it is fast and objective compared to more intuitive forms of benchmarking, like human evals or leaderboards. It is also continuous between pretraining and SFT, so you can compare apples to apples. It also works for any text you think is trustworthy: just feed in some and see if the next predicted word is what your text says is next. That’s quite different from the training and eval data we make here, where we spend a lot of time ensuring quality and diversity and fit to spec and all that. Of course there are advantages to the other approaches and the other data like we make, and we’ll see MAI use them.

So for all these reasons, NLL is the consistent ruler we will see the MAI folks use to measure performance. Although occasionally we will see its cousin, “bits per byte” (BPB), which accounts for the differences in tokenizers.

The Paper

Before we start, just note that I had to skip a lot of things in the interest of time, particularly the serving and training infrastructure. If you want that detail, it’s all in the paper - they were quite thorough.

Let’s start with architecture.

MAI-Base-1 is a 1T parameter model, with 35B parameters active, across 78 layers and with 512 total experts, 8 of which are active at a time, for a sparsity ratio of 64. That is all very similar to many other modern models where we have architecture details. For example, Kimi K2.6 is 1T total parameters and 32B active parameters. DeepSeek V4 Pro is larger but in the same ballpark. So far so good.

Where it starts to go off the beaten path is in the details of the layers.

So for attention, which is where the model forms a holistic picture of the input, they switch between local and global attention, in a 5:1 ratio. So that means for 5 attention layers, the attention is only looking 512 tokens back, and then for the sixth layer it looks all the way back to the first token, assuming we’re within the model’s 256k context window. Switching between local and global is common, but 5:1 is on the more extreme end.

The weirder thing, which I have never seen before, is switching between dense and mixture-of-experts feed-forwards - the layer where all the processing or “thinking” happens. They’re switching it up every time, so in a 1:1 ratio. They claim that interleaving dense and super sparse MoE like this yields the same results as a less-sparse MoE throughout, but faster, which makes sense given how successful interleaving attention types has been.

Speaking of MoE, on the right side they diagram theirs, which is also a bit weird. So they’re showing their 512 experts, and we know they pick 8 of them each time, but they have this down and up projection thing and these yellow boxes around the experts.

Here’s what’s going on with that: they are compressing the inputs before going to the experts. That’s what the down projector does. But routing, which is where you select experts to activate, is super sensitive. So they split out those functions: the router uses the uncompressed input to pick which experts to use, and the dispatcher relays the compressed input to the selected experts. Then they combine all that info and uncompress it.

Here’s a bit about how they settled on their MoE setup.

What they’re showing here is how performance improves on a series of benchmarks given more training and more experts. So the x axis is FLOPs, which is the standard unit of training compute. The y axis is something they call “efficiency gain”, or EG for short, which is basically how much better your experiment is compared to your baseline. And the different colors show different numbers of total experts - they only activate 8 every time.

The upshot is that more total experts improves performance, and training them for longer sometimes improves performance. But more parameters means more cost and more latency, so they compromise on 512.

This one decision per se is not critical for understanding the whole model, but it’s emblematic of how the MAI team approached all of their decisions: empirically, with lots of rigorous optimization experiments. It’s a refreshing paper for its rigor, and also for its openness.

By the way, the weighted average in the bottom-right weights code at 50%, math and STEM at 17.5% each, general at 10%, and multilingual at 5%. So you can really see how much they’re optimizing for code and reasoning compared to everything else.

Worth noting these benchmarks include at least some human-generated data from external vendors — a sign that demand for human data in GenAI isn't going away.

Here’s another example of that optimization loop and rigor.

What they’re showing here is how different mixes of training data impact performance, as measured by NLL on two different validation sets - that’s the x and y axes. Each dot is a different data mix, and each color is a different size of model they trained in order to test the mixes.

They’re only showing a few sizes of model here, so there’s maybe a few dozen dots on here, but they mention that in total they trained several thousand models of between 760M and 4B active parameters.

One particular aspect they pay attention to is how different datasets interact, both within their domain but also across domains.

Like here for example, they’re measuring performance on a graduate-level physics benchmark, which a vendor created for them for this purpose. You’ll see our old friend NLL on the y axis, so lower is better here.

What each of the six graphs show is how adding the given data type in impacts performance. Like before, each dot is a model trained on one specific data mix.

In this example, math and STEM stuff helped, which includes “web PDFs” - those are PDFs of scientific papers. General web content didn’t matter, and code actually hurt. I personally was surprised by that, because generally code data helps with math performance, and vice versa, and physics is mostly math. But that’s why you run the tests! Anyway, hopefully you’re getting a sense of how a team ends up training several thousand models in their pursuit of one final model.

Relatedly, even though this much training takes a lot of infrastructure work, we just don’t have time to cover it. But if you want to know all the gnarly bits about managing GPUs and various forms of parallelism, it’s in there.

So after all that fiddling, here’s where they net out on pretraining mix.

Keep in mind this is all naturally generated - none of it is synthetically made for the purpose of training data. They even take great pains to identify and remove suspected AI-generated text from scraped sources. However, given the high share of code and the pervasiveness of coding agents, I suspect a decent share of the data is ultimately from models.

Speaking of code, it’s over half the data! And it gets one of the highest multiples there in the right corner, meaning they replay the data about two times. That’s kind of a proxy for usefulness. So apparently STEM data, books and journals, and especially math data were all very helpful.

30T tokens in pretraining is pretty good by the way. That’s about the same pretraining corpus as DeepSeek V4. To go much higher than that you’d need to start generating a lot of synthetic data expressly for pretraining purposes, which is a gamble and thus was not where MAI wanted to start.

Now after pretraining there’s midtraining, kind of smaller and more focused pretraining where they only take the cream of the crop. It’s also typically where people extend the context window.

Both cases are true here. They filter heavily for quality and rebalance domains a bit, then they extend context out to 256k tokens. That adds another 3.5T tokens or so.