Originally presented as a live talk on December 17, 2025
Background
So what is AI2, the organization behind this paper? The Allen Institute for AI is the leading US lab doing open source AI.
And what I mean by “open source” is truly, fully open: all the code, all the data, all the processes etc are totally free and available.
That’s actually quite rare; most “open” models are open weights, meaning the finished models are free and available, but the ingredients that made the models are not. So big open weights names like Qwen, DeepSeek, Llama, Gemma - these are all open weights models, not open source models.
Don’t get me wrong, it’s great that they spent all that money on the compute for training! You and me are getting great products for nothing. But it doesn’t advance the science when many of the steps are still secret. For example, the technical reports tend to be light on post-training data details, often not even saying how many examples they used.
AI2 has been doing open source from the start. In fact, their language model, Olmo, stands for Open Language MOdel. They really run with that name by the way, like Molmo is the multimodal version of Olmo, and Dolma is the data for Olmo.
So not only are they established, but they’re nearly peerless. There are a few other outfits doing open source LLMs, like the Marin team at Stanford or the LLM360 team in Abu Dhabi, but no one has the same combination of talent and resources.
Perhaps the most publicly identifiable figure from AI2 is Nathan Lambert, the author of Interconnects on Substack, which I recommend for folks who want relatively technical but still accessible AI news.
He also wrote a book this year called The RLHF Book, which I recommend for anyone who wants to build a deep technical understanding of the topic.
The Paper
That’s actually going to be it for separate background slides, because this whole paper is basically a tutorial. So I’ll weave background knowledge in, as we go from architecture to pretraining to midtraining to post-training to evals.
There are many engineering details we’re going to skip over, like learning rate schedules or GPU utilization, but if you want them they’re all right there in the paper. I just think the science and the data are going to be better uses of our limited time today.
Let me give you a roadmap here to start. We’ll discuss architecture first, which is kind of a prerequisite for this graphic, but then we’ll go as it says:
Pretraining
Midtraining
Long context
SFT
DPO
RLVR
The base model just has the pink pretraining stages, then depending on the greenish post-training stages you get one of three different models:
Think, which is a reasoning model that produces a long chain of thought before giving a final response
Instruct, which just launches into its answer and so is better for general chat
RL Zero, which is more for experimental use and demonstrating the effects of RL directly on a base model. We’re not going to cover that one.
First up is the architecture, which is basically the same across the 7B and 32B variants, except scaled up of course for the bigger model.
We’re working with a fairly traditional Transformer here. It has three main parts:
Embedding, which turns words into numbers
Attention, which takes every piece of input and builds a holistic understanding out of it
Feed-forward, which takes that holistic understanding and processes it, “thinks” about it so to speak
On the embedding side there’s usually not much to report. I’ll point out the embedding dimension here because it shows one of many places where the 7B and 32B differ. The bigger the dimension, the more detail you can capture about a token when you turn it into numbers, specifically into a vector with that dimension. So larger models have a more subtle understanding of each token.
On the attention side there are two things to note. One is this 3:1 ratio between local and global attention. What that means is that 75% of the time, they’re only letting each token talk to a small number of neighbors, and the remaining 25% of the time they’re letting all the tokens talk to each other. Local is less resource-intensive, global is more informative, and empirically a lot of models have settled on this 3:1 ratio.
The other is this note about attention heads. An attention head is like a lens through which the model views the input. Pretty much all models provide lots of attention heads so that the model can view the input in many different ways. Like one head may pick up on sentence structures, another may look at high-level themes, a third could parse code, etc. Those functions aren’t assigned or designed in by the way, the division of labor emerges naturally over the course of training. And the function of each head is often inscrutable to humans.
Anyway, the architecture difference related to attention heads is in that note about key & value heads. In the small model, each attention head gets its own key-value pairs, basically its own set of notes about what it has observed. More notes means more memory required. In the big model, the memory demands are already higher because it’s a bigger model, so to compensate they make attention heads collaborate on notes, in this case one shared set of notes for every five attention heads.
So that’s the attention bit. Then on the last part, feed-forward, this is a dense model, it is not a mixture of experts. You can see they scaled up the intermediate projection size on the big model, basically giving it more space to think, but the rest of the details are shared and not interesting.
The only other thing I’ll note is the number of layers. The 7B model does the Transformer block, which is attention + feed-forward, 32 times. The 32B model does it 64 times. More blocks generally means better reasoning but also makes training more finicky since learning signals have to travel for longer.
So once you’ve got your architecture all coded up, in theory you can start pretraining. But of course, you’re going to need your pretraining data. And getting that in order is no easy task.
Now compared to post-training data, which is what Scale makes, pretraining data is much simpler: it’s just examples of text, including natural language, math, and code. It doesn’t have to be “right” in the sense of a test question, it just has to be a reasonable example of text you’d want your model to produce. The tolerance for errors like incorrect facts or buggy code is high.
The real challenge here is cleanup: of all the text you can find, mostly from the internet, how much of it is even remotely helpful?
Depending on the source, shockingly little! So for Common Crawl for instance, which is basically a scrape of the entire internet, 85% of it is junk: highly repetitive, super spammy content, from known bad sites, boilerplate, error pages etc. Just on that heuristic filtering step they go from 253 billion documents to 39 billion documents.
If you then aggressively deduplicate, including close matches, you reduce by another 75%! So under 4% of the entire internet basically is actually useful.
Now for the 8T English tokens they get that way, they’re able to add another 1T tokens from the other three sources: academic PDFs, GitHub repos, and a couple math pretraining sets. But even with the PDFs they lose just over half their starting pool from filtering.
Filtering takes a lot of iterative work, but after that it’s more straightforward. They tag every document with one of 24 topics and one of 20 different quality levels.
Then they take that dataset and make a mix by sampling based on topics and quality. For topics, they experiment by training very scaled-down versions of Olmo 3 with very scaled-down versions of the pretraining dataset and testing on some benchmarks. So for example they might find that if they sample documents related to programming more, their coding benchmark scores go up. Within a constraint for an overall number of pretraining tokens, you figure out a balance across topics that may look different from the natural distribution.
For quality it’s a different story of course, because you clearly want to sample more from the better documents and less from the worse documents. Through a similar set of experiments to topics, they settle on dropping the bottom 40% of documents by quality, then gradually increasing sampling by percentile, with the top 5% of documents by quality getting sampled seven times.
So here’s the final mix they end up with. Remember the 9T tokens is what they filtered down to, before any sampling decisions. So most of the tokens are from the internet, then academic documents, then code, then a bit of everything else.
One small thing I would note is the number of tokens per document in the original pool compared to the final mix. Just looking at the bottom row for instance, you have an average of 1000 tokens per document to start and about 1500 tokens per document at the end. Long documents tend to be higher quality, although sometimes model and people alike conflate length with quality.
I also wanted to briefly show what pretraining actually looks like for a researcher. The graph here shows the loss as training progresses, with lower loss meaning better predictions. So at the start you see really quick improvement, as the model goes from basically trash to something vaguely coherent. And then the rest of the time you get this slow march of progress, with lots of movement back and forth around a gradually improving trend.
And when I say gradual, I mean gradual. This pretraining run took 45 days on 1024 GPUs, consuming over 90% of their total compute and costing over $2M. By contrast, post-training takes a fraction of the time and an even smaller fraction of the compute.
So with pretraining done, we move on to midtraining. The mechanics are the same - we want the model to get better and predicting the next token - but the data shifts.
Specifically, the document look a lot more like chats, like question-answer pairs or code completions etc. A lot of the work involves taking a source document and turning it into chat-ish data, like having a model write a bunch of questions and answers based on a nice Wikipedia article. There’s also a lot of SFT-like data, like the reasoning traces in the Thinking section here, but used as pretraining rather than post-training data.
Also similar to pretraining, they collect a big corpus and whittle it down with filters and sampling. So the 2T token pool becomes a 100B token mix, focusing on certain abilities and balanced using the exploration + assessment workflow shown on the right. We don’t need to get into the details, but broadly it’s the same idea as in pretraining, using small experiments to then extrapolate into bigger results.
The last stage before we have a complete base model is context extension. Models almost always start out with a very small context window, 8192 tokens in this case. That’s enough for a normal request or a short conversation, but quite limited for tasks involving a lot of input like agentic coding or document analysis.
Mechanically it’s quite easy to increase the amount of context you give to a model, but the model needs training to understand how to deal with all that context. That’s what we do in this stage. So we take documents of increasing length and pretrain on them, which gradually extends the amount of context the model can make sense of. We also keep some short documents in there to balance the learning, otherwise the model will regress on shorter contexts.
The context window of Olmo 3 is 64k tokens, which is why they didn’t include any longer documents in the final mix. But they include them in the starting pool so that other researchers can experiment with them.
So checking back on our roadmap, that gets us to the base model. Now we enter post-training.
Again, across all stages they’re going to spend a fair amount of time gathering and cleaning the data into a mix they call Dolci. Since they don’t have a big data annotation budget, everything is either going to be open source or synthetic.
First they start with the skills they want, tending mostly towards STEM with some baseline chat abilities and that tool use that will be so helpful for agents.
Then comes filtering: non-commercial license, incomplete reasoning chain, non-English, explicit mentions of other models, irrelevant requests like image generation, etc.
After that comes mixing, again using small experiments to extrapolate. Throw in decontamination against the benchmarks and you’ve got your post-training mix.
Here’s the SFT mix. Nothing crazy on here, I would just note the relative size of each category and the overall count of about 2.3M examples.
Now here’s the DPO mix. DPO uses preference ranks, like RLHF but technically not RL under the hood. They make preference ranks by getting one response from a big model and one response from a small model in the same family, in this instance Qwen3, then assume that the big model’s response is always better.
And finally here is the RL mix. For prompts with verifiable answers, they do RLVR as shown in the top-right. For prompts without verifiable answers, they use an LLM judge to assign a score, usually Qwen3 32B.
While we’re on the topic of RL, I wanted to show another example of what researchers are actually watching as training progresses.
All these graphs are showing how the reward given by the verifier or LLM judge changes over the course of training. Each step is where we’ve updated the model and run our suite of RL prompts again to see how performance has changed. Higher reward means the model is getting more prompts right.
As you can see it’s always a noisy trend, sometimes almost more noise than signal. The overall trend for a good run is generally a curve up and then flattening out, although the code reward at the top is a little bit scary.
You’ll notice these graphs have different numbers of training steps. I’m not sure how to square this with what they wrote in the paper though. They mention that Olmo 3 got 750 RL training steps over 5 days, but that they continued up to 2300 steps for 21 additional days and released that version as Olmo 3.1. And apparently they expected further gains with more training, but they ran out of training budget.
Anyway, my guess is these exact curves are from smaller experimental models.
So here’s how each step nets out compared to the open weights and open source competition. The graph generally speaks for itself, but I would note two things.
One, this graph completely depends on the evals you choose. They talk a bit about how they chose their list, but you could easily make a good argument for a different list, and maybe all the sudden Olmo 3 is a lot worse than Qwen3. I do find it a bit suspicious that Olmo 3 is so close to the open weights competition here.
Two, some improvements don’t show up on this graph. An easy example is long context training, which visually seems to make no difference at all, but if you test specifically on long-context tasks you’re going to notice the difference right away.
Certainly this is an improvement on Olmo 2, and almost certainly it’s the best open source model out there. Otherwise I’m not sure what conclusions to really draw.
If you want to break open that aggregate score here’s what you’ll see for the thinking model. Again it seems roughly on par with Qwen3, slightly behind the minor upgrade of Qwen3 VL, but in the ballpark anyway.
Same general story for the instruct model. You’ll see a big jump on the AIME 2024 and 2025 benchmarks, but I think that’s more about Goodhart’s Law than true specific progress.
My Takeaways
The split between SFT and RL has flipped
When RLHF was popular, the rule of thumb was 10x RLHF compared to SFT
The flip is likely due to RLVR (and Rubrics) taking over for RLHF in the areas Olmo focused on
Rubrics are still trickling down
I was surprised not to see any mention of them in here
Agents only somewhat overlap with ASI
The core components of intelligence do not significantly improve with agentic training
Agent work (e.g. environments) is an additional area of work for us, not a substitute for e.g. hard reasoning
Data is a big deal :)