Originally presented as a live talk on December 10, 2025
DeepSeek-V3.2: Pushing the Frontier of Open
or, Papa Bear
DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
or, Baby Bear
Background
The main innovation of DeepSeek-V3.2 is the DeepSeek Attention mechanism, so first I wanted to cover attention more generally.
We’ve covered it before, but to oversimplify somewhat there are three components of a transformer model, which is the basic architecture all your favorite LLMs use.
Embedding, which turns words into numbers, since after all we are ultimately working with numbers in the form of matrix multiplications here
Attention, which calculates how each individual word or token relates to all the other tokens that came before it in order to form a complete, holistic understanding of the input and output so far
Feed-forward, or MLP as it’s noted here, which takes that complete understanding and “thinks” or “processes” it
For the attention layer in particular, you see each input splitting into a query, a key, and a value. The thing to notice here is how the queries and the keys form this matrix, where each query gets to “look at” or calculate against each key of the previous tokens. So like “The” at the start can only attend to itself, whereas “tokens” at the end can attend to all the tokens in the example. And within each row, like on each query, the circles are colored by how strong the relation is. So on the query “to” for example, it’s attending strongly to “The”, weakly to “model”, strongly to “attends”, and weakly to itself. Sometimes those relationships are a bit inscrutable to humans but in this case it generally makes sense.
The only other thing I would briefly note here is that the attention and feed-forward steps or “layers” as they’re called form the “transformer block”, and the transformer block repeats many times before the model is finally ready to predict the next token. What that attention matrix looks like in terms of the strength of relations changes from block to block.
So in the standard transformer model like we just saw, the model pays attention to everything - up to its maximum context window anyway. That would be the first box on the left, “Full n^2 attention”. And it’s called n^2 because the number of green squares goes as the square of the number of tokens. That means if you double the number of tokens in your context window, you’re quadrupling the compute and memory needs. It’s the gold standard, but like gold, it’s pricey.
People spend a lot of time trying to find cheaper attention mechanisms that are nearly as good. One of the more popular ones is Sliding Window Attention, also seen above. Here you specify a window size that is much smaller than the full context length. At first that means you’re only getting information about very nearby tokens, but as you keep going through the layers of the transformer, you start to indirectly get information from tokens that are further and further away. Sort of like a game of telephone: if I initially get information directly from my neighbors one day, and they do the same with their neighbors, then the next day when I check back in with them they’ll have information from their neighbors to share with me too. And if you do more and more layers, more and more days in our metaphor, then eventually you can get information from the whole previous context.
To accelerate this osmosis of information, a lot of models mix in layers of full and sparse attention. Like gpt-oss, the open-weights model OpenAI released earlier this year, alternates between sparse and full, so a 1:1 ratio. Gemma 3, the open-weights model series from Google, does a 5:1 ratio, 5 sparse layers for every 1 full layer.
We don’t need to talk about the third and fourth examples here, it’s just to show there are more and more exotic versions of so-called “sparse attention” mechanisms. Again though, the goal is to greatly reduce resource intensity at minimal quality cost.
I’d also like to briefly discuss proofs and why they get special treatment compared to other types of math and reasoning.
In a normal math problem, you end up with some final answer, something relatively brief that is at least somewhat straightforward to check against a known correct answer. You hope the chain of thought is correct, since users may look at it or care about its correctness, but it’s much harder to check so in practice you just check the final answer since that’s the most important thing.
In a proof however, every step needs to be correct - there is no one final answer to confirm. That’s quite a challenge to verify because one, it’s a lot of stuff to check, and two, a correct proof can take many different forms. Can’t do RLVR like you can with normal math problems.
One way people try to fix this is by writing proofs in a formal proving language like Lean. Writing a proof is like writing code, and if your code runs then your proof works. That’s nice in theory, but Lean is notoriously difficult to write, and a working program depends on all your supporting arguments and lemmas existing already. Something you can assume in your proof because someone else proved it may not exist in Lean, so you have to write their whole proof along with yours, which can spiral. So Lean works but is limited. Of course people are also trying to use AI to write Lean as shown here. Apparently this tool solved an Erdos problem, which is one of the unsolved problems posed by famed mathematician Paul Erdos.
The other thing people try is writing proofs in natural language and training some sort of AI proof verifier. I happened to see recently an example of this tactic in the wild, from this physics professor at Michigan State. Keep that note about his generator-verifier method in mind.
So the natural language approach is much more flexible of course, but it doesn’t provide the same guarantees. Still, if you believe in the increasing abilities of LLMs, you would expect this generator-verifier system to keep improving.
The Papers
So the major innovation of DeepSeek-V3.2 is the attention mechanism: DeepSeek Sparse Attention (DSA). If you understand attention generally it’s a pretty simple conceptual change, although I’m sure it took a lot of experimentation to get right.
The core idea is this: what if instead of manually setting an attention window, we learned the best attention pattern for each token? And that pattern doesn’t have to be just a certain window size, like the last n tokens - it can be just the tokens you’ve learned are likely to be the most informative. A technical person might describe this as a “learned, per-token attention mask”.
The box on the left shows a very small example of this, where “to” skips one token, and “past” and “tokens” skip two tokens. Again though, you can imagine any pattern of skipped vs attended tokens here.
Now in the real paper they always pick the top 2048 tokens to actually attend to, which means if you have less than 2048 tokens so far you’re doing full attention. But then anything after that just looks at what the attention mechanism believes will be the top 2048 tokens for the given token you’re checking against.
For completeness I will quickly acknowledge Multihead Latent Attention, which is the other efficiency measure here, but it’s more an engineering trade-off than a conceptually different approach.
And then the bit in the middle here is the structure of the overall model. If you haven’t seen this diagram before it may be confusing, so just focus on those 61 transformer blocks, the new attention mechanism, and the mixture-of-experts layer. The other stuff like the token embedding layer or RoPE or RMSNorm are standard and relatively minor details.
Anyway, these changes from V3.1 to V3.2 apparently have no negative impact on quality. They report very similar Elo scores on ChatbotArena for example. So you’re getting way cheaper tokens of the same quality, usually faster unless the context is very short.
The DeepSeek-V3.2 paper is mostly not about data, but they do have one section about it I thought would be relevant for us.
Briefly, they care about four use cases:
A code agent, like Claude Code
A search agent, like many deep research agents we’ve seen before that can take many “hops” of reasoning and searching to find answers
A general agent, for day-to-day or personal tasks like planning a travel itinerary
A code interpreter, which uses a Jupyter notebook to solve complex reasoning problems by writing code, like if a problem requires a graph or calculating a square root or doing simulations
For the code agent they mined tons of repos and pull requests from GitHub, which is standard. They were able to set up the environments for those repos, like getting the required packages installed, totally automatically.
For the search agent the environment is just search APIs I think, plus a sub-agent that verifies its answers.
For the code interpreter the environment was just the Jupyter notebook.
But for the general agent, they synthesized environments - over 1800 of them actually. And they synthesized the tasks too. Here you can see an example of a synthetic task and its synthetic environment, which is just a list of tools specific to trip planning.
So here’s how DeepSeek with that modified attention and that agent data stacks up. They group by closed vs open weights, and I’ve also updated the table with Opus results where available.
Since the focus was not quality we shouldn’t be surprised to see other models sometimes besting DeepSeek here. Do note the tool use benchmarks though, where DeepSeek is strictly better than the open-weights competition - that agent data shining through! But apparently the big labs have a better approach.
So that’s DeepSeek-V3.2. Now we turn to the math variant built on it, DeepSeekMath-V2.
As the name implies this is not the first time DeepSeek has produced a model focused on math. Back in early 2024 they released a 7B parameter model with a lot of extra math tokens in the pretraining. The real innovation there though was GRPO, which eventually became the most popular RL algorithm for RLVR, for cases where you can easily check the final answer against a known ground truth. It’s a bit of an aside so if you don’t understand that part don’t worry about it, but if you do it’s a fun little part of DeepSeekMath’s lineage.
Anyway, this second version is bigger and better but also has a different focus, namely proofs. And the innovation isn’t on the training algorithm this time, it’s on the training data.
They start with two key insights: one, the skill of verification is a stepping stone to the skill of generation; two, real mathematicians submit their proofs to many verifications, and even verifications of the verifications. So build a verifier that can also verify verifications, then use it to train a generator, then use the improved generator to make harder proofs for the verifier to train on, in a virtuous cycle. And just like with peer review, running enough different proof attempts with enough verifications on them should generally yield the truth for any problem.
The first goal of a working generator is here. Starting with DeepSeek-V3.2, train a verifier using some human data. In this case, each example contains a problem, a proof, and a score for that proof: 1 if it’s correct, 0.5 if it’s mostly correct, 0 otherwise. The verifier earns a reward if its proof score matches the human’s proof score.
In parallel, train a meta-verifier on some other human data, but this time it’s problem, proof, verification, and a score for that verification: 1 if the verification cites only real defects., 0.5 if the verification cites a few hallucinated or unreasonable defects, 0 otherwise. The meta-verifier earns a reward if its verification score matches the human’s verification score.
Once you have a working verifier and meta-verifier, you use the meta-verifier to reward the verifier, so you don’t need human verifications anymore.
Then you pull the same move again, this time to make the generator. Specifically, you take the verifier and have it generate proofs, using self-verification steps to double-check its own work and fix errors. You also take another copy of your verifier and use it for verification, as a reward signal. With enough of that training you get a good generator.
Finally, with a generator and a verifier in hand, you can kick off the virtuous cycle.
All the details are here, but really the idea is that with enough eyes, all bugs are shallow: throw enough generation and verification attempts at it and usually the truth will emerge. It doesn’t work right away, but after each cycle they checked whether the majority vote agreed with human experts, and eventually it did. So then as long as you have problems to feed in, you should get proofs out. Very Bitter Lesson of them.
Here’s what they end up with. They’re beating basically everyone at proof generation, although GPT-5.1 and Opus 4.5 aren’t on here. Deep Think is a mode of Gemini 3 so that is on here. IMO, CMO, and Putnam are all elite math competitions, where DeepSeek is also cleaning up.
For our last paper slide we actually need to come back to the first paper, for a variant of the model they call Speciale.
Here they made two changes. For one, they reduced the length penalty, and as you can see by the token counts in parentheses the Speciale responses are all much longer than the responses from the thinking mode of the regular model.
For the other, they trained on an extra dataset: the one produced for DeepSeekMath! And indeed it ends up with the same performance on the IMO 2025 benchmark. So the DeepSeekMath paper is really just a part of the DeepSeek-V3.2 paper.
My Takeaways
I like the addition of learning to the attention mask, but it seems more like an optimization than a breakthrough
Full attention is king
I would believe mixing local and global attention layers has no downside within certain ratios
If we believe The Bitter Lesson then the issue of cost or of compute scarcity is temporary anyway
Per-prompt rubrics could improve the verifier
Is there a limit to their verification system? If it’s so reliable shouldn’t it be able to produce new proofs and solve long-outstanding problems?
Math journals are in for a new wave of slop
These tricks could be adapted into data pipelines