DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
or, Necessity Is the Mother of Invention
Originally presented as a live talk on May 6, 2026
Paper · Blog post · Model weights · Demo
Background
To frame all this, we need to acquaint ourselves with DeepSeek, the lab that makes the DeepSeek models.
Or maybe we’re reacquainting, because we’ve covered them a few times before. Specifically, we’ve covered:
DeepSeek V3.2, the last major release from DeepSeek
mHC, a super technical improvement to a very specific part of the model architecture
Engram, a sort of built-in memory for common phrases
I’ve linked all of those talks in the speaker notes if you want to go back. But for those who don’t know or may have forgotten, DeepSeek is the leading scientific lab in China - not always the absolute performance leader, not cranking out models, but always pushing the envelope. In many ways they are what OpenAI set out to be: an open, research-focused organization dedicated to achieving ASI and sharing it with the world.
Unlike OpenAI though, which for a long time subsisted on what it viewed as charity, DeepSeek has always had a sugar daddy: a quant hedge fund called HighFlyer, whose founder decided to spin up this ASI effort with his profits. And that missing profit motive, that journey of discovery, comes through in a lot of what DeepSeek does today.
For one thing, they publish a lot of their innovations. Going back to my list above, mHC and Engram are techniques really only usable at labs pretraining their own models from scratch. From a competitive angle it seems crazy to give away techniques like this, but from a scientific angle it’s completely standard.
For another thing, they don’t put out many models! DeepSeek V3.2 came out on December 1st, 2025. That’s almost half a year ago, which feels like years in model release time. There were rumors they wanted to publish V4 before Chinese New Year, as some other Chinese labs actually did with their newest releases, but apparently they just weren’t ready and didn’t want to put out something not up to their standards.
DeepSeek is also one of the national AI champions in China, which puts it in a tricky position. On the one hand, the power of the Chinese government is behind it in various ways, including a new upcoming funding round I just read about today. But on the flip side, they have to start adopting Chinese chips.
Historically, all the labs in China have used NVIDIA chips basically, because for this type of workload they’ve been the only game in town. But with recent US export controls, China has lost access to cutting-edge chips, and given the importance of AI, the CCP does not want to rely on foreign companies or governments for such a critical resource.
As a result, the Chinese government doesn’t want NVIDIA chips - they’re fine with the export controls! - and instead wants homegrown chips in all the AI labs. Right now that means Huawei chips, specifically the Ascend line. DeepSeek V4 was a major test for the Ascend chips, with Huawei and DeepSeek working closely together to make training and inference work. In fact, there’s some speculation that the delayed release was primarily due to the switch in chips.
It’s okay if the information on this table mostly escapes you - the point is that Ascend chips are behind their NVIDIA counterparts mostly. And the relative underperformance of Ascend chips means DeepSeek has to focus on efficiency, not pure performance, since they’re unlikely to completely match OpenAI and Anthropic and Google.
The other hardware-related thing is at a data center level, which is utilization. Utilization is basically “how often are my GPUs actually doing something” vs waiting for instructions or data to arrive. It’s really, really hard to get high utilization; for example, xAI’s mega data center, Colossus, apparently runs at about 11% utilization. US companies have access to the right hardware and can just scale up; DeepSeek does not - at least for now - so it has to be clever.
In general, the DeepSeek approach has been to accept engineering complexity in exchange for efficiency. They are cracked engineers and they want to push the envelope.
One expression of that crazy engineering and efficiency is the price.
What we’re looking at here is pricing for a bunch of recently released models. You’ll see three types of price: input, output, and cache hit.
Input and output are straightforward, it’s just the price you pay per million tokens you pass in and receive back, respectively. And on those dimensions, DeepSeek V4 is pretty good but not exceptional; other Chinese labs with similarly capable models do about as well, like Kimi K2.6 and GLM 5.1. Of course compared to Opus 4.7 and GPT-5.5, it’s already a bargain - you can see why Anthropic and OpenAI are getting those massive valuations.
But the third bar, the cache hit, is where we really see the engineering prowess and relative altruism of DeepSeek come through. “Cache hit” tokens are tokens that haven’t changed since your previous turn, either from the system prompt or from earlier parts of the same conversation. Those tokens can live in storage, cheap storage like a SSD. And that matters a lot when you’re using agentic scaffolds with big system prompts and doing very long trajectories, rather than a bunch of shorter conversations in a chat window.
Anyway, as you can just barely see on the chart, DeepSeek V4 Pro only charges a single cent for 1M tokens in cache hit. That’s basically free! All these long histories all our agents are generating, they cost practically nothing if you’re using the DeepSeek API.
I don’t want to get into exactly how they do that, partly because it’s in the paper and not part of the background, but it just shows what kind of lab we’re dealing with here.
We also need to do an attention refresher, as attention is the area with the most innovation in DeepSeek-V4.
We’ve covered it before, but to oversimplify somewhat there are three components of a transformer model, which is the basic architecture all your favorite LLMs use.
Embedding, which turns words into numbers, since after all we are ultimately working with numbers in the form of matrix multiplications here
Attention, which calculates how each individual word or token relates to all the other tokens that came before it in order to form a complete, holistic understanding of the input and output so far
Feed-forward, or MLP as it’s noted here, which takes that complete understanding and “thinks” or “processes” it
For the attention layer in particular, you see each input splitting into a query, a key, and a value. The thing to notice here is how the queries and the keys form this matrix, where each query gets to “look at” or calculate against each key of the previous tokens. So like “The” at the start can only attend to itself, whereas “tokens” at the end can attend to all the tokens in the example. And within each row, like on each query, the circles are colored by how strong the relation is. So on the query “to” for example, it’s attending strongly to “The”, weakly to “model”, strongly to “attends”, and weakly to itself. Sometimes those relationships are a bit inscrutable to humans but in this case it generally makes sense.
The only other thing I would briefly note here is that the attention and feed-forward steps or “layers” as they’re called form the “transformer block”, and the transformer block repeats many times before the model is finally ready to predict the next token. What that attention matrix looks like in terms of the strength of relations changes from block to block.
So in the standard transformer model like we just saw, the model pays attention to everything - up to its maximum context window anyway. That would be the first box on the left, “Full n^2 attention”. And it’s called n^2 because the number of green squares goes as the square of the number of tokens. That means if you double the number of tokens in your context window, you’re quadrupling the compute and memory needs. It’s the gold standard, but like gold, it’s pricey.
People spend a lot of time trying to find cheaper attention mechanisms that are nearly as good. Often you’ll hear them called “linear”, since they tend to use compute and memory in direct proportion to the size of the input, which makes a huge difference once you get out to the 100k or 1M token range.
One of the more popular ones is Sliding Window Attention (SWA), also seen above. Here you specify a window size that is much smaller than the full context length. At first that means you’re only getting information about very nearby tokens, but as you keep going through the layers of the transformer, you start to indirectly get information from tokens that are further and further away. Sort of like a game of telephone: if I initially get information directly from my neighbors one day, and they do the same with their neighbors, then the next day when I check back in with them they’ll have information from their neighbors to share with me too. And if you do more and more layers, more and more days in our metaphor, then eventually you can get information from the whole previous context.
There are more exotic forms, like c for example. And then for d, a lot of models have different attention designs interleaved, like one layer with full attention and three layers with sliding window attention for example.
DeepSeek has gotten far more exotic than just a tweak to sliding window as we’ll see. But the key thing to remember is that attention dictates how much information you can actually pull out of the input, and that you might choose different attention mechanisms for different balances of quality and cost and context window size.
Here’s a quick comparison of full and sliding window attention next to DeepSeek Sparse Attention (DSA), which is the foundation on which the most recent model’s attention mechanisms are built.
As the captions highlight, DSA is like SWA in that your query token only attends to a certain number of prior tokens, letting it use far less compute than full attention. However, DSA learns which tokens to attend to, rather than relying on a hard rule like SWA. The component that does this learning, that picks which tokens the query is going to attend to, is called the “lightning indexer”. Keep that in mind for future slides.
The Paper
So high-level, this is what we’re working with: a Pro model with 1.6T parameters and 49B activated, and a Flash model with 284B parameters and 13B activated. I added on a couple other stats that basically explain the size difference - Pro is just bigger on every dimension.
I don’t have a good explanation for the slight mismatch in pretraining tokens, but I think it may be from Pro getting more agent trajectories.
So here is the overall architecture, for both versions of the model. I know it’s a lot, but we’ll go through it node by node. And remember, the important parts of a transformer are the embedding, the attention, and the feed-forward.
The first part at the bottom is where we pass in the input, turning it into numbers, into vectors. That’s the simplest part.
After that you’ll see a blue dotted line around most of the diagram, designating the transformer block. At heart it’s one attention layer and one feed-forward layer, with some supporting players in the mix. As I mentioned in the background slides, the transformer block gets repeated over and over, L times, after which the model is ready to make its prediction at the prediction head. There’s a layer after that about MTP, multi-token prediction, which we don’t need to get into but basically allows the model to predict a few tokens at once instead of just a single token at a time.
Now within the transformer block is the real innovation.
One innovation is mHC, which we covered in a previous paper and which I also don’t want to rehash too much here. But basically, mHC is all the grey stuff - the mixing, the row of circles that expands and condenses etc. It lets the model learn how much to actually care about what happened in the most recent attention or feed-forward layers.
The real show is the attention layer, here labeled as CSA / HCA. Those are two new forms of attention that allow DeepSeek-V4 to be so crazy efficient, and that’s the core technical stuff we’re gonna explore on the next slide. They interleave CSA and HCA, like using CSA in one transformer block and then HCA in the next one etc.
I’ll also note DeepSeekMoE as the feed-forward layer, but that’s also not new; basically all models competing for SOTA are MoEs.
Finally, for anyone who remembers the Engram paper, DeepSeek V4 does NOT use Engram. I expect it to appear in a DeepSeek release eventually, but that may not be until DeepSeek V5, since Engram has to be part of pretraining.
Now we are ready to tackle the big innovations: CSA and HCA.
We’ll start with CSA. It may help your intuition to think of a high-resolution picture you need to reduce the size of. The CSA approach is to keep the pixels in a circle around one spot, then go lower res on all the other pixels, then black most of them out for not being relevant enough. You’re getting some fairly precise but sparse global signal, and keeping the precise local signal.
Back to the model. In this diagram, each little yellow or green bar represents one token, or really one “hidden state” of a token - basically what the token has turned into after some number of transformer layers. The hidden state is like the model equivalent of electrical signals in your brain.
So we have our one query token, and we want to decide which of the KV tokens to attend to, and maybe transform the tokens a bit more first.
The first thing we’re gonna do is compress the KV tokens, in two ways: we’re gonna use smaller vectors, and we’re gonna group tokens together. You can actually see that in the diagram, where the bars are now half the height AND there are fewer of them, something like 4x fewer in this case.
The next thing we’re gonna do is use the lightning indexer to figure out which of these already-compressed tokens we want to attend to. That requires the query token and the KV tokens, and it puts out a list of the top KV tokens to care about. So now we only have three yellow bars at the top.
Finally, we do want to compensate for some of this compression by paying closer attention to nearby tokens. Specifically, we have our sliding window of 128 tokens there towards the top-left, which is gonna add in with the selected compressed KV entries.
So now from the query’s perspective, when it’s attending to all these KV entries, it has some high-fidelity ones in the 128 tokens before it, then a pre-selected group of kinda summarized tokens. And it’s gonna attend to all of them.
Now for HCA. As the name implies, it’s like CSA, but even more compressed. They’re just gonna crank up the compression factor by a lot, like from 4 in CSA to like 128 here. And because everything is so compressed, they don’t need the lightning indexer to trim the number of tokens.
Of course they still have the sliding window KV entries, so nearby information is more preserved. But generally the goal of HCA is a global view of everything. Thinking back to our image and pixels from before, HCA keeps that same radius of original pixels, then makes the rest of the image super low res.
Now if you did only one or the other of CSA and HCA, you probably wouldn’t get good results. The magic is the alternation of these two, kind of superimposing them, so you have your fuzzy global view matching up with your selective global view. Then if a selected CSA block lands on a big fuzzy HCA block, that little higher-res bit gives you a much better idea of what’s going on in that big fuzzy block.
Here’s the upshot of all that innovation. Compared to the previous DeepSeek model, the slopes on the lines for compute (left) and memory (right) are drastically lower for both versions of V4. This is despite Pro having 1.3x the activated parameters of V3.2! And it really starts to matter once you get into the 100k regime.
Speaking of V3.2, here’s where we net out when we compare base models. In case you’ve forgotten, base models have no post-training on them. It’s actually become quite rare to release base models, even when the post-trained models are open weights, so these are by far the best base models available today. Again, it’s the DeepSeek devotion to science shining through.
Anyway, V4 is generally an improvement over V3.2, even when comparing with the much smaller Flash.
Now when you look at the post-trained versions, you can finally compare to the competition that the end user will have in mind.
Before we draw any conclusions about DeepSeek V4, we should look at the distribution of benchmark scores between the other models. Opus 4.6 and GPT-5.4 are generally regarded as the best models on this list, yet they often are not the winners. So we have to take benchmarks with a grain of salt.
Anyway, it seems like DeepSeek V4 is competitive, but rarely on top. And from what I’ve seen of the vibes, that’s about right; DeepSeek has certainly not closed the gap with OpenAI or Anthropic or even Google, and they’re generally competitive with the other top Chinese models.
I wanted to show a bit about the post-training process, but the information is scant. The one interesting thing for this group is that they do use rubrics, although there’s no other information about them. They actually use the same model for generating and judging, and they claim they improve the model’s judging abilities alongside its generating abilities when they’re doing RL.
They also do a somewhat common thing now where they post-train a bunch of different versions of the model that are each experts in one area, then they distill those models down into one final model. I don’t think that would change what post-training data they want, more of a fun fact.
To complement the benchmarks, they do run one eval: a small set of white collar tasks, in Chinese, like writing reports and making nice-looking documents. They don’t give any numbers on contributors or sample size unfortunately.
In the SxS rankings, DeepSeek seems to hold its own against Claude, and on the pointwise criteria it does about the same or a bit better. Maybe the tasks being in Chinese was a major factor though, because I would be surprised if this held up at large n or in English based on what I’ve been hearing.
My Takeaways
Attention innovation still has juice to squeeze
Maybe someone will hit 2M this year
I’m more bearish on alternative attention mechanisms now (e.g. Mamba)
We’re not that far from “tokens too cheap to meter”
The Bitter Lesson strikes again
US and Chinese ecosystems are diverging
The US will probably ban the import of Ascend chips on national security grounds