Paper · Repo

Originally presented as a live talk on May 20, 2026

Background

To understand this paper, we have to understand how a model on a GPU turns your prompt into predicted tokens.

Let’s start with what we already know from our daily use of LLMs: you pass in your prompt all at once, you wait a bit, and then you start getting back tokens one by one.

Already we can start to relate to this diagram. The part where you pass in your prompt is called prefill. During prefill, you are filling up the working memory of the model, what we call the KV cache. The diagram here kinda breaks up “KV” and “cache”, but you can see that the prompt turns into KV vectors, and then those get cached, hence “KV cache”.

So now we have all the input in our working memory. That can take a long time depending on the hardware you’re using and how big your input is, but importantly, prefill happens for all input tokens in parallel. On my GPU at home, prefill runs at several hundred tokens per second.

So once prefill is done, the model is ready to start making predictions. And it makes those predictions one at a time, as the diagram shows: first “jumps”, then “over”, etc etc. This stage is called decode, because we’re decoding the mathematical representations that the model works with into words that humans work with.

Because decode runs one token at a time, it is much slower than prefill. On my GPU, decode runs at something like 30 tokens per second.

Note that when we decode a token, it gets cached too - that’s why the dotted line at the top spans prefill and decode. So the KV cache, that working memory, grows every time the model outputs a new token.

Now let’s look a level deeper, at the hardware. On a GPU, you have your chip and your memory, aka your VRAM. When you start up your model server, the server reads the model from your hard drive and puts it into your VRAM, right next to your chip.

When you actually use your model, like by sending it a prompt and getting back a response, your model server takes one layer of your model at a time from VRAM and sends it to your chip for computation. So if I’m at the very first attention layer, it’s gonna take my input and the matrices that actually make up the first attention layer and send ‘em to the chip for multiplication and so on. Then it takes that first attention layer back from the chip, along with the KV cache created from the computation, and it’s gonna send in the first feed-forward layer for computation.

That process of loading and computing and unloading happens over and over again until you’re at the final layer, where the chip can finally produce the predicted token. Then you gotta do the whole routine over again to predict the next token.

So this shuttling of weights to and from the chip is typically what slows you down - the constraint is your memory bandwidth, not the speed of your chip. If you can somehow compute multiple tokens at once in decode, like you do for prefill, then you can avoid repeating that shuttling. Sure, it costs you a tiny bit of extra time on the chip to deal with more tokens, but that’s peanuts compared to the time you saved in transit.

The question is, how do you predict multiple tokens? If predicting a token requires understanding all the tokens before it, how can you predict more than one?

The answer is in the name of the technique: “speculative decoding”. Instead of making a brand new prediction every time, you speculate about what the next few tokens will be - you take a guess beforehand and then check.

Speculative decoding works for the same reason prefill is faster than decode: inputs get processed in parallel. As we said before, once you load the weights onto the chip, it’s quick to do one or two or three or four calculations. As long as you have a good way to guess tokens, the model can check them all in parallel.

Of course, if the first token fails, then the other ones you guessed after it will likely be wrong and you’ll have to throw them away. But if your method for guessing tokens is cheap enough, and you’re not wrong too often, it can work out.

One common method is to have a version of the model itself make the guesses. Specifically, a much smaller version, ten or a hundred times smaller in fact, so it’s much faster and also can fit on the same GPU. This “draft model” as it’s called is not nearly as smart as the target model that is actually producing tokens, but it’s often smart enough. After all, most tokens are not incredibly complex or subtle; language is chock full of common and supporting words, and a lot of sentences are pretty mundane, meant to support the occasional novel or surprising sentence. It’s even more true for code, which demands predictable structure in a way natural language doesn’t.

As the graphic shows, the draft model quickly produces a few tokens, which all go into the target model in parallel rather than in serial. And thinking back to the last slide about where the bottleneck is, because you now have multiple tokens ready for computation, you save all that shuttling from VRAM to the chip on the second and third and fourth tokens.

In the example here, we have indeed generated four tokens, but the third one gets rejected, and the target model’s predicted token takes its place. The fourth draft token doesn’t get checked at all, because it depends on the third draft token being correct, which it wasn’t.

Let’s zoom in on this example a bit more. We have our four speculated tokens from the draft model, and we’re going to verify them with the target model.

Specifically, we’re going to check if the probability of the speculated token for the target model is at least as high as the probability of that token from the draft model. Like in our case, the target model thought “Brown” was 93% likely, and the draft model though it was 92% likely, and since the target model is smarter, we take the increased probability as a sign that the draft model was pointing us in the right direction. Similar story for “Fox”.

But for “Hopped”, the draft model was more confident in that token than the target model was. That’s a bad sign and means we should reject the draft model’s choice.

Incidentally, when the target model rejects the third token, it substitutes its own - in this case it’s the word “jumped”. That extra token you get from the target model when it rejects the token from the draft model is called a “bonus token”, because you get it “for free” in the process of verification. If you’re really lucky and all your speculated tokens get approved, you get a bonus token after that, directly from the target model. Like if all four tokens had been right in this example, we also would have gotten a fifth token as well, with virtually no extra effort.

Now as you might imagine, the draft model is going to be better at predicting some tokens than others. Like on a hard reasoning problem, the acceptance rate will be quite low, maybe like 25%. But on a more structured and straightforward task, like using a web search tool, it could be near 100%. So speculative decoding isn’t a complete across-the-board speedup, but for a lot of mundane LLM uses it’s helpful. It’s really an empirical question depending on your use cases, your hardware, what model you’re using, stuff like that. Like me personally, on my hardware and for my Hermes Agent, speculative decoding 4 tokens at a time has been helpful and fits on my GPUs.

I should add that there are other forms of speculative decoding, or of trying to predict multiple tokens in one go anyway. We’ll briefly see one called EAGLE-3 in the paper for example.

Another example, which we’re looking at here, is literally called “multi-token prediction” or MTP. The difference with MTP is that is has to be part of a model’s training from the get-go, it’s not an external enhancement.

You can see it right there along the top, at the boxes labeled “Cross-Entropy Loss”. The loss is the single number that tells you how well your model is doing. In a normal model, your pretraining loss is based on how likely you predicted the actual next token in the training data would be. That’s the first box along the top, it has an arrow pointing to L_Main - that’s the symbol for loss.

But here in MTP, there are multiple losses! As you continue along the top, you’ll see L_MTP^1 and L_MTP^2 - the losses for predicting the first and second of the multiple tokens. So now your loss is from the normal token, the first MTP token, and the second MTP token. And if you do training right and minimize the overall loss, you can get pretty good at predicting multiple tokens.

As with so many thing, MTP is an invention of the DeepSeek crew, and all their models since V3 have had it. More recently, Gemma 4 and Qwen3.5 got it, which has been great for local LLM users.

Lastly, one thing I want to remind folks of is what LLMs actually predict. It’s not just one token, it’s actually a distribution of tokens, each with its own likelihood.

Technically the model predicts a likelihood for every single token in its vocabulary, which is usually in the high tens or low hundreds of thousands. Of course nearly all of them will receive virtually 0% odds, so in practice you end up with something more like the distribution here, where a small number sum to almost 100% probability. The shape of the distribution, and how far it extends out, can vary a lot and will be important later on. For now though, just remember that the model usually has a few thoughts on what could or should come next.

The Paper

So now that we’re equipped with the knowledge of speculative decoding, and other techniques like it, we can finally discuss speculative speculative decoding.

For reference, we have SD on the left. Our draft model speculates a few tokens in blue, our target verifies the first one but rejects the second and third, and in the process produces a bonus token, in yellow. A line of connected dots means a sequence of tokens, so we see the end result is two new tokens. Then the cycle begins again, with the next speculated sequence in red.

You’ll notice the draft and target turns happen in serial - one waits while the other works. Surely we could make use of the draft model while the target model is working, right? Like the draft model is fast, it could produce lots of tokens in the time it takes the target model to verify.

There are two problems though. One is that the target model is using up all the memory bandwidth while it’s working. So at least on the same GPU, there is no way for the draft model to work in parallel. That’s fixable with another GPU of course.

The second, bigger problem is that the draft model depends on having all the prior tokens available. If the target model isn’t done verifying, then we don’t know all the prior tokens yet! We know the prior speculated tokens, but of course not all the speculated tokens will be right. Even worse, the bonus token could in theory be any token in the vocabulary. What are we supposed to do?

Well as you probably guessed, there is a solution, shown on the right. Let me walk through it.

First off, you’ll see there are now two parallel paths, separated by a little dotted line down the middle. That’s showing the target model on one GPU and the draft model on another GPU. So they can work simultaneously.

Now the sequence of events is this:

1. The target produces a token, shown in green

2. The draft model returns a few speculated tokens

3. The target model starts verifying. Relatively speaking, that’s gonna take a while

4. On the draft side, we start preparing contingencies, shown as this kinda branching thing with tones of red and yellow. Specifically, at each token position, we are guessing what bonus tokens the target will produce in case the next speculated token gets rejected. So even for the first token, the green one from the target, we could get the first speculated token rejected. So now we have to be prepared for whatever bonus token in yellow the target model will produce

5. Since we have our bonus tokens predicted, we might as well speculate on what comes after those. Again, that’s the chains in red. So as long at the actual bonus token is one we anticipated and speculated on, we’re ready to immediately return that new speculated sequence

6. In this example, the target model verified only the first speculated token, then it produced a bonus token, and that bonus token was one of the three we anticipated. So the draft model immediately returned a new speculated sequence, in that medium shade of red. It’s a little hard to see, but in the tree thing they bolded the line showing the sequence of events

That’s basically it! You now understand the core concept of SSD. Now we look at optimization and performance.

So the big catch on this method is that you have to guess the bonus token, or at least guess it enough of the time that your extra effort isn’t wasted.

On its face, that might seem tough, since the bonus token can in theory be any token in the vocabulary. But of course in practice, it’s not a random choice, and the whole idea of a draft model is it knows how to make reasonable guesses at what the target model will say.

As we know, LLMs actually produce a distribution of tokens, so we have a straightforward way to guess bonus tokens from the start: pick the most likely token in the distribution as the one to speculate, and save a few others for your guesses at the bonus token, in case the speculated token was wrong.

On the surface that seems fine. But because of how verification works, it’s actually a big problem if your other guesses are overconfident, which they tend to be for draft models. Intuitively, since draft models are much smaller than target models, draft model distributions tend to be smaller and thus put a lot of probability on the few tokens they think could be right. By contrast, the smarter target models have wider distributions, accounting for genuinely different thoughts but also things like richer vocab.

So one key to making this work is to fiddle with the raw distributions of the draft model to spread probabilities out more, take out that overconfidence and just hedge a bit. That makes your bonus tokens a better safety net.

The other optimization they work on is how much speculating to do, and where exactly to do it. Like if you had all the time in the world, you would pick tons of bonus tokens at every position in the speculation. But in reality we only have the time it takes for the target model to verify, so the draft model needs to make the best use of it. How exactly they figure out the optimal number of bonus tokens per position is too far in the weeds for this paper, but you can see in the graphic that different positions have different numbers.

Okay, now let’s look at performance. Our target model here is Llama 3.1 70B, and the draft model is Llama 3.2 1B.

Here they compare three methods of decoding, of producing tokens:

1. Autoregressive, which is the baseline of just one token at a time

2. Speculative decoding, which runs draft and target models in serial

3. Speculative speculative decoding, which runs draft and target models in parallel

As you’d expect, SSD wins out, at 4x the speed of AR, and almost twice the speed of SD when using vLLM, which is a very popular LLM server program. SGLang is a relative newcomer and apparently does better with SD than vLLM does, but still, SSD is about a third better.

It’s important to note that the efficacy of speculative decoding, and other techniques like MTP, vary by domain and difficulty.

Like a very easy programming problem is gonna be super predictable, because it has lots of structured language and the draft model’s guesses are gonna be pretty good, because the problem is easy and because programming is a very common use of LLMs.

By contrast, SD will probably not work that well on high-minded creative writing. The draft model is gonna have bad guesses, and there’s just not nearly as much of that stuff in the training data in the first place.

So to that end, it’s important to measure speed gains on different benchmarks. Here we have one code, two chat, and one math benchmark, respectively. As we would expect, the programming and math benchmarks show higher speed gains.

On a related note, temperature also impacts SSD performance. It’s the same intuition, that less predictable text means a lower hit rate for SSD. Higher temperature flattens out the probability distribution and make guessing harder. That’s also why people conflate high temp with creativity, because it makes unlikely tokens relatively more likely, thus creating surprises.

My Takeaways

• There is simultaneously a compute shortage and compute overhang

  • Lots of efforts to squeeze more performance out of existing hardware

• This is a “yes and” approach

  • Innovation will continue on the other constraints (e.g. memory bandwidth)

• ML infra is still in its infancy

  • People have only been serving LLMs at scale for a few years

  • Innovations in architecture (e.g. MoE) and inference (e.g. reasoning/TTC scaling) will present different avenues for optimization

• I am waiting for AI to discover something like this

  • Anyone working on RSI will hit this stuff at some point

  • Great test of creativity in the wild

Paper · Repo

Originally presented as a live talk on May 27, 2026

Background

So to really understand this paper, we need to know a lot about how LLMs predict the next token.

We spend a lot of time talking about the three main parts of the transformer:

1. Embeddings, which turn tokens into vectors

2. Attention, which forms a holistic understanding of the input

3. Feed-forward, which processes or “thinks about” that holistic understanding

What we typically neglect though is what happens at the end of the transformer, after all N transformer blocks. What is that last step that turns the output of the last feed-forward layer into a token? The way it’s shown here, it make it seem like the last feed-forward layer just outputs tokens.

In fact, that’s not the case. After all your transformer blocks, you need some way to turn that final transformer output into whatever it is your model is supposed to output. In the case of an LLM, that’s going to be language.

So we attach and train a language modeling head, or LM head for short. The end result of the LM head is not a single token, but a probability for all tokens in the model’s vocabulary, which usually contains around 100k tokens.

Now the vast majority of them are going to be zero or near-zero probability. But depending on the context, you could have kind of a long tail of possible outcomes. Of course in other cases it’s going to be pretty certain, like if the sentence says “The capital of France is”, the token “Paris” is gonna get 99.9% probability or something.

So that probability distribution is what the LLM produces. But how do we get from a probability distribution to a single, selected token?

There are some simple ways you could come up with, like taking the most likely one every time, or picking randomly based on their probabilities. And people do both.

But there’s a lot more to it than that. And how exactly you sample from this distribution can be a surprisingly big deal. So we need to take a closer look at the mechanics.

So broadly, the knobs to twiddle for sampling from your LLM’s probability distribution are called “inference parameters”. Some other settings are inference parameters too, like the maximum number of tokens per response, but a lot of them are about shaping or sampling from the distribution at each position, each time you need a new token.

The most familiar one is temperature. If you’ve heard of it, you probably heard what it impacts, like creativity or predictability. That’s true, but it will serve us better to understand the math directly.

Temperature is a way to fiddle with the distribution before you sample it. When temperature is 1, you get the unaltered distribution, shown in the middle here. 1 is actually considered a high temperature, which might be surprising given the mathematical impact is to do nothing - like you’d naively expect that to be the baseline or neutral temperature.

The lower the temperature goes, the more uneven things get. So the most likely token gets more likely, and everything else gets less likely, sometimes dramatically so. In the extreme case, T = 0, the most likely token from the original distribution is the only token in the new distribution, so you always pick it. A lot of evals run at T = 0 because it makes results reproducible. Like if you always pick the most likely token, then given the same input you should always get the same sequence of tokens as outputs.

On the flip side, raising T past 1 flattens things out. In the extreme case where T approaches infinity, all tokens are equally likely. Practically speaking you of course would not want that, but even a temp like 1.5 shown here is considered pretty high.

You can see where the conflation of temperature and creativity comes from - if creativity means making unusual or unexpected choices, raising the temperature means choices that started unlikely are now relatively more likely.

So once you have your temperature set, there are a few other ways you can alter your distribution.

One of the most common ways is to keep just the top few token choices, drop the rest, and redistribute that dropped probability equally amongst the token choices you kept. That’s called “top-k”. In this example, k=2, so we kept the top two and dropped the rest and gave their probability mass to the two we kept.

Another common choice is top-p, where you set a cumulative probability and drop everything below that. So let’s say I picked p=0.95, meaning I want to keep adding tokens to my retained set until I cross 95% total probability. As soon as I cross that threshold, I drop the remaining tokens and again redistribute the probability mass.

You can do both by the way. Like you take your original distribution, do top-k so you only keep the top 10 or whatever, redistribute probability mass, and then do top-p. Depending on the shape of your distribution, doing one, the other, or both can make a big difference.

Now once you have your final distribution, after your rebalancing with temperature and truncating with top-k and top-p, you may want to characterize it, to describe it in a few metrics.

You’re likely familiar with the basic stats properties like the mean, the median, the standard deviation and so on. Maybe you even know fancier terms like skewness or kurtosis.

But for LLMs, the key metric for these next token probability distributions is the entropy.

The term originally comes from physics, specifically thermodynamics, where it measures disorder or randomness. From that framing it seems bad, and if you already have some baggage about the term like I once did, you’ll have to put it down for the LLM context.

The term gains a positive valence in information theory, where Claude Shannon adapted it. Here, entropy correlates with the amount of information a message carries, relative to a certain context. So for example if you have a weighted coin that always lands heads, then you’re never going to be surprised when I tell you the coin came up heads. But if I have a fair coin, then you’ll always be relatively surprised, since a priori you have no reason to believe heads vs tails. So in information theory, higher entropy is better, since it means the information is more valuable.

Let’s take that information theory understanding and apply it to LLMs. If my text so far is “The coin flip landed”, most of the probability is going to be split evenly between “heads” and “tails”, with a few other small terms like “on” or “near”. If we ignore those and just give 50% odds to “heads” and “tails”, then we get .5 for p(x). If you use base 2 for the log, which is customary, you get H = 1. The unit when using base 2 is bits, so that’s 1 bit of entropy.

If we work out the math for my example in the previous charts, we see more entropy, about 1.44 bits, since it contains more possibilities, but there is a clear favorite. If all four had the same odds, H would increase to 2 bits. If we had eight options all with the same odds, H would increase to 3 bits. Entropy is rising because we’re adding information about what could realistically happen.

Conversely, if one token had 97% odds and the rest just had 1% each, H would be about 0.24 bits, way lower than before, because the distribution would barely mean anything - you’re almost guaranteed to get the same token every time.

So we can measure the entropy at each position in our response. Some positions will have low entropy, where it was really predictable and basically guaranteed what token we would get at that position. Other positions will have high entropy, where it seems the model had a genuine choice to make and considered many viable options.

Entropy is important not only for measuring, but for training. In fact, for pretraining and for SFT, the entropy is how score your model!

Let me break it down. As we saw before, your model produces a distribution for the next token - what it thinks are the realistic tokens, for the given context, based on all the training data it has seen so far. In the formula above, we’re going to call the model’s distribution q, and we’re going to call the current token position x.

In addition to our model’s distribution, which we can measure and look at, there is also the true distribution, the likelihood of every possible next token given basically all the information in the world: all language ever spoken, the complete physical state of the person speaking it, the time of day, the weather, and so on and so on. On here, we call that p.

Of course that true distribution doesn’t concretely exist, in the sense that it can never be found and calculated. It’s more like a Platonic ideal.

But what we can do is treat any piece of training data as a sample from that true distribution. Like if I write an essay, that’s some real text from some combination of factors from the true distribution. So when we train on that, the model gets one particular view of the true distribution. And if we get lots of different samples, from all sorts of people and places and circumstances, we can get more and more samples from the true distribution. We’ll never fully build it, but we can keep getting closer with more data.

So when we model the true distribution, what we’re really doing is stringing all these individual discrete samples together into one smooth curve, which hopefully is close to the true distribution.

The difference between the samples of the true distribution p and our model q is the loss. And the loss has two parts: the entropy of the true distribution p, since it is changing all the time; and the distance between p and q. Those parts together form the loss. We have no influence on p of course, so loss is never zero, but we can keep trying to shrink the gap between p and q.

The Paper

This is the entire method for the paper. Hopefully some of the math looks a bit familiar, but I’ll break it down into English.

First, they take a model and use it to produce one response per prompt, with certain values for temperature, top-k, and top-p.

Then, they use those prompt-response pairs to do SFT.

Then they test the model, using potentially different values for temperature, top-k, and top-p.

That’s it! They don’t check the responses at all, there is no QA. It’s just raw synthetic data from the same model they train, with certain temperature and top-k and top-p.

Incredibly, this works. We’re going to come back to this table of results once we understand more about why their method works, but for now I wanted to flash the results.

So the core insight of the paper in my view is this: some parts of a response are very predictable, while others are very uncertain.

The most illustrative domain for their insight is code. A lot of code is pretty standard, pretty formulaic, because code has lots of hard and fast rules. Like in Python here, when you define a function, you have to write “def” and then a function name with parenthesis around the arguments and then a colon. If it doesn’t look like that, you’re going to throw an error. There are lots of rules in natural language too, but almost none of them are hard and fast; natural language is famously flexible, it has artistic license and can even contain mistakes like misspellings without destroying the meaning.

Of course, since code is just a way to solve problems and one problem can have many solutions, some code is quite flexible. As the authors show here, there are many different ways to sort a list of things, and which one to pick can be a subtle matter, or may be only a matter of taste depending on the circumstances.

The authors coin two terms for the ends of this spectrum: “fork” for the uncertain cases, and “lock” for the certain ones.

As the charts illustrate, forks and locks have different needs. At a fork, the model needs to explore, to have lots of roughly equal next token probabilities so that it can learn better and so that minor changes to upstream context can tip it towards a different path easily. So there we would like a higher temperature.

By contrast, at a lock, the model needs to be sure. Locks have one right answer, and all other tokens in the distribution are distractors. At locks, we want low temperature.

The problem is that we can’t adjust temperature on the fly like this - it’s a parameter we provide once for the entire inference, for the entire response. In practice we end up compromising, but it’s not optimal.

So you might think we could just make a better rule, right? Like if the distribution looks like this, then change the temperature like that, maybe ignore the tail as defined by a certain probability, things like that. But that’s not very ML of us, right? That’s not very Bitter Lesson, putting human-engineered rules on a thing that learns. Instead, what if we could teach the model to change its distributions? After all, producing the right distribution is already what we train models to do!

What if there was some way to train models to dynamically adjust temperature, top-k, and top-p, so that we get distributions more like this - flatter at forks, more peaked at locks, and always ignoring distractors. Or if it’s not literally learning different T, k, and p values, at least learning to change distributions as if it did learn different T, k, and p.

In the interest of reducing variables, the researchers try out a much simpler model: a finite state machine (FSM). A FSM is not a transformer. It’s barely even a model. Very concretely, each node on here is a set of sixteen numbers, each representing the probability of going in one of sixteen possible directions. They call the directions “tokens” to match LLM terminology, which is why you see “tok” and then a number on all the arrows here.

The idea here is they can construct trajectories to only be made of forks and locks. Like starting from the root, tokens 0 and 1 are both viable, so there is a genuine choice to make. But then after the choice, from either Fork-A or Fork-B, there is only one right choice to make - there are three locks in a row.

The way the model operates is by selecting from its “tokens” in the same way an LLM selects from the tokens in its vocabulary: take the raw probabilities, adjust with temperature and top-k and top-p, then pick based on these final probabilities. So you’re isolating just the impacts of inference parameters and training from all the other LLM stuff, and you have this fake, tightly controlled scenario.

Importantly, you can train this FSM using cross-entropy loss on sequences of tokens, the same way you would train an LLM.

So what is their crazy method, what information exactly are they gonna train their toy model on?

You might think they can use the PASS or FAIL information. That would be perfectly reasonable, given we typically do SFT on good examples, or if you’re still in the RL mindset where we’re thinking in rewards. But that wouldn’t be teaching us anything new!

Instead, they’re just going to self-distill, to take a bunch of trajectories, whether PASS or FAIL, and train on them. The key variables to adjust are the inference parameters: temperature, top-k, and top-p.

What they show is that if you train on trajectories with the right inference parameters, regardless of the quality of the data, you can teach the model to shape its probabilities better.

This is what they actually observe, not just an illustrative example. For the lock nodes, the model learns to drop distractors and put more weight on the dominant token. For the fork node, the model learns to spread out more amongst the options left after truncation.

I want to give a quick intuition on why this should work before we get to the empirical results.

Let’s start with self-distillation at T=1, with no truncation. All we’re doing then is getting unaltered samples from the model and reinforcing those. But since we’re not changing the samples at all, or filtering them like we would if quality were a concern, then training on them should produce no change.

However, with SSD, we specifically avoid T=1, and we do use truncation, i.e. top-k and top-p. So now the synthetic samples are different from what the model normally would say, so there is something to learn from.

So really SSD is just reinforcing and sharpening instincts the model already has. If the distribution looks like a lock, truncating puts even more probability on the top token, and the imbalance is already large enough that temperature shouldn’t change much. If the distribution looks like a fork, temperature spreads out probability more, but truncation ensures you don’t explore too far down the tail.

It’s not gonna help if the model is trash in the first place. And it does bank on there being some untapped potential within the model to then tap. But on the flip side, it’s embarrassingly simple and practically free compared to producing high-quality training data.

Now let’s take a closer look at the empirical results.

For all models they tested in basically every segment on the two version of LiveCodeBench, SSD helps. If you’re not familiar with pass@1 and pass@5, it just means how many chances a model had to get the problem right. Pass@1 measures accuracy in the way an end user would expect, like of course we want the one response we get to be correct. Pass@5 is more a reflection of model potential, like is it able to get the right answer.

Two trends to call out. First, within the instruct models, the smarter models benefited more. The authors don’t explain why, but I think it boils down to the whole “self-distillation” thing. Like if we think SSD’s function is to bring out the potential of the model, then smarter models likely have more potential. It’s actually pretty similar to how RLVR works, if you believe in the elicitation hypothesis, which I do. I wish they had actually done something similar here, where they measured pass@256 or some other really high number in order to see whether SSD was doing the same sort of internal optimization. We know it can’t be learning from the data in the normal way, given how this SFT data is created and how poor the quality can be, at least by traditional measures.

Second, the models generally improved more on the harder problems. Again, the authors don’t provide a direct explanation, but if we think back to the forks vs locks thing, we’d expect harder problems to have more forks. Like on an easy problem, not everything is a lock of course, but the distribution of positions is gonna be more on the lock side. I imagine for harder problems there will be way more forks, so if SSD helps with forks and locks, you can see why it might help with harder problems more.

Of course this doesn’t scale indefinitely; at some point, a problem is just impossibly hard for a model, and no amount of training is gonna help. So that might be starting to bite on the hard problems for Qwen3-4B for example.

To drive home the point about how the training data isn’t operating in the conventional way, they really dial up the temperature on some of it. As you can see on the left, a temperature of 2 can lead to complete gibberish, let alone unrunnable code.

And yet, if you train on this data, where 62% of outputs contain no extractable code at all, you still improve.

To give you an idea of how much better SSD is compared to just picking the optimal temperature, they do a sweep over temperature values on different benchmarks and models. In every case, the best temperature for the untrained model is worse than the performance of the trained model.

I also think this graph is helpful for providing an idea of temperature’s impact. 0.5 to 1.4 is a pretty big range, but for most cases the difference isn’t very big, and the big differences themselves fall in unpredictable patterns. Like why is there a seven-point drop from 0.9 to 1 for the bottom-right graph? Very strange.

My Takeaways

• This is the elicitation hypothesis but with trash data

  • How well it works depends on how much potential the model has left to bring out

• Surely there is a cap to this work

• The result is more instructive than useful

• What are we losing?

  • Leans more into priors

  • So the model may be worse past a certain difficulty - just sharpens the boundary