Reinforcement Learning via Self-Distillation

or, The Student Becomes the Master

Originally presented as a live talk on March 11, 2026

Paper · Website

Group Relative Policy Optimization

+

Reinforcement Learning with Verifiable Rewards

Okay, the first thing I want to do is revisit two foundations of modern RL: group relative policy optimization, or GRPO; and reinforcement learning with verifiable rewards, or RLVR.

GRPO is an algorithm that turns rewards into model weight changes. RLVR is how we get those rewards from responses.

Chronologically, GRPO came first, so we’ll talk about it first. As with many other LLM engineering breakthroughs, GRPO is the work of the DeepSeek team. If you’ve been to some of the recent paper reviews, you know what I’m talking about, but if not I’ll give a quick recap: DeepSeek is one of the top two labs in China, the other one being Qwen, and they have a very R&D-focused culture. They don’t crank out models, but the ones they do put out are always respected, and they often contain new techniques that long outlive the models themselves.

A perfect example is in this paper, nominally about a math-focused LLM called DeepSeekMath, but really a vehicle for this algorithm, GRPO. Like other RL algorithms such as PPO, it’s a method for extracting signals from model outputs, basically turning them into values we can then use to adjust the weights of our model. In PPO, we have to use a special model to estimate that signal, which is complicated and sometimes messy or fuzzy. With GRPO, there is no extra model to train, as you can see by the decrease in yellow boxes in the right-hand graphic. That makes GRPO simpler and often more precise than PPO.

Looking at the GRPO graphic a little more closely, you can see there’s a reward model that turns outputs o into rewards r, that then turn into advantages A, which is ultimately what we need to adjust the weights of our model. But as we’ll see on the next slide, we actually don’t need a reward model per se - we need a system to turn outputs into rewards. The rest of GRPO is just math, completely deterministic.

The other prime example of DeepSeek magic is this paper, nominally about DeepSeek-R1, the model that reproduced the reasoning behavior of o1 and o3 from OpenAI and briefly crashed the stock market.

In this paper, they used GRPO to train their reasoning model, but they didn’t require a reward model. Instead, they calculated rewards deterministically by checking the model’s final answer against a known good answer. So like for a math problem, if the known good answer is sqrt(x), we can easily check if the model’s final answer is also sqrt(x). We may have to check for equivalent forms, like x^½, but there’s no real interpretation or judgment like with a reward model. We know that technique as RLVR.

So just like you don’t need RLVR for GRPO, you also don’t need GRPO for RLVR. Like you could do RLVR instead of having the reward model in PPO. But in practice, these two go well together, because they are basically the simplest possible setup for RL.

Anyway, the DeepSeek-R1 paper found two effects of GRPO + RLVR: you can greatly improve reasoning, and you generally end up using way more tokens. This is exactly what o1 and o3 did. Sometimes people call it “test time compute scaling”, meaning you use more compute when generating your response, and that extra compute is supposed to improve quality.

This combination of GRPO + RLVR is everywhere nowadays. Any prompt where there’s some verifiable final response, like a math problem or some instruction following, is likely trained on with GRPO + RLVR. And rubric training is basically just stacked-up RLVR training, like instead of one final answer you can verify against, you have multiple criteria you can verify against. It’s the same core idea of a method for giving a response a score, which can then go into GRPO to turn into signal on how to adjust the weights.

RLVR has two shortcomings though. For one, you might have incorrect steps but still get the final answer right. With RLVR, you’re going to get full marks, even though the response overall is not correct. For the other, you don’t know why or how you got something wrong. Or if you do, it doesn’t factor into the score, into the training signal. So you’re rewarding at a response level even though parts of the response may be wrong, and you’re leaving a lot of valuable information on the table by just calling a wrong final answer “incorrect. Keep those shortcomings in mind.

Another, more recent finding of RLVR is that it may not be teaching anything. Instead, what some researchers have found is that RLVR merely brings forth solutions paths that are already present in the model, although they may be highly unlikely ones. This is called the elicitation hypothesis.

The image on the left shows what I’m talking about. For Problem A, the base model has all these possible solution paths for a given problem, but only a couple of them are correct, so when you run the model the chances of getting a correct response are low. What RLVR does is make the correct paths much more likely, so that you as an end user are generally getting a correct answer now.

Of course it may be that those other solution paths are involved in other problems, like Problem B here. To oversimplify somewhat, if through RLVR we greatly reduce the likelihood of exploring the left side of this tree, then we can’t reach the correct answer for that problem, even though it actually was there in the base model.

The chart on the right is generally how people show the elicitation hypothesis is true, that any correct solution a model can give is in there from the beginning and not taught wholly new via RLVR. On the x axis is the number of chances you give a model per question to get a set of questions right. As long as any of the k responses is correct, the model gets credit for that question. So that metric is called pass@k.

For end users, we really only care about pass@1, since a person using a chatbot expects the right answer the first time. And you can see there, on the left side of the chart, doing GRPO + RLVR really helps - you’re getting 4x higher accuracy for the yellow line compared to the black line.

However, if you keep giving chances, up to 256 in this case, you see all the lines converge. So it seems that the base model knew the answers all along, it just didn’t know how to produce them efficiently.

Again, elicitation is still good for end users. It just shows RLVR doesn’t teach new things.

Okay, now we need to talk about distillation.

Very broadly, distillation is a family of techniques for transferring some quality or information from one model, the “teacher”, to another, the “student”. Usually the teacher is a more advanced model than the student, which by the way puts a ceiling on how far distillation can take you.

So the naive way to do this is to make training data synthetically, using the teacher model. Like imagine we made the same kinds of post-training data — SFT, RLHF, rubrics — but with GPT and Claude instead of humans.

If all you have access to from the teacher model is the tokens it outputs, that’s the best you can do. That’s true of the Claude API, so if you’re trying to distill off Claude, this is the way to go.

If you have access to the per-token probabilities, sometimes called the logprobs, you can do better. In this setup, you pass the same input to the teacher and the student. Then for each token you need to predict, you compare the probability of each possible token in the student’s distribution with the probability of the same token in the teacher’s distribution. The differences between the probabilities are your loss. So your goal is to teach not only the most likely token, but also how likely all the other tokens are too.

That’s a much richer and more nuanced understanding the teacher is providing to the student, and it’s going to be more effective at passing on the teacher’s wisdom.

Let me draw a human parallel. Let’s say you’re learning to play chess from a master. You’re both playing black, and some opponent is playing white. The prompt is white’s first move. Now you and the master each take the list of all possible next moves and assign each one a probability of being optimal. Then you compare probabilities, then you make the master’s top-rated move.

By doing that over and over, you’re going to get an idea of what set of moves the master realistically considered, what moves he kept in mind but thought were unorthodox, what moves he knew were trash. That’s a lot more wisdom to pass on than just watching him play the opponent!

Of course you need that level of access in the first place. OpenAI actually does offer logprobs through their APIs, which is partly why they’re such a popular target for distillation. And of course any open-weights model includes logprobs too.

So the teacher’s logprobs depend on its innate knowledge of course, but they also depend on the context, the input the teacher has received. In one sense this is obvious - the next predicted token clearly depends on the prompt you’ve given the model - but for other forms of context it’s more subtle.

One easy example is the chunks you provide in RAG. Even if the core answer is the same with and without the extra context RAG provides, you could get some additional details, or smaller effects like a change in tone or vocabulary. That’s all going to show up in the logprobs, the different distributions.

Even more subtly, you could provide different instructions or system prompts to the teacher. That’s what some researchers did here, in a paper called “Learning by Distilling Context”. This paper came out in September 2022, two months before ChatGPT, which explains why they’re focusing on relatively simple queries here.

Anyway, you can see in this example that the teacher receives the schema and four examples before receiving the question. Those four examples help it get the right answer. So then when you do soft-label distillation, even though the student only gets the schema in its context, through distillation it also gets the wisdom of the four examples the teacher got.

This is a very simple example, but in theory you could fill the teacher’s context window for distillation. You won’t be transmitting 100k tokens of information or whatever, but you will be transmitting some compressed version of it, compressed by the teacher model’s intelligence into this low-dimensional space. And that intelligent compression is what makes soft-label distillation generally more effective than just making synthetic data from the teacher.

I think of soft-label distillation as a way to teach process knowledge, stuff that is hard or at least inefficient to just write down as rules. Like yes, you could try to write out the mechanics of SQL and your particular business logic in the student model’s system prompt, but when you use distillation, you’re taking advantage of the teacher’s compressed wisdom and having it guide the student directly. Again drawing a human parallel, I think of it like a golf teacher standing behind you and putting his hands over yours, guiding you through the entire motion of driving or putting. Sure, he could try to explain how to do it, but the hands-on feedback and guidance as you complete a swing is going to be so much richer.

There’s a decent body of work out there about context distillation. I’ve heard Anthropic uses context distillation to take very long prompts about what Claude’s character should be and distill it into the weights. I also read a paper late last year about distilling lots of documents into a kv cache that you can then pop into a model, to basically give it memories of those documents when you then want to ask about related matters. So this general idea of using a model to effectively and intelligently compress context down into the densest possible signal is pretty robust.

Self-Distillation Policy Optimization

+

Reinforcement Learning with Rich Feedback

So this paper coins two new terms: self-distillation policy optimization, or “SDPO”; and reinforcement learning with rich feedback, or “RLRF”. Those are the parallels for GRPO and RLVR, which we discussed earlier.

The key insight for RLRF is that RLVR feedback is incomplete. Yes, if you’re only going to provide one bit of feedback, it’s going to be whether the solution was right or wrong. But there’s no reason we have to limit ourselves! We can provide more, richer feedback.

RLRF says, let’s use that feedback. Maybe we have runtime errors when our program fails. Maybe we have written feedback from an LLM judge. Maybe this is real user data and they yelled at the agent about why its actions were wrong. That’s all crucial stuff! It’s important to know how you were wrong, not just if you were wrong. And there are way more ways to be wrong than right. So just returning a reward of zero leaves something on the table.

Of course some environments are more feedback-y than others. They pick coding here because you do get those runtime errors and so on. But this is not a code-specific technique - anything you can put into tokens can be feedback here.

This here is where the feedback goes. This is the prompt for the teacher, which we bolster with as much context as possible so that it can guide and distill into the student. Remember, the goal of the teacher is to make adjustments to probabilities at each token, so we want the teacher to always be right.

So we start with our prompt of course, whatever problem we’re trying to solve.

Next, if we have one, we can provide an example of a correct solution from a previous rollout. That’s definitely going to help the teacher guide the student if it exists. If not, we skip this part.

After that is our feedback, from the compiler or the LLM judge or whatever. We should basically always have that, except for on the first run, but again if it doesn’t exist we skip it.

Then we provide the student’s original response. And you might be surprised that we’re providing an assistant response. Like, isn’t the model supposed to provide a new response?

But they’re using the teacher model differently. They’re not using the teacher to give its own answer, they’re using it to give its own probabilities at each token of the student’s response. You never get a complete response from the teacher. We’ll see more about that on the next slide.

So that’s what they explain in the paper, but I want to call out two things. One, you may be wondering why we’re doing any more training if the student was able to produce a correct solution. Well as we saw earlier with those pass@k graphs of increasingly high k, there’s a difference between “can produce a right answer” and “probably will produce a right answer”, and that difference may be enormous. We want to shrink that gap so that end users get better pass@1 results.

Two, why can’t we use someone else’s correct solution? The theoretical answer is that it will change the teacher’s guidance to bias toward the style of the correct solution as well as the contents of the correct solution. So if the model naturally is verbose, but our correct solution is concise, then the teacher will punish the student for being verbose, even though that doesn’t impact correctness. We want a clean signal from the teacher on correctness, which can only happen if the correct solution is from the same model we’re using for the teacher and the student.

However, practically they don’t test it. You could give them a pass and say it’s out of scope, but I think it would have been nice to try having another model family write the solutions and see what the penalty really is. If it’s small, it could be worth having humans write solutions to really hard problems and then having a SOTA model do this self-distillation thing. But then again, if we really believe in the elicitation hypothesis, you can’t teach new things with GRPO + RLVR or related methods. Still, I wish they had done the test.

Okay, so here’s how it looks when the teacher provides feedback to the student. In this example we have a question, then an answer from the student, then some text feedback. We’re gonna take all that and give it to the teacher.

The teacher is going to go token by token through the student’s answer, that bit in step 2, and compare its likelihood of that token vs the student’s likelihood of that token.

So in this example, for most of the program, the teacher and the student gave similar probabilities for the tokens: for the three tick marks, for the word “python”, for the function name etc.

But then you get to this plus symbol and you see the teacher disagree. Specifically, the teacher gives a very low probability for “+”, but the student gives a relatively high probability. Since the teacher knows the feedback and the student didn’t, the teacher is probably right, and we penalize the student.

Here’s a closer look at that same example. At the top they have the baseline approach of GRPO, which assigns equal credit to all tokens in a response. Red means less likely here, so because the initial response was wrong, GRPO gives low credit to all the tokens.

By contrast, SDPO assigns credit per token. We can see that for each token the student actually picked, in the first row, we also have alternative tokens it considered. So like in the first position, we have parenthesis + “range” as the selected token. But the student also considered a leading space and then “range”, “range” and a space afterwards, and a dot then “range”. The teacher agreed with the token the student chose, so the selected token has a white background.

Now if we look at the blue tokens, where the teacher assigned more credit, we see two things. One, in the second position, the student’s second pick gets high credit from the teacher, so it seems like the student was close to making the right decision.

Two, we see that plus in the sixth position is really the main problem, and that removing it and the “1” after was the right choice. The most blue token, with two close parentheses and then “\n”, is actually the final token that the student actually selected.

So again, the teacher is not generating a correct response we compare to; instead, the teacher is evaluating the student’s choice of the immediate next token given all the previous tokens chosen.

So that loss, that mismatch between teacher and student probabilities, is signal for SDPO. SDPO is the algorithm that turns the signal from the mismatch into changes of the model.

And because the teacher and the student are the same model, you can keep doing this over and over until performance saturates. So let’s say on the first round you did eight rollouts and got none right. But you got feedback, so you give that to the teacher and measure that mismatch between teacher and student probabilities. You have your loss, which you plug into SDPO, you train your model with that feedback, then you try again. Now two of your eight rollouts are correct. But the rest are still wrong. So now you take the feedback and the correct solution, give it to the teacher, calculate loss etc. Ideally you end up with all eight rollouts right, but maybe you top out at seven.

With standard GRPO + RLVR you just have to keep trying and hoping that the answer is in there somewhere, or that changes to the weights due to good signal on one problems helps you on this other problem.

With SDPO + RLRF, by contrast, you’re getting this extra feedback that can nudge you closer to a good solution. So it’s at least more likely that if you keep training like this, you’ll get there.

Here’s their first test of SDPO vs GRPO, with the Olmo3-7B model, working on a chemistry benchmark.

A few things to note. First, of course the accuracy improved, and improved more than with GRPO. They probably wouldn’t have published this paper otherwise.

Second, the x axis is in wall time, how long training took. That’s not usually a metric we see in these papers, they’re usually measuring performance against training steps or perhaps amount of data. Frankly I think it’s a little bit suspicious, like the accuracy improvements don’t stand on their own so they have to tout some other benefit.

Finally, and perhaps most underratedly, the response length went down rather than up. It went way down actually, 11x compared to GRPO, which as we saw incentivizes more tokens. “Longer thinking” as some would say, or “test-time compute scaling” - the idea that more computation to produce more tokens means more accurate answers.

But if you actually look at the thinking, it’s often repetitive or looping, and sometimes contradictory. When DeepSeek R1 came out people made a lot of observations about GRPO + RLVR getting models to say things like “Wait” or “Let’s try again” or “On the other hand”. Of course it’s good to confirm your answer from a different perspective, fox-like thinking as Isaiah Berlin would say, but doing the same calculations three times or ping-ponging between the same two points repeatedly isn’t productive.

So perhaps SDPO + RLRF is a more direct way to bring forth these reasoning paths than GRPO + RLVR.

Now here’s an experiment where they narrow their training set way down, just to hard problems, to see if that rich feedback really can guide you to the right solution.

One is exactly what they’re measuring. They’re taking a couple sets of hard tasks, where the initial model gets the answer right almost never for “very hard”, or where it gets the answer right less than half the time for just “hard”. That’s only 9 and 19 tasks, respectively, which is why the error bars are so big. Anyway, what they’re measuring is basically pass@k for increasingly large k, but with a twist: for SDPO, since they’re training after each round of a certain number of rollouts, what the x axis really measures is how many rounds of SDPO it takes to get a solution. That’s why they call the y axis “discovery@k” instead.

Two is what they’re comparing. We know SDPO of course, and best-of-k just means we take k samples all at once and count it if at least one of the k samples is correct. The other trendline, “multi-turn”, is like a hacky version of SDPO: instead of training after each round, they just keep adding feedback to the prompt. So each incorrect attempt and its feedback get added to the prompt, which could be like hundreds of examples. They do have to cap the number of examples due to context limits, but that’s still a ton of information they’re providing. Also they run a variant where they kept the attempt and the feedback, not just the feedback, and that did much worse. So that’s a practical tip for your prompting: try to boil down the lessons of failed examples if you can.

And three of course is the results! SDPO outperforms the best-of-k baseline, and also outperforms multi-turn. It’s really not surprising that sequentially trying and then updating outperforms just making more and more parallel attempts, but I was surprised how bad multi-turn was. To me that really highlights how much more effective feedback becomes when you train on it. Similar to how in theory you can carefully follow documentation to get the same outcome as someone with lots of experience, but in practice there’s something about learning that makes the knowledge fit into your own sort of mental schemas in a way that reading external text misses.

One other piece to remember about the results is that from an end user perspective, the model at the end of SDPO is going to be much more useful, because it’s much more likely to get you the correct result on the first try. Pass@k for a huge k is important scientifically, but it’s irrelevant for end users.

So one complaint you might raise is that SDPO + RLRF requires additional feedback compared to GRPO + RLVR. Remember, the “RF” in “RLRF” stands for “rich feedback”.

To address that, they separate out the RLRF part and just test SDPO vs GRPO. So now we don’t have to make any changes to our training data or our environments, just to the training process. Remember, GRPO and SDPO both depend on there being at least some correct rollouts to get some signal, they just use that signal differently. In the case of no correct rollouts, both algorithms do nothing.

The story here is again not a clean victory for SDPO. It wins most of the time, sometimes by a lot, but other times it is basically tied or even loses by a few points. Also keep in mind the scores are average accuracy across 16 attempts. I would have liked to see pass@1 to reflect the user experience.

Another way you might compare SDPO to GRPO is by making the binary more of a spectrum, like a smoother transition from GRPO to SDPO.

The authors do that here when they make the SDPO feedback increasingly granular. Recall that GRPO is feedback over the entire response, what researchers often call the “sequence”. So sequence-level SDPO is basically GRPO but with feedback given to the teacher prompt, which apparently helps, given the separation of the grey and blue lines.

The next step in granularity is token-level, just a binary about whether the teacher thought the student’s token was the best choice or not.

And then the final step is what we saw earlier, the logits, which are basically the probabilities of any possible token for the given position. That’s the full SDPO. We called that “soft-label distillation” in the background slides.

So intuitively you might think bigger models are gonna be better self-teachers, because they can take feedback into account better. Like anyone who has used AI consistently over the years knows models used to be terrible at taking feedback in conversation, missing out on subtleties or getting confused etc.

Turns out that does hold up in SDPO, at least for the range of sizes they looked at. SDPO is no better than GRPO at 0.6B, but consistently widens the gap up to 8B. Unfortunately they didn’t test the 32B model, which would have really made for a nicer story, but my guess is it does keep working at least up to a certain point. One wonders what the big labs with 1T parameter models can achieve.

My Takeaways

• RLRF could be a good use of human qualitative feedback

  • RLHF already uses human quantitative feedback (the Likert scale)

  • Justifications from existing RLHF data could become RLRF data

• Seems like a straightforward replacement for GRPO

  • There was some Twitter conversation about big labs using a better version of GRPO, perhaps this is it

  • There is a sense we are “due” for a new technical breakthrough (although maybe that was Engram)

• This all depends on the raw intelligence being in there from pretraining

  • Elicitation hypothesis

  • Does that mean we need to focus more on pretraining to get the most out of post-training?

• RSI is coming

  • Anything you can verify can self-improve

  • You don’t need a smarter teacher, just a student with good feedback looking over their old results