Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
or, Fox Math
Originally presented as a live talk on March 18, 2026
Background
The framing I want to use this week comes from Isaiah Berlin, a 20th century philosopher and historian of ideas. He wrote many works and essays, but perhaps his most popular and enduring is this one, The Hedgehog and the Fox.
The title is a reference to a quote from an ancient Greek poet: “a fox knows many things, but a hedgehog knows one big thing.”
Berlin spends the essay expanding on that idea, classifying thinkers into one camp or the other. So for example he puts Plato, Pascal, and Proust in the hedgehog camp, while Aristotle and Shakespeare go in the fox camp. Heavy hitters on both sides, to be sure.
In more modern times, a lot of forecasters have taken the fox’s side, asserting that fox-like thinking produces better predictions than hedgehog-like thinking. So for example Philip Tetlock - famous for his work on superforecasters, people who are consistently good at predicting global and political events - has said foxes outdo hedgehogs here. Similarly, Nate Silver’s old forecasting team, FiveThirtyEight, adopted the fox as their logo in allusion to this distinction.
Two things make this framing relevant to LLMs. The first is that they are very widely read. In fact, they have read basically the whole internet, minus the absolute cruft of course. So they contain many and diverse ideas by nature.
The second is that an LLM’s job literally is prediction. We call the act of using an LLM “inference”, but really the job of the model is to predict with various likelihoods what the right next token is. So we seem to be in very foxy territory here.
Switching to more technical matters, we need to discuss entropy.
The term originally comes from physics, specifically thermodynamics, where it measures disorder or randomness. From that framing it seems bad, and if you already have some baggage about the term like I once did, you’ll have to put it down for the LLM context.
The term gains a positive valence in information theory, where Claude Shannon adapted it. Here, entropy correlates with the amount of information a message carries, relative to a certain context. So for example if you have a weighted coin that always lands heads, then you’re never going to be surprised when I tell you the coin came up heads. But if I have a fair coin, then you’ll always be relatively surprised, since a priori you have no reason to believe heads vs tails. So in information theory, higher entropy is better, since it means the information is more valuable.
Let’s take that information theory understanding and apply it to LLMs. If my text so far is “The United States of”, you’re basically guaranteed to see “America” next. So our LLM is going to predict the token “America” with almost 100% probability, with basically every other token at 0%. If you use the formula here, then H for this position, this next token, is going to be very close to zero; p(x) is basically 1, log(p(x)) is basically 0, and there’s only one term in the sum that matters.
We can go back to my fair coin flip example on the other end. If my text so far is “The coin flip landed”, most of the probability is going to be split evenly between “heads” and “tails”, with a few other small terms like “on” or “near”. If we ignore those and just give 50% odds to “heads” and “tails”, then we get .5 for p(x). If you use base 2 for the log, which is customary, you get H = 1. The unit when using base 2 is bits, so that’s 1 bit of entropy.
H can go much higher than that of course. Like for a fair six-sided die, H is about 2.6 bits. The more events you plausibly could have, the higher H is going to be.
Here’s a related paper from last year that I really enjoyed. It’s from the Qwen team, and it looks at the relationship between RLVR and entropy.
If we think back to the elicitation hypothesis from last week, which says that RLVR just optimizes and brings forth good solutions rather than teaching new solutions, that should have implications for entropy. Like if RLVR is improving the odds of getting a good solution out of your model, it must be changing the distribution of token likelihoods at each step, probably making some of them much more likely at the expense of all the others. That should reduce entropy.
But as we established earlier, entropy is actually a good thing, something we want to preserve. The framing you sometimes hear is exploration vs exploitation - you need to be able to explore different areas of solution space as well as exploit the most promising areas. Higher entropy means more exploration. There’s such a thing as too high of course, but the concern with RLVR is getting entropy too low, basically cutting off all the exploring.
You can actually see this difference in the word clouds they assembled on the right. The high-entropy tokens are at the start of ideas, often new ones or representing changes in direction. The low-entropy ones, by contrast, are often part of calculation or totally deterministic, like a closing parenthesis that you know has to be there because you had an open parenthesis earlier.
The finding in this paper is that if you focus on the small share of high-entropy tokens for training and updates, you end up better off than if you made updates based on all the tokens. That’s not as relevant for us, but it does show there is value in preserving a certain level of entropy.
The Paper
So now that we’re up to speed on the value of diversity and entropy, we can dive in.
The basic idea of this paper is simple: your model stays more fox-like if you collect and emphasize less common solutions during training.
They give an example here and diagram it out. Let’s say you have to compute this series, summing the cubes of the integers from 1 to 100. This is RLVR, so we have a golden final answer, in this case it’s 25,502,500. What we want to train on, to reward, is the model getting that right final answer.
So we take our model, we have it generate a bunch of solutions, then we group those solutions together, using a more powerful LLM. That lets you see past mere changes in wording or variable names etc and just focus on the different strategies represented.
Then you assign advantages to each group. Here an advantage is a reward times a weight. The reward is going to be either 1 or 0 depending on correctness, classic RLVR. The weight, however, will depend inversely on the number of solutions in the group.
So as the diagram shows here, any group of wrong solutions is going to get zero advantage. A group of correct and common solutions is going to get a positive advantage, but on the low side. A group of correct and rare solutions will receive a high advantage, with the highest of all going to any “group” that only has one example.
That’s pretty much it! All you need for GRPO, for the algorithm that adjusts the model’s weights, is these advantages.
Here are a few examples of solution diversity. The have three problems, each with a varying number of solutions, plotted as shapes.
Now to measure coverage, they happened to have a substantial number of human examples to cluster, from web and textbook sources. They imply that having many human solutions means they likely covered all the possible strategies, but they weren’t rigorous about it imo.
Anyway, the result they’re showing here is how the initial model expands its coverage. So like in the middle example, there are apparently four different strategies, only one of which they saw in the initial model. But after some training, they saw two additional strategies, although they still didn’t observe one that humans got.
So does it work?
To answer that, they train three models with their technique: Qwen2.5, Qwen3, and Olmo-3, all in the same size range.
What they’re measuring here is AUC, short for “area under curve”. It’s a way to capture performance of pass@k for a series of k values as a single number, instead of having to report pass@k for each value of k. The higher the AUC, the more question you got right at lower values of k. And remember, the pass@k curve can never go down, because increasing k just means giving the model more chances to get something right. One consequence is that lower AUC numbers actually show differences better, since if you’re already doing well on pass@1, there’s not much room for improvement. So here for instance they drop AIME and Medicine after Qwen2.5 and focus on the more discriminating benchmarks, HLE and Physics.
Anyway, as you’d expect from a published paper, their method wins out. The baseline model, which they label Instruct, does the worst most of the time, although not always.
The other comparison they do is against what they call “SimpleRL”, which is your standard GRPO + RLVR, the foundation this paper builds on.
For Qwen3 you’ll notice two other methods, DAPO and Forking Token. DAPO is like a tweaked version of the GRPO algorithm that we don’t need to worry about. Forking Token is actually from the paper in the background slides, the ones about high-entropy tokens. I was pleased to see they liked that one as much as I did. Still, the simpler method of rewarding diverse strategies does best.
Of course we have to cover entropy too, since we know how valuable it is. Here they show the classic trend for standard GRPO + RLVR, where the entropy trends down as the accuracy trends up, regardless of the model.
For their method, by contrast, you preserve or maybe even increase entropy. So then you’re doing better exploitation while maintaining good exploration, exactly as desired.
Oh, and if you noticed the relative smoothness of the latter two graphs, I think that’s just from a change in plotting rather than a genuine difference between Qwen2.5 and the other two models.
My Takeaways
Entropy is a first-class metric
Any training method that only reports accuracy (or pass@k for low k) is cheating
Maybe we can use this for reasoning data
Their coverage evaluation used human-generated strategies as the ground truth
We could verify all the clustered strategies AND try to come up with alternate strategies not represented in the clusters
Could you be more purposeful in exploration?
Example: ask SOTA for diverse strategies and then only grant max advantage if you display all of them