Originally presented as a live talk on October 22, 2025
Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
or, A Few Ways to Be Right, Infinite Ways to Be Wrong
Online Rubrics Elicitation from Pairwise Comparisons
or, One Lesson Ahead of the Student
Background
Let’s briefly recap the motivation for rubrics.
At the start of 2025, DeepSeek R1 made big waves by revealing how OpenAI made their reasoning models like o1 and o3. The key was a technique called RLVR, reinforcement learning with verifiable rewards. You have a problem with a ground truth final answer that is easy to check, like a number or a short expression. You let your model try to solve the problem, and then you check the model’s final answer against the ground truth final answer. Sometimes you have to deal with equivalent expressions, like “square root of x” and “x to the power of one half”, but on the whole verification is straightforward.
The reason RLVR is nice is because it’s immune to reward hacking, which is where the model gets the credit of a good response even though the response is actually bad. That’s an unavoidable problem in RLHF, where you’re training a reward model using preference rank data based on fuzzy implicit human judgments. But with RLVR, because the final answer is verifiable against the ground truth, the model only gets credit for an actually correct answer.
Of course the set of prompts with a verifiable answer like that is quite small, mostly STEM problems and code using unit tests. But surely many more prompts have many aspects of quality that are objective and verifiable if your judge has enough intelligence.
That’s where rubrics come in. A rubric is a set of objective, atomic, self-contained, and MECE criteria that describes the ideal response to a prompt. You use a LLM judge to evaluate each criterion, then you total up all the points from all the fulfilled criteria. Sometimes the criteria are weighted, sometimes a criterion is worth negative points if it’s for a bad feature. But in the end you should have a verifiable score, just like in RLVR.
Of course not every aspect of quality can fit in criteria, and some prompts may allow radically different good responses. Like a creative writing prompt for example, where two responses may take totally different tacts but both be great. In that case you’re still going to need RLHF. But rubrics have really taken a bite out of RLHF because so many open response questions do entail responses with shared, definite aspects of quality.
One other advantage of rubrics over RLHF is you can reuse the rubric any time you’re getting a response to the rubric’s prompt. That’s not the case in RLHF, where you have to make a new judgment from scratch every time you get a new pair of responses. So the reusability plus the verifiability means customers want to use rubrics wherever possible.
Now for a long time rubrics weren’t really published about - they were secrets of the frontier labs. But slowly word has gotten out, and around early summer 2025 we started seeing rubrics papers coming out.
One of them was by Scale, which I covered in an earlier Friendly Paper Review. Here are two others that came out over the summer that I’ll cover briefly now.
The first one is from Google DeepMind. They frame the main motivation as avoiding reward hacking, as we discussed in the previous slide, it’s not about reusability here. They also focus on what they call “causal attributes”, the attributes of a response that cause it to be good, as opposed to “neutral attributes”. They use a LLM and pairs of responses to suss out these causal attributes - what we would call criteria. However, this is more of a proto-rubrics paper, since they ultimately still use preference ranks. It’s just that they use their causal attributes to synthetically alter responses, adding the causal attributes to one copy of a response and then making it the preferred version in the preference rank compared to the unedited version. In the end they’re still training a reward model like in RLHF.
The second one is from Ant Group, which is an affiliate of Alibaba, so distantly related to the Qwen folks. They have some pretty exotic stuff in their rubrics, like criteria with different levels of satisfaction rather than yes/no, diminishing returns on a criterion, and interactions between criteria, like negative criteria that veto all the other criteria. If you’re on a rubrics queue it’s worth taking a look at their examples, they’re pretty different from the rubrics data I’m familiar with. I think this is a result of disconnected ecosystems, where rubrics for Chinese labs look different than rubrics for American labs because they have evolved independently. But I acknowledge that is speculation.
The second paper is also interesting because it shows rubrics can exhibit reward hacking, which I think would surprise the first paper’s authors. Really it’s because the user of the rubric is a model, and just like with a reward model, the LLM judge using the rubric can be tricked. One specific way they show models do reward hacking is “self-praise”, saying stuff like “The following answer is accurate and thorough” to bias the LLM judge.
The Paper
This graph encapsulates the main finding of the first paper, which is that distinguishing between the top responses is really what matters for rubrics.
I should warn you though that these are theoretical results, which is why the curves look so nice and smooth. We’ll see empirical results later.
Here we see what happens to models trained on various alterations to scoring. The x axis, the KL Divergence, is a measure of how much the model you’re training has changed, so a higher divergence means a more different model, separate from how many training steps or how much training compute etc. The y axis, the Win Rate, is how often a response from the trained model is going to beat a response from the initial model. That’s why at KL Divergence = 0, all the curves start at 50% - that would be just two copies of the initial model and of course that’s going to be a coin flip.
Anyway, the different curves correspond to the different alterations to rubric scores, like what errors are in your scoring system compared to the true scores that responses would get using the true rubrics. Correct means the right scores of course, and Reversed means the exact opposite scores. Correct produces the best results and Reversed produces the worst results.
The interesting bit is the other four cases. If you take the worst 50% and flip those rankings, so like the absolute worst response gets the median true score and vice versa, all the way through the bottom 50%, you see virtually no difference with the Correct case - the lines almost totally overlap. Even if you flip the worst 90%, where the absolute worst response gets the 90th percentile true score and vice versa, again you’re able to recover the performance of Correct, although it will take more training.
Contrast that to flipping the top 25%, where your absolute best response gets the 75th percentile true score and vice versa. With much less incorrect data you’re already seeing worse results. And if you flip the top 50% then your model isn’t going to improve at all!
So there are two crucial steps when writing a rubric: one, get some truly great responses so you know where the ceiling is; two, make sure your rubric can accurately distinguish between those top responses.
Then the question becomes, how do you get these truly great responses? Because if your prompt is worth training on, you’re very unlikely to get a truly great response from your model; prompts where a truly great response is even somewhat likely are probably too easy.
If you’re training a weaker model, where a SOTA model is able to produce great responses on your prompts, then you can just write a rubric based on the SOTA responses.
However, if you’re training a SOTA model, you’re going to need humans to write your great responses, possibly by editing the SOTA model responses. So that’s the opportunity for our data.
So how do they propose making these rubrics that are so good at distinguishing between top responses?
The basic idea is simple: look at the best responses you currently have available and make sure your rubric can distinguish between them. If it can’t, update your rubric until it can. Then regenerate responses and repeat the cycle until your rubric has stabilized, until it can handle any pair of responses you throw at it.
To do that you need two things: a source of new responses, and a source of rankings. New responses can come from anywhere, since the rubric is model-independent. That means you can use a bunch of models, or even humans, to generate responses. For a really tough task where even SOTA is good but not great, a human response could go a long way.
The rankings typically come from a LLM judge using the existing rubric, but it doesn’t have to be that way. For example you could have a human pick the top two, perhaps from a smaller pool of top candidates as chosen by the LLM judge with rubric.
How many great response pairs do you need, like how many cycles? Well this table sheds some light on the question. Using Gemini 2.5 Flash as a good response writer and Gemini 2.5 Pro as a great response writer, the authors see how many pairs of responses it takes to distill a maximally effective rubric out of them. They’re training Qwen3-8B Base here, then comparing it against Qwen3-8B Instruct for the win rates. Remember, the Instruct model is going to have all that post-training on it, so that’s a lot of ground for our rubrics to cover, hence the low win rates even in the best version. But it’s the trend that matters, and moving from 5% or 10% up to almost 40% win rate is pretty good.
So unsurprisingly, distilling rubrics from better responses helps more. That fits right in with this thesis about chasing the tail.
And of course more great responses is better than fewer great responses. They don’t report numbers other than 1 and 4 pairs, but I assume greatly diminished returns after 4 based on them not reporting.
The interesting twist here is that last line: great and diverse pairs. That actually mirrors a finding in reasoning data, where showing the model multiple different pathways to the same correct solution is more valuable than just training on one or the other pathway. That robustness from diversity apparently holds up in rubrics too, which makes sense to me - it gives the rubric greater coverage, and really there’s no one perfect response to most questions. So your rubric needs to be able to recognize greatness in whatever form it takes. Which is a pretty tall order, but at least it’s a north star to shoot for.
So individual response quality is the most important thing, then number of responses, then diversity of responses. But they all matter for your rubric to distinguish top responses.
Now what exactly are rubrics gaining from these comparisons? Here they show buckets of criteria, ranked by frequency, and highlighted in cases where a significant number of the refinements came from comparing either two good responses OR two great responses, i.e. not comparing good to great.
So for example in the first highlighted row, refinements that “break down complex criteria into sub-components” come from comparing two great responses significantly more often than comparing two good responses.
And then the next row, refinements that “introduce penalties, prohibitions, or negative scoring” come from comparing two good responses much more often.
So it seems great responses introduce subtlety into the rubric, while good responses highlight common errors. Given how much distinguishing the best responses matters compared to distinguishing good or mediocre responses, it seems we should spend most of our rubrics time on the frontier of performance and let some of the lower scores be noisier if we have to.
Personally I would like to see this same analysis in fewer buckets. I would also like to see a split between positive and negative features, or like criteria that came from a good feature of a response vs a bad feature. So like if a response mentions something smart that no previous one has, that to me is different from a response making a mistake that the new criterion makes sure won’t happen again.
Intermission
So the first paper shows how we can generate rubrics from pairs of great responses. This one shows how we can adapt rubrics as training goes on.
The motivation is pretty simple: as you train the model, it gets better. And as we know from the previous paper, a better model producing better responses is going to need finer and subtler rubric criteria. So if we can use responses from the most recently trained version of the model in our response pairs, we should be able to keep finding relevant criteria.
That’s really all they’re illustrating here. In step 1, you get one response from your current version of the model and one response from some other reference model, like the initial version of your model or even a completely different model from another provider, as long as the control doesn’t change. Then you compare them to find new criteria, what the authors call “elicited criteria” shown in green, that you can add to your existing criteria shown in red.
After you score your responses with the newly expanded rubrics, you train your current version and then repeat the process in step 2. They chose to show a different prompt in step 2, but the three dots above the prompt in blue means you’re doing lots of prompts in parallel, including the one from step 1.
This type of data generation, where you’re using responses from the latest version of the model you’re training, is called online. When you use data from any other source, including older versions of your model, it’s called offline. For anyone who has done an RLHF project, RLHF data is most effective when it is online, which is why it’s a good line of business for us; once you retrain the model you need to generate new responses again.
Online is more work to collect, but it’s more effective when done correctly. It’s kind of like having a personal tutor instead of being in a classroom or reading a textbook. In the lingo you’d say it has better exploration and distribution matching.
Here’s the main result. If you compare any form of rubrics to just using a good LLM judge with a simple Likert scale, rubrics is gonna win every time. But within rubrics, it seems the online approach outdoes offline. And within online, it seems to not matter much whether you pick the control model to be the original one or an earlier iteration in the online cycle. Note that these are scores on the held-out eval sets, we’ll see benchmarks on the next slide.
Now these results are from training Qwen2.5-7B, which is a smaller and older model, but I suspect the results hold up on bigger and better models.
So here’s a kind of buildup of techniques, on generalist and expert data.
They start with the initial model, then show scores for the versions we saw on the previous chart: LLM judge, offline rubrics, and then both versions of online rubrics.
The new bits are the two techniques added on to offline. The first one, Universal Requirements, is from another paper, showing that a fixed set of criteria added to all rubrics improves stability and mitigates reward hacking. The second one, Pointwise Extraction, is in contrast to the pairwise extraction of criteria in online rubrics. Here they’re just looking at the new model’s response to find elicited criteria, rather than looking at new vs control. So even though Pointwise Extraction is grouped under Offline Rubrics, really it is an online technique.
Anyway, you can see that for both domains Online Rubrics is the best or tied, within some small margin of error. I would also note that the human rubrics are also the best or tied within the margin of error, compared to synthetic rubrics.
And in case you’re wondering about the win rate: it’s against the base model, Qwen-2.5-7B-Instruct. The reason the WR isn’t 50% on the first line is because they allow ties.
As with the previous paper, they use a LLM to bucket the criteria they found by looking at response pairs.
And a lot of these line up with the other list actually, even though they are somewhat different domains; healthcare for the other list, STEM for this one. I matched up some of the buckets from the other list here on the right. Some of them are nearly identical.
My Takeaways
Distinguishing the top responses is the most important job of rubrics
Rubrics are an abstraction that avoids artifacts and off-policy issues
Human-written responses, rewrites, responses from other models - all fair game
But like all abstractions, they eliminate some detail in exchange for generality
Pairwise > pointwise, specific > universal, human > synthetic (to start)
Other fun tips:
Watch out for reward hacking by “self-praise”, e.g. “The following advice is the most relevant”
o4-mini beat GPT-5 (and all other models) as a judge, regardless of generalist/expert