RLAnything + OpenClaw-RL
Originally presented as a live talk on March 25, 2026
RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System
or, Theory
OpenClaw-RL: Train Any Agent Simply by Talking
or, Practice
Background
So the first thing we need to talk about, and really the framing we should have for almost all AI progress, is this graph here.
It’s from a research organization called METR, and it shows the increasing duration of tasks that AI can handle autonomously. The duration is in human time, so for example they’re saying it takes competent humans about 10 minutes to find a certain obscure fact on the web, and just under an hour to train a simple type of ML model called a classifier. METR has put together a bunch of tasks with varying human durations, and it administers that suite of tasks to new models, then graphs the result.
There are two toggles on the bottom here. One adjusts the scale of the y axis, so right now it’s a log scale, meaning that our straight line here is actually an exponential line on a linear scale. The other toggle adjusts the quality threshold. Here I have it on the more conservative 80% Success setting. So like Claude Opus 4.6 can do stuff that normally takes humans about an hour, completely autonomously, at 80% success rate.
Now this graph shows one trendline over the whole period, from GPT-2 through Mythos. But if you look at the green dots, which represent SOTA at the time, really it looks like two trendlines: one trend from GPT-2 to GPT-4o, in that quick run of three green dots below the line; and then another trend from o1-preview onwards. That break is where reasoning models came in, using test-time compute scaling, using very long chains of thought to arrive at better answers.
If you believe there really are two trendlines, then the second one is steeper than the first. And on that newer steeper trendline, the time horizon is 10x’ing every year. So Opus 4.6 for example came out in February 2026 and hits ~1 hr on this graph. That means in February 2027 we’ll be at ~10 hrs. That’s an entire day of work, done completely autonomously, and largely successfully.
So a lot of the work between now and then, and after then I suppose, is about long horizon - increasing the duration of task that a model or agent can do, not necessarily making it smarter or more knowledgeable.
Of course the longer the horizon gets, the longer we have to wait and the more things that have to go right in order to give out a positive reward. That latter problem, where you have more and more possible points of failure, is what makes long horizon training so difficult.
Now if we think about a human setting, where you’re trying to teach someone a skill or maybe learn one yourself, the person learning probably isn’t going to do a whole practice run themselves and then check back with the teacher or the instruction manual or whatever at the end. Instead, we usually check in at certain steps or milestones. Or like in cooking, you might taste test along the way rather than tasting only at the end.
The equivalent in machine learning is process supervision. In process supervision, we give a reward for every step, judging two things: one, is this step accurate; two, is this step on the right path. So looking at a math example like this, which is from an OpenAI paper that we made data for, you have the accuracy of calculation and the helpfulness of that calculation in getting the right answer. Of course in this example the final step is the incorrect one, so all we have to judge is the accuracy of the calculation, but in earlier steps you implicitly have to judge both with just one score.
Process supervision works for anything that happens in discrete steps. Math is like this, but creative writing is not. Even programming mostly is not, since you can often change the order of code blocks without impacting functionality. But one big chunk of the ML world is like this: agent trajectories. The observe-think-act loop is inherently sequential, and changes to the environment often demand a certain sequence. For example, navigating through a website requires a series of clicks in a certain order.
So it seems we can add process supervision onto agents pretty naturally. That means we’ll have a process reward model, a PRM, that gives out rewards at each step. That’s in addition to whatever reward we give at the end of the trajectory, which people sometimes call an outcome reward model or ORM, but often is just called a reward model, since outcomes are the default thing to give rewards on.
So if intermediate rewards are so valuable, and maybe increasingly necessary, why doesn’t everyone use them? We’ve covered agent RL papers before, and none of them have had intermediate rewards. Why not?
In my view, there are two reasons.
One is, it’s expensive to make process supervision data. There are just way more annotations per task. If you can avoid process supervision and just use outcome labels, that’s way cheaper per task, so you can get more tasks and more diversity etc. This is partly why RLVR data has largely displaced PRM data for reasoning. Of course, if you could get that data for cheap or even free, the balance might change.
The other, thornier reason is reward hacking.
Briefly, reward hacking is when the stuff you reward is only a proxy for what you actually want, and that gap between the rewarded thing and the actual goal causes unexpected or bad behavior. For example, you might reward a model for passing unit tests on a coding problem, but the model could learn to just hardcode passing behavior instead of writing the real program. Or even worse, it could learn to edit the tests to just always make them pass.
Reward hacking can happen whenever there are rewards. If you just have an outcome reward, then there’s only one possible hacking target per task. But if you add in intermediate rewards, you’ve added new hacking targets. So intermediate rewards already add some risk.
But there’s a deeper problem with intermediate rewards: they can distract or derail the agent entirely. The gif on the slide here shows one such example. It’s from a 2016 blog post from OpenAI, by Dario Amodei and Jack Clark actually, who later went on to found Anthropic. The post is called “Faulty reward functions in the wild”. In it, they describe some RL experiments they did in a boat racing game called CoastRunners. As humans, we know the goal of a boat-racing game is to place first. But actually, the game only tracks points, which you get from hitting these floating green blocks and then from your finish order. The most points wins.
When they set their RL agent loose on this game, they saw something funny: the agent learned to ignore the race and just steer the boat in a circle over and over again, racking up points from the same three green blocks. Often it just circled forever and never ended the race at all.
That is the danger of intermediate rewards. You have to pick something, often arbitrary or not strictly required for your end goal, and reward on it. Even with domain expertise and careful design, it can still go sideways.
On a different note, we also need to cover a training technique called “self-distillation”.
Let’s start with plain ol’ distillation, which is where you use one model to train another model, “distilling” the wisdom from the teacher model into the student model. Distillation has been in the news from time to time because OpenAI, Anthropic, and Google have all caught the major Chinese labs distilling off their models, which violates the terms of service.
The naive way to distill is to have the student copy each output token from the teacher in response to a prompt. Like if you penalize the student for outputting anything except what the teacher output, then the student will learn to sound more like the teacher.
But if you can go one level deeper, to the probability distributions of the teacher rather than just the one final token it spit out, that teaches much more richly. That’s what we’re looking at here. Sometimes people call that “soft-label distillation”, as opposed to the “hard-label distillation” I described earlier as the naive approach.
So the Chinese labs have been distilling off the American models by passing the same input to the teacher and the student. The teacher is better than the student, so the student learns and improves.
But it doesn’t have to be that way. In the paper from two weeks ago, they actually kept the model the same in both cases - same model for teacher and student - but they gave the teacher extra context. Just like having your friend quiz you, with them looking at the answer key so they can guide you to the right answer.
In the example here, the teacher for this SQL bot gets some examples in its context, whereas the student does not. So when you do distillation, the student is receiving the extra wisdom the teacher had due to its extra context, without actually needing that context in the prompt anymore. Since you’re using the same model in both cases, it’s called “self-distillation”.
Notice how flexible this can be. Anything you can put into words can be distilled. For example, apparently Anthropic uses distillation to teach Claude how to be Claude, with a big document in the teacher’s context containing Claude’s principles and operating procedure etc.
The paper from two weeks ago used context distillation to incorporate feedback from the environment. So the model would attempt a task, get some feedback like failing unit tests or a critique from an LLM judge, then put that into its context in the role of teacher. Then the model with its teacher hat on would basically grade every token probability in the original response, then train to minimize that loss. They called that “self-distillation policy optimization”, SDPO, and unlike other techniques like GRPO, it doesn’t require rewards that boil down to numbers. It’s a richer signal that naturally occurs in any system that provides feedback.
The Paper: RLAnything
Okay, so our first paper is called RLAnything. That’s because their proposed system is fully generic, across any sort of task, or at least that’s the claim.
They break down RL into three parts:
The policy model, which is the model actually generating responses or actions etc
The reward model, which judges process and outcome rewards, except for any verifiable rewards you might have that don’t need judging
The environment, which includes the prompt as well as the tools and state the agent has to navigate to accomplish whatever task
They’re going to be adjusting all three of these dynamically to maximize performance, which we’ll see in the next few slides.
While we’re here though, a couple things to sort out. One is some terminology: they often say “step-wise” instead of “process”, and they have a “critic” model that is actually a separate LLM not involved in training at all.
Second is the notation. I know the math looks very scary, but it’s just a lot of annoying symbols to keep track of, no exotic operations. Like for the left side, all they’re saying is the total reward is the outcome reward plus the average of the process rewards, which they request from the PRM m different times just for consistency. And then O and S, the outcome and step-wise rewards, are gonna be either 1 or -1, nothing in between. Also P stands for the policy model, R stands for the process reward model, and E stands for the environment.
Third, the PRM is going to return a score but also its reasoning, like a justification our contributors would write in an RLHF task. That will be helpful down the line as we’ll see, especially for separating the correctness of the individual step from the helpfulness towards a good outcome.
Now for their experiments they’re primarily testing on two benchmarks of a couple hundred tasks each:
AlfWorld, which is basically a text adventure game
OSWorld, the gold standard for computer use agents
Sometimes they report the results for AlfWorld as “LLM agent” and the results for OSWorld as “GUI agent”.
As for the models, they’re using Qwen2.5-7B for AlfWorld and Qwen3-VL-8B for OSWorld. Not the newest models, and not the biggest either, but quite common for academic work.
Here’s an illustration of how they adjust the task and environment to balance difficulty and novelty and hit that Goldilocks zone. Remember, the PRM is basically a classifier, and classifiers do best when the classes in their training data are roughly balanced. So we roughly want to have an equal number of good and bad steps to train on and thus improve our PRM, which is key to this whole approach.
On the top is an OSWorld example, where the reward model gives two accurate criticisms of the policy model. An outside LLM then reviews the trajectory and the criticisms and makes a modification, which allows the policy model to improve from 0% to 25% accuracy.
They also have AlfWorld and coding examples with the same idea, although in both cases it’s to make the task harder.
This graphic encapsulates their results, so let’s go through each:
On the first one, they’re showing that you get the best results when you make all three components of your system dynamic. So you want to train both your policy and reward model, and you want to adjust your environment to be neither too hard nor too easy, so that you’re always getting some signal but always leaving room to improve
On the second one, they’re comparing the value of process rewards and outcome rewards. So for the yellow line, it’s just outcome rewards, with human labels. Then on the blue line they’ve added a PRM, but they don’t train it, which maybe helps a little but shows a PRM per se is not the big difference. The red line however is only a PRM, but trained, and that ends up doing the best. So that shows the value of a good PRM if you can get it.
On the third one, they’re just showing overall improvement from their method. LiveBench is actually a coding benchmark and isn’t an agent thing, yet their method even works there
Finally, on the fourth one, they’re showing how their adjustment to environment and task difficulty progresses over time and correlates with the quality of the policy and reward models
No mind-blowing results, but again, the emphasis is on how general and automatic the method is. They didn’t need to make any data, including those risky intermediate rewards we talked about earlier.
And they didn’t need to customize the setup much. They did have to think a bit about what inputs the PRM received in each case - the prompt + a summary of past actions and observations for both, the last two images for OSWorld - and they did have to know where the metaphorical “knobs” on the environments were for difficulty adjustment. But that’s about it.
Here’s a closer look at the results on the benchmarks, with different components allowed to change in different rows. A couple quick pieces of terminology:
OOD is “out of distribution”, like eval tasks that weren’t similar to the training tasks
UT stands for “unit test”, how accurate the PRM was at writing unit tests for judging step quality
“Detect” means accuracy at detecting bugs
So they’re improving across the board, with more dynamic components being better. A couple thoughts.
First, look at the difference between process and outcome reward accuracy between GUI Agent and LLM Agent. That’s a clean demonstration of how process rewards have two jobs: checking the correctness of the immediate step, and estimating how much that step helps get the right final result. So we see in LLM Agent they’re similar - it’s roughly as hard to do one as the other. These are short trajectories and each one needs to have the end goal in mind. But for GUI Agent, they’re quite different; it’s apparently quite easy to tell whether the agent actually performed the right action, but it’s often not obvious whether that action will lead to a good outcome. That makes sense for computer tasks, where you have to click through to see everything, it’s not all laid out for you at once. Like sometimes you have to click around to check out all the options and menus and so on.
The second thing is that you don’t need a perfectly accurate PRM to improve results. You can see for both agents that the improvement in the policy model is far greater than the improvement in the reward model in most cases. Even noisy signal is better than nothing, at least at these noise levels. Below 50% accuracy for the PRM I expect the policy model would suffer.
The Paper: OpenClaw-RL
Okay, so we’ve seen how process supervision can help agents. Now we’re going to see how to collect process supervision data naturally.
Because if you think about it, natural use of chatbots and agents doesn’t quite look like training data. Like if you’ve worked with agents at all, you know you’re often in dialog with them, providing feedback on their work and often not just leaving them to work on their own.
And nobody is really going to make nice training data out of their own usage. It’s a lot of work to get nicely scoped tasks and neat boundaries on trajectories and final outcomes and rewards. That’s why we get paid!
But surely there is a way to use the conversational feedback users often give, whether positive (like saying thanks or showing excitement) or negative (like scolding the model or making a correction).
Now this paper has much more of an engineering focus than the previous paper. So you’re going to encounter some training and hosting tools we don’t tend to cover here, especially because Scale focuses on the data side. So RL servers, Megatron, SGLang, slime, these are not important for us frankly. Just know they are the software that makes a lot of this all work.
So we’re going to be collecting and using feedback in three different ways here.
The first is basically a PRM, like in the previous paper. The rewards are binary, i.e. just -1 or 1 or 0, even though this graphic doesn’t say it. And they come in the course of conversation rather than waiting until the end. New words but same meaning really.
The genuinely new thing here is what context they give the PRM. Specifically, they’re providing the agent’s action, environment feedback, and user response. So if the agent makes a tool call and it returns some error about missing arguments, that’s probably gonna earn a -1. Or if the tool call works but the user complains that the agent is fetching the wrong information, also -1. Conversely, if the user seems happy, that could earn a 1. And if there’s no clear feedback or too little to judge on, the PRM can return 0. So this method rewards the user for more natural or expressive use of the agent.
The second method is also like a PRM, but with distillation instead of rewards. So it’s operating on the same single-step scale, and it’s using the same sources of feedback, but it’s putting them to work in different ways. First, the distillation method includes a lot of filtering and only uses clear mistakes for feedback, since the teacher is only helpful if the student has something to learn. Also, there’s some consolidation, where they use a model to summarize the mistake rather than just dumping the raw interaction into the teacher’s context.
The third method is pretty much what we saw in RLAnything: combining process and outcome rewards, PRM and RLVR basically.
The graphic doesn’t make the difference clear, but it’s important to separate the first two from the third. The first two methods are only stepwise, so they can run continuously, alongside all your interactions. The third one, by contrast, requires a definite outcome to then reward.
It’s a bit of an artificial marriage then, these continuous vs task-based methods. And actually the things they improve and the way they measure are different too.
For the continuous methods, the feedback from conversations, the researchers used two related examples: a student who wants to use OpenClaw to do his homework, and a teacher who wants OpenClaw to do grading in a certain way. Pretty subjective stuff, which makes sense with the more qualitative feedback their methods are able to capture, although the results are less compelling in my view. They have a scoring method, and you can see the improvement in scores on the right, but the scoring is just an LLM judge scoring the agent’s response given the user’s preferences.
This is the type of change you could prompt engineer, rather than something that absolutely has to be trained. So the value isn’t results per se, it’s how you can get these results “for free” just by incorporating natural signals.
They also have the more traditional objective results on benchmarks. Keep in mind they’re using Qwen3-4B across the board, so the gains are respectable. I don’t have much to add because this part is basically a rehash of RLAnything, but it seems their two new methods may be helping too.
My Takeaways
Where is the role for our training data?
Extracting training data from natural use means it’s way cheaper (even though lower quality/hit rate) and already from the natural distribution
So perhaps the economics favor the low-volume, high-utility cases (e.g. medicine, frontier STEM)
Can PRMs replace human-written intermediate rewards?
We know intermediate rewards can incentivize weird behavior
Another step on the path to continual learning
Also similar in spirit to Karpathy’s Autoresearch
This is probably already happening in some form
Big labs I’m sure
Possibly anyone with an agent and access to the weights