Originally presented as a live talk on October 8, 2025
Background
Computer use agents are what most people picture when they hear “agent” — a JARVIS sort of figure who clicks through a bunch of things for you when you give it a task.
In practice it’s much easier to give agents access to code tools like APIs and MCPs, since navigating around a GUI is more of a human thing, but some tools don’t have APIs or MCPs. So we have to use CUAs.
Now here’s the background on computer use agents:
Sep 2022: Adept Action Transformer 1, or ACT-1. Adept saw that transformers worked by predicting the next thing in a sequence and decided computer actions like clicks and scrolls could be that thing, the native language so to speak of a transformer. ACT-1 was trained directly on these computer actions. Unfortunately they were heading down the wrong path; the CUAs we’re discussing today are LLMs with computer use capabilities added in, making use of the raw intelligence that language modeling seems to impart. They were also well ahead of their time too as you can see from the rest of the timeline. ACT-1 never got good enough and was not widely adopted after release. The company had a fake acquisition like Inflection and Character in 2024, and now they focus on agents.
Jun 2023: Mind2Web, an early and widely adopted benchmark for “LLM-based web agents” as the description says. It has over 2k tasks across 137 real websites.
Oct 2023: SWE-bench, the first agentic coding benchmark. This isn’t web-based, it’s command-line based like Claude Code, but it does appear in the UI-TARS paper and it’s a benchmark that’s still with us today, like our SWE-Bench-Pro leaderboard, which we recently released.
Apr 2024: OSWorld, the gold standard of CUA benchmarks. 369 tasks for desktop and web, with multi-app workflows.
Oct 2024: Claude Computer Use, the first CUA from a major lab
Jan 2025: Operator, OpenAI’s CUA, now part of ChatGPT Agent which honestly I had never heard of until writing this. Also, UI-TARS.
Apr 2025: BrowseComp, also from OpenAI, containing requests that require many steps of web research to answer. Also, UI-TARS-1.5
May 2025: LMGame-Bench; Terminal-Bench
Sep 2025: UI-TARS-2
Now by complete coincidence, Anthropic release Claude Sonnet 4.5 last week with significant bumps in a few agentic areas. Look at that jump on OSWorld: nearly 20%! Sonnet 4.5 already blows past UI-TARS-2, although UI-TARS-2 does surpass Opus 4.1 and Sonnet 4.
There have been lots of other papers published here recently too.
The one on the left came out on September 19th but is already a little outdated - you can see it gives results from a previous version of UI-TARS. It has a lot of similarities with the paper we’re reviewing and goes into a lot of detail about how they collected data.
[2509.15221] ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
The one on the right came out only a week ago, from our friends at Apple. It’s actually the perfect bit of background, because it covers small agents like the NVIDIA paper we talked about last week, but for GUIs, like the paper we’re covering today.
Of course since it’s Apple they’re thinking about iOS use, but they test their agent on a lot of the same benchmarks as ScaleCUA and UI-TARS do. They are more buttoned-up on the details, but they also do a lot of the same work as the other two papers on unifying interfaces, creating and annotating data, doing SFT and RL etc.
[2509.26539] Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Note that all three papers use some human annotation!
Broadly, you’ll see actions, observations, and thoughts. For example, there are “plan” and “replan” nodes for the model to think about how to accomplish the prompt, and the whole Browser State section is basically the observation.
Now CUAs cover more than just browser use, but since browsers can do so many things and are more standardized than OS-dependent apps, browser use is probably the biggest component of computer use. That’s also why Perplexity released a browser and why OpenAI is rumored to be developing one too.
At the same time, browser and computer-use agent data has drawn less commercial demand than coding-agent data. One component I think is the more mature, commercially valuable progress on coding agents. Like Anthropic clearly spent a lot of effort getting to that 60%+ result on OSWorld, but what really rakes in the tokens is Claude Code.
Another issue I suspect is standards, or really lack thereof. We do see this in coding agent world, where different scaffolds like Claude Code or Aider demand different data. For computer use agents it’s an even more basic, scientific question, where it’s not obvious what combination of vision and text like a DOM or an a11y tree is going to yield the best results, and whether one may be better in the short-term but more limiting in the long-term. And that type of indecision or lack of clarity causes commercial paralysis. But I think standards and science have progressed a lot over the last year, and the quickening pace of CUA releases may finally drive real demand for CUA data.
Now I do have to mention one specific way in which CUAs are really not ready, and that is safety.
This is a frame from a short video showing how to hack Perplexity’s Comet browser, which has an agent built right in, as you can see in the right panel there.
The user is currently viewing a Reddit post, where one comment started with all its text covered, using the no spoilers feature. Now even though the user and any purely vision-based agent wouldn’t be able to read the text, since Comet looks at the DOM or a11y tree as well, it can read this visually obscured text.
And the text is basically a prompt injection, a set of instructions for the model that the user definitely would not want. Here it’s asking the agent to post the user’s email and one-time password for their Perplexity account to Reddit, which involves navigating Perplexity account detail pages and checking Gmail for the one-time password.
If a leading product in the space can get you hacked just by reading Reddit, then clearly we are not ready for CUAs in daily life. Useful for enterprise maybe where malicious content is unlikely to appear, but then again this is just one method. Lots of work for red teamers to do to discover other vulnerabilities.
Lastly, let’s recap this lab, Seed. It’s the AI development arm of ByteDance, which makes TikTok.
As far as I can tell it’s the product of some reorganization and centralization in 2023, which I suspect was a response to the ChatGPT moment in late 2022.
It’s quite a diverse group. Here is a list of models or products the Seed team makes:
Seed, a series of LLMs, including a multimodal variant called SeedVL
BAGEL, which does multimodal input and generation natively
Seedream, for image gen
SeedEdit, for image editing
Seedance, for video gen
Seed LiveInterpret, Realtime Voice, and Music, for everything audio
GR-3, a robotics model that we covered in a previous session
UI-TARS, the CUA we’re covering today
Of course being the parent company for TikTok, the multimodal and translation models are not surprising. But the robotics and computer use agent models take a little more digging. ByteDance also owns Douyin, which is like TikTok for China, but has a heavy ecommerce component, including warehouses with robots. They also make software with workflows instead of just consumption, like various creator and internal tools.
One major reason you’ve probably never heard of Seed is they only just started releasing open-weights, permissively licensed models - right at the same time as OpenAI did with gpt-oss, and in fact they called theirs Seed-OSS.
The Paper
So here’s an overview of what we’re looking at with this paper, which again is pretty similar to the other two papers and fairly representative of CUAs.
At the top of the diagram the user provides the prompt. Over on the left we have the different environments - desktop, mobile, browser, game - and the different actions the agent can take. There are GUI actions, like clicking and typing, and there are tool actions, like using the terminal or calling an API. Depending on the environment and the task, you may have access to different tools.
Now the way the agent proceeds is with our familiar observe-think-act loop. Looking at the example in the diagram, the agent looks at the game and thinks about how to play it. It looks up instructions and strategies, then contemplates how to apply them here on this particular board, then clicks somewhere on the screen. That loop happens over and over again until the game is over. So we’ve taken one turn, since the user has only given one prompt, but there have been many steps, many iterations through the loop by the end of the first turn.
Now we see the second turn where the user asks the agent to make a website about what just happened. Pretty contrived, but it works. They don’t bother showing much more of the loop here, just showing that they added in the terminal tool really. So apparently this was a two-turn task, with N steps for the first turn and 2 steps for the second turn.
So knowing how the agent works concretely, we should have a good idea of how the data looks and how to collect and use it. That’s what these images show.
The top image shows how they use the data. They describe it as a flywheel but they show it more like a staircase, which I actually think fits better.
They start with the base model, Seed1.6, which is 230B-A23B, ~500M vision encoder, has its own post-training. Apparently on par with Qwen3-235B-A22B.
They kick off the flywheel with some data they already have on hand from prior versions. Some of it is unlabeled, pretraining-like data, which they call “continual pre-training data” - that would be stuff like tutorials or guides. The rest of it is SFT data, a combination of synthetic and human-generated prompt-trajectory pairs.
The flywheel really gets going with their first bit of RL, which we’ll cover on the next slide. But the key to the cycle is that cube labeled “Rejection Sampling”. It sounds fancy, but basically what they’re saying is to take the good trajectories, where the model got a positive reward, and use those as SFT examples in the next iteration. The remainder, the failed trajectories, go in the CT data, since they’re still valuable in showing how computer use works, just not valuable for showing what to do exactly for a certain prompt.
Now how do they get the annotated CT and SFT data?
For CT, they actually have a program installed on the personal computers of their contributors. While the contributors perform their tasks, they also think out loud and narrate what they’re doing, and the program captures that audio, which they then clean up with a LLM. So they get good trajectories and the reasoning behind each individual step or decision in the trajectory, which is really helpful for the model. They actually note in the paper that models trained only on actions, without thoughts, tend to mimic the training data rather than understanding it, which of course doesn’t generalize well.
They also tried retroactively adding reasoning to trajectories, but that didn’t work either - they said it was “nearly impossible to reconstruct the annotator’s original thought process.”
As for the contributors themselves, we don’t get any numbers, and the only detail about them is that they are a mix of novices and experts. The novices teach exploration, the experts teach efficient usage.
For diversity, they pick applications based on “industry coverage, user engagement, and market penetration” across web and desktop. Then they synthetically generate a representative array of tasks. Unfortunately they’re quite vague about exactly how they do this, but apparently there is some human input and also some input based on “normalized measures of usage frequency, user benefit, and cross-scenario transferability.” They also ensure a range of difficulties based on step count, number of pages required, and error handling.
So that’s CT. For SFT, they want the data to be “on-policy”, i.e. using model actions. On-policy is always more effective than off-policy, where a human or perhaps another model has composed the response or trajectory, but on-policy is often harder to generate.
So to get on-policy data, they have the model generate a few different thought + proposed action pairs, and then the annotator picks the best one OR writes their own. Once the annotator approves a thought + candidate action pair, the action happens and the observation follows, in this case represented by a screenshot. By the end of the task, you have a trajectory composed mostly or solely of thoughts and actions from the model, i.e. on-policy.
For RL data things were a bit different.
First off is their three different environments
Web browsing, to search for information
Web app usage, to accomplish tasks
Web games, to reach a certain state or score
For web browsing, they synthetically generate questions that use lots of partial or obfuscated information. Here’s an example:
““Discovered by a representative from the Music And Cabaret talent agency, this group had a founding lineup—initially under another name—that included members from Dreghorn and Irvine, plus a lead guitarist and drummer. The lead vocalist joined after being recommended by a founding member who saw them perform with a Kilmaurs-based band, and their lead guitarist left to form another ensemble before late 1975. Which record label did this group sign with?”
So as you can tell that’s going to involve a lot of searching to figure out. And the agent has to do the searching in a browser, no APIs.
For the web app one, again they’re synthetically generating prompts, this time by getting a more powerful model to look at screenshots from 690 different web apps and determine what functionality different apps have. Then they put those functions together in various ways to make sufficiently complex tasks.
Finally for the game one, it’s actually very similar to old-school RL, from before LLMs were even a thing. The model plays the game and you check the score or level or whatever.
So as you can imagine, these outcomes are varying amounts of verifiable. Like for web browsing, it’s pretty objective, and for games it definitely is. For web app usage though it’s tough, so actually they train UI-TARS to also be a good judge model for UI stuff and have a separate instance to judge outcomes.
They also occasionally reward proper formatting and penalize excessive length, but they provide no specifics and my general understanding is you take an accuracy hit by having other goals like this.
So they do this RL training on three separate models, all initialized from the same SFT checkpoint. Once they have their three RL’d models for web browsing, web apps, and web games, they then merge the specialists together into one model again. And then that new model is what generates the next round of trajectories for SFT, then that SFT’d model gets RL’d into three and merged again, in cycles until performance levels off.
Now let’s look at a couple impacts of training and scaling.
The top graph is showing the amount of thinking per cycle, per think/act/observe cycle. The “step” label on the x axis refers to the number of training steps, NOT the number of steps per turn.
Anyway, as you can see, with more training the model actually needs less thinking. That’s the opposite of what we see in so-called “thinking models” or “reasoning models” like o1 or R1; usually the more you train, the longer the chain of thought gets.
But CUAs make progress by acting more, not thinking more apparently. And it makes sense, since most actions on a computer don’t need tons of thought - it’s usually fairly clear in a well-designed program what button a user needs to press next for example. An initial plan might need an increasing amount of thought, but probably not the thoughts for each individual action.
That’s the pattern for web browsing and web app use anyway. For games, it’s actually kind of a funny pattern, where thinking goes up and down with training. The reason there is that as the model trains more, it gets better at the game, and it progresses to harder stages or levels. Then it needs more time to think, but then it gets used to that next stage or level, then it needs less time. So that inherent difficulty progression works against the baseline trend of increasing efficiency we see with more general computer use.
Now for the bottom graph, we’re scaling up test-time compute but in terms of actions, not in terms of thinking. So the x axis here shows the maximum number of steps or cycles we allow the agent to take, and the y axis shows how performance on a given benchmark improves.
For OSWorld on the left, allowing more steps slowly increases performance, but probably isn’t worth it given how much longer 100 steps is going to take compared to 20 or 50 steps where you’re getting most of the gains. And even then I somewhat struggle to imagine as a user tolerating 50 steps.
For their suite of games on the right, we see steadily increasing scores for UI-TARS-2, all the way up to a perfect 100 if you allow 1000 steps. Obviously the steps are going to be qualitatively different for games compared to browsing or apps, and also you as a user probably don’t care about a maximum number because a longer game session means your agent is doing better. Although why you’d want to employ an agent like this to play a game for you I struggle to imagine. Still, game annotation is a real if niche area, so it’s not totally obscure.
And finally, here’s where we end up. UI-TARS-2 is probably the best open-weights CUA right now, although given the pace of progress I don’t expect them to hold that crown long.
Also as a reminder, Sonnet 4.5 got over 60% on OSWorld, so even though UI-TARS-2 is better than some pretty recent Claude models, Anthropic is still king of CUA.
Also also, Gemini apparently loves Candy Crush.
So What?
2026 probably is the Year of Computer Use Agents
CUAs on the edge for phones should arrive next year, at least on the research side
More broadly, software will be made with non-human users in mind
Similar to how websites are now responsive to different viewports and have accessibility features
Eventually it may seem odd to interact with much software directly at all
CUA red teaming could be big