Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
or, How To Evaluate Your Agent
Originally presented as a live talk on March 4, 2026
Background
So the obvious place to start is going to be Gaia, the original benchmark that Gaia2 follows. I have a few things to say about it.
First let’s just look at what it is, what it’s testing. We’ve got 466 prompts split into three levels of difficulty. The prompts all require internet research, and sometimes require an additional skill: multimodality, reasoning, or file reading.
All of that is bog standard today, and even looking at the toughest prompt here, your gut is probably telling you a SOTA model can handle that easily. I actually tested all three of these on Gemini Pro. It got Level 1 right, was slightly off for Level 2, and completely off for Level 3. So Gaia was pretty ahead of its time, which you can see in the results section of the paper; the one model that could really do internet search and multimodal etc at the time was GPT-4, and in a few different setups, it got basically 0% of Level 3 questions.
There’s actually still an active leaderboard, and the best system gets 87% of Level 3 questions right. At that point I suspect many of the remaining examples are actually wrong or somehow broken, like the internet resources they require are gone or something. Also on the topic of leaderboards, our old friend ToolOrchestra from a paper review back in January is actually in 4th place.
Now looking at the ground truth of each example, you’ll see they’re all verifiable: easy to check, format and even decimal places specified in the prompt. This paper came out before RLVR was a thing, but you could definitely use it for RLVR today if you wanted.
Second thing to note is how little the paper mentions about agents. The word does appear a couple times, but it’s not the dominant framing. We’re firmly in model territory, not much extra conceptual or code structure on top of them. This benchmark came out only a year after ChatGPT, only eight months after GPT-4, and using tools was still pretty exotic; GPT-4 only got tools and code execution in June 2023. Now everything is agents and all the models are tool-native. Gaia2 even has the word “agents” in the title of the paper!
Finally, I will note the one thing that hasn’t changed, which is the need for tough, creative, and objectively verifiable evals. Every benchmark has an expiration date, but benchmarking in general has a long life ahead of it - basically until we reach ASI. In a way benchmarks actually set the path that the big labs then walk.
Continuing on benchmarks, about six months after Gaia came the standard-bearer of computer use agent benchmarks, OSWorld. It has 369 tasks across dozens of apps on three different operating systems. They intended most tasks to require using a GUI, since that’s mostly what people think of with the term “computer use agent”, but as anyone with Claude Code experience knows, you can do a surprising amount of stuff just from the terminal and with Python. Actually some tools are leaning into that fact now, like Obsidian recently released a command line interface. Of course there’s a long tail of legacy software that will not be adding these agent-friendly interfaces any time soon, so being able to navigate a GUI is pretty helpful.
Gaia and Gaia2 both do not require GUI use, but I wanted to show OSWorld for a few reasons. One, it’s a good barometer of CUA progress, regardless of the modality, which is really taking off in 2026 and I think will become part of our daily lives soon. Two, the part about surprisingly powerful command line interfaces is relevant to Gaia2, which lets agents interact with apps via command line, via tool use, as we’ll see. And three, the environment bit in the bottom third of the slide is quite relevant. This setup of an agent in an environment with an initial state is how agentic benchmarks like this work. And all the way on the right you’ll see the environment checks for completion in the final state, looking for signs the agent completed the task. It’s not asking for a certain text output like in the original Gaia benchmark, or most other benchmarks for that matter. So in this age of agentic benchmarks, you’re going to see this final state thing more and more, and a final answer much less.
The Paper
Let’s start with the environment Gaia2 is testing agents in, which I think will make the tasks in the benchmark and the results easier to understand.
So they have this open source package called Agent Research Environments, ARE, that lets you create specific environments with this setup here. The user and the agent deal with it through some tools, and they get feedback from those tools plus also some notifications, which can come from other sources like an app saying you got a new email or whatever.
Now inside the environment, we have a few things. The biggest is the apps, things like email or spreadsheets that the agent could interact with. Keep in mind this is all by API, nothing graphical. Anyway, the tools the agent uses correspond to APIs in the app. The app also has a state, all the data in it basically, like what’s in the inbox or what’s in the spreadsheet and all the settings and filters that are currently on.
Inside the environment you also see this thing called event queue, pointing to event loop and then event log. That’s because their environments have a concept of time. For example, you could have an email appear at 9 AM with the day’s work agenda, or you could have a food delivery notification appear 30 minutes after you put in the order, or if you got two texts from a friend the notification for the first text would appear before the second notification. That temporal aspect is actually pretty rare in these types of evals, it’s a key differentiator for Gaia2.
The last bit is the Scenario box up top, kinda floating above the environment. The scenario is basically the particular way you’ve set up the environment. So like in one scenario you might be starting your work day with a busy inbox, in another scenario you might be submitting health insurance claims and texting your wife for details, and those are going to require different initial states for your apps. The different goals of course are going to require different verifiers, like responding to an email and providing certain verifiable information, or submitting the claims with the correct information typed into the form. In Gaia2 they are checking specifically for changes to state, rather than a message from the agent to the user with a final answer.
The scenarios can get pretty complex actually. Almost like being a dungeon master, where you have contingent plans and events you initiate, rather than just a passive environment that only moves because of the agent’s actions.
So that’s environments in general. For this benchmark specifically, they decided to make and use a specific environment they called Mobile, which of course reflects a smartphone OS. They have 12 apps: the basic stuff, so Messages, Chats, Emails, Calendar, Contacts, Files, System; then general consumer apps they made up, called City, Shopping, Cabs, and RentAFlat. The final one is called AgentUserInterface, a very unwieldy name. That one is for the user to text with the agent. Again, this is all text-based, nothing graphical, so you could translate a lot of this stuff to an environment you call Desktop or whatever. You’d probably just take out some of these apps and add in others.
The apps have tools like search_emails or book_cab or wait_for_next_notification. All together there are 101 tools scattered across the 12 apps. All the apps and tools get fed into the system prompt so the model knows how to use them.
The data in the apps is of course synthetic. But it’s coherent across the many apps, reflecting a specific persona. So for example, if you have messages with a person, that person will appear in your contacts, and maybe you’ll also have an email from them. That collection of data across all the apps is usually between 400k and 800k tokens, way bigger than almost any model’s context window, so you can’t just feed all the data into the model at once and have it work from memory - it has to use the tools and read from state.
As for the agents themselves, they’re actually set up in kind of an old-school way.
They use something called ReAct, which is from this paper from October 2022, right before ChatGPT came out. I view it as kind of a stepping stone to full-on agents, and one of the main authors of this paper actually has been on the forefront of agent progress for a while: he was on the SWE-bench paper, which came out in October 2023; and he was on the Tau-bench paper, which came out in June 2024. Those are both important agent benchmarks, SWE-bench in particular, which kind of established the genre alongside the original Gaia paper. The first real agents from the big labs didn’t come out until late 2024 and early 2025, stuff like Claude Code and Gemini Deep Research and OpenAI Operator, so in a way those benchmarks set the target for the labs to aim at.
Anyway, back before agents really worked, people tried different ways of getting chatbots to do more than just chats. ReAct was probably the most successful effort, I think because it’s so straightforward. All they do is prompt the model with a few examples of this thought/action/observation loop, and they have a special format for the actions that some helper software picks up on to execute the action and return the observation.
You can see it in action for these two example questions from the benchmarks Hotpot QA and AlfWorld. For the first question in particular, they show just asking, then asking with CoT prompting, then asking with access to a search tool, all of which yield the wrong answer. Over on the right you get the full ReAct paradigm of think, act, observe. When they lay it out like this it’s a pretty natural marriage of CoT and tool use, which is still what you see when you’re using something like Claude Code today - that think-act-observe loop is at the heart of all agents.
Remember, at the time they published ReAct, tool use by models was not really a thing, like ChatGPT couldn’t do anything except return you text. Of course now tool use is second nature for all models, so you don’t have to hammer on that in the system prompt. But it does help standardize things to all use this same basic prompt, rather than relying on more complex or model-specific scaffolds. That type of variability is a killer for benchmarks.
Now for the tasks. They used human annotators to make 800 of them, split evenly across the first 5 capabilities in this table. So that’s 160 each.
These tasks all happen within 10 different “universes”, basically a user persona and their collection of user-specific data. In the paper they give the example of a persona named Helena Mueller, a 43-year-old Marketing Manager living in Berlin. So then the contacts, emails, calendar etc all make sense for Helena and her imagined life - individual and group messages, calendar events for the week, a ride hailing history, pretty detailed stuff. That’s separate from the Mobile environment, which details in general the apps, the event triggers etc. You fill up the environment with the data to get a universe.
The tasks are all verifiable. Specifically, the verifiers are mostly going to look for write commands that show the agent actually took the necessary actions. So like the first one, which is kind of a dumb task, that’s going to look for writes to all the contacts. Unfortunately, making them verifiable usually means being weirdly specific, like the Search example where the user anticipates there could be a tie and says how to break it. I think the authors realized that and made the Ambiguity capability to compensate, but I suspect real users will be ambiguous much more than a fifth of the time.
Anyway, if the agent takes more than 200 steps or overflows its context or misses a deadline for a temporal task, it automatically fails.
The tasks specify an initial state and also a chain of events, in some cases based on earlier events, in other cases based on times.
Now for the final two capabilities, they’re actually tasks from earlier capabilities, but with a twist. For Agent2Agent, that’s going to allow the agent to call a sub-agent, either using the same model or using a different model, via Google’s Agent2Agent protocol. For Noise, they’re going to throw in some errors and see how robust the agent is.
One other note about capabilities: while each task has a predominant capability for categorization, it’s really impractical to have a task with just one capability and not any of the others. I know that’s a problem we often encounter when we’re writing prompts, that we can’t categorize them cleanly, and I think the authors made the right choice here to prioritize realism over clean categorization.
Putting all of this together, here’s what their annotation and reviewing software looks like, it’s that open source package I mentioned earlier. They have user input in blue, some agent actions in purple, events in green, all in a flow chart. Then further down they have outputs from the model in a couple different forms, like the thinking and final response in the bottom-left, the list of steps and their components in bottom-middle, and the raw contents of a particular component in the bottom-right. Also on the left is the list of apps for the Mobile environment.
So here’s where we net out in terms of raw performance. A few things to note here.
One is, we’re using older models. Putting aside the fact that it’s a Meta paper so they have to throw in Llama 4 and even Llama 3.3, we’re only seeing GPT-5 and Claude 4 and Gemini 2.5. Every one of those has meaningfully more skilled versions available today, with particular focus on agentic work. So I wouldn’t be surprised if SOTA models now did meaningfully better on at least the first two capabilities.
Second, you’ll notice Time has universally poor performance. I think that’s a bit artificial, because by definition all tasks except Agent2Agent only used one agent. For time-sensitive work, you’d definitely want one orchestrator keeping an eye on the time while sub-agents work. What they observed with the Time tasks is that because models can only process one thing at a time, they’d often be in the middle of something else when some crucial event came and went. It’s especially tough for the thinking variants of some of these models.
Overall though, I like that there’s room to move on most of these capabilities, I like that the prompts are human-made, and I like that they’ve made the tasks more real by incorporating elements like time and ambiguity. It’s a real test of agent utility.
Continuing on the realism trend, I like that they reported out cost and wall time too; if you’ve been following any conversations about Openclaw and the like, you know people are complaining about burning through tokens or taking too long. Cost is also quite relevant for enterprise, wall time less so since jobs may run in the background. But for the Mobile environment that reflects consumer usage, wall time is definitely going to matter, maybe as much as quality.
So on cost, there’s a pretty clear trendline, which only Grok is really below. Not a surprise.
On time, it’s not surprising that models can beat humans on a lot of tasks, but I do take issue with this graph. Notice the title is “Time per Solved Gaia2 Scenario”. It’s defensible to exclude the failures, since a lot of them probably involved looping and artificially long times, but if you only report on the successful tasks then you’re biased towards the easy ones. By contrast, since humans are able to solve pretty much all these tasks, you’re going to get the full spectrum of difficulty or inherent length in those human results. This is also why some of the most successful models seem to take the longest, like GPT-5. So the most I would take from this is that models definitely can be faster than humans - nothing more.
Now for the bottom graph, calls and tokens of course are the drivers of cost and time ultimately, so it’s good to look at both; you want to know what to change in order to alter your end metrics.
For LLM calls, for a given amount of performance you want fewer of them, because each call is going to add time, even for the same total number of tokens. One trend here is that thinking variants make fewer calls and also do better. They’re going to use more tokens per call, since using extra tokens to think is by definition what thinking models do, but apparently the thinking actually does result in better thoughts and actions. If you look back above, you’ll see thinking variants cost the same or less and take similar amounts of time.
One related point they mention is that which apps the agents use and how often is pretty similar across all the models. So the difference really is the quality of thought, how wisely you’re using the apps and what exactly you’re doing with them.
Do note that the x axis is logarithmic on the output tokens graph on the bottom-right. Also it’s not shown here, but they note all models flatline past a certain point, like scaling up tokens indefinitely is not going to bring better performance.
So we looked at scaling the LLM calls and scaling the tokens for the one agent. Now we look at scaling the number of agents.
They’re using the Agent2Agent protocol, shown top-left. Multi-agent stuff is still pretty new so I haven’t seen much of this protocol in the wild. I will say though, I prefer the terms “orchestrator” and “sub-agent” compared to “Main-Agent” and “App-Agent”. Anyway, if you haven’t been on any paper talks where we’ve discussed agent swarms, there are two advantages: one, you can do work in parallel and thus save on wall time; and two, having a sub-agent do a task keeps the context of the orchestrator clear. Given the push on long-horizon tasks, keeping the orchestrator focused for longer is a big deal.
Now they’ve done two experiments here on multi-agent setups. For the first one, the table on the top-right shows what happens when you upgrade the orchestrator model and the sub-agent model. In this case, they’re making a major upgrade, from Llama 4 Maverick to Claude 4 Sonnet. In both cases, performance approximately doubles, and the effect seems to be independent. My hunch is that you always want the smartest orchestrator and that you should pick the right model for the sub-agent’s job, like if it’s simple you can pick a cheap model but if it’s tough you pick an expensive one.
The second one, shown in the graphs on the bottom, is the more interesting one. The goal here was to control the amount of delegation the agent could do, with this ratio “r”. r = 0 means it’s just the one agent, using all the tools directly. r = 1 means the agent must use sub-agents, it can’t call any tools directly. And then r = 0.5 means half the apps can be used directly, half have to be through sub-agents.
They found two things. One, the more you can delegate, the fewer tokens get used overall. You might think more communication overhead would lead to more tokens, but really what happens is that keeping context clean and work compartmentalized allows all the agents to focus and work more effectively. For example, since the orchestrator doesn’t have to juggle as many tools, it makes fewer syntax errors and enters fewer loops. It’s a stronger effect in weaker models, and the communications overhead is still there, so it can become a more marginal trade. But it still holds true with Sonnet at least.
The other finding is that it’s not always obvious what the impact on quality will be. For Llama it helps to delegate in terms of pass@k, but for Claude it hurts, at least for this task in this framework. I do wonder how generalizable this finding is, since models are trained with certain scaffolds and expectations about sub-agents now. Like if you open Claude Code right now, Opus 4.6 is going to expect certain sub-agents available. And modern models likely have their own preferences on how much they want to outsource vs handle themselves, whereas when Llama 4 came out agents were only just becoming a thing, so they wouldn’t really have figured into training.
My Takeaways
Swarms are the future
As the authors point out, a variety of subagents is optimal, for cost but also for latency
How agents work together and communicate is nascent
First time seeing Agent2Agent in the wild
Authors point out that many models are trained to work alone, maybe not good at organizing other agents (just like many humans are bad managers)
On the flip side, models trained to work in a swarm of other agents with models in their family may be harder to benchmark in neutral tooling or with other models as agents
There is room for a GUI version