Originally presented as a live talk on January 21, 2026
Background
Let’s start today with a look at the lab behind this paper, namely NVIDIA. As we all know, NVIDIA is practically synonymous with AI these days. Here’s a graph from Epoch AI showing the total amount of AI compute capacity sold, starting in Q1 2020 and ending in Q4 2025. They normalize the amount of compute to equivalents of the H100, an advanced NVIDIA GPU.
While there are other chip makers in the game, all of them combined have sold just over 30% of the world’s compute. The remaining 70% is all from NVIDIA. It’s no surprise then that NVIDIA is the most valuable company in the world, and has been for some time now. It’s also no surprise that Alphabet, the parent company of Google, recently became the second-most valuable company in the world, in part due to their tensor processing units or “TPUs” starting to seriously compete with NVIDIA’s GPUs.
Relatedly, Amazon is the 5th most valuable company in the world as of writing. All those companies are worth $2-4T apiece. AMD is around $330B and Huawei is estimated at $100B, although they’re not a public company.
Anyway, that stock value is largely pricing in future sales, with a price-to-earnings ratio of around 46. For Alphabet is about 33, similar to Amazon and other tech companies like Microsoft. AMD’s P/E ratio is over 100! So the market expects lots of NVIDIA chips to be sold in the future, but also expects some other players to make a dent.
So where is all that demand for AI chips gonna come from? Well part of it is definitely training, so that’s the big foundation model labs like OpenAI and Anthropic and xAI that don’t make their own chips, i.e. excluding Google and Amazon. But a bigger part of the demand is inference, i.e. people using models. Even OpenAI and Anthropic spend a significant amount of their total compute on inference, since their revenue is roughly proportional to how much use their models get. But anyone who’s only serving models, or who maybe is doing some post-training, is using zero or nearly zero training compute, and that’s going to be the vast majority of organizations. In business lingo, the total addressable market or “TAM” of inference compute is bigger than the TAM of training compute.
So if you’re NVIDIA, you of course want to drive all forms of GPU demand, but the business wisdom here is to “commoditize your complement” - to make the other stuff in your bundle of goods as cheap as possible to drive the overall cost down, which increases demand due to lowered costs, but by definition doesn’t hurt your prices.
The most famous example of this is the PC market from the 90s. Microsoft, being the maker of Windows, wanted everybody to have a PC and to buy a copy of Windows. But PCs used to be specialty items, with differences in quality and architecture etc. In that non-commodity PC world, consumers had to spend more, and Microsoft had to make different versions of Windows to work on those differentiated PCs. So Microsoft worked hard to commoditize the PC market, making everything standard for Windows and forcing manufacturers to compete primarily on price.
NVIDIA wants the same thing. They want AI to be a commodity, they want model providers to compete on price, they want the inference market to grow. And one way they do that is by permissively releasing open-weights models, which gives anyone with a GPU a no-cost way to use AI, thus driving up usage. That setup isn’t going to compete with SOTA models on raw performance, but they can compete on performance per price.
NVIDIA has released a lot of models, including a lot of fine-tunes of other open-weights models like we’ll see in this paper, but their main family of releases is called Nemotron. They’re on version 3 of the series, all completely trained from scratch. They have released the Nano size, which is a 30B mixture-of-experts model with 3B parameters active, and plan to release two larger sizes soon. You can also see a post-train of Qwen3 here, for use as a reward model.
Now while the Nemotron series will compete with the likes of Qwen3 for the open-weights SOTA crown, NVIDIA has also released a lot of smaller models focused on reasoning and tool calling. These small language models or “SLMs” are great for enterprise work where the logic is custom but often not complex. For example, it doesn’t take PhD-level reasoning to understand how to join a couple of tables in a company’s database, but it does take understanding and implementation of that business logic, and familiarity with SQL or perhaps a specific analytics tool.
NVIDIA made waves in certain circles last year with a paper called “Small Language Models are the Future of Agentic AI”, laying out this exact case theoretically but not running any experiments. Here’s an excerpt from the abstract: “Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice.”
In case you didn’t catch that, what they’re saying in layman’s terms is that LLMs are usually overkill in enterprise work, and that if you really need that SOTA LLM level of intelligence, you can just call it when you need it.
The featured diagram in their paper shows two approaches to agentic systems. On the left is one language model for the user to interface with that is also calling some tools directly and also employing another language model as needed. On the right is a language model for the user to interact with, which then offloads coordination work to a controller, which then uses tools and the other language model. Keep both examples in mind.
I also should quickly describe the key benchmarks the paper uses. One is HLE, which hopefully we’re all familiar with, but if not it’s basically hard reasoning with optional research tool use.
The next one is FRAMES, by folks at Google. It involves reasoning but leans much more on tool use, for checking facts and doing research. Basically a RAG benchmark.
The last one is Tau Bench, or more specifically its successor, Tau-squared Bench. This one simulates conversations between users and agents in retail, airline, and telecom settings. You can see a telecom example here and some stats from each domain below. This benchmark is by some folks at Sierra, a big customer support agent company, and includes one of the authors from SWE-bench.
The Paper
Okay, so here’s what they’re up to in this paper. They’re basically applying the theoretical argument they made in the Small Language Models paper and training Qwen3-8B to be an orchestrator, which will talk to the user and use tools plus other models to provide an answer.
I want to note a few details here. As the graphic notes, they give a positive reward for correctness, a negative reward for cost, a negative reward for wall time, and a positive reward for any additional user preferences. In the example the user’s preference is also about cost, which makes it kind of a lame example honestly, but you could imagine other preferences like sticking to basic tools for maximum determinism. They use GPT-5 as a judge and GRPO as their RL algorithm.
The second detail to note is the three categories of tools and their partial lists of examples:
Basic tools, which is what you normally think of when discussing tool use with LLMs - any sort of command or API or MCP etc. If it’s an external service, the orchestrator will get information about its cost per query
Specialized LLMs, which are models either trained or prompted to do specific stuff. Like the Qwen Math series or Codestral for example, those are versions of Qwen and Mistral that have received extra training in math and code, respectively. In my view this is somewhat of a fake category, except for coding specialist models, which a lot of folks still regularly release
Generalist LLMs, which are just other models the orchestrator can use like you or I can. These “tools” are going to be the most expensive
The researchers made prompts that naturally require multiple steps to address, giving the orchestrator many opportunities to pick the optimal tool. We’ll see more about that on the next slide.
So being NVIDIA and having basically infinite access to compute, they made a lot of synthetic data. Here they track an example of creating synthetic data for a single domain, “movie booking”.
First thing they do is generate a database, including the fields like movie time and total price, and of course the data within the database.
Then they generate tools that would operate on that data. Here they give examples like a cancel tool and a refund tool.
Then for a random sample of tools, they generate “intentions”, or what we would call “prompts” - things the agent should do with the tools and the data.
After that they turn intentions into tasks by randomly sampling database rows to go with the tools and intention, and then solving the task. That golden set of functions calls and key information, plus the prompt and data and tools, is one sample in the dataset.
They also do some enhancing and filtering, like adding constraints and checking that the task actually requires at least one tool call to solve.
Once they have that complete eval dataset, they can check an agent’s trajectory against the golden tool calls and key info, in addition to checking the final response or database change etc.
One last thing they don’t show in here, but which factors in at test time, is a random pricing schedule for all the tools. That’s not “golden” info since prices may change in the real world, so it doesn’t go in the eval dataset directly.
So with this dataset and training in mind, they RL Qwen3-8B into Orchestrator-8B, with results shown here.
It’s a little bit silly, but they show comparisons between Qwen3-8B and more powerful models with no tool access or just basic tool access, i.e. no LLMs available to call. Obviously the bigger models blow Qwen3-8B out of the water generally.
The real results are in the bottom row, with the full suite of tools and with Orchestrator listed. Here we see the power of SLMs. Orchestrator has the best performance, the lowest cost, and the lowest latency. On the next slide we’ll see how and why, but I want to call out a couple other results in the bottom section.
First, Qwen3-8B is already a good tool-user. The clearest way to see that is the Tau-squared Bench results, where Qwen beats GPT-5 and isn’t far off Claude Opus 4.1.
Second, the cost is the biggest improvement from Qwen to Orchestrator. So Orchestrator is using better tools, but the stronger lesson it apparently learned is to incorporate pricing information. And latency I view as a consequence of that, since cheaper tools are also likely to be faster tools.
This here is a good summary of how Orchestrator beat out the other models. Basically, it learned to use the right tool for the job. I put the cost numbers relative to Qwen3-8B at the top for easy comparison, so for example Orchestrator is only a third the cost of Qwen3-8B. And remember, Orchestrator wins on quality against every other model.
Since Orchestrator is a Qwen model, let’s compare relative changes from Qwen3-235B-A22B since they don’t give this same breakdown for Qwen3-8B:
GPT-5 reduced by over half, mostly replaced with GPT-5-mini - this is probably the biggest cost saver. GPT-5 also employs this strategy. In fact, OpenAI already knows this strategy - that’s what the router in ChatGPT is for!
Far more Qwen3-32B use, unclear why but if I had to guess it’s because Qwen3-8B knows Qwen3-32B is its bigger brother and thus should be able to help. Certainly it’s well known that models prefer outputs by other models in their family.
More local search, using cheap lookup instead of thinking longer themselves or asking another model - both expensive in tokens!
Note that this is not at the expense of web search
This is very relevant for FRAMES since it’s a RAG benchmark, and fairly relevant for Tau-squared Bench since it’s in a corporate environment
No Llama use, although the real question is why the big Qwen uses Llama at all - none of the other models use it really
So that’s the formula for success: use tools instead of models when possible, use small models instead of big models when possible, and always use the best model within the size you pick.
My Takeaways
We will continue stacking models - “It’s models all the way down”
Very cheap and small models will be ubiquitous, similar to how very cheap and small chips are in things as simple as USB cables
It will be rare to interact with a bare tool
Models still aren’t that business-savvy
As demonstrated by Vending Bench
ToolOrchestrator is a cousin of Claude Code
I don’t touch a command line anymore, Claude Code does it all. Let it decide what resource to use.