Originally presented as a live talk on February 10, 2026
Background
Let’s start by recapping recent Chinese model releases and progress.
The top two labs in China are DeepSeek and Qwen. DeepSeek produces the most original work, like with R1 at the start of 2025 or with recent hardcore engineering papers. Qwen produces the most work overall, with models in most varieties you could ask for: small or large, dense or MoE, general or coder, text or image, etc. Qwen models are the default choice for most work involving training.
Below that S tier is the A tier, companies putting out highly capable models but not quite matching the originality of DeepSeek or the volume of Qwen. Here I would put Z.ai (the makers of GLM), Moonshot (the makers of Kimi), and MiniMax. We’re talking about the recent Kimi release today of course, but by sheer coincidence both Z.ai and MiniMax have released major updates today: GLM-5 and MiniMax M2.5. Really incredible pace by all the big Chinese labs.
Now we’ll see some Kimi benchmarks that compare it to recent SOTA, but the vibe I get is Chinese models are only a few months behind the best American models. And for an increasing share of use cases, “recent SOTA” is good enough. So I think it will become increasingly common to be using Chinese models, although more often through a third-party host like Fireworks rather than directly calling the China-based servers.
Now in terms of focus, I think the lightning success of Claude Code non-code use and then more recently OpenClaw gives a good idea of where we’re heading and what the underlying models in agents will need to train for. We’ve reached an inflection point where models have enough smarts and skills to do most generalist computer work: browsing, researching and the like. Domain-specific programs, spreadsheets, decks etc I think need a lot of work or specialized harnesses and training data for now, but everyday computer use is happening now. As we’ll see in this paper, those particular skills but also the general approach to agentic work are hot areas of research and progress.
Finally, I’d like to harken back to a paper I covered last month called ToolOrchestra. The good people at Nvidia took a pretty lightweight model, Qwen3-8B, and trained it to tackle complex queries by calling other models and some dedicated tools. Even a simple model trained for the specific task of orchestrating bested all other models, even far larger and smarter ones.
So it seems orchestration per se is a skill, not something we can take for granted as part of general intelligence, and that it has significant advantages for cost and latency in addition to at least marginal benefits for quality.
The Paper
Let’s talk about architecture first. This is a big mixture of experts model, as all frontier models are nowadays. 1T is on the high end, maybe the biggest open weights model available today. 32B active parameters is proportionate. 61 layers is pretty normal, the more layers you have the more thinking you do but also the longer latency is. There’s also a practical limit on depth because signals you get during training get attenuated over all those layers, so if you have too many of them you basically never teach the earlier layers anything.
Another stat or ratio to check out is the number of experts compared to the number of selected experts. That’s the sparsity, which is 48 here and is on the pretty high end. I also want to call out the 256k context length, also on the high end - 128k is more common and is also where Kimi K2 topped out at. So really big, super sparse, pretty long context - that’s Kimi K2.5 from an architecture POV.
Now Kimi K2.5 is actually an update to Kimi K2, meaning they took the K2 base model and trained it more.
Specifically, they got a vision encoder working first, then added it to K2 and trained the new system for 15T tokens. That’s on top of the 15.5T tokens Kimi K2 already got, meaning we’re just over 30T tokens total. The third stage here is about lengthening the context out from 32k to 256k, which happens in steps and adds another few hundred billion tokens. So all told it’s like 32T tokens to make this beast.
In the paper they talk about how the conventional wisdom is to train only on text for most of the time, then throw in a heavy dose of vision tokens at the end, like a 50/50 split. In the extreme, you can even take a fully trained LLM and add a vision encoder on afterwards, onto a model that never knew it was going to be multimodal.
What the Kimi folks did instead is to start training both vision and text from the start, but with a smaller dose. Specifically, they do a 90/10 split between text and vision tokens the whole time. They also don’t group their training data by modality, just by skill, so in post-training you’re getting a sprinkling of vision throughout.
Also note the types of data in each stage. First off, you’ll note they do video as well as image, although no audio. Also they’re looking at a lot of computer use imagery, definitely angling for a computer use agent here.
For the SFT after this pretraining, they only spend a single paragraph on it, but they mention lots of synthetic examples from K2 and some “in-house expert models”, and also some human annotation. No sense of numbers here sadly.
Now let’s talk RL. First we’ll look at vision, then the agent stuff.
They’re training in a lot of capabilities: OCR, captioning, segmentation, counting, grounding etc. The higher-level goals of understanding and processing sit on top of that.
In terms of how they reward, they do RLVR where possible, which covers many cases for vision. Where they can’t do RLVR, they do rubrics, although they don’t use that word. I believe the rubrics are generic rather than prompt-specific, but they don’t say. Kimi K2 is the LLM judging the outputs.
Now again, they’re not sharing anything about their data, but they do show the effect of RL. They see improvement across the board on their vision benchmarks, in pretty classic-looking curves of training vs performance improvement.
So that’s vision taken care off, which was mostly par for the course other than their insight about when to start vision in pretraining. Now we move to the more forward-looking stuff, which is agent swarms.
Here’s their overview of an agent swarm. The examples are not especially inventive but you get the gist: an orchestrator inventing subagents to give tasks to. The orchestrator manages while the subagents work. The user is only ever talking to the orchestrator.
Now for training, we’re only ever rewarding the orchestrator; it’s going to take credit for the final outcome, as well as some other decisions we’ll see in a bit. Because the subagents are “frozen” like this, we can think of them as tools in the environment rather than as part of the main agent, the orchestrator. Of course below it all every subagent was also Kimi K2.5 in this case, but like with the ToolOrchestrator paper you can imagine the subagents using different models, including API-based ones that you don’t have the ability to train anyway.
As an aside, I find it amusing in a Dilbert sort of way that the manager at the top is the only one who gets rewarded. I suspect we will discover many organizational principles and tropes don’t depend on having humans in the loop.
Now as for those rewards, I came into this paper not knowing what to expect. I think we’re just at the beginning of multi-agent RL, so I take this as just one idea rather than gospel.
Anyway, as you can see at the top they reward three things: the outcome, r_perf, which is the most important thing; the creation of subagents, r_parallel, which they call “instantiation”; and the completion of subagent tasks, r_finish. R_parallel and r_finish together are a good example of how tough reward design can be. On the one hand, you do want the orchestrator to spin up subagents rather than doing work itself, but on the other hand it’s not helpful to spin up subagents and just let them hang there.
By the way, even OpenAI can make mistakes in reward design. It recently came out that due to over-rewarding of tool use, a prior version of ChatGPT would open the calculator tool, do 1+1, then close the tool and answer the user’s query. Apparently that was happening on something like 5% of all queries at one point, which is huge in absolute terms.
Anyway, how much of each reward you need in order to balance the other out and avoid reward hacking is an empirical matter, hence the weights lambda_1 and lambda_2. Over the course of training, lambda_1 starts at one and goes to zero as the orchestrator durably learns to call subagents. Lambda_2 starts at zero and goes to one as the model gets better at giving work to subagents, so the tolerance for poor subagent use goes down.
The second formula calculates how parallelizable a task is by measuring the critical path. If you’re not familiar with that term, the critical path is the set of steps that determines the length of an overall project. So let’s say I want to make dinner, and my entree needs 45 minutes in the oven. The critical path is likely just the time it takes to prepare the entree, then that 45 minutes in the oven, then any cooling and plating time after. Other courses can happen in parallel, like making the salad while the entree is in the oven, so those are not on the critical path.
Bringing it back to agent swarms, you have hit maximum parallelism if the overall time is equal to the critical path time. To measure the critical path time, for each round of decisions, you find the longest-lasting individual subagent task. Then you add those up for every round to get the critical path of the entire task. If your time matches the critical path time, you get full points on parallelism.
Note that the prompts they’re going to train on don’t say anything about parallelism per se. There are tools implying parallelism is a good idea, but the system prompt or whatever doesn’t say like “make sure to divide up your work”.
So here’s the payoff for all this reward design. On the left we see accuracy on their training set increasing with compute. They don’t give exact amounts of training data but they hint it’s in the tens of thousands of examples. It’s all synthetic prompts, emphasizing “wide search” (many independent tasks) or “deep search” (multiple branches and then aggregation).
On the right we see the level of parallelism, which they measure as the number of overall steps divided by the critical steps we saw before. So if all your steps are critical, your parallelism would be 1. Note that different tasks vary in their maximum parallelism, like some can be totally run in parallel while others may have only a few parallelizable steps. But since we’re seeing the same tasks many times, overall you can accurately draw conclusions about changes in parallelism during training.
Anyway, two things to note: one, the x axis does not start at zero or one even; you’re getting good parallelism right out of the box. Two, the trend has two phases: flat–ish in the first part, up in the second. The way I interpret that is the model starts its learning on more generic things like designing better individual agents or understanding agents results better. Once it hits a ceiling on correctness, it then has to learn to improve parallelism if it wants to get every available reward. Like it only learns later on how to reason about parallelism, which it is not naturally going to be skilled at since it’s not naturally part of the pretraining data.
Of course parallelism per se isn’t valuable, but the effects are. You could expect better quality as the orchestrator maintains clean context and abstracts tasks away to subagents. You could expect lower cost from using cheaper models, or from more efficient thinking and thus fewer tokens to pay for. But I think the most sure effect is savings in time, “wall clock time” as they say. There’s gonna be two reasons for that: one, obviously parallel agents can do multiple things concurrently; but then two, models get slower as their context length grows.
Here’s a graph quantifying that speedup, with single agent vs agent swarm performance on the WideSearch benchmark, which requires gathering information from a wide variety of sources. Swarms are always significantly faster, but the trend seems to grow superlinearly. So it seems like if we want agents doing all our work for us in the future, they’re gonna be working in swarms, just like how most work is done by companies instead of individuals these days.
Then just a couple other quick remarks here. They have a word cloud on the left showing all the different subagents the orchestrator made, which are all pretty similar in my opinion and also have a lot of duplicates, like the most common one is “Biography Researcher” but they also have “Biograph Investigator” and “Biographical Researcher”. Kind of a lazy chart honestly, and they don’t provide a table of the data. I asked Gemini to extract and dedupe and group, and it gave four high-level themes: research, technical/coding, content processing, and orchestration.
On the right they compare swarms with a technique for managing a single agent’s context. Basically they’re trying to see how much of the benefit of swarms is just due to subagents abstracting work away and keeping the orchestrator’s context clean. If you only have one agent, you can basically have it remove the intermediate work from its own context. Cleaning up context like that helps for sure, but the orchestrator-subagent setup is still superior. The authors claim it’s because the orchestrator can proactively plan which context to keep and which to give away to the subagent, whereas retroactively wiping intermediate work may erase stuff the model was planning to refer back to.
Personally I think the swarm results are kind of a lowball measurement, and that the potential for swarms is much higher, whereas the context management stuff for a single agent is more like an optimization closer to the inherent ceiling. Both valuable but not comparable in the long-run.
Finally, here’s where we net out. Kimi K2.5 is, according to benchmarks, on par with Opus 4.5, GPT-5.2, and Gemini 3 Pro. That’s an extraordinary claim, and while it’s increasingly difficult to compare SOTA models, I think it’s at least arguable. I have seen one respected ML researcher say he has switched to Kimi from Opus, although he always uses one in combination with GPT 5.2. And who knows now with Opus 4.6 and GPT 5.3 Codex already out.
On the data side, one unfortunate but likely true theory as to how Kimi pulls even with SOTA on benchmarks is that the Chinese models are distilling from the American models. In other words, the Chinese labs are creating synthetic data from the American models and then training on it. The easiest evidence for this is if you ask the Chinese models who they are, you often hear they are GPT or Claude. Personally I suspect that’s part of why we don’t hear much about the data in many of thee tech reports.
Anyway, I encourage you all to experiment with Kimi. It’s available on OpenRouter. If you want it for coding you can use Opencode, which is like Claude Code, or you can hack it into Claude Code directly and it works pretty well. But it’s also supposed to be good at creative writing, so don’t restrict yourself.
Last note on this table: I’ll call out HLE and SWE-Bench Pro as Scale benchmarks, and GDPVal as basically identical to our Remote Labor Index benchmark, which I think is better and seems to be harder for models to crack.
My Takeaways
Number of agents is another axis for scaling
Swarm training aka MARL is in its infancy
Setups, rewards etc
But also data
Home-runnable models may be good enough for generalist work by EOY