ZAYA1-8B Technical Report
or, Swapping Space for Time
Originally presented as a live talk on May 13, 2026
Background
I want to start with a high-level view of where AI is headed and what that looks like to the observer on the ground here, the folks living it like you and me.
All the curves so far say we’re on track for some super powerful AI. I don’t necessarily want to call it AGI, or ASI, or anything else like that, because frankly the labels are not helpful. There is no bright line for these events, and there isn’t even really an objective definition for such labels. People put a lot of stock in the Turing test, we blew past that pretty quickly in the modern AI era, and since then it’s been more instructive to follow specific and objective measurements, like the METR graph or the scores on Humanity’s Last Exam.
And if you watch the trend on those specific and objective measurements, you find that progress at the frontier has not stopped. But at the same time, people are always pointing out weird and dumb behaviors, things that models do that a human of that intelligence never would. Similarly, a lot of our specific and objective measurements cover only a partial set of characteristics we as humans would expect from a strong intelligence. So people call machine intelligence “jagged”, and part of that jaggedness is because researchers focus on improving the abilities they can best measure.
For the big labs, who want to build truly general intelligences, that jaggedness is something to fix. Anthropic wants Claude to be a good philosopher. OpenAI wants GPT to provide good relationship advice. xAI wants Grok to praise Elon more believably.
But there is another set of labs leaning into the jagged intelligence, spiking on code and reasoning and letting the more well-rounded skills languish. With a much narrower focus and different success criteria, there may indeed be room for players besides the big boys. Newer labs like Arcee and Poolside are positioning their models as agent-first, recognizing that enterprises don’t much need amazing writing or impressive creativity - just sharp analysis and reliable decision-making.
One of these is Zyphra, the makers of ZAYA1-8B.
Unlike many so-called “neolabs”, outfits like Thinking Machines or Safe Superintelligence that can trace their lineage back to the big labs, Zyphra is on its own evolutionary branch. Founded in the Bay Area in 2021, before ChatGPT and the generative AI explosion, they have been working on neural networks and architectural innovations for some time now. For example, they released a 7B parameter model called Zamba in 2024 that used Mamba rather than the traditional attention mechanism. They also released a model called ZUNA trained entirely on EEG data, for brain-computer interfaces. And just for kicks they released a music generation model called Zonos. So they’ve got range.
Staying on the divergent evolutionary branch idea, they also live in a different hardware ecosystem: they train and run on AMD, not NVIDIA. That’s why AMD led their unicorn round back in October 2025, and why their cloud offering stresses “Deep integration across the AMD stack”. They offer inference on Kimi, DeepSeek, and GLM, as well as their own model of course, and they offer training, which I assume is the same infrastructure they used to train their models. So they’re a full-stack AI company.
Flipping to the science side, we need to talk about a few bits of prior art that ZAYA1 makes use of.
By now we’re all familiar with reasoning mode or reasoning models. These are models that use a lot of tokens to “think” about a solution before giving a shorter, condensed final answer. The first of these was o1 from OpenAI, R1 from DeepSeek made the whole method public, and now pretty much every model comes with a thinking mode.
That ability to think more, to spend more compute and more tokens on a problem to get a better solution, is called “test-time compute scaling”, or TTC scaling for short. “Test-time” is just a fancy term for “at the time you’re using the model”, so basically the term is saying you can scale up the amount of tokens generated just like you can scale up the number of parameters or the amount of pretraining data.
The version I just described, where a model authors a long CoT before responding to the user, is serial TTC scaling - it’s about a single response growing longer.
You can also have parallel TTC scaling, where you have the model generate multiple responses to the same prompt, then use some technique to review or aggregate those responses. A simple example would be majority vote, where you get k responses and just pick one that gave the most common answer.
This is actually what people thought o1 was doing behind the scenes before R1 came out. If you were paying attention back then, you may remember talk of Q* and Project Strawberry. The rumor was that o1 basically searched a tree of possible responses and had a way to learn which next steps were most promising to further explore. You also may remember a paper we did involving Monte Carlo Tree Search (MCTS), that again would be more like parallel TTC scaling.
Anyway, the point is that you can go wider in addition to going longer.
Here’s an example one such technique, called Parallel Coordinated Reasoning (PaCoRe).
The authors start from one insight: the limit to serial TTC scaling is the model’s context window. If we really believe in scaling, we need to break past that limit.
So they do two things. First is, they have multiple reasoning trajectories in parallel. So that’s already doing some parallel TTC scaling.
But second, instead of just aggregating and picking from the reasoning trajectories, they compact them - basically use the model to boil them down to their essentials - and then feed them back into the model along with the original prompt. You can run that many times before all the compactions take up too much of the context window. And then at the end, you do whatever aggregation system, or you just have the model pick the one it thinks is best, etc. So you’re getting the wisdom of many millions of tokens, but squeezed into one context window.
We will see PaCoRe in the ZAYA1 paper, but as training data - the input is a problem and a set of reasoning trajectories, and the ideal output is a compaction.
ZAYA1 does something similar to PaCoRe, a technique they call “Markovian RSA”. It’s a combination of two papers. One is called “The Markovian Thinker”, and the other is called “Recursive Self-Aggregation”, or RSA.
This graphic is from The Markovian Thinker. If you’re not familiar, a Markov process is a process where all you need to know is the current state in order to predict the next state. For example, if you throw a ball in the air, you don’t need to know where it previously was in order to predict where it’s going - you just need its mass and its velocity and gravity. By contrast, text generation from an LLM is not a Markov process, since the next word depends on all the previous words that came before it.
If you’re again trying to scale test-time compute past the context window, that’s going to cause problems as earlier text leaves the context window. But if you can make text generation Markov, then you don’t have to worry about earlier text leaving the context window, because your prediction only needs the text from the step before.
As you can imagine, that’s probably not going to work at the single-token level, like it’s gonna be impossible to have one token have everything you need to predict the next token. But if you break up a CoT into bigger units, like steps or sections or lines of inquiry, then maybe you have a shot.
And in fact that’s what they did here. They have a prompt and the start of a response, composed of two steps. One prompt plus two steps is one chunk. Then they summarize the two steps, feed the prompt and the summary into the model, and generate another step. That’s chunk 2. They keep going with this over and over until in the final chunk, chunk L, the new step has the final response. As long as the summary is good enough, you have a Markov process for generating text.
The other half of ZAYA1’s “Markovian RSA” technique is the RSA, the recursive self-aggregation.
The basic idea is to generate a bunch of trajectories, sample a few of them, and synthesize something from that sample. If you do that over and over, eventually you should converge on the right solution.
The graphic here I found a bit confusing at first, so let me walk through it. First we have a model and a question. We generate a population of solutions, where each solution has a CoT and a final response. In this case there are eight solutions in the population.
And then eight different times, we sample from the population and synthesize a new solution from the sampled ones. So like in the first row of blue, we have sampled solutions two, six, one, and four. The synthesized solution from that combination is T_1. Since we’re doing that sample-and-synthesize process eight times, we again end up with eight solutions.
You can do that over and over again, as many times as you like. And then in the final step you aggregate them somehow, like with majority voting or by just having a model pick the best one.
You can see how this is complementary to the Markovian technique on the previous slide, which is just a way to shorten any one CoT. Put ‘em together, and boom - Markovian RSA.
The Paper
Let’s start with the architecture. Fundamentally it’s still a transformer, so we have our three transformer components:
Embedding, for turning words into vectors
Attention, for understanding all the input
Feed-forward, for thinking about that input
You may not recognize the attention and feed-forward bits given all that’s going on here and given some of the nomenclature, so I’m going to break out each one.
This part here is the attention. Actually it’s attention with an extra step right before it, something they call Compressed Convolutional Attention (CCA).
I don’t want to look much at the guts of the thing, but the upshot is you end up with compressed versions of the normal inputs to attention. Like normally in attention you split up the input into queries, keys, and values, q k and v. That’s still happening here, BUT you do all this stuff on the right to make q k and v a lot smaller. To even out some of the quality loss, you basically look at the previous token in addition to the current token when you do that compression. That’s the convolution part.
This part here is the feed-forward network. It’s a mixture of experts, but they’ve show the individual components broken out instead of just one block that says “MoE” on it.
So you can see there’s a router that picks the expert, and there are 16 MLPs, those are the individual experts, the little sub-networks. That’s actually quite low; even small MoEs these days will have 128 or 256 experts.
The trick is specifically in the router portion, which they’ve shown is actually four components that I again don’t want to get too far into. Basically they’ve made the router smarter, which lets it spread the load in training across all 16 experts more evenly, which improves performance.
It also allows them to pick just one expert, which again is out of step with the rest of the field; usually there are multiple experts active, and one of those is usually what’s called a “shared expert”, i.e. it is always active.
Here’s a summary of the whole thing. With 40 of those transformer layers we end up with about 8B parameters, less than 1B active, for a sparsity ratio of about 11. That’s pretty normal at this size, although bigger MoEs tend to be more sparse.
Note also the call-out in the last row of AMD hardware. You would never see that mentioned if they were training on NVIDIA hardware.
Overall, while they’re making a couple unusual choices, they’re not the stars of the show. The main thing is the size, as we’ll see when we get to the benchmarks and look at what models ZAYA1-8B is competitive with.
Speaking of size: it’s not covered in the paper, but they have a 74B version in preview now. The finished version should be out soon.
Now, on to the training.
For pretraining, they do about 13T tokens, which is actually not a ton. For comparison, Qwen3.5 and DeepSeek V4 did over 30T. Part of that is probably down to language support - I think ZAYA1 only supports English, although it’s not mentioned - but I can think of two other causes.
One is synthetic data. For reasoning and code in particular, synthetic data is quite common, but you need a big operation to create enough tokens at sufficient quality to help out.
Similar deal for agentic trajectory data, which is by definition synthetic. They mention a few times that they haven’t trained ZAYA1 for use in agents yet, and agent trajectories can be a significant part of the pretraining corpus.
The other of course is that they’re a smaller operation, and it takes time to accrue a big pretraining corpus, whether scraped or generated. So they could catch up on performance with more time there.
Now for post-training, they provide this handy graphic showing all their stages: one for SFT, four for reasoning RL, and one for behavioral RL.
For SFT, they have three goals: one, get it to chat; two, lengthen the context to 131k tokens; and three, show it how to aggregate old chains of thought to produce a new one. That third one is key for ZAYA1 and relates to the RSA technique we saw in the background slides.
For the RL stages, they of course want to improve reasoning, but they also want to improve the various TTC scaling techniques. Like in the two blue boxes, they call out TTC, RSA, and PaCoRe as techniques the model will use and be rewarded for.
Now let’s highlight the data. We don’t get number of examples for SFT, but based on the token count and the maximum response length at that stage, I would guess around 20k. Then for the green box, it’s not in the graphic for some reason, but in the paper they mention about 85k examples. Then for the rest of the boxes, you can see they mention having 400 environments in the purple box, over 30k RLVR examples between the two blue boxes, and over 80k RLVR examples in the orange box. Overall that’s roughly 20k SFT, 200k RLVR, and 400 environments.
Also on the topic of data, they mention in a couple places how they try to balance out difficulty as the model trains and improves. We don’t need to get into all the stuff they do with the reward function to capture that and how they dynamically filter the data, but it’s good for us to keep in mind how valuable a distribution of difficulties is.
Finally, after all that training, we are ready for some inference. And that’s where ZAYA1 shines.
This tangle of boxes and arrows is their Markovian RSA, combining the aggregation and synthesis of RSA with the context window workaround of Markovian Thinker.
We can see our model producing N different responses, which by default they set to 16. Then they cut off the last Tau tokens of each response and throw away the rest, which they can do because the model has been trained to think in these Markovian chunks. Usually they set Tau as a percent of the overall response length, like half or less.
Next, they randomly sample C of them, by default 4, and they put em in the aggregation prompt. Feed that prompt to your model and you get a new response. And remember, you’re gonna do that N times so you end up with N new responses.
In theory you can keep doing that forever, but by default they go through two rounds of aggregation. They hint that it’s because they only do RL for a single aggregation step, and they claim they will train on multiple rounds in future work.
Now let’s see where we net out. This is the hero graph, where ZAYA1-8B competes with all these much bigger models. Of course many of these are closed models, so we don’t know how big they are or their architecture, but they’re likely all MoEs in the 100B-1T range.
For this model in particular, I think the shaded part, with Markovian RSA activated, is the right comp. And it seems like it’s holding up against these previous-gen models. Actually Sonnet 4.5 isn’t even that old, we’re only on Sonnet 4.6 now. So, pretty impressive.
Another way to look at performance is against more recent models from the same size range. Qwen3.5 and Gemma 4 are both pretty recent, and both have about 4x more active parameters, being dense models instead of MoEs. It’s apples-to-oranges but they’re in the same ballpark on net.
As we might expect given the focus on reasoning in training, ZAYA1-8B is the best for math and code and competitive on knowledge and instruction-following. The worst performance is on Tau^2 Bench, a recent agentic benchmark, which again is no surprise given where the researchers chose to focus. I suspect they could improve greatly here.
So since this paper’s primary focus is TTC scaling, of course they’re going to sweep those parameters and show how far TTC scaling can take you.
Here’s one view of that, just looking at overall tokens per final response you get and how it correlates with pass rate. It shows a typical diminishing-returns pattern, but take caution, because that y axis is already pretty high when they start the curve. Usually the last few percent are the hardest to get. I would have liked to see similar graphs for benchmarks with more headroom.
Here’s a similar look into a different benchmark, but with the different inference parameters laid out. Again, T is the number of aggregation rounds, N is the number of responses generated, and C is the number of responses sampled for aggregation.
With no TTC scaling at all, The model hits 32%. Adding just two rounds with a pretty short maximum response length barely moves the needle, adding 1%. It’s only when you really crank the tokens that you see big gains - 14% from quadrupling the aggregations rounds and doubling the response length, and another ~20% from doubling the number of responses and doubling the response length again.
I’ve added the rough multipliers on maximum token usage to the table, but keep in mind you don’t always use the maximum per response. So if you want to be conservative, maybe cut the multipliers in half for what actually gets used on average. It’s still a lot, like millions per final response on average for the highest TTC setting. Again, that’s going to be tens of dollars per response at Sonnet 4.6 pricing. This type of thing is only feasible for smaller models.
I also added ROI, basically the gain on the benchmark relative to the token multiplier. Weird pattern, I suspect somewhat specific to the benchmark, so I wish they did this for more benchmarks and maybe a finer-grained sweep.
So if Markovian RSA is such a great form of TTC scaling, can we just apply it to any old model?
Well they ran that experiment here, and the answer seems to be no, not really. Remember, ZAYA1 is trained to aggregate and synthesize solutions. It’s not a super common workflow, so at least at this size, models aren’t very good at it.
I do wonder if bigger models would work though, since they are generally more flexible. Again, an experiment I wish they did.
My Takeaways
TTC scaling may substitute for parameter scaling
If you don’t have the space, you can instead pay in time
Goes hand-in-hand with faster, cheaper tokens
Groq, Cerebras, Taalas
“Tokens too cheap to meter”
Could be promising for consumer hardware
Are small reasoning models “disruptive technology”?
Amenable to context engineering and orchestration
Ex. have a bigger model pick the final response
Ex. use deterministic checks to stop and regenerate bad solutions
Decoupling improvement from ML knowledge/tools/control of weights means more accessible innovation and customization
I’m not sure if the architecture mattered