Originally presented as a live talk on April 8, 2026
👂
We need to start with an overview of audio in AI, because there are some unique considerations and we haven’t really touched on the modality before.
First off, I want to distinguish the different types of audio input and output. For input, I think of it as three categories: speech, music, and ambient - like sounds from the environment, “noise” you might say, but without the negative connotation.
Music presents a fairly distinct set of challenges from speech and ambient, so I’m going to put that one aside. For the rest of this talk, we’ll be focusing on speech but also allowing for some ambient sound, like someone asking to identify a bird’s chirp or noticing that the person is at a crowded venue.
So sound in general has a few interesting properties. For one, it’s continuous and analog, like vision. That means we have to sample sound, with the equivalent of frame rate for video, and we have to capture it with dedicated hardware. That’s all very different from text, which is natively digital and doesn’t lose anything from compression or sampling or hardware limitations etc.
Audio is also hard to tokenize. Like with video, the raw data is too information-dense, and a naive tokenization strategy would blow out the context window in an impractically short amount of time. We don’t need to get into the details, but it involves compression and mapping to the closest token the model builders have built in.
Lastly for audio in general, there’s just less work overall in the domain: less literature, fewer models, fewer companies working on it etc. I think that’s mostly for the reasons above, that text is just much easier to work with than audio, and that text is the universal interface.
🗣
Now for speech in particular, there are further challenges.
One is the nature of spoken interaction: it is realtime, and it is dynamic. Like when you’re speaking to someone else, you generally expect them to understand as you’re speaking, to understand when you’ve finish, and to respond in short order. You also have to be prepared for them to jump in early, or to jump in early yourself if needed. And you expect them to understand easily when different people are speaking without relying on visual cues.
This is all very different from text-based interactions. If I’m chatting with a model, I can tolerate some thinking before a response. If I need to interrupt I can stop the response from processing, but conversely I know the model cannot interrupt me. And I’m pretty much only chatting one-on-one, although even if it were multiplayer like in a Discord, the model would get my username when it got my message.
So realtime responsiveness, interruption, and diarization (the term for tracking who is speaking) are all new challenges for speech models. And those challenges present constraints. For example, a realtime speech model cannot be the size of a SOTA text model - the latency is just too high. That’s why you usually see speech on the smaller versions of models, like Gemini Flash instead of Gemini Pro or GPT-4o instead of GPT-5.
But a lot of that is on the engineering side. On the data side, the biggest challenge to address is the stuff in speech you can’t really capture in text: the paralinguistics. That’s gonna include stuff like tone, pacing, volume, pronunciation etc. All the stuff that makes speech so much richer than text.
In all multimodal data, the emphasis is on the aspects that are inherent to the medium. The anti-example I often give for vision is a picture of a quantum mechanics problem - the difficulty of quantum mechanics has nothing to do with the difficulty of visually parsing the text, which is a very easy problem these days.
It’s the same thing for speech data. Like audiobooks are gonna be poor speech training data, because the text already lacks the paralinguistics. What helps is natural conversation, with all the variation we expect any time we talk to someone. It takes a lot of data for native speech models to get what all the paralinguistics mean.
We should also talk briefly about the two different architectures for speech models: native and cascaded.
Native means that the model goes directly from audio to latent representations, the inner stuff of the AI mind basically, and then back out to speech. Sometimes you hear this called speech2speech, voice2voice or audio2audio. Actually in this paper we’re going to hear “E2E” or “end-to-end”, which isn’t specific to audio, but does describe a model where there’s no intermediary or change in modality.
Cascaded means that the model is actually three models in a trenchcoat: one to turn speech into text, another to process the text into a response, and a third to turn the response back into speech. The first step is called ASR, automated speech recognition. The second step is usually an LLM these days. And the third step is TTS, text-to-speech.
Native is typically the best quality because it preserves the paralinguistics. Cascaded has to compress the speech into text, so you lose the richness of speech. And then also the TTS model often isn’t good at adding paralinguistics to the response. If you’ve ever talked to a phone menu, you know what I’m talking about.
Native is also typically faster than cascaded, because it doesn’t have to translate from speech to text and then text to speech.
However, native is also much less common as we’ve discussed, and it also has less language support. If you’re working in English, you have good options for native models, but for most languages in the world you probably have to go cascaded.
Cascaded can also be better if you need the smarts of a SOTA model. Since native speech models need to be approximately realtime, you can’t have parameter counts in the 100 billions or trillions. But with cascaded, each part is independent. This can also make debugging and guardrails easier too. So there’s room in the world for both architectures right now, but ultimately we’d like native speech to be more common, since we’re built for speech.
And actually I do think we’re headed in this direction. Bit of a spoiler, but as you can see from our Audio MC leaderboard, Google recently released an updated realtime speech model, Gemini 3.1 Flash Live, that may now be best in class.
We’re also seeing more adoption of speech with AI. A friend of mine works at Wispr Flow, which does dictation for AI, and they’re growing fast.
And I think when people imagine working with agents, they think about something like JARVIS from Iron Man or Cortana from Halo. Part of that is the autonomy and natural learning we discussed a couple weeks ago with our OpenClaw papers, but part of that I think is speech. So long-term I’m bullish.
Finally, I want to briefly touch on MultiChallenge, which Audio MultiChallenge is based on. MultiChallenge is a multi-turn benchmark that evaluates four skills:
Instruction retention - continuing to comply with earlier instructions
Inference memory - recalling details from earlier in the conversation
Versioned editing - iterating on a shared work product
Self-coherence - avoiding contradiction in the absence of new facts
The areas of investigation are still valid, and Audio MC is going to take inspiration from them, especially since speech is naturally multiturn.
Also, it’s quite common to make audio versions of text benchmarks. For example, the knowledge and reasoning benchmark MMLU has MMAU. We’re going to be working in that tradition here.
The Paper
So high-level, this is what we’re looking at for the benchmark: about 450 conversations of 3-8 turns each, totalling 15 hours, split across four different skills to probe and a wide variety of topics.
Each conversation ends in a failure, from either GPT-4o Audio or Gemini 2.5 Pro. All the prior turns form the input for whatever model you’re benchmarking. And then there’s a rubric for the final turn, with criteria that fit both the skill being evaluated and the final turn, not just one or the other.
To give you a more concrete idea of the four skills, here are some examples.
For Inference Memory, they have two types: semantic and audio-cue. Semantic would be memories of the words, of the meaning conveyed by them. That’s something you could get from text as well, as the blue highlights show. But for audio-cue, a non-word sound has to provide the memory. So here the model has to understand that the meowing implies a cat is present, and the words about Mochi are likely in reference to that cat.
For Instruction Retention, it’s the same as in the original MultiChallenge, where the model has to keep following an earlier instruction. So in the first turn the user gives an instruction, and then later presents an opportunity for the model to follow the instruction.
For Self Coherence, it’s not obvious just from the highlighted text, but in this example the human asks about a fourth component of shoulder bones even though the model said earlier there are only three. So the human is teeing up the model to choose between hallucinating to please the user or sticking to the facts from before.
And then for Voice Editing, they again split into two types. Mid-Utterance is when you revise within the same turn, as the first blue highlight shows in changing the number of weeping willow trees. You can also have Previous-Turn, where a turn changes something “in memory” so to speak from a previous turn.
And because this is a Scale paper, you know there’s gonna be some good content about how they produced the data.
High-level, the main things we want are natural conversations, producing the given failure mode. So right away we know we can’t script out the whole thing, but also it probably wouldn’t be very efficient to just put contributors in a metaphorical room with a model and let them poke away.
So instead they synthetically generate conversational blueprints, which the human will then roughly follow, improvising the details and probing for that failure.
There are a few different models involved in the process:
The planner is o3, which was the best model available at the time - that’s what you want for finding these likely failure patterns
The planner then gives a candidate blueprint to the tester, which is going to basically do a synthetic attempt, to make a conversation from the blueprint and see if it induces a failure. That’s GPT-4o
Even though GPT-4o has an audio version, they need a text transcript at the end of the loop for o3 to review. So they have the tester agent output text, and then they use a separate TTS engine to speak, in this case gpt-4o-mini-tts
The conversations happen with one of two target agents, either GPT-4o Audio Preview or Gemini 2.5 Pro, it’s random which one you get. The idea is that if there’s a failure on one of those models, it’s a case worth testing against all the models you want to put on your leaderboard
The conversation history goes back to o3 for review against the initial blueprint, to generate a revised blueprint for the human contributor to use
Now you have your blueprints. The next phase is to turn those blueprints into the conversations that will be the benchmark. So we’re going to pass these vetted blueprints to humans and have them test against the same target agents. Although the human doesn’t have to use the synthetic blueprint, and in fact in 35% of cases they don’t, which I still think is a decent hit rate. Anyway if that works and induces a failure on the desired skill, then the humans write rubrics.
After that there’s a standard quality-control process. And at the end you have a dataset of multiturn conversations + rubrics for judging the final model turn, which you can use to run your eval.
Here’s how those evals work. We feed in the entire history except the failing final assistant turn, with audio from the user and text from the assistant, so that they can also evaluate models that only do speech input, not speech output. Then we ask the model we’re evaluating to produce a final assistant turn.
Once it does, we score it using an LLM judge and a rubric. We already know that LLM judges work well given a good rubric, but they do spend a bit of time showing how LLM judges generally agree with human judges when supplied with the same rubric.
Anyway, the scoring results in two different metrics:
Average Pass Rate (APR), which is per-rubric rather than per-criterion. This indicates whether or not the response was acceptable, which is ultimately what the user cares about. In the example shown, some criteria failed, so the APR is zero.
Average Rubric Score (ARS), which is per-criterion. This shows how close the model gets to acceptable, although personally I think it’s not that helpful without knowing the distribution. Like you could have the same ARS for a totally bimodal distribution and for a normal distribution, but the user experience will be quite different
And here’s where we land. I want to call out a few things.
First, we’re looking at both text and audio outputs. Every model here natively takes audio input, but only 6 of the 20 can talk back.
Second, the best scores vary considerably based on the output modality, but that’s at least partly due to model size. Like Qwen 3 Omni is only 30B parameters. GPT-4o isn’t publicly disclosed, but GPT-4o Mini is around the same size as Qwen 3 Omni. Whereas Gemini 3 Pro is more like 1T.
Relatedly, if you look at the text vs audio output scores for the same model, they’re often pretty close. Like GPT Realtime, GPT-4o Audio, GPT-4o Mini Audio, Kimi Audio, most of the scores are close overall and per skill. Text scores a bit better on average, but not always.
Finally, if you cut by axis and see which skill is strongest and weakest per model, Voice Edit is clearly the weakest skill, and Self Coherence is clearly the strongest skill. We’ll see a bit more about Voice Edit later.
—
1These models previously report scores on a TTS set of MultiChallenge (referring to it as MultiChallenge Audio) which we distinguish from.
On the topic of text vs audio, they do a test of human speech vs TTS output to see how some of the natural hesitations and disfluencies of human speech impact performance.
Broadly speaking, the cleaner, slower, and more even voice of TTS makes the tasks easier, which is no surprise. What is surprising is how the output modality impacts performance. Like if you look on the left, the only models that do worse with TTS input compared to human speech input are the audio-output versions of the GPT models. And then on the right, you see the same trend for the Instruction Retention and Voice Editing skills. However, you see an improvement on the Inference Memory and Self Coherence skills!
It not clear at all why the assistant’s output modality would impact quality so significantly. Even the authors note that they require “further investigation”. They speculate that at least for the left chart, the models are optimized for human speech rather than TTS inputs, but to me the whole thing is a big question mark.
Now digging into the skills a bit more, in figure 5, we see that audio cues are more challenging than semantic cues - audio models do better when the audio is more text-like, unsurprisingly.
And in figure 6, they examine fixed vs conditional instructions, where again the results line up with intuition: conditional instructions are harder to follow.
Now looking at that Voice Edit skill, which was generally the toughest one, they give us the same breakout we saw in the example tasks: previous turn edit vs mid-utterance edit. I actually was surprised to see how similar the scores for all three bar were on many of the models, like for Gemini 3 Pro the error rates are nearly identical. It seems like editing skill degrades faster than overall performance though, as seen by the steep changes on weaker models like Gemma 3n and GPT-4o Mini.
Still, I would have expected worse performance for mid-utterance compared to previous turn edit across the board, and that just wasn’t true.
One last thing: the authors note that “most failures on Voice Editing occur when users introduce multiple… edits throughout a conversation, with the final prompt typically requesting a summary or revision that integrates all previously specified changes.” So I guess one tip for using these models is to recap the current state regularly, rather than waiting until the end of your thoughts to review the final output. That’s good practice for text models too.
So because this is a multi-turn benchmark, we can look at the impact of both turns and duration on quality.
If you look at turn count on the left, you see kind of a funny trend - or really no trend at all. Like For Gemini 2.5 Flash Thinking for example, the second row, 3 turns is less than half the score of 6 turns, but is virtually equal to the score of 7 turns! Conversely, the score at 3 turns is almost double the score at 6 turns. So apparently turns per se doesn’t strongly influence quality. I think the real issue here is that the distribution of turns wasn’t designed, only bounded. So like humans had to induce a failure between turns 3 and 8, but within there it just depended on the flow of the conversation and when humans felt was the right time to strike.
For total duration though, which we see on the right, there’s kind of a trend downwards with duration, but the very small sample size for the longest buckets makes the whole thing a bit questionable.
Overall, the authors conclude that audio perception is the quality bottleneck, and that length doesn’t really have a chance to impact quality much yet.
One last note on length: longer model responses did correlate generally with quality.
Now that initially is a red flag, since length bias is a well known issue for human and machine judges alike. For rubrics in particular, models learn to just throw everything at the wall and see what sticks, to maximize their chances of hitting criteria even at the cost of readability or wrong or contradictory information. That’s classic reward hacking - abusing the scoring system while producing trash.
But the researchers did two things here. First, they just looked at some examples by hand and saw that yes, a lot of the longer responses actually were better. They focused particularly on the Gemini family since it clustered at the high end of this trend, and indeed the responses were good.
Second, they added negative criteria for common errors. That prevents the flooding approach from before where the model just says everything it can think of. That’s a good tip for all rubrics by the way, not just for this benchmark.
My Takeaways
Native speech models still have a long way to go
No wonder Siri and Alexa failed
It’s probably a good long-term bet for Scale
People are built for talking over writing
“We are still early” as they say
Evals will have a paralinguistic component that humans will be best equipped to judge