AI audio understanding comparison
Bee Box's agent can't hear. When a question about a voice message needs more than the transcript ("did that sound annoyed?", "how did I pronounce mañana?"), the box sends the recording to a model that accepts audio and asks it. Choosing that model is a recurring decision, so the repository keeps a small benchmark for it. It has run twice, in August and October 2026.
The models
Models that understand audio (list price per million tokens, input / output):
- Gemini 3.1 Pro August and October. $2 / $12.
- Gemini 3.8 Flash October. $0.75 / $3.75, doubling in 2027.
- Gemini 3.7 Flash August. $0.75 / $3.75. Requests for it are now answered by 3.8 Flash; see below.
- Gemini 3.5 Flash-Lite October. $0.30 / $2.50.
- Gemini 2.5 Flash August. $1 / $2.50.
- GPT Audio 1.5 August. $32 per million audio input tokens, $10 per million text output.
Transcription only (asked only for the words):
- Whisper August. $0.006 per minute.
- Gemini 3.5 Transcribe October. $0.003 per minute.
Gemini counts 32 tokens per second of audio, so $1 per million input tokens is about 0.2¢ per minute. In practice, one short question cost about 2¢ with Gemini 3.1 Pro and under 0.1¢ with Gemini 3.8 Flash.
How it is tested
One speaker recorded 29 short clips. Each clip is designed to test one thing. Related clips go to a model together, labeled only "Recording A", "Recording B", and so on, with the same questions every time: a literal transcript, the language, the emotion and intent, pronunciation, timing and emphasis, and any non-speech sounds. The model is never told what the clips were meant to show. A separate text-only model scores the answers against fixed pass/fail checks without knowing which model wrote them. The recordings are not published.
Each result below is graded:
- ✅ Correct The model heard what the sample was designed to show.
- 🟡 Okay Partly: some of it, or in some tries and not others.
- ❌ Failed It missed the point, transcribed it wrongly, or made something up.
Results by sample
The same words, a different meaning
Samples 1–2 · Emotion carried only by delivery
Recorded: "That's fine, really," once warmly and sincerely, once with reluctant resignation (with an "uh" in front).
Tests: whether the model hears the emotion when the words are identical.
- Gemini 3.1 Pro ✅ Correct Both runs. October: "weary, resigned, or perhaps slightly passive-aggressive, suggesting that things are not actually fine but they are giving up the argument."
- Gemini 3.7 Flash ✅ Correct "Resigned, weary, and slightly exasperated."
- Gemini 3.8 Flash ✅ Correct "Weary, resigned, and fatigued … reluctant acceptance."
- Gemini 3.5 Flash-Lite ✅ Correct "Hesitant, reluctant, or disappointed."
- GPT Audio 1.5 🟡 Okay Heard hesitation ("hesitant but still accepting"), not resignation.
- Gemini 2.5 Flash ❌ Failed Called both takes neutral acceptance.
Happy or sad
Samples 3–4 · Happiness and sadness carried only by delivery
Recorded: "I'll see you tomorrow," once happily, once as a painful goodbye.
Tests: whether the model hears each emotion. No model in either run heard the first take as happy, so the take itself probably isn't; only the sad take is graded.
- Gemini 3.1 Pro ✅ Correct Both runs. "Tired, resigned, or reluctant … the initial heavy exhale."
- Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.5 Flash-Lite ✅ Correct Weary or exhausted.
- Gemini 2.5 Flash ✅ Correct "Weary or resigned."
- GPT Audio 1.5 ❌ Failed "Neutral to hesitant."
One wrong sound
Samples 8–9 · A single changed consonant
Recorded: "Quiero reservar una mesa para mañana a las siete," twice. In the second take only mañana changes: it is said manana, with a plain n in place of ñ.
Tests: whether the model notices one changed sound and says where it is.
- Gemini 3.1 Pro ✅ Correct Both runs. "The 'ñ' in 'mañana' is mispronounced as an English alveolar nasal 'n' (sounding like 'manana')."
- Gemini 3.7 Flash ✅ Correct Named both sounds: [ɲ] in the first take, [n] in the second.
- Gemini 3.8 Flash 🟡 Okay Three of four tries, naming both sounds. In the fourth it described only the pauses.
- Whisper 🟡 Okay Can't be asked, but spelled the second take manana, which gives the change away.
- GPT Audio 1.5 ❌ Failed Mañana both times: "no noticeable differences in content or delivery."
- Gemini 2.5 Flash ❌ Failed Noticed pacing, not the sound.
- Gemini 3.5 Flash-Lite ❌ Failed "Standard Spanish pronunciation" both times, and claimed the first take was whispered. It wasn't.
- Gemini 3.5 Transcribe ❌ Failed Mañana both times.
A rolled rr
Samples 24–25 · A localized consonant change
Recorded: "El perro corre rápido por el parque," once with a trilled rr in perro, once with a single tap, as in pero. The change was deliberately exaggerated.
Tests: whether the model hears the missing trill.
- Gemini 3.1 Pro ✅ Correct Both runs, the only model to hear it. It described the second take as English-accented throughout rather than one changed sound.
- Gemini 3.7 Flash ❌ Failed "Acoustically identical."
- Gemini 3.8 Flash ❌ Failed Heard "strongly articulated alveolar trills" in both takes, and "a very deep, theatrical, and gravelly voice". Neither is true.
- GPT Audio 1.5, Gemini 2.5 Flash, Gemini 3.5 Flash-Lite ❌ Failed Called the takes the same.
A sounded h
Samples 26–27 · A silent letter pronounced
Recorded: "Hoy hablamos de historia en el hotel," once with every h silent as in Spanish, once with an English h added to hotel only.
Tests: whether the model hears the one added sound and places it.
- Gemini 3.1 Pro ✅ Correct Both runs. "The speaker incorrectly articulates an English aspirated 'h' ([ho'tel])."
- Gemini 3.7 Flash, Gemini 3.8 Flash ✅ Correct Same localization, with [o'tel] versus [ho'tel].
- Gemini 3.5 Flash-Lite 🟡 Okay Found it in one of two tries.
- GPT Audio 1.5, Gemini 2.5 Flash ❌ Failed Called both takes standard.
- Whisper, Gemini 3.5 Transcribe ❌ Failed Identical spelling for both; no sign of the change.
Which word was stressed
Samples 16–20 · Contrastive stress
Recorded: "I didn't say she stole the money," five times, stressing I, say, she, stole, and money in turn. Each stress changes the meaning.
Tests: whether the model finds the stressed word in each take.
- Gemini 3.1 Pro, Gemini 3.7 Flash, Gemini 3.8 Flash ✅ Correct Five of five, every try, with timing evidence: "Primary stress on 'say' with a pitch peak and elongated vowel."
- Gemini 3.5 Flash-Lite 🟡 Okay Nine of ten across two tries.
- GPT Audio 1.5 🟡 Okay Three of five; heard the first take as a restart, "I... I didn't".
- Gemini 2.5 Flash 🟡 Okay Three of five.
- Whisper ❌ Failed The same sentence five times, once with STOLE capitalized.
- Gemini 3.5 Transcribe ❌ Failed The same sentence five times.
A pause that means something
Sample 14 · Silence as meaning
Recorded: "Sure … I guess," with a reluctant pause measured at about three seconds.
Tests: whether the model keeps the pause, estimates it, and reads it as reluctance.
- Gemini 3.1 Pro ✅ Correct Both runs. August: "approximately three seconds." October: about two to two and a half.
- Gemini 3.8 Flash, Gemini 3.5 Flash-Lite ✅ Correct "~3.5s", "~2.5s", and "a three-second pause".
- Gemini 3.7 Flash 🟡 Okay About two seconds, and claimed it was filled by a sigh.
- GPT Audio 1.5 🟡 Okay "A notably long pause", with no estimate.
- Gemini 2.5 Flash ❌ Failed Missed the pause and reported a creaking sound.
- Whisper, Gemini 3.5 Transcribe ❌ Failed "Sure. I guess." and "Sure, I guess." The words survive; the meaning doesn't.
Coughs and laughs
Samples 12–13 · Non-speech sounds
Recorded: two short coughs; then two short laughs. No words.
Tests: whether the model tells the sounds apart, and doesn't invent words.
- Gemini 3.1 Pro ✅ Correct Both runs. "Throat clearing (1), cough (2)" and "laugh (2)".
- Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.5 Flash-Lite, GPT Audio 1.5 ✅ Correct Coughs and laughter, no words.
- Gemini 3.5 Transcribe ✅ Correct Returned nothing for either, which is right for a transcriber.
- Gemini 2.5 Flash ❌ Failed A throat clear and a slow "aah", and the same "sigh-like" sound for both.
- Whisper ❌ Failed Transcribed the coughs as "Thank you for watching."
Unusual names
Samples 10–11 · Names, careful and fast
Recorded: "Shaniqua asked Priyanka whether Sanjay had called," once carefully, once fast.
Tests: name spelling, and whether the model notices the change of pace.
- Gemini 3.7 Flash, Gemini 3.8 Flash ✅ Correct Every name, every try, and the pace: "around 5.3 seconds" versus "3.5 seconds".
- Gemini 3.1 Pro 🟡 Okay Wrote Shiniqua once and Shaniqua once.
- GPT Audio 1.5 🟡 Okay Every name, but "no major audible differences" in pace.
- Whisper 🟡 Okay Sheniqua in the careful take, exact in the fast one.
- Gemini 2.5 Flash ❌ Failed Shequwa both times.
- Gemini 3.5 Flash-Lite ❌ Failed Shanique both times.
- Gemini 3.5 Transcribe ❌ Failed Shinqua and Shinika.
A transcript with stage directions
Sample 29 · Everything beyond the words
Recorded: a sigh, "Okay", a pause, "I was going to say Tuesday — no, Wednesday", a small awkward laugh, a whispered "Don't tell Maya yet", normal voice for "I'll call her after lunch", and one cough.
Tests: asked for one transcript with bracketed stage directions, whether the model captures every event, including the switch to a whisper and back.
Gemini 3.1 Pro ✅ Correct October produced every designed event:
[Heavy sigh] Okay. [Pause 1.5s] I was gonna say Tuesday, [Pause 0.5s] no, Wednesday. [Laugh] [Sigh] [whispering] Don't tell Maya yet. [Pause 1s] [normal voice] I'll call her after lunch. [Pause 0.5s] [Cough]
Gemini 3.8 Flash 🟡 Okay Marked the whisper, but called the cough a throat clear.
Gemini 3.7 Flash 🟡 Okay The sigh, pauses, chuckle, and cough, but not the whisper, and it normalized "gonna" to "going to".
Gemini 3.5 Flash-Lite 🟡 Okay Marked the events but neither voice change.
GPT Audio 1.5 ❌ Failed Missed the laugh and the whisper, and misplaced the pause.
Gemini 2.5 Flash ❌ Failed Heard "Maya" as "my … yet", and no whisper.
Whisper, Gemini 3.5 Transcribe ❌ Failed Only the words.
Plain transcription
Samples 5, 6, 21 and others · Words on the page
Recorded: sentences such as "The rural juror rarely worries about the brewery," Spanish, a code-switch, and an ordinary late-for-dinner message.
Tests: getting the words right. This is where transcription-only models compete.
- GPT Audio 1.5, Whisper ✅ Correct The lowest error rates in August.
- Gemini 3.5 Transcribe 🟡 Okay Matched Whisper's error rate in October, but wrote "The world-weary rarely worries about the brewery."
- Gemini 3.1 Pro, Gemini 3.7 Flash, Gemini 3.8 Flash ✅ Correct Close behind, and more likely to keep fillers and hesitations that were really spoken.
- Gemini 3.5 Flash-Lite ❌ Failed "The roolger rarely worries about the brewery", Deepak for Deepa, perro for pero.
Invented messages
Generated clips · Audio with no speech
Recorded: nothing. Three generated clips: a short electronic beep, three seconds of silence, and three seconds of low hiss.
Tests: each clip went to the model alone, inside the prompt Bee Box really sends, with the question "What exactly did the user say in this message, and how did they sound?" The right answer is that nobody said anything.
- Gemini 3.1 Pro ✅ Correct Quoted nothing in 9 of 9. It said there was no speech, or, less helpfully, that no audio was attached.
- Gemini 3.5 Transcribe ✅ Correct Returned an empty transcript.
- Gemini 3.8 Flash ❌ Failed Quoted a message every time, 18 of 18, and described the speaker. None of the answers hedged:
- Beep: "Send that email."
- Beep: "Uh, hey, can you just delete that last voice message?"
- Silence: "It's raining a lot outside, so remember to bring your umbrella." ("The speaker (a male voice) sounded calm, clear, and helpful.")
- Hiss: "Keep your voice down." ("She speaks in a hushed, urgent whisper.")
- Gemini 3.5 Flash-Lite ❌ Failed 8 of 9: "Set a timer for 15 minutes.", "Add milk to my grocery list."
Bee Box's agent can't hear, so it takes the audio model's answer as what the user said. An accidental send or a pocket recording looks like these clips. The agent would get a confident quote of a request nobody made. How often that happens with real recordings isn't measured yet.
Rankings
August 2026: Gemini 3.1 Pro first, Gemini 3.7 Flash a close second, then GPT Audio 1.5 (good transcription, weak on pronunciation and prosody), then Gemini 2.5 Flash. Bee Box moved its audio questions to Gemini 3.7 Flash.
October 2026: Gemini 3.1 Pro first again, on every capability. Gemini 3.8 Flash second. Gemini 3.5 Flash-Lite third: fast and very cheap, but it misses most pronunciation changes and invents messages.
Gemini 3.7 Flash is no longer what answers. Every request for it in October was answered by Gemini 3.8 Flash, according to the model version in each response. Google's model list and changelog don't mention the substitution. The October report recommends moving audio questions to Gemini 3.1 Pro.
Re-running it
The benchmark, its pass/fail checks, every model answer, and the full reports are in the repository at research/audio-understanding-bakeoff. When a new model ships, the update is mechanical: add the model, run it next to the current one, and write a new dated report.
generated from the repository