In AssemblyAI’s test, new model transcribed less background speech and dropped fewer short replies
In AssemblyAI’s own test of 12,460 calls, its new speech recognition model for voice agents transcribed 47 per cent less background speech than its predecessor and dropped 38 per cent fewer one- or two-word replies. The company said that reducing one of these errors usually increases the other.
AssemblyAI has released AssemblyAI-STT-3.7-Realtime-Agent, a speech recognition model for voice agents: programs that handle a call and answer the caller. On the company’s research blog, Ilya Sklyar explains why the two error types must be measured together.
A quiet ‘yes’ and a single word from a background speaker sound alike: short and faint. If a system raises its speech-detection threshold or suppresses noise, it filters out both. Background speech therefore usually falls only if more replies are dropped. The industry’s most common metric, word error rate (the share of words misrecognised, missed or added), reveals neither error, because both affect only a fraction of a per cent of words.
AssemblyAI’s answer is context. For each turn, the model receives relevant audio from earlier in the call and earlier lines of dialogue, such as the question the agent has just asked. Background voices were audible in the training data recordings but were left out of the target transcripts. The company said suppression works in most call environments without configuration but recommends the near-field voice focus feature when another speaker is close to the microphone.
The company’s results compared with the model’s predecessor, Universal-3.6 Pro Realtime:
| Metric | Universal-3.6 Pro Realtime | STT-3.7-Realtime-Agent |
|---|---|---|
| Background-speech words per 1,000 words | 2.53 | 1.34 |
| Dropped short replies | 0.29% | 0.18% |
| Word error rate | 5.26% | 4.53% |
| Final-transcript latency after speech ends | 172 ms | 86 ms |
Source: AssemblyAI. 12,460 English-language calls; the predecessor’s accuracy figures from a September evaluation; latency measured in a separate test in which the end of each turn was supplied, so 86 ms excludes the time the model takes to decide that a turn has ended.
The figures come from the company’s own test, in which 522 speakers read scripted scenarios as telephone audio. The company said the differences in background speech, replies and word error rate are statistically significant. The Voice Arena benchmark ranked the model first among streaming models for US English, with a semantic word error rate of 2.45 per cent, the blog said. On the Speko benchmark, the model beat ElevenLabs Scribe v3 Live, Deepgram Flux, Soniox Realtime v6, Fish Transcribe Stream 1 and Cartesia Ink-2 on word error rate and two latency metrics, the blog reported.
The model is not the fastest: Soniox Realtime v6, Deepgram Nova-3 and a few others return the final transcript sooner but make at least 30 per cent more word errors, AssemblyAI said. In a number-heavy test of 300 recordings, the share of number errors fell from 11.28 per cent to 2.27 per cent.
The model is available now in the company’s streaming API with speech_model="assemblyai-stt-3-7-realtime-agent" and costs $0.45 an hour, the same as its predecessor. The No Code MBA website says the model supports 32 languages with automatic language detection, but the company’s tests cover English. The changelog of Pipecat, a voice agent library, says the library supports the model but keeps Universal-3.6 Pro as its default until ‘official release’, even though AssemblyAI says the model is already available.