ElevenLabs’ own test: Scribe v3 makes 27 per cent fewer word errors than Scribe v2
Written by AI
ElevenLabs says its new speech recognition model Scribe v3 makes 27 per cent fewer word errors than its predecessor in a 102-language test. The real-time Scribe v3 Live now separates speakers and uses the content of the conversation to judge when a turn has ended.
ElevenLabs announced its new Scribe models on its blog on 10 November. Scribe v3 is for recorded audio. Scribe v3 Live runs with low latency, which suits it to voice agents, for example.
Speakers and turns in real time
Scribe v3 Live is the company’s first real-time Scribe model with speaker diarization, which labels who said what.
A voice agent also has to recognise when the speaker has finished. Traditional systems wait for silence. A short timeout cuts off a speaker who pauses mid-sentence; a long one makes the agent slow. Scribe v3 Live judges the end of a turn from the content of the conversation as well as the audio. As a result, a caller dictating a number can pause without the agent interrupting, the company said. The same detection already runs on the ElevenAgents voice agent platform and is now coming to ElevenAPI.
The figures are the company’s own
ElevenLabs measured the results itself or reported them on its blog. The Runtimewire website based its report on the company’s X thread and blog, so it does not independently confirm the figures either.
The company measured accuracy across languages with its own version of the FLEURS benchmark, which contains short, clearly read speech samples in 102 languages. Word error rate (WER) shows the share of words transcribed incorrectly. Semantic WER counts only errors that change the meaning.
The table shows how far the new models cut errors compared with their predecessors. The rows use different metrics, so they cannot be compared with one another.
| Model | Compared with | Metric | Fewer errors |
|---|---|---|---|
| Scribe v3 | Scribe v2 | WER, 102 languages | 27% |
| Scribe v3 | Scribe v2 | speaker diarization, at least 3 speakers | 17% |
| Scribe v3 Live | Scribe v2 Realtime | semantic WER, 102 languages | 43% |
| Scribe v3 Live | Scribe v2 Realtime | WER, 102 languages | 31% |
Source: ElevenLabs blog, the company’s own measurements.
Each FLEURS sample has only one speaker. To measure diarization, ElevenLabs therefore built its own dataset from real meetings, agent calls and voice messages in 16 languages. The company says it will release the dataset openly soon. On the English portion of the dataset, Scribe v3 Live had a diarization error rate of 3.1 per cent, compared with 5.9–7.3 per cent for the other real-time models tested. The scoring measured only whether each word was attributed to the right speaker. This method lowers every model’s error rate, so the figures cannot be compared with the diarization results for recorded audio.
ElevenLabs said that in a pre-release test on real customer calls, a voice agent at Eneco, a Dutch energy company, recognised postcodes on the first attempt 40 per cent more often and house numbers 15 per cent more often with Scribe v3 Live.
Term lists and availability
Users can give Scribe v3 up to 1,000 key terms, such as brand names and abbreviations, and formatting rules in plain English. The Live version also has a multilingual mode for conversations that switch language mid-sentence, and a term library that the model uses only when it hears a likely match. The blog says the library holds thousands of terms; Runtimewire puts the figure at 10,000.
Scribe v3 Live is available in ElevenAgents, and both models are the default in the ElevenLabs app. In ElevenAPI, an existing integration switches to the new model when the model ID is changed to scribe_v3 or scribe_v3_live. Enterprise customers can also use separate environments in the EU, India and Singapore.