Nvidia’s Nemotron 3.5 Transcribe most accurate in Artificial Analysis diarization benchmark
Artificial Analysis’s new Diarization Index measures the share of transcript words assigned to the wrong speaker or to no speaker. In English conversations of two to eight people, Nvidia’s model had an error rate of 3.3 per cent.
Artificial Analysis, which publishes comparisons of AI models, announced its new Diarization Index benchmark on X on 13 November. The benchmark measures speaker diarization: whether a speech-to-text service can mark in a transcript who said each word.
How errors are counted
The metric is WDER (Word Diarization Error Rate). It gives the share of scored reference words that are assigned to the wrong speaker or left without a speaker.
The dataset contains 97 English conversations with two to eight speakers. The index is the average of three groups, so conversations with two, three to four and five to eight speakers carry equal weight. Some of the conversations are Artificial Analysis’s own held-out conversations, both real and synthetic, and some are meetings from the NOTSOFAR-1 dataset.
The index primarily measures speaker labels, but missing words also count as errors. Artificial Analysis measures word recognition accuracy separately with its AA-WER error rate.
Results
Nvidia’s Nemotron 3.5 Transcribe had the lowest error rate, at 3.3 per cent. Mistral’s Voxtral Small Transcribe 3 came second with 7.5 per cent. The table shows the top five from the announcement chart.
| Model | WDER |
|---|---|
| Nemotron 3.5 Transcribe | 3.3% |
| Voxtral Small Transcribe 3 | 7.5% |
| Universal | 8.6% |
| Muse Voice Transcribe | 8.6% |
| MAI-Transcribe-2 | 8.8% |
Source: Artificial Analysis announcement chart, 13 November 2026 (24 models), English conversations of 2–8 people. Lower is better.
Behind the top two, the differences are small: in the announcement chart, places three to seven range from 8.6 to 8.9 per cent, and their error bars overlap. Artificial Analysis itself says MAI-Transcribe-2 is level with the group just behind the top two.
Speed and availability
Artificial Analysis highlights Microsoft AI’s MAI-Transcribe-2 as a fast option close to the top: it processes 93.7 seconds of audio per second. In the site’s transcription-only table, the same model runs at 313.3 seconds of audio per second. Transcription speed was measured on ten-minute recordings and may differ for other lengths, especially files under a minute.
In the announcement chart, Nemotron 3.5 Transcribe processes about 30 seconds of audio per second. The fastest model is Deepgram’s Nova-3, at almost 200 seconds of audio per second, but the chart puts its error rate at 16.7 per cent.
The site lists Nemotron 3.5 Transcribe as an open-weight model. In the transcription table, DeepInfra offers it at $2 per 1,000 minutes of audio; the page shows diarization prices in a separate chart.
Read more: Models