Hugging Face said in a blog post on 6 November that its leaderboard of speech recognition models is getting unpublished test data for two new English accents. Liva AI is supplying the data for the Open ASR Leaderboard: 1.34 hours of Egyptian English and 1.55 hours of Nigerian English.

Both datasets consist of spontaneous conversational speech. The transcripts include punctuation and capitalisation, and they also mark disfluencies such as stumbles and filler words. Tech Beat, a site summarising the blog post, regards the small size of the datasets as a risk.

The datasets will not be published. Hugging Face said the aim is to reduce benchmaxxing, the tuning of models to known test sets. New data providers also increase trust in the evaluations, the company said.

A few months ago, Appen and DataOceanAI supplied 11 private splits to the leaderboard, covering scripted and conversational speech in five accents. Voice Arena later supplied an Indian English split.

With the new accents, the leaderboard’s Private tab breaks down results in more detail. The breakdown follows Kachru’s Three Circles model, which groups English-speaking countries by whether English is the primary first language, an official second language or a foreign language there. Hugging Face said it took the idea from Hume AI’s leaderboard.

The Private tab now offers two comparisons. The first compares the US accent with the others: the UK, Australia, Canada, India, Nigeria and Egypt. The second compares the inner circle (the US, the UK, Australia, Canada) with the outer and expanding circles (India, Nigeria, Egypt). The latter two circles share a group for now, so users cannot yet compare them with each other in this breakdown. Hugging Face plans to separate them when more accents arrive.

Liva AI collects interaction data in several formats from users of its own consumer products, the blog said. Liva AI says it can design products around specific weaknesses in models, so the data captures natural behaviour rather than specially requested tasks.