kya bola?

Every speech recognition vendor publishes one accuracy number per language. Hindi in Delhi and Hindi in Araria are not the same problem, and one number hides the difference. This measures it district by district.

It also asks something nobody has published an answer to. The corpus behind this holds 64 languages. Commercial speech APIs officially support 19. Here is what happens to the other 45.

languages in the corpus

with no API support

150

districts measured

9.4%

human disagreement floor

The map

Error rate per district. Greener is better. Grey is not a gap in the measurement: Project Vaani has recorded 165 of India’s districts so far, and the rest of the country has no transcribed audio in this corpus to score against.

model

language

Reading the map

at the human floor
15 to 25%
25 to 40%
40 to 60%
over 60%
no audio in this corpus

The first band stops at 15.0% because two independent human transcribers of this same audio disagree with each other by 10.0% to 15.0%. Below that line no model is measurably better than another. Darker bands carry a hatch as well as a colour.

This view

districts measured: 147
districts in the corpus: 165
clips scored: 4450
excluded: 297

Measured but not drawn

These districts have Vaani audio but no polygon in the 2021 boundary release. Named here rather than left as a silent gap.

Annamaya, AndhraPradesh
created 2022 from Kadapa/Chittoor/Nellore; postdates the boundary file
Manyam, AndhraPradesh
Parvathipuram Manyam, created 2022 from Vizianagaram/Srikakulam
SriSatyaSai, AndhraPradesh
Sri Sathya Sai, created 2022 from Anantapur
CharkhiDadri, Haryana
created 2016 from Bhiwani; absent from the 2021 ADM2 release

The coverage gap

Languages an API does not claim to support are still sent to it, with language detection on. A refusal or an empty answer counts as a failed clip, because that is what a developer building for those speakers would experience.

sort

Error rate by language and provider, with whether each language is officially supported.
language	official support	clips	indicconformer:ctc	sarvam:saaras:v3	sarvam:saaras:v4
Angami	not listed	39	100.0%wer	97.4%wer	95.2%wer
Angika	not listed	98	100.0%wer	29.5%wer	29.8%wer
Ao	not listed	38	100.0%wer	89.3%wer	85.2%wer
Awadhi	not listed	64	100.0%wer	33.2%wer	30.0%wer
Bajjika	not listed	39	100.0%wer	43.3%wer	41.8%wer
Bearybashe	not listed	32	100.0%cer	73.0%cer	75.5%cer
Bhili	not listed	15	100.0%wer	86.6%wer	85.6%wer
Bhojpuri	not listed	212	100.0%wer	45.7%wer	40.4%wer
Bundeli	not listed	36	100.0%wer	25.3%wer	21.4%wer
Chakhesang	not listed	47	100.0%wer	100.8%wer	90.8%wer
Chakma	not listed	41	100.0%wer	108.7%wer	99.1%wer
Chhattisgarhi	not listed	174	100.0%wer	49.0%wer	44.4%wer
English	listed	191	100.0%wer	11.4%wer	11.6%wer
Garhwali	not listed	38	100.0%wer	81.7%wer	73.3%wer
Garo	not listed	56	100.0%wer	133.0%wer	103.7%wer
Gondi	not listed	20	100.0%cer	96.4%cer	87.4%cer
Halbi	not listed	18	100.0%wer	88.6%wer	72.6%wer
Haryanvi	not listed	50	100.0%wer	57.9%wer	51.3%wer
IduMishmi	not listed	20	100.0%wer	110.9%wer	99.4%wer
Jaipuri	not listed	36	100.0%wer	29.6%wer	28.2%wer
Karbi	not listed	20	100.0%wer	90.5%wer	87.4%wer
Khariboli	not listed	103	100.0%wer	24.3%wer	24.3%wer
Khortha	not listed	73	100.0%wer	44.4%wer	44.5%wer
Kokborok	not listed	55	100.0%wer	104.3%wer	100.3%wer
Kumaoni	not listed	35	100.0%wer	43.0%wer	39.0%wer
Kurukh	not listed	19	100.0%cer	65.8%cer	41.1%cer
Magadhi	not listed	63	100.0%wer	55.1%wer	52.0%wer
Magahi	not listed	47	100.0%wer	29.3%wer	28.0%wer
Malvani	not listed	46	100.0%wer	49.9%wer	48.2%wer
Marwari	not listed	32	100.0%wer	55.4%wer	54.8%wer
Mizo	not listed	19	100.0%wer	95.1%wer	95.1%wer
Nagamese	not listed	35	100.0%wer	96.0%wer	95.8%wer
Nagpuri	not listed	19	100.0%wer	69.1%wer	62.6%wer
Nyishi	not listed	19	100.0%wer	120.2%wer	103.9%wer
Rajasthani	not listed	35	100.0%wer	51.1%wer	50.7%wer
Rengma	not listed	20	100.0%wer	96.3%wer	95.1%wer
Rongmei	not listed	49	100.0%wer	96.2%wer	95.5%wer
Sadri	not listed	40	100.0%wer	58.3%wer	56.4%wer
Sambalpuri	not listed	20	100.0%wer	59.1%wer	59.8%wer
Sumi	not listed	36	100.0%wer	116.2%wer	95.7%wer
Surgujia	not listed	20	100.0%wer	57.2%wer	47.6%wer
Surjapuri	not listed	19	100.0%wer	110.1%wer	75.6%wer
Tagin	not listed	20	100.0%wer	122.7%wer	110.1%wer
Thethi	not listed	20	100.0%wer	46.2%wer	48.0%wer
Tulu	not listed	18	100.0%cer	60.8%cer	79.5%cer
Wancho	not listed	20	100.0%wer	106.5%wer	99.4%wer
Santali	listed	17	83.2%wer	84.2%wer	58.6%wer
Odia	listed	120	56.3%wer	60.5%wer	41.9%wer
Marathi	listed	263	40.0%wer	40.0%wer	40.2%wer
Urdu	listed	105	36.5%wer	48.8%wer	47.8%wer
Konkani	listed	26	36.5%wer	36.9%wer	36.3%wer
Maithili	listed	254	36.5%wer	35.5%wer	34.8%wer
Telugu	listed	340	34.1%cer	28.9%cer	28.8%cer
Kashmiri	listed	50	33.8%wer	35.7%wer	31.4%wer
Nepali	listed	79	33.3%wer	34.0%wer	33.8%wer
Tamil	listed	154	27.0%cer	32.0%cer	22.7%cer
Gujarati	listed	56	26.6%wer	26.6%wer	25.7%wer
Punjabi	listed	136	25.7%wer	24.1%wer	22.4%wer
Assamese	listed	55	24.0%wer	22.2%wer	22.4%wer
Bengali	listed	405	23.3%wer	23.7%wer	22.8%wer
Manipuri	listed	21	21.4%wer	26.7%wer	26.7%wer
Hindi	listed	899	19.5%wer	22.2%wer	21.0%wer
Kannada	listed	222	17.2%cer	16.4%cer	16.9%cer
Malayalam	listed	119	12.2%cer	12.9%cer	12.5%cer

CER is reported as primary for Dravidian languages, where one long agglutinated token can carry a whole clause and word error rate over-punishes a single wrong morpheme.

Rates above 100% are real, not a bug. Word error rate counts insertions as well as substitutions and deletions, so a model returning more words than were spoken can exceed the length of the reference. It usually means the model is producing fluent text in the wrong language rather than declining to answer.

Is this trustworthy?

Before measuring anything new, this harness reproduced a published result. The Vaani team benchmarked Sarvam Saaras v3 on their own Hindi evaluation set and reported a district mean word error rate of 18.3% with a standard deviation of 4.6. Running their clips through this pipeline gives:

district mean, ours: 17.6% ± 7.7
district mean, published: 18.3% ± 4.6
two humans, same audio: 9.4%

That last row is the floor. Two people transcribing the same audio disagree by that much, so no model can score below it and no gap narrower than it is a real difference between two systems.