A benchmark built on Project Vaani · ARTPARK and IISc
kya bola?
Every speech recognition vendor publishes one accuracy number per language. Hindi in Delhi and Hindi in Araria are not the same problem, and one number hides the difference. This measures it district by district.
It also asks something nobody has published an answer to. The corpus behind this holds 64 languages. Commercial speech APIs officially support 19. Here is what happens to the other 45.
15,424 clips scored across IndicConformer CTC, Sarvam Saaras v3, Sarvam Saaras v4.
The support list, against what happened
District view
The map
Error rate per district. Greener is better. Grey is not a gap in the measurement: Project Vaani has recorded 165 of India’s districts so far, and the rest of the country has no transcribed audio in this corpus to score against.
Language view
The coverage gap
Languages an API does not claim to support are still sent to it, with language detection on. A refusal or an empty answer counts as a failed clip, because that is what a developer building for those speakers would experience.
| support | |||||
|---|---|---|---|---|---|
| Angami | not listed | 39 | 100.0%wer | 97.4%wer | 95.2% (best of the models)wer |
| Angika | not listed | 98 | 100.0%wer | 29.5% (best of the models)wer | 29.8%wer |
| Ao | not listed | 38 | 100.0%wer | 89.3%wer | 85.2% (best of the models)wer |
| Awadhi | not listed | 64 | 100.0%wer | 33.2%wer | 30.0% (best of the models)wer |
| Bajjika | not listed | 39 | 100.0%wer | 43.3%wer | 41.8% (best of the models)wer |
| Bearybashe | not listed | 32 | 100.0%cer | 73.0% (best of the models)cer | 75.5%cer |
| Bhili | not listed | 15 | 100.0%wer | 86.6%wer | 85.6% (best of the models)wer |
| Bhojpuri | not listed | 212 | 100.0%wer | 45.7%wer | 40.4% (best of the models)wer |
| Bundeli | not listed | 36 | 100.0%wer | 25.3%wer | 21.4% (best of the models)wer |
| Chakhesang | not listed | 47 | 100.0%wer | 100.8%wer | 90.8% (best of the models)wer |
| Chakma | not listed | 41 | 100.0%wer | 108.7%wer | 99.1% (best of the models)wer |
| Chhattisgarhi | not listed | 174 | 100.0%wer | 49.0%wer | 44.4% (best of the models)wer |
| English | listed | 191 | 100.0%wer | 11.4% (best of the models)wer | 11.6%wer |
| Garhwali | not listed | 38 | 100.0%wer | 81.7%wer | 73.3% (best of the models)wer |
| Garo | not listed | 56 | 100.0% (best of the models)wer | 133.0%wer | 103.7%wer |
| Gondi | not listed | 20 | 100.0%cer | 96.4%cer | 87.4% (best of the models)cer |
| Halbi | not listed | 18 | 100.0%wer | 88.6%wer | 72.6% (best of the models)wer |
| Haryanvi | not listed | 50 | 100.0%wer | 57.9%wer | 51.3% (best of the models)wer |
| IduMishmi | not listed | 20 | 100.0%wer | 110.9%wer | 99.4% (best of the models)wer |
| Jaipuri | not listed | 36 | 100.0%wer | 29.6%wer | 28.2% (best of the models)wer |
| Karbi | not listed | 20 | 100.0%wer | 90.5%wer | 87.4% (best of the models)wer |
| Khariboli | not listed | 103 | 100.0%wer | 24.3% (best of the models)wer | 24.3%wer |
| Khortha | not listed | 73 | 100.0%wer | 44.4% (best of the models)wer | 44.5%wer |
| Kokborok | not listed | 55 | 100.0% (best of the models)wer | 104.3%wer | 100.3%wer |
| Kumaoni | not listed | 35 | 100.0%wer | 43.0%wer | 39.0% (best of the models)wer |
| Kurukh | not listed | 19 | 100.0%cer | 65.8%cer | 41.1% (best of the models)cer |
| Magadhi | not listed | 63 | 100.0%wer | 55.1%wer | 52.0% (best of the models)wer |
| Magahi | not listed | 47 | 100.0%wer | 29.3%wer | 28.0% (best of the models)wer |
| Malvani | not listed | 46 | 100.0%wer | 49.9%wer | 48.2% (best of the models)wer |
| Marwari | not listed | 32 | 100.0%wer | 55.4%wer | 54.8% (best of the models)wer |
| Mizo | not listed | 19 | 100.0%wer | 95.1% (best of the models)wer | 95.1%wer |
| Nagamese | not listed | 35 | 100.0%wer | 96.0%wer | 95.8% (best of the models)wer |
| Nagpuri | not listed | 19 | 100.0%wer | 69.1%wer | 62.6% (best of the models)wer |
| Nyishi | not listed | 19 | 100.0% (best of the models)wer | 120.2%wer | 103.9%wer |
| Rajasthani | not listed | 35 | 100.0%wer | 51.1%wer | 50.7% (best of the models)wer |
| Rengma | not listed | 20 | 100.0%wer | 96.3%wer | 95.1% (best of the models)wer |
| Rongmei | not listed | 49 | 100.0%wer | 96.2%wer | 95.5% (best of the models)wer |
| Sadri | not listed | 40 | 100.0%wer | 58.3%wer | 56.4% (best of the models)wer |
| Sambalpuri | not listed | 20 | 100.0%wer | 59.1% (best of the models)wer | 59.8%wer |
| Sumi | not listed | 36 | 100.0%wer | 116.2%wer | 95.7% (best of the models)wer |
| Surgujia | not listed | 20 | 100.0%wer | 57.2%wer | 47.6% (best of the models)wer |
| Surjapuri | not listed | 19 | 100.0%wer | 110.1%wer | 75.6% (best of the models)wer |
| Tagin | not listed | 20 | 100.0% (best of the models)wer | 122.7%wer | 110.1%wer |
| Thethi | not listed | 20 | 100.0%wer | 46.2% (best of the models)wer | 48.0%wer |
| Tulu | not listed | 18 | 100.0%cer | 60.8% (best of the models)cer | 79.5%cer |
| Wancho | not listed | 20 | 100.0%wer | 106.5%wer | 99.4% (best of the models)wer |
| Santali | listed | 17 | 83.2%wer | 84.2%wer | 58.6% (best of the models)wer |
| Odia | listed | 120 | 56.3%wer | 60.5%wer | 41.9% (best of the models)wer |
| Marathi | listed | 263 | 40.0% (best of the models)wer | 40.0%wer | 40.2%wer |
| Urdu | listed | 105 | 36.5% (best of the models)wer | 48.8%wer | 47.8%wer |
| Konkani | listed | 26 | 36.5%wer | 36.9%wer | 36.3% (best of the models)wer |
| Maithili | listed | 254 | 36.5%wer | 35.5%wer | 34.8% (best of the models)wer |
| Telugu | listed | 340 | 34.1%cer | 28.9%cer | 28.8% (best of the models)cer |
| Kashmiri | listed | 50 | 33.8%wer | 35.7%wer | 31.4% (best of the models)wer |
| Nepali | listed | 79 | 33.3% (best of the models)wer | 34.0%wer | 33.8%wer |
| Tamil | listed | 154 | 27.0%cer | 32.0%cer | 22.7% (best of the models)cer |
| Gujarati | listed | 56 | 26.6%wer | 26.6%wer | 25.7% (best of the models)wer |
| Punjabi | listed | 136 | 25.7%wer | 24.1%wer | 22.4% (best of the models)wer |
| Assamese | listed | 55 | 24.0%wer | 22.2% (best of the models)wer | 22.4%wer |
| Bengali | listed | 405 | 23.3%wer | 23.7%wer | 22.8% (best of the models)wer |
| Manipuri | listed | 21 | 21.4% (best of the models)wer | 26.7%wer | 26.7%wer |
| Hindi | listed | 899 | 19.5% (best of the models)wer | 22.2%wer | 21.0%wer |
| Kannada | listed | 222 | 17.2%cer | 16.4% (best of the models)cer | 16.9%cer |
| Malayalam | listed | 119 | 12.2% (best of the models)cer | 12.9%cer | 12.5%cer |
The bolder figure in each row is the lowest error rate of the models tested. CER is reported as primary for Dravidian languages, where one long agglutinated token can carry a whole clause and word error rate over-punishes a single wrong morpheme.
Rates above 100% are real, not a bug. Word error rate counts insertions as well as substitutions and deletions, so a model returning more words than were spoken can exceed the length of the reference. It usually means the model is producing fluent text in the wrong language rather than declining to answer.
Calibration
Is this trustworthy?
Before measuring anything new, this harness reproduced a published result. The Vaani team benchmarked Sarvam Saaras v3 on their own Hindi evaluation set and reported a district mean word error rate of 18.3% with a standard deviation of 4.6. Running their clips through this pipeline gives:
- district mean, this harness
- 17.6% ± 7.7
- district mean, published
- 18.3% ± 4.6
- two humans, same audiothe floor
- 9.4%
That last row is the floor. Two people transcribing the same audio disagree by that much, so no model can score below it and no gap narrower than it is a real difference between two systems. Measured over 350 clips across 90 districts.