Interspeech · 2026 · Conference
Srija Anand et al. (IIT Madras) · → Paper · Demo: ? · Code: ?
A controlled, multidimensional pairwise listening study of 7 TTS systems across 10 Indian languages (120K comparisons from 1,915 native raters) that builds a Bradley-Terry leaderboard, explains overall preference through six perceptual axes, and measures how many raters and sentences a stable ranking needs.
Problem
Crowdsourced pairwise “arena” evaluation has become a common way to rank foundation models, but for TTS it inherits high variance from uncontrolled text (raters hear different sentences, languages, and input types) and from the multidimensional nature of speech perception. Absolute-rating protocols such as MOS and MUSHRA are sensitive to rater calibration and usually report aggregate scores that do not explain why one system is preferred. Existing evaluations of Indic TTS are mostly small MOS/CMOS studies that predate current commercial systems and do not cover code-mixing, raw numerals and symbols, or deployment-realistic domains.
Method
The framework has four components.
Benchmark. 5,357 sentences across Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Tamil, Telugu, and Urdu, spanning 16 domains. Sentences are collected from public sources or translated and generated with Gemini 3 Pro Preview (including stress categories such as tongue twisters, extreme repetition, and dense STEM content), plus 100 expressive utterances from RASA-test. Native language experts accept, correct, or discard every sentence. The set is split into three input-condition subsets: normalized (numerals, equations, acronyms verbalized), symbolic (raw numerals, formulas, operators; 1,752 sentences), and code-mixed (intra-sentential English, transliteration, mixed script; 4,164 sentences).
Raters. A multistage screen: an auditory task distinguishing a degraded from a clean recording, then a justification task using the study’s perceptual criteria, then training on guidelines and the platform. The final pool is 1,915 native raters (767 male, 1,148 female) from 22 states.
Annotation protocol. Each rater judges 150 randomly sampled sentences. For each pair of anonymized, randomized samples, the rater first gives a holistic overall preference (Model A, Model B, Both Good, Both Bad). That choice is then locked before a second screen asks for the same four-way judgment on six axes: intelligibility, expressiveness, voice quality, liveliness, hallucinations (text fidelity), and presence of noise. The two-step design is meant to keep the overall judgment from being rationalized after the fact.
Ranking. A maximum-likelihood Bradley-Terry model is fitted to the preferences and mapped to an Elo-like scale. 95% confidence intervals come from 500 bootstrap refits. One system counts as strictly better than another only when its interval lies entirely above the other’s. Systems are run with identical text, no style conditioning, default voices, and non-streaming inference. Where a system offers several voices, comparisons are restricted to same-gender pairs.
The seven systems are Gemini 2.5 Pro TTS, GPT-4o-mini TTS, ElevenLabs v3, Sonic 3, Speech 2.8 HD, Bulbul v3 Beta, and the open-source Indic F5. The paper contributes no synthesis model of its own.
Key Results
Overall leaderboard (Table 3). Gemini 2.5 Pro TTS ranks first (1128.53 ± 3, 70% win rate). ElevenLabs v3 (1056.28 ± 2) and Sonic 3 (1050.83 ± 3) share second place. They are followed by Bulbul v3 Beta (1021.91 ± 3), Speech 2.8 HD (993.94 ± 6, evaluated in only 2 languages), GPT-4o-mini TTS (942.76 ± 4), and Indic F5 last (805.75 ± 3, 19% win rate) despite covering all 10 languages and 42K comparisons. Gemini ranks first in 9 of 10 languages, with near-parity against ElevenLabs v3 in Marathi (Figure 1). It also ranks first in all 16 domains, though Speech 2.8 HD leads the Stress Test domain and several systems tie on Tongue Twisters (Figure 2).
Input-type sensitivity (Table 4). Rankings shift only modestly across the normalized, symbolic, and code-mixed subsets. The clearest shift is Bulbul v3 Beta, which rises on symbolic inputs (1048.20, against 1012.58 normalized). Indic F5 and Speech 2.8 HD both drop most on symbolic inputs (785.42 and 958.15), which points to weak text normalization of raw numerals and operators.
Perceptual drivers (§4.3, Figures 3-4). An XGBoost classifier trained on binary axis-level outcomes predicts the overall preference with 86.1% accuracy on held-out languages (Bengali, Kannada, Malayalam, Marathi, Urdu; per-language range 83.6-91.0%). SHAP attributions rank expressiveness and intelligibility as the strongest predictors, followed by liveliness and voice quality. Hallucinations and noise contribute least. The authors attribute this to most systems already being robust on those axes, which leaves little variance for them to explain. ElevenLabs v3, Sonic 3, and Bulbul v3 Beta keep strong intelligibility and robustness but trail Gemini on expressiveness and liveliness.
Reliability (§4.4, Figure 5). With Spearman ρ ≥ 0.95 against the full leaderboard as the target, stable rankings emerge at roughly 100-200 raters, depending on how many systems are compared. Mean CI width at that point is still 17-21 Elo points. With 200 raters fixed, about 1,000 sentences are needed for stable 5- and 7-system rankings and about 500 for 3 systems. Adding sentences beyond that mainly narrows the intervals rather than changing the order.
The tie between ElevenLabs v3 and Sonic 3 is declared with rounded intervals (±2 and ±3) around scores 5.45 points apart. The shared rank therefore presumably rests on the unrounded bootstrap intervals, and readers should treat the ordering in this band as fragile.
Novelty Assessment
None of the individual tools is new: pairwise preference, Bradley-Terry with bootstrap intervals, Elo-scale leaderboards, and SHAP over a tree classifier are all standard (the ranking criterion follows Chatbot Arena). The contribution lies in the controls and the scale. Text is fixed and expert-validated, input conditions are stratified, raters are screened native speakers rather than anonymous crowd users, and the overall judgment is locked before the per-axis judgments. Run over 120K comparisons in 10 Indian languages, this is a much larger and more diagnostic study than prior Indic TTS evaluations. The rater- and sentence-count reliability analysis gives practical sizing guidance, but it is specific to this pool of systems and languages. The XGBoost/SHAP analysis is interpretive rather than causal. Because the axis judgments come from the same rater immediately after the overall choice, the 86.1% predictability partly measures within-rater consistency. The benchmark and preference data are promised but not yet released.
Field Significance
Moderate. The paper provides a large, well-controlled human preference dataset for multilingual TTS in under-evaluated languages, together with concrete sample-size guidance for pairwise leaderboards. Its finding that expressiveness and intelligibility, rather than noise or hallucination, separate current commercial systems is useful evidence on what drives listener preference once baseline robustness is reached. The leaderboard itself is a time-bound snapshot of commercial APIs.
Claims
- supports: Pairwise preference judgments aggregated with a Bradley-Terry model can resolve most quality differences among modern TTS systems with statistically separable confidence intervals.
Evidence: Over 120K comparisons, most adjacent leaderboard score gaps exceed their 500-sample bootstrap 95% intervals; only one pair of systems in the 7-system ranking is declared tied. (§3.4, §4.1, Table 3)
- refines: In pairwise TTS evaluation, rank order stabilizes with fewer raters and sentences than precise score estimation, so leaderboard order is more trustworthy than small score gaps.
Evidence: Spearman ρ ≥ 0.95 is reached at about 100-200 raters while mean CI width remains 17.36-21.47 points; beyond about 1,000 sentences, extra sentences mostly shrink intervals rather than change ranks. (§4.4, Figure 5)
- supports: Once systems are reasonably robust to noise and content errors, listener preference among high-quality TTS systems is driven mainly by expressiveness and intelligibility.
Evidence: SHAP on an XGBoost classifier that predicts overall preference from six axis-level judgments ranks expressiveness and intelligibility highest and hallucinations and noise lowest; the classifier reaches 86.1% accuracy on held-out languages (83.6-91.0%). (§4.3, Figure 4)
- complicates: Aggregate TTS leaderboards hide condition-specific weaknesses, notably in handling raw numerals and symbols and in domain-specific content.
Evidence: Bulbul v3 Beta rises on symbolic inputs while Indic F5 and Speech 2.8 HD fall most there; Speech 2.8 HD ranks first on the Stress Test domain despite placing fifth overall. (§4.2, Table 4, Figure 2)
- complicates: Open-source TTS models adapted to low-resource languages can still trail commercial multilingual systems by a wide margin in native-listener preference.
Evidence: Indic F5 ranks last overall (805.75 against 1128.53 for the top system, 19% win rate) and at or near the bottom in every language and domain, despite the largest language coverage and 42K comparisons. (§4.1, Table 3, Figure 1)
Limitations and Open Questions
Warning
System coverage is uneven: Speech 2.8 HD is evaluated in only 2 languages (7.8K comparisons) and GPT-4o-mini TTS in 5, so their overall scores rest on different language mixes than those of systems covering 9-10 languages. Overall rank comparisons across these systems are therefore not like-for-like.
- Six of the seven systems are closed commercial APIs evaluated with default voices at a single point in time. The leaderboard cannot be reproduced once providers update their models, and default-voice choice is a confound.
- All systems are run without style conditioning and in non-streaming mode. Results do not speak to expressive control or streaming deployment.
- The axis-level judgments are binarized (better or both-good versus worse or both-bad) and collected from the same rater right after the overall choice. Both choices limit how far the SHAP attributions can be read as independent perceptual drivers.
- The reliability thresholds (100-200 raters, about 1,000 sentences) are measured on this specific system pool and rater population, and may not transfer to closer-matched systems or other languages.
- The benchmark and preference data are announced for release but were not available at the time of the paper.
Wiki Connections
- Subjective Evaluation — The paper is a large-scale human listening study (120K pairwise judgments from 1,915 screened native raters) with a locked two-step overall-then-axis protocol.
- Evaluation Metrics — The paper contributes a multidimensional pairwise evaluation methodology with Bradley-Terry ranking and an empirical analysis of how many raters and sentences a stable leaderboard needs.
- The State Of TTS: Human Fooling Rates — Earlier human evaluation of TTS by the same group; this paper moves from fooling rates to multidimensional pairwise preference across Indian languages.
- Pairwise Evaluation of Accent Similarity — Prior pairwise TTS evaluation study that the paper cites as focusing on a single dimension; this paper extends pairwise judgment to six perceptual axes.
- F5-TTS — Cited as a recent neural TTS advance; the evaluated open-source Indic F5 system comes from the group’s Indic adaptation work in this lineage and ranks last in the study.