arXiv · 2026 · Preprint
Venkata Pushpak Teja Menta (Praxel Ventures) · → Paper · Demo: ✗ · Code: ✓
Proposes PSP (Phoneme Substitution Profile), an automatic accent benchmark for Hindi, Telugu, and Tamil TTS that splits accent into six named dimensions (retroflex collapse, aspiration, vowel length, Tamil zha, FAD, and prosodic signature divergence), and applies it to four commercial and open-source systems on 10-utterance pilot sets.
Problem
Automatic TTS evaluation for Indic languages relies on intelligibility (WER, CER) and overall naturalness predictors (MOS, UTMOS), none of which target accent. A system can transcribe perfectly and still sound non-native because it collapses contrasts that are phonemic in the target language: retroflex versus dental consonants, aspirated versus unaspirated stops, long versus short vowels, and the Tamil retroflex approximant (zha). Existing accent-oriented metrics either produce a single scalar (Fréchet-style distribution distances, pairwise accent similarity) or are rule-based and English-specific, so they cannot say which phonological feature a system gets wrong.
Method
PSP defines accent as a vector of per-feature fidelity scores. For each phonological dimension, a native phoneme set and its typical substitute set are defined (for example retroflex {ṭ, ḍ, ṇ, ṣ, ḷ} versus dental {t, d, n, s, l}). A generated utterance is force-aligned with a language-specific Wav2Vec2 CTC aligner (torchaudio forced_align), the Wav2Vec2-XLS-R 300M layer-9 embedding is extracted over each expected native-phoneme span, and fidelity is the rectified cosine similarity to a native centroid normalised by the sum of similarities to the native and substitute centroids. A token counts as collapsed below a threshold of 0.5. Four per-phoneme dimensions are instantiated: retroflex (RR), aspiration (AF; Hindi primarily, sparse in Telugu, not applicable in Tamil), vowel length (LF; a long/short duration-ratio comparison against a native prior of about 1.90), and Tamil zha (ZF).
Centroids come from 500 studio-recorded native clips per language (IndicTTS for Telugu and Tamil, Rasa for Hindi), sampled across speakers with a 25-clip per-speaker cap. The substitute centroid is built from the cognate dental, unaspirated, or short-vowel graphemes in the same utterances so that microphone, room, and speaker timbre are matched. Two corpus-level dimensions complete the profile: FAD between generated and 1000 native utterance-level XLS-R layer-9 embeddings, and prosodic signature divergence (PSD), a Fréchet distance over a 5-D prosodic feature vector (pitch range, log-F0 mean, speech rate, nPVI, log-duration) against about 500 native clips. Code is released under MIT and centroids under CC-BY, together with 300-utterance held-out golden sets per language that are not yet used in this version’s experiments.
Key Results
All results are on 10-utterance pilot sets (20 wavs per commercial system with two voices, 10 per Praxy configuration) and are labelled preliminary by the author; no ranking-significance claims are made.
- Difficulty gradient (Tables III, V, VII). Mean retroflex collapse over the four non-Praxy systems is about 1% on Hindi (only Indic Parler-TTS collapses, at 0.045), about 40% on Telugu (0.333 to 0.500), and about 68% on Tamil (64.3% to 70.5%). Hindi aspiration collapse is 0 for all four systems.
- WER and FAD orderings diverge on Hindi (Table VI). Sarvam Bulbul has the lowest FAD (211.8) and Cartesia Sonic-3 the highest (267.4), while the WER ordering cited from prior benchmarks puts ElevenLabs first and Cartesia second.
- Telugu (Tables III, IV). Sarvam Bulbul and Indic Parler-TTS tie for the lowest collapse rate (0.333); Cartesia is worst (0.500). Retroflex fidelity and collapse rate do not rank systems identically: Praxy R5 has the highest mean fidelity (0.891) but a 0.400 collapse rate. PSD isolates a prosodic failure in ElevenLabs v3 (154.4 versus 11.1 for Sarvam and 10.4 for Parler), traced to a narrower log-F0 range (0.87 versus 1.44 native). Sarvam leads FAD (250.4) and ties Cartesia on LLM-WER (0.029, a Qwen-2.5-72B semantic judge over Whisper transcripts).
- Tamil (Table VII). Indic Parler-TTS is best on RR (64.3%), ZF (61.5% collapse), LF (0.30), and PSD (27.1), while Sarvam leads FAD (200.3). Three systems share an 85.7% zha collapse, which the paper itself attributes partly to a 7-token sample. Sarvam, described as Indic-first, has the highest Tamil retroflex collapse of the four systems (70.5%).
- Praxy Voice case study. From R5 to R6 (about 85 h to about 1,220 h of LoRA training data), Telugu retroflex collapse stays at 0.400, FAD falls from 534.4 to 355.0, and PSD rises from 14.1 to 61.7. Supplying a commercial Telugu clip as a zero-shot voice prompt lowers collapse to 0.267 (Sarvam reference) or 0.333 (Cartesia reference) and PSD to 13.1 or 26.5 (Tables III, IV).
- Native-audio sanity check (Table I). On 50 held-out native utterances per language, Hindi scores RR and AF of 1.00, but Telugu and Tamil native audio score RR of only 0.54 and 0.47; the empirical native Telugu collapse rate is 0.43 (Table III), inside the 0.267 to 0.500 range of the synthetic systems. Native FAD (31.9 to 43.5) and PSD (2.1 to 5.6) are far below every synthetic system.
Validity evidence consists of five internal-consistency signals and one informal native-listener ear test; correlation with human accent ratings is deferred to a v2.
Novelty Assessment
The contribution is an evaluation design rather than a new modelling idea. Its components are established (XLS-R embeddings, CTC forced alignment, prototype-centroid probing, Fréchet distances, nPVI), and the novelty lies in assembling them into named, per-phonological-dimension accent scores for Indic languages, with a substitute centroid constructed to share acoustic channel conditions with the native centroid. Compared with the English, rule-based phoneme shift rate of Quantifying Speaker Embedding Phonological Rule Interactions, PSP is acoustic-probe-based and decomposed. The empirical side is thin: pilot sets are tiny, token counts per cell are 15 to 30, Praxy Voice is the author’s own system, and the per-phoneme probes do not separate native Telugu and Tamil audio from synthetic output. The released centroids and golden sets are the more durable part of the contribution.
Field Significance
Low — PSP offers a concrete, inspectable recipe for measuring specific Indic phonological contrasts in TTS output and shows on Hindi data that distributional accent distance (FAD) can rank systems differently from intelligibility. Without human-rating correlation and with per-phoneme probes whose native-audio scores overlap the synthetic systems on Telugu and Tamil, it currently serves as a diagnostic prototype rather than validated evidence about which systems sound more native.
Claims
- supports: Intelligibility metrics such as WER do not capture accent, so systems ranked by intelligibility can rank differently on distributional distance from native speech.
Evidence: On the Hindi pilot set, Sarvam Bulbul has the lowest XLS-R FAD (211.8) and Cartesia Sonic-3 the highest (267.4), while prior-benchmark WER places ElevenLabs first and Cartesia second; on Telugu, ElevenLabs v3 has a 0.041 LLM-WER but a PSD of 154.4 versus 11.1 for Sarvam. (§VI.B, §VI.C, Table IV, Table VI)
- supports: Phonemic contrasts specific to the target language degrade faster than intelligibility as the target language moves further from the well-resourced core of a TTS system’s training data.
Evidence: Mean retroflex collapse across four commercial and open-source systems is about 1% on Hindi, about 40% on Telugu, and about 68% on Tamil, and Tamil vowel-length fidelity is 0.13 to 0.30 for all four systems. (§VI.B to §VI.E, Tables III, V, VII)
- complicates: Per-phoneme acoustic-probe accent metrics inherit the quality of the forced aligner and can fail to separate native from synthetic speech in lower-resource languages.
Evidence: Held-out native Telugu audio has a retroflex collapse rate of 0.43 (fidelity 0.538, 221 tokens), inside the 0.267 to 0.500 range of the synthetic systems, and native Tamil scores RR 0.47, while native Hindi with a stronger aligner scores 1.00. (§V Signal 5, §VII, Table I, Table III)
- refines: Increasing training data for a lightly adapted multilingual TTS model can move utterance-level spectral distance towards native speech while worsening prosody and leaving segmental contrasts unchanged.
Evidence: Moving Praxy Voice from about 85 h (R5) to about 1,220 h (R6) of LoRA training data leaves Telugu retroflex collapse at 0.400, reduces FAD from 534.4 to 355.0, and raises PSD from 14.1 to 61.7. (§VI.C, Table III, Table IV)
- supports: Conditioning a TTS model on a native-speaker voice prompt can recover prosodic and segmental accent features that text-side adaptation alone does not.
Evidence: Supplying an 8 to 9 s commercial Telugu clip as Chatterbox’s zero-shot voice prompt lowers Praxy R6 retroflex collapse from 0.400 to 0.267 (Sarvam reference) and PSD from 61.7 to 13.1, with LLM-WER unchanged (0.033 to 0.034). (§VI.C, Table III, Table IV)
Limitations and Open Questions
Warning
PSP is not yet validated against human accent judgements, all system comparisons use 10-utterance pilot sets (15 to 30 retroflex tokens per cell), and on Telugu and Tamil the per-phoneme probes score held-out native audio no better than the synthetic systems, so per-phoneme Telugu and Tamil rankings are not interpretable as absolute accent quality.
- The author recommends reading Telugu and Tamil per-phoneme scores only as relative rankings; even that use is limited because 5-point differences are, by the paper’s own account, not statistically distinguishable at these sample sizes.
- Centroid and test corpora share IndicTTS and Rasa speaker pools, and the PSD feature vector is not normalised across dimensions of very different scale (nPVI around 100, log-F0 around 1), so PSD may be dominated by the rhythm feature.
- The FAD reference set is drawn from studio recordings, so FAD may also reflect channel and speaker-pool mismatch rather than accent alone.
- Each commercial system is represented by its own vendor voices (two genders) and each Praxy configuration by a single voice, so voice choice is confounded with system.
- The Praxy Voice rows are the author’s own in-progress system, and the LLM-WER and intent metrics depend on a Qwen-2.5-72B judge rather than a standard WER computation.
- Code-mixed input, conjunct epenthesis, and normalised per-phoneme scores are deferred to v2.
Wiki Connections
- Evaluation Metrics — Introduces a per-dimension automatic accent metric for Indic TTS, combining phoneme-level probes with corpus-level FAD and a prosodic Fréchet distance.
- Self-Supervised Speech — All per-phoneme probes and the FAD dimension are computed in Wav2Vec2-XLS-R 300M layer-9 embedding space, with Wav2Vec2 CTC models providing the forced alignment.
- Praxy Voice — Companion paper by the same author whose system is evaluated here as the R5, R6, and voice-prompt configurations, using PSP as a development diagnostic.
- Quantifying Speaker Embedding Phonological Rule Interactions — Positioned as the closest prior metric (phoneme shift rate for American and British English); PSP is presented as an acoustic, Indic, per-dimension complement.
- Learning-free L2-Accented Speech Generation — Cited as related phonological-rule machinery used for accent generation rather than evaluation.
- Pairwise Evaluation of Accent Similarity — Cited as a single-scalar accent-similarity approach (PPG and formant distances) that PSP contrasts with its decomposed scores.
- UTMOS — Cited as an overall-quality predictor that does not target accent, motivating a dedicated accent metric.