arXiv · 2026 · Preprint

Ali Jafar et al. (National University of Computer and Emerging Sciences (FAST-NUCES)) · → Paper · Demo: ? · Code: ✓

Evaluates four publicly accessible Urdu TTS systems across four communicative domains (Formal, Conversational, Literary/Storytelling, Emotional) with a MUSHRA-style listening test, an ABX test and three reference-distance metrics, and finds that the listener-preferred system and the acoustically closest system are different.

Problem

Comparative TTS studies, and Urdu TTS studies in particular, mostly score short neutral read-speech prompts with a single metric, usually MOS. That design cannot show whether a system that sounds natural on read news text stays acceptable in conversation, storytelling or emotional speech, and prior Urdu evaluations (including the Tacotron-versus-MMS comparison of Naseem et al.) used read speech only. The authors argue that no published benchmark measures how synthesis quality varies across communicative styles for any South Asian language, so practitioners have no principled basis for choosing a system for a given application.

Method

The study is a benchmark of four existing systems chosen by three criteria: public programmatic Urdu synthesis at collection time (November-December 2024), direct Perso-Arabic script input without romanisation, and coverage of distinct paradigms (§3.1). The systems are MMS-TTS (facebook/mms-tts-urd-script_arabic, a VITS model with a fixed single male voice, 16 kHz), Indic-Parler-TTS (ai4bharat/indic-parler-tts, the description-conditioned autoregressive codec LM of Parler-TTS extended to 22 Indic languages, run with one fixed description selecting the male speaker “Rohit”, sampling temperature 0.8, 44.1 kHz), Microsoft Edge TTS (the ur-PK-AsadNeural voice at default prosody, 24 kHz) and Google Gemini TTS (gemini-2.5-flash-preview-tts with the Charon preset, 24 kHz) (§3.2, §6.1). Each system’s voice configuration is held fixed across domains so that the domain is the only manipulated factor, and each utterance is synthesised once from its original transcript.

The evaluation corpus has 60 male-speaker reference utterances per domain: Formal draws 30 from FLEURS Urdu and 30 Wikipedia-style utterances from UrduSpeech; Conversational and Literary/Storytelling use the conversational and BOOK styles of UrduSpeech; Emotional uses UrSEC, a drama-derived emotion-recognition corpus, balanced over six emotions with Neutral excluded (§4). That gives 4 systems × 4 domains × 60 = 960 reference-synthetic pairs for the objective metrics (§5.1, §6.1). Because every system speaks with its own fixed voice rather than cloning the reference, every objective comparison is between different speakers.

Five measurements are applied (§5, Table 2). A MUSHRA-inspired test, run in Google Forms, shows a reference, a degraded anchor and the four system outputs together and asks for 1-5 ratings of naturalness, clarity and overall quality; the reference and anchor are not rated. It uses 20 native Urdu listeners, each covering two domains (10 per domain) on a 30-utterance subset per domain (§5.2.1, §6.1). An ABX test is then run on only the two MUSHRA leaders, Edge TTS and Gemini TTS: listeners hear the reference twice (A and B) and an unknown X that is either the reference or a synthetic output (§5.2.2). The objective measures are Resemblyzer GE2E d-vector cosine similarity at 16 kHz, MCD over 13 WORLD/SPTK mel-cepstral coefficients after DTW at 22.05 kHz, and F0 RMSE in cents over DTW-aligned voiced frames using pyworld harvest at 16 kHz (§5.3, §6.2). Scripts, result tables and Colab notebooks are released on GitHub.

Key Results

Listening test (Table 4). Gemini TTS has the highest overall MUSHRA-style score (3.040), narrowly ahead of Edge TTS (2.947), with MMS-TTS (2.322) and Indic-Parler-TTS (2.099) well behind. Gemini leads Conversational (3.148 vs. 3.093), Literary/Storytelling (3.602 vs. 3.361) and Emotional (2.167 vs. 1.963); Edge leads Formal (3.373 vs. 3.244). Every system scores lowest on Emotional. Per-cell standard deviations are large (often above 1.0 on Literary and Formal), and no significance tests are reported, so the 0.093 overall gap between Gemini and Edge is not shown to be reliable.

ABX (Table 5). Listeners chose correctly in 90.7% of trials (98/108; 49/54 on each of Literary/Storytelling and Formal), well above the 50% chance level, so the two strongest systems remain distinguishable from natural Urdu speech.

Speaker similarity (Tables 6-8). At domain level, Literary/Storytelling has the highest mean similarity (0.6903) and Emotional the lowest (0.5437). At system level, Edge TTS has the highest overall mean (0.6329), with Indic-Parler-TTS close behind (0.6312) and Gemini TTS lowest (0.5956). Rankings flip by domain: Indic-Parler-TTS is highest on Conversational (0.7293) and Literary (0.7542) but lowest on Formal (0.5172), while MMS-TTS is highest on Formal (0.6814) and Emotional (0.5700) but lowest on Conversational and Literary. The authors caution that MMS-TTS’s scores mostly reflect the mismatch between its fixed voice and each reference speaker.

MCD (Tables 9-10). The domain means run from Conversational (6.90 dB, the lowest) through Literary (7.87) and Formal (9.12) to Emotional (12.03 dB). Edge TTS has the lowest overall MCD (7.91 dB) and the lowest value in four of five domain-subset conditions. Indic-Parler-TTS has the lowest Emotional MCD (10.62 dB), Gemini TTS the highest Emotional MCD (13.58 dB), and MMS-TTS the highest overall MCD (9.93 dB vs. 9.79 for Gemini).

F0 RMSE (Tables 11-12). Emotional is the clear outlier (888.54 cents, against 520.98 for Formal, 521.24 for Literary and 546.02 for Conversational). Edge TTS has the lowest overall mean (565.00 cents), just ahead of MMS-TTS (573.17), and Gemini TTS the highest (646.10; 1027.44 on Emotional). Per condition, Edge is lowest on Conversational, Formal (UrduSpeech) and Emotional, MMS-TTS on Literary (492.08), and Indic-Parler-TTS and MMS-TTS tie on Formal (FLEURS) (470.66).

Cross-metric reading (§7.6, §8.3). Every domain-level metric puts Emotional last, and MUSHRA, MCD and F0 RMSE are each worst on Emotional for every individual system. At system level the metrics disagree: listeners prefer Gemini TTS, which has the worst overall F0 RMSE and speaker similarity, while Edge TTS leads the reference-distance metrics and Indic-Parler-TTS combines near-top speaker similarity with the lowest overall MUSHRA-style score.

The prose and the tables disagree in several places; this page follows the tables, as checked against the PDF text. §7.3.2, §7.6.2 and §8.1 call Indic-Parler-TTS’s 0.6312 the highest overall similarity, but Edge TTS’s 0.6329 in Table 7 is higher. §7.5.2 says Edge has the lowest F0 RMSE in four of five conditions, but Table 12 supports three. §8.1 says MMS-TTS is worst on MCD in every domain, but Table 10 shows Gemini worst on Emotional and Formal (FLEURS). §7.6.2 says Gemini is worst on every objective measure, but MMS-TTS has the higher overall MCD. The ABX reporting also conflicts with itself, as described under Limitations.

Novelty Assessment

None of the systems, metrics or datasets is new. The contribution is the experimental design: stratifying an Urdu evaluation by communicative domain, running subjective and objective measures side by side, and releasing the pipeline. Applied to Urdu this design is new, and including Gemini TTS gives what the authors call the first published evaluation of that system on Urdu. Methodologically the study is thin. The MUSHRA adaptation drops the 0-100 continuous scale and rates only the four systems. The listener pool (10 per domain, recruited from the authors’ networks) is small and no significance testing is reported. The ABX design, with both A and B being the reference, departs from the standard paradigm. Because the systems never clone the reference voice, the reference-distance metrics mix synthesis quality with speaker and register mismatch, which undercuts the subjective-objective inversion the paper presents as its central finding.

Field Significance

Low — The paper adds a single-language data point showing that system rankings for publicly available Urdu TTS change across domains and between listener-preference and reference-distance metrics, and that emotional speech is the hardest condition on every domain-level measure. With a small listener panel, no significance tests, and reference-distance metrics confounded by fixed non-reference voices, it is mainly a practitioner-facing comparison and a reusable evaluation pipeline for Urdu, not new evaluation methodology.

Claims

  • complicates: Listener-preference ratings and reference-distance acoustic metrics can rank TTS systems in opposite orders, so a single evaluation paradigm can pick a different best system.

    Evidence: On Urdu, Gemini TTS has the highest overall MUSHRA-style score (3.040) but the highest overall F0 RMSE (646.10 cents) and the lowest overall Resemblyzer similarity (0.5956), while Edge TTS has the lowest overall MCD (7.91 dB) and F0 RMSE (565.00 cents) and Indic-Parler-TTS pairs near-top similarity (0.6312) with the lowest overall MUSHRA-style score (2.099). Every system speaks in a fixed voice different from the reference, which confounds the acoustic metrics. (§7.6.2, §8.3, Tables 4, 7, 10, 12)

  • supports: Emotional speech is a harder evaluation condition for TTS than formal, conversational or narrative speech, both perceptually and in distance to natural references.

    Evidence: On Urdu, the Emotional domain has the lowest MUSHRA-style score for every system, the highest domain-level MCD (12.03 dB vs. 6.90-9.12) and F0 RMSE (888.54 cents vs. 520.98-546.02 cents), and the lowest domain-level speaker similarity (0.5437). The emotional references come from a drama-derived emotion corpus, while every system is run at its default or fixed neutral voice setting. (§7.6.1, Tables 4, 6, 9, 11)

  • complicates: TTS system rankings measured on one speech domain do not transfer reliably to other domains in a low-resource language.

    Evidence: Edge TTS leads the Formal MUSHRA-style column while Gemini TTS leads the other three; Indic-Parler-TTS has the highest speaker similarity on Conversational and Literary but the lowest on Formal; Indic-Parler-TTS has the lowest Emotional MCD despite trailing Edge TTS overall; even the two Formal source subsets order systems differently. (§7.3.3, §7.4.2, §7.6.4, Tables 4, 7, 10)

  • supports: High subjective quality ratings do not imply that synthetic speech is perceptually indistinguishable from natural speech.

    Evidence: The two highest-rated Urdu systems in the MUSHRA-style test (Gemini TTS and Edge TTS) are still identified correctly against the natural reference in 90.7% of ABX trials (98/108) on the Literary/Storytelling and Formal domains, far above the 50% chance level. (§7.2, §7.6.2, Table 5)

Limitations and Open Questions

Small, non-standard listening tests with inconsistent ABX reporting

The MUSHRA-style test uses 10 listeners per domain on a discrete 1-5 Google Forms scale, with no reported significance testing, and the ABX section contradicts itself. Table 5’s caption and §5.2.2 name Edge TTS and Gemini TTS, but §7.2’s text names MMS-TTS and Gemini TTS. Table 5 covers Literary/Storytelling and Formal (90.7% each), but §8.3 cites “87.5-88.9% of Emotional and Literary/Storytelling trials” and §8.2 mentions a reduced Emotional ABX rate for Edge TTS that no table reports. Table 5’s trial count assumes 9 listeners per domain, against the 10 stated in §5.2. Treat the subjective rankings and ABX conclusions as indicative only.

  • Reference-distance metrics under voice mismatch. No system clones the reference speaker, so Resemblyzer similarity, MCD and F0 RMSE all measure distance to a different speaker’s recording. The authors acknowledge this for MMS-TTS and for F0 register offsets (§5.3.3), but it applies to all four systems and limits what the subjective-objective inversion can show.
  • Single synthesis per utterance. Indic-Parler-TTS is sampled at temperature 0.8 and the cloud services are not seed-controllable, so the variance from repeated synthesis is unmeasured (§6.1).
  • Formal domain heterogeneity. The FLEURS and UrduSpeech subsets order the systems differently. The identical Indic-Parler-TTS and MMS-TTS values on Formal (FLEURS) for both MCD (10.37 dB) and F0 RMSE (470.66 cents) are reported without verification (Tables 10, 12).
  • Scope. The study covers male speakers only, one fixed voice per system, no intelligibility metric (WER or CER) and no code-switched text. Gemini TTS is a preview endpoint and the cloud systems are opaque, so results may not reproduce as the services change.
  • Emotion is a domain, not a target. Systems are not prompted for emotion, so the Emotional results measure how far neutral-configured synthesis is from acted emotional references, not emotion rendering.

Wiki Connections

  • Subjective Evaluation — Runs a MUSHRA-inspired listening test (20 native Urdu listeners, 10 per domain) and an ABX discrimination test on real system output, and documents its deviations from ITU-R BS.1534.
  • Parler-TTS — The description-conditioned architecture behind Indic-Parler-TTS, one of the four evaluated systems; it receives the lowest overall MUSHRA-style score on Urdu despite near-top speaker similarity.
  • Developing High-Quality TTS for Punjabi and Urdu — Prior Urdu evaluation of Tacotron-based systems against MMS-TTS on read speech only; this study extends that kind of comparison to four domains.
  • PashtoTTS-Bench — Cited as evidence that ASR round-trip intelligibility checks miss language-identification and script-fidelity failures in non-Latin-script low-resource TTS, motivating a multi-metric design.