arXiv · 2026 · Preprint

Hanif Rahman (Independent Researcher) · → Paper · Demo: ? · Code: ✓

Proposes INSV, a four-part reporting framework (intelligibility, naturalness, script fidelity, language verification) for TTS in low-resource non-Latin-script languages, and releases its automated subset as PashtoTTS-Bench, the first dated multi-system Pashto TTS screening benchmark.

Problem

TTS evaluation for low-resource languages usually collapses quality into a single ASR round-trip word error rate. For non-Latin-script languages such as Pashto this hides distinct failures: a system may produce no audio, speak a neighbouring language (Urdu or Dari), yield a transcript in the wrong script, collapse language-specific phonemes (Pashto lateral fricatives and retroflexes), or sound unnatural to native listeners. The available automatic tools are themselves unreliable for Pashto: Whisper does not identify Pashto, generic multilingual recognisers decode natural Pashto poorly, and MOS predictors are calibrated on English and Japanese. No published multi-system Pashto TTS evaluation existed.

Method

INSV separates four dimensions: Intelligibility (ASR round-trip WER, CER, share of perfect utterances), Naturalness (native-listener MOS under ITU-T P.800/P.808), Script fidelity (Script Fidelity Rate, SFR: the fraction of countable transcript characters inside the target script’s Unicode blocks, adopted from the author’s prior work), and language Verification (automatic language ID, with native adjudication for disputes). The framework imposes a dependency order: script fidelity and language verification must be checked before WER is interpreted, because WER on wrong-language or wrong-script output measures a category error. This release reports only INSV-A, the automated subset (synthesis completion, S, V, I); the MOS protocol (50-sentence stratified subset, 16 target native raters across four counterbalanced forms, Krippendorff’s alpha) is specified but not run.

PashtoTTS-Bench instantiates INSV-A on 200 FLEURS Pashto prompts and 200 filtered Common Voice 24 Pashto prompts. The April-May 2026 run covers Microsoft Edge neural voices GulNawaz and Latifa, the open OmniVoice model in two modes (zero-shot cloning with Edge GulNawaz reference audio, and an automatic voice mode with no reference), and an Urdu Edge voice as a neighbouring-language negative control. Fish Speech S2-Pro was only smoke-tested, and Google, OpenAI, ElevenLabs, XTTS v2, and MMS-TTS were excluded for lacking a verified Pashto path (Table 3). Intelligibility uses two recognisers: omniASR_CTC_300M_v2 as the independent primary metric (O-WER) and the author-trained pashto-asr-v3 (a W2V-BERT 2.0 CTC model) as a language-specific secondary (P-WER), chosen because the author also built the Edge Pashto TTS training data. Language ID uses MMS-LID-4017 and SpeechBrain VoxLingua107; Whisper Large V3 is kept as a diagnostic only. WER carries 1,000-resample bootstrap 95% confidence intervals. The paper also proposes a five-mode failure typology (F1 pre-synthesis rejection, F2 language substitution, F3 phoneme collapse, F4 prosodic disfluency, F5 grapheme ambiguity) mapped to INSV dimensions (Table 4), plus a grapheme-class WER screen.

Key Results

Under the independent O-WER (Table 5), OmniVoice auto has the lowest WER on both sets (24.1% FLEURS, 27.4% CV24), followed on FLEURS by Edge GulNawaz (32.8%), Edge Latifa (35.6%), and OmniVoice clone (45.4%). On CV24 the order differs: OmniVoice clone (34.8%) sits between OmniVoice auto and GulNawaz (39.5%), with Latifa at 47.7%. P-WER gives the same per-set orderings at lower absolute values (FLEURS: 20.4 / 26.3 / 27.0 / 41.2). OmniVoice clone produced only 195/200 FLEURS files (partial F1); the other Pashto systems produced all files.

Natural FLEURS speech scores 47.9% O-WER, so OmniVoice auto is 23.8 points below the natural-speech baseline (natural minus synthetic). Under pashto-asr-v3 the same difference is 14.2 points (34.6% vs. 20.4%). The paper attributes this to the recogniser favouring clean synthetic audio, not to superior intelligibility.

All four Pashto-targeted systems and the Urdu control reach transcript SFR = 1.000, so SFR did not discriminate in this run. The Urdu control is separated by WER (90.1% / 92.0% O-WER) and LID (MMS Pashto-ID 9.0% on both sets, SpeechBrain 3.0% / 11.5%). Whisper Large V3 returned 0.0% Pashto labels on manually checked Pashto TTS audio, against 89.4% for MMS-LID-4017 and 99.1% for SpeechBrain on the same audio (§6.2). MMS and SpeechBrain disagree on Edge GulNawaz CV24 (65.0% vs. 98.0%), which the paper leaves unresolved. Grapheme-class WER (Table 6) shows no class-specific spike for any system. Edge GulNawaz shows a candidate F5 pattern: both recognisers write U+06D0 where the reference uses U+06CC in one verb family across three FLEURS sentences (Table 9). No MOS is reported.

Novelty Assessment

The contribution is evaluation methodology and a dated benchmark, not a model. The INSV framing (gate WER behind script and language checks, require multi-model LID, treat disagreement as unresolved) is a sensible codification rather than a new metric; SFR itself comes from the author’s earlier work. The genuinely new artifacts are the Pashto prompt sets with per-sentence scores, the provider metadata, the failure typology, and the empirical findings about tooling (Whisper’s Pashto LID failure, synthetic speech beating natural speech on an unconditioned recogniser). The study is small (four Pashto systems, two of them from one provider) and single-author, with no human evaluation in this release.

Field Significance

Low. The evidence comes from a single-author automated screen of four Pashto-targeted conditions with no human evaluation, so its rankings are narrow. The paper does provide the first dated multi-system screening benchmark for Pashto TTS and a reusable recipe for porting the same automated checks to other non-Latin-script languages (Table 2). Its most transferable contribution is cautionary: it demonstrates on concrete data that round-trip WER can rank synthetic speech above natural speech, that a widely used LID model can fail outright on a low-resource language, and that reference orthography can inflate WER independently of pronunciation.

Claims

  • complicates: ASR round-trip WER on synthetic speech can fall well below the WER of natural speech under the same recogniser, so a sub-baseline WER reflects acoustic cleanliness rather than better-than-human intelligibility.

    Evidence: Under omniASR_CTC_300M_v2, OmniVoice auto scores 24.1% WER on FLEURS Pashto while natural FLEURS speech scores 47.9%; under the Pashto-tuned recogniser the gap narrows to 20.4% vs. 34.6%, consistent with recogniser sensitivity to natural-speech variability. (§6.1, §7.3, Table 5)

  • supports: Verifying the output language of TTS in low-resource languages requires multiple automatic language-ID models, because a single LID model can fail systematically on the target language.

    Evidence: Whisper Large V3 assigns 0.0% Pashto labels to checked Pashto TTS audio, while MMS-LID-4017 and SpeechBrain VoxLingua107 assign 89.4% and 99.1%; MMS and SpeechBrain also disagree on Edge GulNawaz Common Voice output (65.0% vs. 98.0%). (§6.2, §7.4, Table 5)

  • complicates: In scripts with ambiguous graphemes, round-trip WER can count phonologically correct synthesis as errors when the reference text and the recogniser use different codepoint conventions.

    Evidence: For Edge GulNawaz, two independent recognisers both transcribe U+06D0 where the FLEURS reference uses the ambiguous U+06CC in one verb family across three sentences, inflating WER; the paper treats this as a candidate pending native adjudication. (§5.5, §6.5, Supplementary §C, Table 9)

  • complicates: Transcript script-fidelity checks only discriminate when systems lack a target-language path; among systems sharing the target script they add no signal and cannot detect neighbouring-language speech in the same script.

    Evidence: All four Pashto-targeted systems and the Urdu-voice negative control reach SFR = 1.000 on both test sets, while the Urdu control is separated only by WER (90.1% / 92.0%) and LID (MMS 9.0%). (§6.2, §7.2, Table 5)

  • refines: A neighbouring-language negative control provides a useful sanity check that automatic intelligibility and language-ID screens separate target-language from substituted-language output.

    Evidence: An Urdu Edge voice given Pashto prompts produces audio for every prompt but receives 77.8-80.4% P-WER, 90.1-92.0% O-WER, and low Pashto-ID rates from MMS and SpeechBrain. (§6.1, Table 5)

Limitations and Open Questions

No human evaluation and an author conflict of interest

No MOS, native language adjudication, or phonetic annotation is reported, so naturalness and phoneme-level correctness (F3, F4) remain unmeasured. The author trained the secondary recogniser (pashto-asr-v3) and created the Edge Pashto TTS training data; the paper mitigates this by making an independent recogniser the primary metric.

  • Only four Pashto-targeted conditions are ranked, two from one commercial provider and two modes of one open model; Fish Speech S2-Pro is excluded for an incomplete run.
  • The CV24 natural-speech P-WER baseline (32.5%) is taken from prior work because the audio is not redistributable, and pashto-asr-v3 was fine-tuned on FLEURS training data, so its FLEURS numbers may be optimistic.
  • The failure typology is derived from one language and manual inspection; the paper states it is not a validated cross-language taxonomy.
  • Dialect variation (for example Yousafzai or Waziri Pashto) is not controlled in voices, recognisers, or text.
  • The MMS vs. SpeechBrain disagreement on GulNawaz CV24 output is unexplained, and the LID thresholds in the report card (Table 8) are not empirically calibrated.
  • Commercial audio is not redistributed, so reproduction depends on regenerating audio from providers whose models change over time.

Wiki Connections

  • Evaluation Metrics — proposes a gated multi-dimension reporting framework and documents concrete failure cases of round-trip WER, script fidelity, and single-model LID for low-resource TTS.
  • Omnilingual ASR — supplies omniASR_CTC_300M_v2, the independent primary recogniser for round-trip WER.
  • Whisper — Whisper Large V3 is retained only as a diagnostic after it returns 0.0% Pashto language labels on Pashto TTS audio.
  • SpeechBrain — provides the VoxLingua107 ECAPA language-ID model used as one of two LID signals.
  • OmniVoice — evaluated in zero-shot cloning and automatic voice modes; the automatic mode gives the lowest WER in the benchmark.
  • UTMOS — deliberately excluded as an automatic naturalness proxy because it is not validated for Pashto.