arXiv · 2026 · Preprint
David Guzmán et al. (McGill University / Mila - Quebec AI Institute / AIMS Research and Innovation Centre / NM-AIST / Saarland University) · → Paper · Demo: ? · Code: ✓
Releases an openly licensed, verse-aligned TTS corpus of 3,469 hours across 37 genuinely low-resource languages and uses it to compare per-language from-scratch TTS (FastSpeech 2, VITS, F5-TTS) against off-the-shelf multilingual systems (OmniVoice, Gemini-TTS), finding that no system wins on every language and metric.
Problem
Low-resource TTS research often simulates scarcity by downsampling high-resource corpora, which hides the real difficulties of underrepresented languages: orthographic variation, limited phonetic coverage, and noisy alignment. Bible recordings are the most available speech source for many such languages, but earlier Bible-derived corpora (CMU Wilderness, BibleTTS) either cover a single region or face licensing and alignment barriers to scaling. It is also unclear whether large multilingual speech generation models, which advertise coverage of hundreds of languages, actually beat small models trained from scratch on the target language.
Method
Corpus construction. OpenBibleTTS is built from CC BY-SA Open Bible recordings (chapter-level MP3s) and USFM/USX transcripts. For 28 languages with word-level timing metadata, verse boundaries are cut directly from the timestamps; for the remaining 9, the ReadAlongs Studio zero-shot forced aligner (CTC-based, with a universal und g2p fallback) produces word boundaries, with a dummy transcript line absorbing narrator preambles (§3.3). Speaker labels come from pyannote diarization run on a concatenation of the first verse of each Bible book, propagated book-wide (§3.4). Quality filtering follows the BibleTTS framework: a 30-second duration ceiling, a 10-character transcript floor, a CTC feasibility check, and removal of audio-to-text ratio outliers beyond three standard deviations (§3.5). The result is 3,469 hours and 1,121,956 utterances over 37 languages in five regions (19 African, 13 South Asian), nine families and seven scripts, roughly 25-30k training utterances per language (Table 1).
Systems. Three architectures are trained monolingually from scratch for each language under a matched budget of 500,000 optimizer updates (§4.2, Appendix B):
- EveryVoice: a modified FastSpeech 2 acoustic model (18.2M parameters, transformer-enc-dec) with learned multi-speaker embeddings, plus an iSTFTNet vocoder (a lightweight HiFi-GAN variant) fine-tuned from a pretrained universal checkpoint for 100,000 steps on predicted mels.
- VITS (Coqui implementation, 36.3M parameters): end-to-end conditional VAE with flow prior and adversarial (GAN) training, with learned speaker embeddings.
- F5-TTS (base DiT configuration, 335.8M parameters): flow-matching mel generator with a language-specific character vocabulary; waveforms come from the pretrained Vocos vocoder without fine-tuning.
Two systems are used off the shelf with no per-language training: OmniVoice (discrete non-autoregressive diffusion LM initialized from Qwen3-0.6B, one reference clip per language drawn from the Open Bible train split) and Gemini-TTS (gemini-2.5-pro-preview-tts, Kore voice) (§4.1, Table 2).
Evaluation. Automatic metrics cover all 37 languages on the first 500 held-out Open Bible test utterances: WER from the Omnilingual ASR 1B LLM model with a Flores-style language code, and UTMOSv2 naturalness, which the authors treat only as a within-language ranking signal because the predictor was trained mainly on English (§4.2). Northern Ndebele has no ASR language code and is excluded from WER. A human listening test covers 10 languages with three native-speaker raters each, 20 utterances each from Open Bible, FLEURS (Wikipedia) and BOUQuET (conversational), rated on a single 1-5 scale combining naturalness and intelligibility (340 clips per language, 320 for Haitian Creole) (§4.2, Appendix C).
Key Results
Intelligibility across 36 languages (Table 4). Mean WER is 16.95% for EveryVoice, 21.50% for OmniVoice, 26.86% for Gemini, 31.13% for VITS and 44.51% for F5-TTS, against a ground-truth ASR floor of 18.74% (§5.1, Table 4). Per language, EveryVoice has the lowest synthetic-system WER in 25 of 36 languages; OmniVoice is lowest in 8 (Assamese, Central Kurdish, Chhattisgarhi, Hiligaynon, Hindi, Kannada, Luganda, Telugu) and Gemini in 3 (Haitian Creole, Marathi, Nepali). That the EveryVoice mean sits below the ground-truth mean reflects how noisy the ASR floor is on these languages, so the comparisons are better read as rankings within a language than as absolute intelligibility. Among the from-scratch systems, WER rises with parameter count (EveryVoice < VITS < F5-TTS); the authors read this as favoring parameter-efficient architectures at this data scale, though the three also differ in paradigm and vocoder.
Coverage versus support. Gemini’s WER on African languages averages 3.88% on languages it supports and 40.59% on unsupported ones; in South Asia the split is 17.10% versus 24.86% (§5.1). OmniVoice is roughly flat across supported (15.36%) and unsupported (14.97%) South Asian languages. African languages show higher WER than other regions for every system (§5.1, Figure 2).
Naturalness and human ratings (Table 3). Gemini has the highest mean UTMOSv2 (3.41, versus 3.00 OmniVoice, 2.89 EveryVoice, 2.89 F5-TTS, 2.57 VITS and 3.08 ground truth) and the highest UTMOS on all ten human-rated languages. Human raters give Gemini the highest synthetic MOS in six languages (Haitian Creole 4.82, Hindi 4.83, Swahili 4.88, Telugu 4.77, Turkish 4.81, Vietnamese 4.54). EveryVoice is rated highest overall in Oromo (4.18 against Gemini 2.50), and ground truth is rated highest in Hausa, Shona and Yoruba, where EveryVoice is the top synthetic system (3.72, 3.13, 2.93) and Gemini drops to 2.92 in Shona and 2.36 in Yoruba (§5.2, Table 3). Within the 10-language subset, Table 3 shows OmniVoice with lower WER than EveryVoice in Hindi (9.94 vs 12.38) and Telugu (13.91 vs 14.26), and Gemini lower in Haitian Creole. The text of §5.2 says EveryVoice has the lowest WER in every language except Haitian Creole, which does not match these two rows.
Metric validity. Per-language Spearman correlation with human MOS reaches |ρ| ≥ 0.7 for WER in 7 of 10 languages but for UTMOSv2 in only 3 (Haitian Creole 0.94, Swahili 0.71, Vietnamese 0.89). UTMOSv2 correlates negatively with MOS in Oromo (-0.61) (§5.2, Table 3).
Domain shift (Figure 3). Pooled over ten languages, Gemini holds its MOS across Open Bible, FLEURS and BOUQuET. EveryVoice drops out of domain, most visibly on FLEURS, while OmniVoice is roughly stable and closes the gap; VITS and F5-TTS trail. In the four low-resource languages (Hausa, Yoruba, Shona, Oromo), EveryVoice stays ahead of Gemini and OmniVoice in all three domains (§5.3, Figure 3). These domain-level readings come from a bar chart; no numeric domain-level MOS table is reported.
Novelty Assessment
The contribution is a dataset plus a controlled benchmark. The modelling is entirely existing architectures run with upstream default recipes, and the alignment pipeline reuses ReadAlongs Studio, pyannote and the BibleTTS filtering criteria. What is new is scale and licensing (37 languages, about 100 hours each, under CC BY-SA, with alignments and trained models released) and a matched-budget comparison across five TTS paradigms on genuinely low-resource languages, evaluated by both automatic metrics and native-speaker listeners. Every trained model is monolingual, so the benchmark does not test multilingual or cross-lingual training on the new data. The comparison also has confounds the authors acknowledge (different vocoders, tokenization and toolchain defaults), and the F5-TTS from-scratch results may reflect a data-scale mismatch for a 336M-parameter model rather than a weakness of flow matching itself.
Field Significance
Moderate — The paper provides an openly licensed, verse-aligned speech resource for 37 underrepresented languages and evidence that small per-language models trained on about 100 hours can match or beat large multilingual systems on intelligibility, while commercial multilingual TTS wins human preference mainly on the languages it supports. Its evaluation is broad, but intelligibility rests on a single ASR model, and human ratings cover only 10 languages with three raters each.
Claims
- supports: Small single-language TTS models trained from scratch on about 100 hours of target-language read speech can match or exceed large multilingual pretrained TTS systems on intelligibility for low-resource languages.
Evidence: Monolingual EveryVoice (FastSpeech 2, 18.2M) reaches 16.95% mean WER over 36 languages, against 21.50% for OmniVoice and 26.86% for Gemini-TTS, and has the lowest synthetic-system WER in 25 of 36 languages. (§5.1, Table 4)
- complicates: Nominal language coverage in large multilingual TTS models does not guarantee usable synthesis on low-resource languages outside their explicitly supported set.
Evidence: Gemini-TTS averages 3.88% WER on supported African languages but 40.59% on unsupported ones, and receives MOS 2.36-2.92 in Yoruba, Shona and Oromo while scoring above 4.5 on supported mid-resource languages. (§5.1, §5.2, Table 3)
- complicates: English-trained automatic MOS predictors are unreliable proxies for human naturalness judgments on low-resource languages, where ASR-based WER tracks listener ratings more closely.
Evidence: Spearman |ρ| ≥ 0.7 with human MOS holds for WER in 7 of 10 languages but for UTMOSv2 in only 3; UTMOSv2 ranks Gemini first in all ten languages, whereas native listeners prefer it in six, and UTMOSv2 correlates negatively with MOS in Oromo (-0.61). (§5.2, Table 3)
- complicates: TTS models trained on narrow-domain read speech lose quality on out-of-domain text even when their in-domain intelligibility is high, while large pretrained systems degrade less.
Evidence: Pooled human MOS for EveryVoice drops from Open Bible to FLEURS and BOUQuET, while Gemini stays roughly flat and OmniVoice catches up out of domain; in the four low-resource languages EveryVoice still leads across domains. (§5.3, Figure 3)
- refines: At a fixed update budget on about 100 hours per language, larger from-scratch TTS architectures do not yield better intelligibility.
Evidence: Mean WER grows with parameter count across the from-scratch systems: EveryVoice 18.2M at 16.95%, VITS 36.3M at 31.13%, F5-TTS 335.8M at 44.51%, all trained for 500k updates; the systems also differ in paradigm and vocoder, so size is not isolated. (§5.1, Table 4, Table 5)
Limitations and Open Questions
Intelligibility rests on one ASR model with a noisy floor
All WER comes from omniASR_LLM_1B_v2. Ground-truth WER exceeds 20% for several languages (Dawro 49.32%, Haitian Creole 50.52%, Gofa 38.19%), Northern Ndebele cannot be scored at all, and the mean synthetic WER for EveryVoice falls below the ground-truth mean, so cross-system WER differences are meaningful only as within-language rankings (§Limitations, Table 4).
- Training and in-domain evaluation are scripture-only: narrow vocabulary, formal style and repetitive phrasing. Out-of-domain evidence is limited to 20 FLEURS and 20 BOUQuET utterances per language in the 10-language human study (§Limitations, §5.3).
- System comparisons are confounded by toolchain defaults: EveryVoice fine-tunes its vocoder, F5-TTS uses a frozen Vocos, and tokenization and inference schedules differ. Upstream default hyperparameters may also not suit a 336M-parameter F5-TTS trained on about 100 hours (§Limitations, Appendix B).
- Human evaluation uses three raters per language and a single combined naturalness-plus-intelligibility score, so the two dimensions cannot be separated. No inter-rater agreement statistic is reported (§4.2, Appendix C).
- All trained models are monolingual. Whether pooled multilingual or cross-lingual training on OpenBibleTTS would help the weakest (African, tonal) languages remains untested; the authors attribute the African-language gap to tonal contrasts and orthographic variation without testing that hypothesis (§5.1).
- OmniVoice and Gemini prompts use a single voice or reference clip per language. Speaker similarity is not measured for any system.
Wiki Connections
- Transformer Encoder-Decoder TTS — The FastSpeech 2-based EveryVoice cascade, trained from scratch per language, gives the lowest mean intelligibility error of all systems evaluated.
- GAN Vocoder — VITS is trained end-to-end with adversarial losses and the iSTFTNet (HiFi-GAN variant) vocoder is fine-tuned per language on predicted mels.
- Flow Matching — F5-TTS trained from scratch on about 100 hours per language gives the highest mean WER of the five systems, a data-scale cautionary point for large flow-matching models.
- Subjective Evaluation — Native-speaker MOS across 10 low-resource languages and three text domains, with three raters per language.
- Evaluation Metrics — Contributes a 37-language benchmark and shows WER tracks human MOS better than UTMOSv2 on low-resource languages.
- F5-TTS — Trained from scratch per language as the flow-matching representative; it is the weakest system on intelligibility here.
- OmniVoice — Evaluated off the shelf as the massively multilingual zero-shot baseline; second-best mean WER and roughly stable across supported and unsupported South Asian languages.
- FastSpeech 2 — The acoustic model inside EveryVoice, retrained from scratch for each of the 37 languages.
- HiFi-GAN — iSTFTNet, the vocoder fine-tuned in the EveryVoice cascade, is a lightweight HiFi-GAN variant.
- Vocos — Pretrained vocoder used without fine-tuning for F5-TTS outputs, one of the confounds the authors flag.
- Omnilingual ASR — The 1B LLM-ASR model that supplies every WER measurement in the benchmark.
- XTTS — Cited as a massively multilingual TTS system built on high-resource data, motivating the low-resource gap the paper targets.