arXiv · 2026 · Preprint
Răgman et al. (POLITEHNICA Bucharest / Technical University of Cluj-Napoca) · → Paper · Demo: ✗ · Code: ✓
Trains and adapts four popular open-source TTS architectures (FastPitch, VITS, Grad-TTS, Matcha-TTS) from scratch for Romanian and evaluates them on installation friction, documentation quality, and synthesis quality using both objective metrics and listening tests.
Problem
Open-source TTS frameworks are frequently presented as broadly reusable, but most published comparisons rely on results reported by the original authors on their own (typically English) benchmarks rather than on hands-on reproduction in a new language. For under-resourced languages such as Romanian, practitioners face a double burden: scarce high-quality training data and toolchains whose installation, data-preparation, and hardware requirements were never validated outside their original release conditions. The paper argues that existing surveys of open-source TTS systems catalogue architectures and cite documented results but do not test whether those systems can actually be installed, adapted, and trained end-to-end for a new language.
Method
The authors select four widely-referenced open-source TTS architectures spanning different generative paradigms: FastPitch (non-autoregressive Transformer encoder-decoder with explicit phoneme-level F0 conditioning and a duration predictor), VITS (conditional VAE with normalizing flows and adversarial training, end-to-end waveform generation via monotonic alignment search), Grad-TTS (score-based diffusion decoder over Mel-spectrograms with monotonic alignment search), and Matcha-TTS (conditional flow-matching decoder, ODE-based, with a stochastic duration predictor). For all systems except VITS, a HiFi-GAN vocoder converts predicted Mel-spectrograms to waveform.
Each system was trained from scratch on the Romanian SWARA corpus (21 hours, 17-18 speakers, 44.1kHz/16-bit), using a 16-speaker parallel “eigen-voice” subset (no explicit speaker conditioning) as the shared baseline training set, standardized to a common Phonemizer/eSpeak-NG-based phoneme pipeline adapted to Romanian (§3.1-3.2). Two held-out speakers, selected for comparatively low recording quality to emulate realistic conditions, were used for speaker adaptation: each was fine-tuned with either 10 samples (<50 seconds) or 1000 samples (~1 hour), at 120, 400, and 4000 iterations, to probe both extreme low-resource and moderate-resource fine-tuning regimes (§3.3-3.4).
Evaluation combines a qualitative usability assessment (installation and dependency complexity, documentation quality, reproducibility, training/finetuning experience, community support, based on GitHub issue activity) with an objective quality assessment (WER/CER via Whisper transcription, UTMOS-based automatic naturalness, Speaker Encoder Cosine Similarity via Resemblyzer, PESQ/STOI/SI-SDR via TorchAudio-Squim, log-F0 RMSE and MCD) computed over 42 held-out synthesized utterances per condition, and a subjective listening test (31 listeners rating naturalness and speaker similarity on a 0-100 scale for the 1000-sample/4000-iteration models, analyzed with one-way ANOVA and Tukey HSD post-hoc tests) (§4.1-4.3). The authors also attempted to adapt Parler-TTS to Romanian as a fifth system but excluded it after it failed to train from scratch and produced only silence, training successfully only when fine-tuning from an existing English checkpoint (§5).
Key Results
Objectively, FastPitch achieved the lowest WER across all conditions (best case 2.1% for the female speaker at 1000 samples), while Matcha-TTS stayed close behind and more stable (WER consistently below 6% for both speakers regardless of adaptation data volume). VITS and Grad-TTS showed markedly weaker intelligibility, with Grad-TTS producing the highest WERs (32% dropping to 14% for the female speaker as adaptation data grew) and VITS showing an unexpected WER increase for the male speaker with additional training, traced to truncated utterances (§4.2.1, Figure 1). At the most data-constrained setting (10 samples, 120 iterations), FastPitch reached WER 0.03, SPK-SIM 0.72, and UTMOS 2.94 for the male speaker, and Matcha-TTS reached WER 0.05, SPK-SIM 0.87, and UTMOS 2.98 for the female speaker, both matching or exceeding their own higher-resource baselines (§4.2.6).
For speaker similarity, VITS led overall (SPK-SIM peaking at 0.92) and preserved identity even with minimal adaptation data, while Grad-TTS scored lowest (§4.2.3, Figure 3). On PESQ/STOI/SI-SDR, VITS and Matcha-TTS consistently outperformed FastPitch and Grad-TTS, showing that perceptual fidelity does not always track WER or speaker-similarity rankings (§4.2.4). In the subjective listening test, however, VITS was rated best for both naturalness and speaker similarity by human listeners, with FastPitch second, despite VITS’s comparatively poor objective intelligibility scores; ANOVA plus Tukey HSD showed FastPitch and Matcha-TTS were not significantly different from each other on either naturalness or speaker similarity, while Grad-TTS and VITS differed significantly from the rest (§4.3). Matcha-TTS scored highest on automatic UTMOS naturalness but was ranked only third by human listeners, a case where the objective metric and subjective judgment diverge (§4.3, §5).
Novelty Assessment
The contribution is empirical rather than architectural: no new model, training objective, or evaluation metric is proposed. The paper’s value lies in a controlled, reproducible, hands-on comparison of four existing architectures trained from scratch on identical low-resource data, combined with a usability audit (setup friction, documentation, community support) that published benchmarks of these tools rarely report. The Parler-TTS training failure and the systematic WER/UTMOS-versus-listening-test divergences are useful negative and cautionary data points rather than incremental performance gains.
Field Significance
moderate — the paper is a practical, reproducible case study of adapting four commonly cited open-source TTS architectures to a genuinely under-resourced language, with code and data released for verification. Its contribution is scoped to a single target language and a modest listening-test panel, so its main value is as a grounded reference point for low-resource TTS toolchain selection rather than a result that reshapes how the field builds or evaluates TTS systems.
Claims
- supports: Objective automatic naturalness predictors (e.g. UTMOS-style MOS estimators) can rank systems differently than human listeners in direct listening tests.
Evidence: Matcha-TTS scored highest on automatic UTMOS naturalness but was ranked only third by the 31-listener subjective panel for both naturalness and speaker similarity. (§4.3, §5)
- complicates: Word Error Rate computed via automatic speech recognition transcription can misrepresent the intelligibility of TTS systems whose output contains recognizer-unfriendly artifacts.
Evidence: VITS and Grad-TTS produced Whisper-transcription WERs frequently above 10%, yet none of the 31 listeners spontaneously reported intelligibility problems with these systems’ audio. (§4.2.1, §5)
- refines: Under extreme low-resource speaker adaptation (tens of seconds of target-speaker audio), architecture choice matters more than further increasing adaptation data or training iterations.
Evidence: With only 10 adaptation samples and 120 finetuning steps, FastPitch (WER 0.03, SPK-SIM 0.72, UTMOS 2.94) and Matcha-TTS (WER 0.05, SPK-SIM 0.87, UTMOS 2.98) already matched or exceeded their own higher-data baselines, while VITS and Grad-TTS required substantially more data and iterations to reach comparable quality. (§4.2.6)
- complicates: Reported ease-of-use and community support for open-source TTS toolkits do not necessarily track their measured synthesis quality.
Evidence: Grad-TTS required manual patching of an incomplete monotonic-alignment module and unmerged multi-speaker fixes before it could be trained for a new language, and it ranked last in both objective and subjective quality among the four systems. (§4.1, Table 2, §4.3)
Limitations and Open Questions
The study evaluates a single under-resourced language (Romanian) with a single speech corpus (SWARA) and a 31-listener subjective panel per condition; the ranking of architectures may not generalize to other low-resource languages, phonological systems, or larger listener pools.
The paper also does not run the systems’ original English training recipes as a control, so it cannot isolate how much of the observed quality gap versus published English results is attributable to the Romanian text-processing adaptation versus the smaller Romanian training corpus. The Parler-TTS adaptation attempt was abandoned after training-from-scratch failures, so the paper’s usability findings are limited to architectures that could successfully be trained under the intended protocol.
Wiki Connections
- Evaluation Metrics — reports a broad battery of objective TTS metrics (WER/CER, UTMOS, SPK-SIM/SECS, PESQ, STOI, SI-SDR, F0-RMSE, MCD) side by side and documents where they diverge from human judgments.
- Subjective Evaluation — runs a 31-listener naturalness and speaker-similarity test with ANOVA/Tukey HSD analysis, directly contrasted against the automatic metrics.
- Speaker Adaptation — fine-tunes each architecture to two held-out speakers under 10-sample and 1000-sample adaptation budgets to probe extreme low-resource adaptation.
- Diffusion TTS — trains and evaluates Grad-TTS, a diffusion-based decoder, as one of the four compared systems, finding it the weakest on both objective and subjective metrics.
- Flow Matching — trains and evaluates Matcha-TTS, a conditional flow-matching architecture, finding it robust under low-resource adaptation despite an objective/subjective naturalness mismatch.
- GAN Vocoder — relies on a HiFi-GAN vocoder for three of the four evaluated systems and evaluates VITS’s integrated adversarial waveform generation directly.
- Transformer Encoder-Decoder TTS — trains and evaluates FastPitch, a non-autoregressive Transformer encoder-decoder with explicit duration and pitch prediction, finding it the strongest on objective intelligibility.
- Grad-TTS — one of the four TTS architectures trained from scratch and evaluated in this study.
- HiFi-GAN — used as the vocoder for FastPitch, Grad-TTS, and Matcha-TTS to convert Mel-spectrograms to waveform.
- UTMOS — used as the automatic non-intrusive naturalness predictor for the objective quality assessment.