CHiPSAL 2025 · 2025 · Workshop
Chhetri et al. · → Paper · Demo: ✗ · Code: ✗
A comparative evaluation of WaveNet and MelGAN vocoders paired with Tacotron2 for Nepali text-to-speech synthesis, finding MelGAN consistently superior in both quality and inference speed on two Nepali datasets.
Problem
Nepali is a low-resource language with little prior study of vocoder selection for TTS. While WaveNet and MelGAN have been benchmarked extensively for English, their relative performance when paired with Tacotron2 for Nepali synthesis was unknown. Prior Nepali TTS work either used a single vocoder without comparison or relied on concatenation-based methods that predate the neural vocoder era.
Method
The study trains Tacotron2 as the acoustic model on two Nepali datasets independently: the publicly available OpenSLR Nepali corpus (predominantly female speech) and a male voice dataset collected from Nepal Television news broadcasts (approximately 2,100 segments at 22.05 kHz). Nepali text is preprocessed into Unicode using the unidecode function before being fed as character embeddings to Tacotron2.
Two vocoders are evaluated in the mel-spectrogram-to-waveform stage. WaveNet is used autoregressively, conditioning on Tacotron2’s mel-spectrogram output to forecast audio samples one step at a time. MelGAN uses a GAN framework (generator plus discriminator) and processes input in parallel, bypassing WaveNet’s sequential bottleneck. Both vocoders are trained separately on each dataset using standard hyperparameters from their respective original papers.
Evaluation uses two complementary methods: Mel Cepstral Distortion (MCD) as an objective spectral measure, and MOS collected from 40 participants via Google Forms using four synthesized samples per model, rated 1-5 for naturalness and accuracy.
Key Results
On the OpenSLR dataset, Tacotron2 + MelGAN achieves an average MOS of 4.245 against 3.65 for Tacotron2 + WaveNet (Table 7). On the male voice dataset, both systems score lower overall (2.885 vs. 2.31), reflecting the added difficulty of multi-speaker male voice data with limited training samples.
MCD results are directionally consistent: Tacotron2 + MelGAN records a lower average MCD (928.37) than Tacotron2 + WaveNet (1131.42) on the OpenSLR dataset, while the two models converge on the male voice data (both approximately 987.87) (§3.4.2, Figure 2).
Inference time reveals a decisive practical difference: MelGAN requires 0.142 seconds per synthesis versus 1,320 seconds for WaveNet on the same GPU (NVIDIA GTX 960M), a difference of roughly four orders of magnitude attributable to MelGAN’s parallel generation versus WaveNet’s sequential autoregressive decoding (§5.1, Table 4).
When compared against prior Nepali TTS work, the Tacotron2 + WaveNet configuration (3.65 MOS on OpenSLR) surpasses the 3.07 MOS reported by Basnet (2021) but falls below the 4.53 MOS WaveNet achieves on English data, illustrating the challenge of low-resource conditions (§5.3, Table 8).
Novelty Assessment
The contribution is primarily empirical: Tacotron2, MelGAN, and WaveNet are all established models used here without architectural modification. The novelty lies in applying this combination to Nepali, a language for which vocoder comparison studies did not previously exist. The evaluation methodology is straightforward, the datasets are small (OpenSLR plus roughly 2,100 self-collected samples), and the participant pool for MOS (40 raters, four samples each) is modest. The results confirm known properties of MelGAN (parallel inference, competitive or superior quality relative to WaveNet) in a new language context. The paper is best read as a confirmatory evidence point establishing baseline vocoder behaviour for Nepali TTS, not as a generalizable claim about optimal vocoder design.
Field Significance
Low — This paper fills a documentation gap for Nepali TTS by providing the first systematic vocoder comparison in that language. Its findings largely replicate what was already known from English evaluations: MelGAN offers better naturalness and dramatically lower inference latency than WaveNet when paired with Tacotron2. The contribution is language-specific evidence rather than a new methodological or architectural advance.
Claims
-
supports: GAN-based vocoders offer substantially lower inference latency than autoregressive vocoders when generating speech from mel-spectrograms.
Evidence: Tacotron2 + MelGAN synthesises speech in 0.142 s versus 1,320 s for Tacotron2 + WaveNet on an NVIDIA GTX 960M GPU, owing to MelGAN’s parallel waveform generation. (§5.1, Table 4)
-
supports: GAN-based vocoders achieve comparable or superior perceptual quality to autoregressive vocoders in low-resource language TTS.
Evidence: Tacotron2 + MelGAN outperforms Tacotron2 + WaveNet in MOS on both Nepali test sets (4.245 vs. 3.65 on OpenSLR; 2.885 vs. 2.31 on male voice data), confirmed by lower MCD on the OpenSLR set. (§5.2, Tables 5–7; §3.4.2, Figure 2)
-
complicates: Naturalness scores for Nepali TTS systems trained on multi-speaker male voice data are substantially lower than those achieved on curated single-accent female corpora.
Evidence: Both Tacotron2 + MelGAN and Tacotron2 + WaveNet score more than 1 MOS point lower on the male voice dataset relative to OpenSLR, attributable to speaker variability and limited data volume, with the gap persisting across both objective (MCD) and subjective (MOS) metrics. (§5.2, Tables 5–7)
-
complicates: Performance of well-studied vocoders on low-resource languages lags behind their English-language benchmarks, even when the acoustic model and training procedure are unchanged.
Evidence: Tacotron2 + WaveNet achieves 3.65 MOS on Nepali OpenSLR versus 4.53 MOS reported for WaveNet on US English data; Tacotron2 + MelGAN achieves 2.885 MOS on the male voice set, below MelGAN’s 3.49-3.61 MOS on English (LJSpeech, VCTK). (§5.3, Table 8)
Limitations and Open Questions
Warning
The MOS evaluation is based on only 40 participants rating four samples per model. This is too small a listener pool for statistically robust conclusions, and no statistical significance tests are reported. The MOS advantage of MelGAN should be interpreted with caution.
The male voice dataset is self-collected from Nepal Television (approximately 2,100 segments) and is not publicly released, limiting reproducibility. Both models struggle with numbers and symbols in Nepali text, an acknowledged limitation with no proposed fix. The study does not explore fine-tuning pre-trained vocoders from English to Nepali rather than training from scratch, which could be a more data-efficient baseline for future work.
Wiki Connections
- GAN Vocoder — This paper evaluates MelGAN as a GAN vocoder for Nepali TTS, finding it superior to WaveNet in both perceptual quality and inference speed on low-resource data.
- Evaluation Metrics — The study applies both MCD as an objective measure and MOS as a subjective measure, illustrating their complementary roles in vocoder comparison.
- Subjective Evaluation — MOS scores are collected from 40 participants via Google Forms; the paper’s results illustrate the sensitivity of small-pool evaluations to dataset characteristics.