arXiv · 2025 · Preprint
Qingliang Meng et al. (Megatronix) · → Paper · Demo: ? · Code: ?
FNH-TTS replaces VITS’s duration predictor with a Mixture-of-Experts variant (MoE-DP) and its HiFi-GAN vocoder with VOCOS plus two advanced multi-scale discriminators, achieving more human-like phoneme duration distributions and higher MOS scores at a compact model size.
Problem
Non-autoregressive TTS systems require an explicit duration predictor to expand phoneme sequences to frame-level representations, yet existing deterministic and stochastic duration predictors fail to capture the full diversity of natural prosodic patterns — particularly speaker-specific duration variation in multi-speaker settings. A related but distinct issue is that HiFi-GAN, the dominant vocoder in NAR TTS, introduces spectral artifacts (disharmony components) when fed richer, more varied prosodic information. These two problems compound each other: improving the duration predictor alone tends to degrade vocoder output quality, creating a tension that prior work had not resolved jointly.
Method
FNH-TTS extends VITS by replacing two components while retaining the text encoder, speaker encoder, posterior encoder, and normalizing flow unchanged.
MoE Duration Predictor (MoE-DP). The duration predictor is rebuilt around a 1D convolution front-end followed by two Switch-Transformer blocks. Each Switch-Transformer block applies a sparse Mixture-of-Experts feed-forward network with 8 experts and a learned router that incorporates the speaker embedding alongside the text hidden state. During training, a load-balancing auxiliary loss (L_aux, scaled by alpha) encourages uniform expert utilization so that different experts specialize in distinct prosodic patterns rather than collapsing. The routing decision at inference uses top-1 expert selection (standard Switch-Transformer). The duration prediction loss (L_dur) is the sum of the MAS-derived alignment loss and L_aux. The convolution blocks use kernel size 3 with a hidden dimension of 192; the MoE blocks have 4 attention heads and a hidden dimension of 192.
VOCOS Vocoder with CoMBD + SBD. HiFi-GAN is replaced by VOCOS, a ConvNeXt-based vocoder that operates in the Fourier domain and reconstructs the waveform via ISTFT. To integrate VOCOS into the end-to-end pipeline, the original Mel-spectrum input is replaced by the posterior encoder’s latent variable z concatenated with the speaker embedding s (Eq. 7 in the paper). The backbone contains 8 ConvNeXt blocks with intermediate dimension 1536 and hidden dimension 512 (hop length 256, FFT size 1024). Two adversarial discriminators are added: a Collaborative Multi-Band Discriminator (CoMBD) that applies multi-scale discrimination at multiple waveform resolutions for temporal coherence, and a Sub-Band Discriminator (SBD) that uses PQMF analysis to decompose into frequency sub-bands, each processed with multi-scale dilated convolutions. These discriminators replace HiFi-GAN’s multi-period discriminator and reduce to 27.07M total discriminator parameters (vs. 46.75M for HiFi-GAN’s discriminators).
The total training objective is L_rec + L_kl + L_dur + L_adv + L_gen. Training uses AdamW (beta_1=0.8, beta_2=0.99, lr=2e-4 with 0.999 per-epoch decay) on 4 NVIDIA RTX 3090 GPUs, batch size 24.
Key Results
Synthesis quality (Table I — LJSpeech and VCTK): FNH-TTS achieves MOS 4.48 on LJSpeech (vs. 4.26 for VITS, 4.35 for StyleTTS2, 3.87 for F5-TTS) and MOS 4.63 on VCTK (vs. 4.34 for VITS, 4.49 for F5-TTS). On WER, FNH-TTS scores 2.59% on LJSpeech (VITS: 3.41%, StyleTTS2: 5.78%) and 3.88% on VCTK. FNH-TTS’s generator is only 47.73M parameters, far smaller than F5-TTS (337M) and SparkTTS (507M).
Ablation: MoE-DP alone with HiFi-GAN degrades MOS to 3.92 on LJSpeech, confirming that the vocoder upgrade is necessary to realize MoE-DP’s prosodic gains. The combination of MoE-DP + VOCOS + CoMBD + SBD yields the full FNH-TTS result.
Vocoder evaluation (Table II): FNH-TTS’s vocoder achieves the lowest M-STFT (1.207) and MCD (1.654) among the compared vocoders, with a GPU RTF of 0.0046 (vs. 0.00748 for HiFi-GAN), a 1.6x speedup. PESQ (2.425) is marginally below HiFi-GAN (2.441).
Prosody accuracy (Table III — Libri460): FNH-TTS achieves 67.07% phoneme duration classification accuracy (high/normal/low categories via MFA forced alignment), the highest among all systems evaluated, including SparkTTS (66.77%), VITS+SDP (65.76%), FastSpeech2 (59.80%), and F5-TTS (11.17%). Distribution visualizations (Fig. 2 and 3) show FNH-TTS’s duration histogram follows the ground-truth distribution most closely on both single-speaker (LJSpeech) and multi-speaker (VCTK) data, with visible speaker-specific variation across VCTK speakers that other systems fail to reproduce.
Jensen-Shannon divergence analysis (Table IV): The JS divergence between each ablation variant’s DP output and FNH-TTS’s output confirms that MoE-DP is the dominant contributor to prosodic diversity improvement, especially in multi-speaker settings.
Novelty Assessment
The primary contribution is applying the Mixture-of-Experts mechanism (specifically Switch-Transformer-style sparse MoE) to the duration predictor of a VITS-based system, which the paper claims is the first such application. The joint redesign of both the duration predictor and vocoder to address their coupled failure mode is conceptually sound. However, each individual component (VOCOS, CoMBD, SBD, MoE) is borrowed from prior work; the novelty lies in the integration and the identification of the interaction between prosodic complexity and vocoder artifact generation. The prosody visualization methodology and the vocoder evaluation scheme that avoids length-mismatch artifacts are useful methodological contributions. The overall contribution is primarily engineering and integration rather than fundamental architectural innovation.
Field Significance
Moderate — FNH-TTS demonstrates that MoE routing applied to phoneme duration prediction can improve prosodic diversity in NAR TTS while remaining at compact model sizes, providing a practical counterpoint to the trend toward large autoregressive systems. The key finding that improving the duration predictor alone degrades synthesis quality unless the vocoder is co-upgraded provides useful evidence about the coupled nature of these two subsystems.
Claims
- Applying Mixture-of-Experts routing to the duration predictor of a non-autoregressive TTS system produces phoneme duration distributions that more closely match ground-truth speaker-specific prosodic variation than deterministic or stochastic single-path predictors. (§IV-C, Table III)
- Improving duration prediction diversity in NAR TTS without upgrading the vocoder degrades synthesis quality, because richer prosodic information exceeds the artifact-rejection capacity of standard GAN discriminators. (§IV-A, Table I)
- Frequency-domain and multi-band discriminators (CoMBD and SBD) are more effective than HiFi-GAN’s multi-period discriminator at suppressing spectral artifacts when a more expressive duration predictor is used. (§V-B, Table II)
- Word Error Rate is an unreliable proxy for prosody modeling quality in TTS, as simpler prosodic patterns can yield lower WER by reducing ASR system disruption. (§IV-C, §III)
Limitations and Open Questions
- The MoE-DP requires the discriminator upgrade to yield any net benefit; neither module alone suffices, which raises questions about robustness when either component is replaced or updated independently.
- Evaluation is limited to English datasets; whether MoE-DP’s speaker-specific prosody gains transfer to more typologically diverse languages with different prosodic structures is untested.
- The paper notes that current vocoders still struggle under highly diverse prosodic patterns (disharmony components persist even with CoMBD+SBD), pointing to an open challenge for future vocoder designs.
- WER is shown to be an unreliable proxy for prosody quality, but the proposed duration-accuracy metric (MFA-based classification) requires ground-truth phoneme duration annotations, limiting its generalizability.
- No code or demo links are provided, limiting reproducibility.
- Inference RTF on CPU (0.046) is not compared directly against autoregressive or diffusion-based competitors that operate on the same hardware, making efficiency claims difficult to contextualize.
Wiki Connections
This paper advances research on prosody-control by introducing MoE-based duration modeling with load-balanced expert specialization for speaker-specific prosody. It informs gan-vocoder through the replacement of HiFi-GAN with VOCOS paired with CoMBD and SBD discriminators, and the analysis of how discriminator design affects spectral artifact handling. The VITS foundation links it to transformer-enc-dec-tts (VITS’s flow-based architecture). The paper’s critique of WER as a prosody metric and its proposed MFA-based duration accuracy measure are relevant to evaluation-metrics.