arXiv · 2025 · Preprint
Ziyang Zhang et al. · → Paper · Demo: ✓ · Code: ✓
Replaces the fixed-variance Gaussian sampling used in continuous-valued autoregressive TTS with a Bayesian evidential (Normal-Inverse-Gamma) formulation that models data-dependent uncertainty, letting a model trained on ~5k hours outperform 50k-hour baselines.
Problem
Text-to-speech is a one-to-many mapping: the same text can be realised with many valid prosodies, rhythms, and acoustic details, yet standard TTS datasets provide only a single recording per sentence and most models treat synthesis as deterministic regression. Continuous-valued autoregressive models (e.g. MELLE) improved on discrete-codec approaches by avoiding quantisation artefacts, but they still sample from a Gaussian with a fixed unit-variance prior, which forces a homoscedastic (uniform-uncertainty) assumption onto every acoustic frame regardless of how ambiguous or deterministic it actually is. This constrains generation to something close to a point estimate and is a plausible source of the oversmoothing and moderate stochasticity commonly reported in continuous AR TTS. A second, related obstacle is data: rigorously estimating a per-frame variance from evidence normally requires multiple realisations of the same text, which single-reference corpora like LibriSpeech do not provide.
Method
BELLE (Bayesian Evidential Learning with Language modelling) keeps the general architecture of mel-based autoregressive TTS unchanged: a Prenet embeds text and previous mel frames, a decoder-only Transformer AR LM produces hidden states, a Sampling Module introduces stochasticity, a Postnet refines the output, a Stop-Prediction Module ends generation, and a pretrained HiFi-GAN vocoder converts mel-spectrograms to waveform. The change is entirely inside the Sampling Module and its training objective. Instead of projecting the AR hidden state to a Gaussian mean and log-variance and training with a KL term against a fixed-variance prior (as in MELLE), BELLE projects each hidden state to the four parameters of a Normal-Inverse-Gamma (NIG) distribution over the acoustic target. Generation proceeds by first sampling a variance from an Inverse-Gamma distribution, then a mean from a Gaussian conditioned on that variance, and finally the output sample from a Gaussian with the sampled mean and variance; marginalising over the NIG prior yields a Student-t posterior predictive distribution that strictly subsumes the Gaussian as a limiting case. Training combines an evidential loss (negative log-likelihood plus a regulariser that penalises incorrect evidence) with an L1/L2 regression loss on both the coarse and Postnet-refined mel outputs and a “spectrogram flux” loss that encourages the predicted distribution location to differ from the previous ground-truth frame, which is intended to discourage oversmoothed, near-static output.
Because a NIG posterior needs several realisations of a given text to be estimated reliably, and standard datasets provide only one, the paper introduces a “one-to-many” training strategy: for each training text, mel-spectrograms are synthesised from six publicly available pretrained TTS systems (CosyVoice2, IndexTTS, SparkTTS, F5-TTS, MaskGCT, XTTS-v2) in addition to the original human recording, and the full evidential-plus-regression loss is computed against each of the resulting N+1 targets and combined in a fixed weighted sum (original LibriSpeech audio weighted roughly twice as heavily as any single synthetic source). The synthetic samples are explicitly not treated as imitation targets; they serve only to characterise the variance structure the model should learn, so errors specific to any one teacher are diluted across the ensemble. BELLE-stream reuses the identical architecture and initialises from the trained BELLE checkpoint, then fine-tunes on text and mel-spectrogram sequences split into interleaved fixed-size chunks (20 phonemes, 50 mel frames ≈ 0.8s) under a causal mask, allowing chunk-by-chunk generation at inference. The AR backbone is a 12-block, 16-head decoder-only Transformer with hidden size 1024 and feed-forward dimension 4096, trained on Librispeech audio filtered to 0.5–14s (706 hours) plus the six-teacher synthetic augmentation (4,817 hours total).
Key Results
On LibriSpeech test-clean zero-shot TTS (Continuation and Cross-Sentence protocols), BELLE trained on the full 4,817-hour multi-teacher dataset achieves the lowest WER among all compared systems (WER-C 1.63%, WER-H 2.13% for Continuation; WER-C 2.45%, WER-H 2.99% for Cross-Sentence), outperforming F5-TTS, MaskGCT, VALL-E, RALL-E, CLAM-TTS, and ELLA-V, all of which the paper reports as trained on around 50,000 hours of data (a claim not independently re-verified here but stated as the comparison basis). Under identical training data and architecture, BELLE consistently beats a reproduced MELLE baseline: in the Cross-Sentence setting, BELLE reduces WER-H from 3.83% to 2.99% and raises SIM-r from 0.652 to 0.679. Subjectively, BELLE reaches an MOS of 4.21 (comparable to Ground Truth’s 4.20 and F5-TTS’s 4.25) with the lowest standard deviation of any system tested, and the highest SMOS (4.13). A controlled ablation (BELLE trained on human-only LibriSpeech data, no synthetic augmentation) still outperforms MELLE trained the same way (WER-C 3.81% vs. 7.78%), indicating the evidential objective itself, not just the added synthetic data, drives much of the gain; adding synthetic teachers then improves results further and monotonically with teacher count, while a naive pooled-data-augmentation baseline using the same total data volume underperforms the principled multi-teacher weighting. In the streaming setting, BELLE-stream attains the lowest WER among compared streaming systems (WER-C 3.54%, WER-H 4.44%, versus SMLLE and IST-LM baselines) with a real-time factor of 0.55 and a first-packet latency of about 440ms.
Novelty Assessment
The architectural backbone is unchanged from prior continuous AR TTS work (a standard decoder-only Transformer with Prenet/Postnet/Stop-Prediction modules very close to MELLE’s design); the genuine contribution is in the probabilistic formulation and the accompanying training recipe. Replacing a fixed-variance Gaussian KL objective with an evidential NIG posterior is a real methodological shift, not a superficial modification: it changes what quantity the model is trained to predict (calibrated, data-dependent uncertainty rather than a point estimate regularised toward unit variance) while adding no parameters or inference-time cost. The multi-teacher “one-to-many” training strategy is also a genuine contribution specific to this Bayesian framing, since it is motivated by and designed to solve a variance-estimation problem that a purely regression-based objective would not need to face. The paper is honest that the streaming extension (BELLE-stream) is not a novel streaming architecture but a demonstration that the evidential framework composes cleanly with an existing chunk-based streaming recipe.
Field Significance
High
High — the paper demonstrates a concrete, drop-in alternative to the fixed-variance sampling assumption that underlies recent continuous-valued autoregressive TTS models, showing meaningful data efficiency and quality gains without added inference cost, and it provides a principled reason (an explicit uncertainty framework) for why synthetic multi-teacher data helps a single-reference training set beyond simple volume, which reframes how such augmentation should be weighted and used.
Claims
- supports: Replacing a fixed-variance Gaussian sampling prior with a dynamically estimated Bayesian evidential distribution over acoustic targets improves the robustness and speaker similarity of continuous-valued autoregressive TTS without adding parameters or inference latency.
Evidence: Under identical training data, architecture, and evaluation pipeline, replacing the Gaussian KL sampling objective with the Normal-Inverse-Gamma evidential loss reduces Cross-Sentence WER-H from 3.83% to 2.99% and raises SIM-r from 0.652 to 0.679. (§6.1, Table 2)
- supports: Synthetic speech from multiple diverse pretrained TTS systems can serve as a statistical support set for variance estimation in single-reference TTS corpora, rather than as imitation targets, and generation quality improves as teacher diversity increases.
Evidence: Under the proposed weighted “one-to-many” loss, WER and speaker-similarity metrics improve monotonically as the number of teacher models grows from 0 (GT only) to 2 to 6 on a fixed reduced-scale dataset. (§6.3, Table 4)
- complicates: Increasing the volume of pooled multi-source synthetic training data does not by itself explain quality gains in continuous AR TTS, and naive augmentation without a variance-aware weighting scheme can underperform a smaller, principled multi-teacher training set.
Evidence: A data-augmentation baseline trained on the same total pooled data volume as GT+6 TTS underperforms the smaller GT+2 TTS configuration on multiple WER and similarity metrics. (§6.3, Table 4)
- supports: A continuous-valued autoregressive TTS architecture with variance-aware evidential sampling can be adapted to chunked streaming generation while remaining competitive with dedicated streaming baselines.
Evidence: BELLE-stream attains the lowest WER among compared streaming systems (WER-C 3.54%, WER-H 4.44%), outperforming SMLLE and IST-LM, while reaching a real-time factor of 0.55 and a first-packet latency of about 440ms with a 0.8-second audio chunk. (§6.4, Table 5)
- refines: The degree of stochastic diversity in evidential autoregressive speech generation can be controlled at inference time via a single sampling-distribution hyperparameter, trading diversity against robustness.
Evidence: Doubling the Normal-Inverse-Gamma scale parameter β increases within-prompt diversity metrics (cosine distance from 0.0037 to 0.0053) but raises Cross-Sentence WER-C from 2.45% to 3.33% and slightly lowers SIM-o. (§6.2, Table 3, Table 7)
Limitations and Open Questions
The paper validates only on English speech (LibriSpeech); cross-lingual generalisation of the evidential framework is untested.
The MELLE baseline is a re-implementation rather than the original authors’ released model, since no official MELLE code is public, which the authors argue keeps the BELLE-vs-MELLE comparison fair (identical pipeline) but leaves the reported MELLE numbers only approximately comparable to the original MELLE paper. The ablation studies isolating individual components (teacher count, weight allocation, sampling module vs. flux loss) were run on a reduced-scale dataset (up to ~455 hours) rather than the full 4,817-hour setting, so their absolute numbers are not directly comparable to the main results table. As an autoregressive model, BELLE’s real-time factor remains higher than non-autoregressive alternatives, and the authors note that predicting multiple mel frames per step is a plausible direction to close this gap. Finally, the training data scale (~5k hours) is deliberately far smaller than the ~50k-hour baselines it is compared against, which the paper frames as demonstrating data efficiency but which also means direct architecture-for-architecture comparisons at matched scale are not reported.
Wiki Connections
- Zero-Shot TTS — evaluates BELLE under the standard continuation and cross-sentence zero-shot prompting protocols on LibriSpeech test-clean.
- Streaming TTS — extends the evidential AR framework to chunked, causal streaming generation (BELLE-stream) with reported real-time factor and first-packet latency.
- Subjective Evaluation — reports crowdsourced MOS and SMOS listening tests comparing BELLE against Ground Truth and open-source baselines.
- Evaluation Metrics — introduces a WavLM-embedding-based diversity analysis (cosine distance, L1/L2) to quantify sample-to-sample variability under repeated generation, alongside standard WER and speaker-similarity metrics.
- MELLE — BELLE replaces MELLE’s fixed-variance Gaussian sampling and KL loss with a Bayesian evidential formulation, using an identical architecture and training pipeline for direct comparison.
- F5-TTS — used both as a ~50,000-hour-trained baseline in the main comparison tables and as one of six teacher models providing synthetic training data.
- VALL-E — cited as a foundational discrete-codec AR TTS baseline and as the source of the zero-shot evaluation protocol (continuation/cross-sentence) adopted here.
- RALL-E — included as a strong codec-LM baseline in the objective evaluation table.
- CosyVoice 2 — one of six pretrained TTS systems used to generate multi-teacher synthetic training data for the “one-to-many” strategy.
- MaskGCT — serves both as a teacher model for synthetic training data and as a baseline compared on MOS/SMOS and WER/similarity metrics.
- IndexTTS — one of six pretrained TTS systems used to generate multi-teacher synthetic training data.
- HiFi-GAN — used as the pretrained vocoder that converts BELLE’s predicted mel-spectrograms into waveform.