arXiv · 2026 · Preprint
Hien Ohnaka et al. (Nara Institute of Science and Technology, LY Corporation) · → Paper · Demo: ✓ · Code: ✓
Introduces WaveTrainerFit, a fixed-point iteration vocoder that replaces Gaussian initial noise and implicit gain estimation with a VAE-based trainable prior, improving waveform generation from self-supervised speech features with fewer inference iterations.
Problem
Neural vocoders that generate waveforms from data-driven self-supervised learning (SSL) features (rather than signal-processing-derived Mel-spectrograms) have become important for speech restoration, high-quality dataset creation, and other data-driven pipelines. WaveFit, a fixed-point iteration vocoder combining diffusion-style inference with GAN losses, can vocode SSL features, but it was originally designed around Mel-spectrograms. When conditioned on SSL features it loses two mechanisms that make Mel-spectrogram vocoding easier: (1) it can no longer sample initial noise from a hand-crafted, spectral-envelope-informed prior, so it falls back to plain Gaussian noise; and (2) it can no longer read gain directly from the Mel-spectrogram’s power, so the network must implicitly learn to predict signal energy during denoising. Both compromises add difficulty to the waveform modeling task and increase the number of iterations needed for high-quality output.
Method
WaveTrainerFit keeps the WaveFit fixed-point iteration backbone (a GAN-and-diffusion-style denoiser trained with a multi-resolution STFT loss) but replaces its noise-sampling and gain-adjustment mechanisms with a trainable, VAE-based prior adapted from RestoreGrad. A prior encoder and a posterior encoder, both conditioned on the SSL feature (up-sampled 2x via a transposed convolution), produce variance parameters for a noise distribution: during training, the posterior encoder additionally sees the target waveform and provides the sampling distribution; at inference, the prior encoder (conditioned only on the SSL feature) supplies it. The two encoders are trained to minimize their KL divergence, so at inference the prior can approximate an informative, target-like noise distribution instead of standard Gaussian noise.
Unlike RestoreGrad, which models the trainable prior in the waveform domain, WaveTrainerFit models the variance in the time-frequency domain (an F x K spectrogram-shaped tensor) and converts it to time-domain initial noise via inverse-STFT reparameterization, which the authors argue shortens the effective sequence length and reduces modeling complexity relative to a full waveform-domain VAE. A second addition, reference-aware gain adjustment, imposes a loss term that pushes the posterior encoder’s output energy to match the power spectrogram of the target waveform; this lets the fixed-point iteration use the prior’s own energy to normalize each denoising step, replacing WaveFit’s self-gain-adjustment operator (which must implicitly infer the correct energy) with the prior’s directly learned energy estimate.
The encoders use a real-valued DCUnet-10 architecture with two U-Nets in the posterior encoder (one for the target power spectrogram, one for the SSL feature) whose intermediate activations are summed block-by-block; the prior encoder is a single such U-Net conditioned on the SSL feature alone. The posterior and prior encoders total 2.61M and 2.59M parameters respectively, added on top of the WaveFit denoiser. The full loss combines the original WaveFit adversarial-plus-STFT loss, the prior/posterior KL term, and the energy-matching guidance term. The model was trained on LibriTTS-R (585 hours, 2,456 speakers) for 400k steps at batch size 8, upsampling SSL features 480x to 24 kHz waveforms.
Key Results
Across three conditional SSL features (WavLM-large, XLS-R-0.3B, and the Whisper-medium encoder, extracted at the 8th layer) and against HiFi-GAN and WaveFit baselines, WaveTrainerFit improved every reference-aware objective metric (SpeechBERTScore, MCD, Log-F0-RMSE, speaker-similarity cosine) over WaveFit, and improved similarity MOS (S-MOS) for all three feature types (Table 1). For naturalness MOS (N-MOS), WaveTrainerFit beat both baselines with the Whisper encoder (3.87 vs. WaveFit’s 3.23 and HiFi-GAN’s 2.41) but underperformed WaveFit with XLS-R features (3.21 vs. 3.79) and WavLM features (3.50 vs. 3.76), which the authors attribute to possible hyperparameter sensitivity inherited from RestoreGrad and to known degradation effects of XLS-R on English tasks.
A layer-depth ablation (Table 2) using WavLM features from the 2nd, 8th, and 24th transformer layers showed WaveTrainerFit’s advantage over WaveFit growing with feature depth: at the 24th layer (which carries limited acoustic detail), WaveTrainerFit improved SpeechBERTScore from 89.08 to 92.19, MCD from 5.501 to 5.066, and Log-F0-RMSE from 0.3623 to 0.2301, without degrading reference-free UTMOS. An iteration-count sweep (Figure 3) showed WaveTrainerFit outperforming WaveFit on nearly all objective metrics at every tested iteration count, with the largest SpeechBERTScore gap at a single iteration, at the cost of a modestly higher real-time factor from the added prior encoder and sampling step.
Novelty Assessment
The contribution is architectural but incremental relative to two existing pieces: WaveFit (the vocoder backbone) and RestoreGrad (the VAE-based trainable-prior mechanism for diffusion models). The paper’s genuine additions are (1) reformulating RestoreGrad’s waveform-domain trainable prior in the time-frequency domain specifically to keep sequence length and encoder cost manageable for a vocoder, and (2) deriving a reference-aware gain-adjustment operator from the same trainable prior, which is a new mechanism not present in RestoreGrad (whose target task, signal restoration, does not need explicit gain adjustment). The combination is purpose-built for the SSL-feature vocoding setting rather than a direct application of RestoreGrad, but the core idea, an informative learned noise prior replacing Gaussian sampling, is adapted rather than newly invented.
Field Significance
moderate — This paper provides a concrete, evaluated fix for a specific known limitation of SSL-feature vocoding (loss of hand-crafted noise priors and gain cues that Mel-spectrogram vocoders can exploit), and its layer-depth robustness result is a useful data point for anyone selecting SSL feature depth for a downstream vocoder. The scope is narrow: a single vocoder family (WaveFit) evaluated on one corpus (LibriTTS-R) with a 5-page ICASSP-style paper’s worth of ablations.
Claims
- supports: Trainable, target-informed noise priors can substitute for hand-crafted signal-processing priors in fixed-point iteration and diffusion-style vocoders when conditioning on features that are not derived from signal processing.
Evidence: Replacing WaveFit’s Gaussian initial noise with a VAE-based prior trained to approximate the posterior over the target waveform improved SpeechBERTScore, MCD, Log-F0-RMSE, and speaker-similarity across all three tested SSL feature types on LibriTTS-R test-clean. (§4.2.1, Table 1)
- supports: Reference-aware gain adjustment derived from a learned prior can replace an implicit, self-inferred energy-normalization step in iterative vocoders.
Evidence: The gain-adjustment operator computed from the prior’s learned energy (Eq. 10) consistently improved reference-aware objective metrics over WaveFit’s self-gain-adjustment operator across SSL feature types and iteration counts. (§3.4, §4.2.2, Figure 3)
- complicates: Improvements in reference-aware objective metrics from a trainable-prior vocoder do not guarantee consistent gains in naturalness MOS across different SSL feature types.
Evidence: WaveTrainerFit’s naturalness MOS (N-MOS) exceeded WaveFit’s when conditioned on Whisper-encoder features (3.87 vs. 3.23) but fell below WaveFit’s when conditioned on WavLM (3.50 vs. 3.76) or XLS-R (3.21 vs. 3.79) features, despite objective metric gains in all three cases. (§4.2.1, Table 1)
- refines: The benefit of a learned, target-informed initial noise prior over Gaussian sampling in iterative vocoders is more pronounced when the conditioning feature carries less acoustic detail.
Evidence: On WavLM features from the 24th transformer layer (deep, low-acoustic-detail), WaveTrainerFit’s gains over WaveFit were substantially larger (SpeechBERTScore +3.11, Log-F0-RMSE -0.1322) than on 2nd-layer features (SpeechBERTScore +0.18, Log-F0-RMSE +0.0002), where the two methods were nearly tied. (§4.2.3, Table 2)
Limitations and Open Questions
The evaluation is confined to a single training corpus (LibriTTS-R, English-only) and a single vocoder backbone (WaveFit); the authors’ own future-work note flags scaling to larger datasets and multilingual models as unaddressed. The N-MOS regressions with WavLM and XLS-R conditioning (§4.2.1) are not fully explained: the authors offer two untested hypotheses (hyperparameter sensitivity inherited from RestoreGrad, and XLS-R’s known weaker English performance) rather than a diagnosed cause. The added prior encoder and sampling step also increase real-time factor relative to WaveFit, a trade-off the paper reports but does not otherwise mitigate.
Wiki Connections
- GAN Vocoder — extends the WaveFit/HiFi-GAN family of GAN-based iterative vocoders with a trainable noise prior and reference-aware gain adjustment for SSL-conditioned vocoding.
- Self-Supervised Speech — vocodes directly from frozen pretrained SSL representations (WavLM-large, XLS-R-0.3B) and studies robustness to the SSL feature extraction layer.
- Subjective Evaluation — reports human-rated naturalness (N-MOS) and similarity (S-MOS) listening tests with 15 participants across 450 samples per method.
- HiFi-GAN — used as a non-iterative GAN vocoder baseline to isolate the effect of the WaveFit architecture itself when vocoding SSL features.
- UTMOS — used as a reference-free naturalness metric in the layer-depth robustness ablation.