arXiv · 2026 · Preprint

Wangzixi Zhou et al. (Nara Institute of Science and Technology / National Institute of Information and Communications Technology) · → Paper · Demo: ✓ · Code: ✗

Recasts the ConvNeXt-based WaveNeXt generator as an iterative residual-denoising sub-model, fed by the STFT of the previous waveform estimate, so that one generator design serves both a GAN vocoder (fixed-point iteration) and a few-step diffusion vocoder (noise-level sub-models), with much faster CPU inference than HiFi-GAN, WaveFit and FastDiff on multi-speaker LibriTTS-R. Accepted at ICASSP 2026.

Problem

ConvNeXt-based vocoders such as Vocos and WaveNeXt are fast on CPU, but they had only been trained as GAN generators, and the original single-pass WaveNeXt degrades in multi-speaker settings (here it scores well below HiFi-GAN on LibriTTS-R). Iterative GAN vocoders like WaveFit recover quality but are slower than real time on a single CPU core, while diffusion vocoders train faster than GANs yet remain slow at inference even with reduced step counts. The authors want one fast generator design usable under either training paradigm.

Method

The building block is a sub-model that refines a waveform estimate. Given the previous estimate y_{t-1}, an STFT module (Hann window, centred) computes a complex spectrogram, truncates it to the mel-spectrogram length, and forms a real-valued “STFT-spec” by concatenating the full real part with the imaginary part minus the DC and Nyquist bins. This STFT-spec and the target mel-spectrogram are fed to a WaveNeXt-based generator (Conv1D, layer normalisation, n = 8 ConvNeXt blocks, two linear layers and a reshape that emits waveform samples directly, as in WaveNeXt without an iSTFT layer). Instead of producing the waveform outright, the generator predicts the noise (residual) component at each step, which is what makes it usable in both frameworks. Judging from the reported parameter counts (about 15M per step), each iteration uses its own sub-model weights.

Fig. 2: Overview of proposed architectures: (a) WaveNeXt-based generator where n is the number of ConvNeXt blocks. n = 8 is used in all the proposed models. (b) Sub-model for both GAN-WaveNeXt 2 and Diff-WaveNeXt 2, where t is the number of iterations steps.

GAN-WaveNeXt 2 follows the fixed-point iteration training of WaveFit: each sub-model takes the mel-spectrogram and the current waveform and outputs the next refined waveform, repeated for T steps (2 to 5 are evaluated). Compared with WaveFit, the authors drop the initial input noise (the “denoising” constraint in the loss) and the gain adjustment modules, which preliminary experiments showed were unnecessary given the STFT loss. Discriminators and losses are taken unchanged from WaveFit, with hop size 300 and 128-dimensional mel-spectrograms.

Diff-WaveNeXt 2 adopts noise-level limited sub-modeling from the same group’s earlier work: four sub-models are trained independently, each conditioned on the mel-spectrogram and a noisy waveform x_t = sqrt(a_t) x_0 + sqrt(1 - a_t) epsilon within its own noise range, with an MSE noise-prediction loss. The four-step noise schedule ([1.0e-4, 2.8e-2, 5.6e-1, 9.1e-1]) comes from a noise schedule predictor adapted from BDDM. At inference the four sub-models run sequentially from Gaussian noise, followed by a time-invariant spectral enhancement post-filter to restore high-frequency detail. Hop size is 256 to match FastDiff.

All systems are trained on the LibriTTS-R train-clean-100 and train-clean-360 subsets at 24 kHz on A100 GPUs; the WaveNeXt generator is implemented by replacing the iSTFT layer of the official Vocos code with a linear layer.

Key Results

Evaluation uses 4,824 LibriTTS-R test utterances for objective metrics (NISQA, UTMOS, MCD, log F0 RMSE) and RTF on an A100 GPU and a single AMD EPYC 7542 core, plus a 20-listener MOS test on 20 utterances (Table 1, Figure 4).

  • GAN family. At matched iteration counts, GAN-WaveNeXt 2 roughly ties WaveFit on automatic quality while being much faster: at 4 iterations, UTMOS 4.04 vs 3.99, NISQA 4.01 vs 3.97, RTF 0.0066 vs 0.0213 (GPU) and 0.20 vs 4.28 (CPU). WaveFit at 5 iterations has the best NISQA (4.02) and best MCD (0.90) of all systems, and HiFi-GAN V1 has the best UTMOS (4.05, vs 4.04 for GAN-WaveNeXt 2 at 4 or 5 iterations). In the MOS test, GAN-WaveNeXt 2 (4 iterations) scores 4.24, against 4.21 for WaveFit (5 iterations), 4.04 for HiFi-GAN and 4.25 for ground truth; no significance test is reported. Against HiFi-GAN it is faster (0.0066 vs 0.0110 GPU, 0.20 vs 0.80 CPU) and has lower MCD (0.95 vs 2.34), but higher log F0 RMSE (0.30 vs 0.16).
  • Single-pass baseline. The original WaveNeXt remains the fastest system (RTF 0.0022 GPU, 0.06 CPU) but scores only UTMOS 3.20 and NISQA 3.16 on this multi-speaker data, which is the gap the iterative design closes.
  • Diffusion family. With four steps, Diff-WaveNeXt 2 beats FastDiff with sub-modeling on every quality metric (UTMOS 3.87 vs 3.78, NISQA 3.81 vs 3.67, MCD 4.16 vs 4.32, log F0 RMSE 0.12 vs 0.24, MOS 3.81 vs 3.66) and is faster (RTF 0.0164 vs 0.0282 GPU, 0.16 vs 0.80 CPU). Its log F0 RMSE of 0.12 is the lowest of all systems. Its MOS (3.81) is, however, well below the GAN systems and HiFi-GAN.
  • Sub-modeling ablation. Without sub-modeling, the diffusion variant of the WaveNeXt generator is no better than FastDiff (UTMOS 3.55 vs 3.50) and has much worse MCD (7.34 vs 4.76); the sub-model training strategy accounts for most of Diff-WaveNeXt 2’s quality.
  • Training cost and size (Table 2, Table 1). Diff-WaveNeXt 2 trains in 32 GPU-hours vs 96 for FastDiff; GAN-WaveNeXt 2 needs 410 hours, the same as WaveFit and more than HiFi-GAN’s 270. Model size grows with each step: 59.94M parameters for GAN-WaveNeXt 2 at 4 iterations vs 15.51M for WaveFit and 13.9M for HiFi-GAN.

Cross-family comparisons (GAN vs diffusion) are confounded by different hop sizes (300 vs 256) and baseline implementations; comparisons within each family are matched.

Novelty Assessment

The new element is the residual sub-model: a WaveNeXt generator that takes the STFT of its own previous output alongside the mel-spectrogram and predicts a residual, which lets the same block be plugged into WaveFit-style fixed-point GAN training and into sub-model diffusion training. Most of the surrounding machinery is inherited: the GAN training recipe, discriminators and losses come from WaveFit (simplified by removing the input noise and gain adjustment), and the diffusion side reuses the authors’ earlier noise-level limited sub-modeling, a BDDM-derived schedule and their spectral enhancement post-filter. The contribution is therefore a modest architectural change combined with careful integration. The headline speed gains come mostly from swapping WaveFit’s and FastDiff’s generators for a ConvNeXt backbone; the cost is a parameter count that scales linearly with step count, which the paper acknowledges. Evaluation covers one English dataset with a small MOS test and no significance analysis.

Field Significance

Moderate: the paper shows that a ConvNeXt waveform generator can serve as the per-step denoiser in both iterative GAN and few-step diffusion vocoders, with the iterative form recovering the multi-speaker quality that single-pass WaveNeXt lacks while staying well under real time on one CPU core. It provides a controlled within-family comparison of generator backbones under fixed training recipes, rather than a new training paradigm.

Claims

  • supports: Iterative refinement with a lightweight convolutional generator can recover multi-speaker vocoding quality that a single-pass generator of the same family loses, while keeping CPU inference well under real time.

    Evidence: Single-pass WaveNeXt scores UTMOS 3.20 on LibriTTS-R, while the 4-iteration GAN-WaveNeXt 2 reaches UTMOS 4.04 and MOS 4.24 (ground truth 4.25) at CPU RTF 0.20, against 4.28 for WaveFit with the same iteration count. (§4.2, Table 1, Figure 4)

  • supports: Training a separate denoiser for each noise-level range is critical for few-step diffusion vocoders; a single shared denoiser at four steps loses much of the attainable quality.

    Evidence: Without sub-modeling, the diffusion WaveNeXt variant reaches UTMOS 3.55 and MCD 7.34; with four independent sub-models it reaches UTMOS 3.87 and MCD 4.16, and FastDiff shows the same direction (UTMOS 3.50 to 3.78). (§4.1 “Ablation study”, Table 1)

  • complicates: Per-step sub-models trade parameter count for inference speed and quality, so fast iterative vocoders can be several times larger than single-pass GAN vocoders.

    Evidence: GAN-WaveNeXt 2 grows from 29.97M parameters at 2 iterations to 74.93M at 5, and Diff-WaveNeXt 2 has 57.68M, against 13.9M for HiFi-GAN V1 and 15.51M for WaveFit; the authors list this growth as a limitation. (§4.2, Table 1)

  • refines: Diffusion vocoders train far faster than GAN vocoders but, at very few steps, still trail them in perceived naturalness even when sharing a generator backbone.

    Evidence: Diff-WaveNeXt 2 trains in 32 GPU-hours vs 410 for GAN-WaveNeXt 2, but scores MOS 3.81 vs 4.24 (HiFi-GAN 4.04), although the two variants also differ in hop size (256 vs 300). (§4.2, Table 2, Figure 4)

  • complicates: Objective vocoder metrics can rank systems inconsistently, so spectral distortion and pitch error do not track each other or perceived quality.

    Evidence: HiFi-GAN V1 has far higher MCD than GAN-WaveNeXt 2 (2.34 vs 0.95) yet lower log F0 RMSE (0.16 vs 0.30), and the highest UTMOS (4.05) while trailing GAN-WaveNeXt 2 in MOS (4.04 vs 4.24). (§4.2, Table 1, Figure 4)

Limitations and Open Questions

Model size scales with the number of iterations or sub-models, reaching roughly four times the parameters of WaveFit or HiFi-GAN at four steps, which offsets part of the deployment benefit of the faster CPU inference; the authors acknowledge this as an open issue.

  • Evaluation is limited to one English read-speech corpus (LibriTTS-R) with copy-synthesis from ground-truth mel-spectrograms; there is no TTS pipeline test, no out-of-domain or unseen-recording-condition evaluation, and no other languages.
  • The MOS test uses 20 utterances and 20 listeners with no reported significance testing, so the small MOS differences between GAN-WaveNeXt 2, WaveFit and ground truth are not resolved.
  • The diffusion comparison is against FastDiff at four steps only; stronger recent diffusion or flow-matching vocoders, and other ConvNeXt vocoders such as Vocos, are not compared directly on this data.
  • GAN-WaveNeXt 2 offers no training-cost advantage over WaveFit (410 hours each) and costs more than HiFi-GAN (270 hours).
  • No code release is mentioned; baselines rely on unofficial HiFi-GAN and WaveFit implementations.

Wiki Connections

  • GAN Vocoder — GAN-WaveNeXt 2 is an iterative GAN vocoder that matches WaveFit and HiFi-GAN quality on multi-speaker data with much lower CPU RTF.
  • Diffusion TTS — Diff-WaveNeXt 2 is a four-step diffusion vocoder whose quality depends on per-noise-level sub-models and a learned noise schedule.
  • Subjective Evaluation — Naturalness is assessed with a 20-listener five-point MOS test alongside automatic MOS predictors.
  • HiFi-GAN — Used as the main single-pass GAN baseline, which GAN-WaveNeXt 2 matches in quality while running faster on GPU and CPU.
  • Vocos — The official Vocos code is the implementation base for the WaveNeXt generator, with its iSTFT layer replaced by a linear layer.
  • UTMOS — Used as one of the two automatic MOS predictors in the objective evaluation.