arXiv · 2025 · Preprint
Du et al. · → Paper · Demo: ? · Code: ?
DLL-APNet replaces standard convolutions in the APNet2 GAN vocoder with causal convolutions and applies knowledge distillation from a non-causal teacher to recover the resulting quality drop, achieving synthesis quality competitive with non-causal vocoders while eliminating future-context latency for real-time streaming applications.
Problem
Most mainstream GAN-based neural vocoders optimise for synthesis quality and throughput while ignoring model-intrinsic latency. Standard convolutional layers use symmetric padding, giving the kernel access to future input frames; as network depth increases through stacked convolutional blocks, the receptive field accumulates future context and latency grows proportionally. This prevents frame-by-frame concurrent transmission and computation, a hard requirement at the receiving end of real-time communication pipelines where mel spectrograms are transmitted and decoded on the fly. Replacing standard convolutions with causal counterparts removes future-context access, but this modification degrades synthesis quality, and no targeted remedy for this degradation had been applied to all-frame-level amplitude-and-phase-spectrum vocoders.
Method
DLL-APNet extends APNet2, an all-frame-level GAN vocoder that predicts amplitude and phase spectra in parallel from a mel spectrogram and reconstructs the waveform via inverse short-time Fourier transform (iSTFT). The primary architectural change is replacing every standard convolutional layer with a causal convolution that applies asymmetric padding, ensuring that the output at time step t depends only on inputs at t and earlier frames.
For amplitude prediction, an input causal convolution is followed by K=8 causal ConvNeXt v2 blocks (depth-wise convolution converted to causal, with point-wise convolutions unchanged since they introduce no latency) and a causal output convolution. Phase prediction uses a causal parallel estimation architecture (PEA): two parallel causal convolutional layers estimate pseudo-real and pseudo-imaginary components, which are combined via the two-argument arctangent to produce the wrapped phase directly.
To recover quality lost to causality constraints, a pre-trained non-causal APNet2 serves as a teacher model. Knowledge distillation is applied after the input convolution and after each of the K ConvNeXt v2 blocks, minimising the L1 distance between teacher and student intermediate features. This allows the causal student to learn temporal dependencies it cannot directly observe within its limited causal receptive field. The full training objective combines amplitude loss, phase loss, reconstructed STFT loss, final waveform loss, and distillation loss, weighted by per-term hyperparameters, along with multi-period discriminator (MPD) and multi-resolution discriminator (MRD) adversarial supervision.
STFT settings: frame length 320, frame shift 80, FFT size 1024, 80-dimensional mel spectrograms at 16 kHz. Training ran for up to 500k steps using AdamW, with the distillation weight lambda_KD set to 5.
Key Results
Evaluation uses 2,937 utterances from 8 unseen VCTK speakers (Table I). DLL-APNet achieves UTMOS 3.98, SNR 6.07 dB, F0-RMSE 20.53 cent, MCD 1.04 dB, and V/UV error 3.16%, at 6.30 GFLOPS per second of audio.
Among causal baselines, DLL-APNet outperforms causal Vocos (UTMOS 3.87), causal APNet2 without distillation (3.90), causal HiFi-GAN (3.88), and causal iSTFTNet (3.75). Notably, the causal BigVGAN variant failed to converge entirely, highlighting that causal adaptation is not universally applicable.
Compared to non-causal vocoders, DLL-APNet’s UTMOS 3.98 is second only to non-causal APNet2 (4.00) and exceeds non-causal BigVGAN (3.97), HiFi-GAN (3.93), iSTFTNet (3.93), and Vocos (3.91), while maintaining the same computational cost as APNet2 (6.30 GFLOPS).
Ablation of distillation weight lambda_KD shows peak UTMOS at lambda_KD=5; both lower and higher values degrade quality, indicating a balance between distillation and direct natural-speech supervision. Ablation of distillation depth (number of ConvNeXt v2 blocks involved) shows monotonically increasing UTMOS from 3.73 (no blocks distilled) to 3.98 (all 8 blocks), supporting full-depth distillation as the optimal configuration (§V-B, Table II).
Novelty Assessment
The contribution is primarily engineering integration. Causal convolutions for low-latency processing are established in speech coding and enhancement; knowledge distillation from a non-causal teacher is standard practice in model compression and causal model training. DLL-APNet applies both techniques to the APNet2 amplitude-and-phase vocoder framework without introducing new architectural primitives. The causal PEA for phase prediction is a direct adaptation of the authors’ prior phase estimation work rather than a new design. The paper’s genuine contribution is demonstrating that this combination nearly closes the quality gap to non-causal vocoders in the all-frame-level spectral prediction setting, and that distillation depth matters monotonically. The negative result for causal BigVGAN (convergence failure under the same causal substitution procedure) is informative. The evaluation relies entirely on objective metrics and automated MOS prediction (UTMOS); no human listening test is reported, which limits the strength of the perceptual quality claims.
Field Significance
Moderate — DLL-APNet addresses a concrete and underexplored gap in the GAN vocoder literature: the absence of a causal model that approaches non-causal quality without autoregressive or diffusion-based inference. The paper provides systematic evidence that knowledge distillation from a non-causal teacher is an effective strategy for causal adaptation, and that all-frame-level spectral vocoders are more robust to this adaptation than waveform-upsampling architectures. The convergence failure of causal BigVGAN provides a useful cautionary data point for practitioners targeting real-time deployment.
Claims
-
supports: Knowledge distillation from a non-causal teacher can substantially recover the quality degradation introduced by causal convolutions in GAN-based neural vocoders.
Evidence: DLL-APNet with full distillation reaches UTMOS 3.98, compared to 3.90 for causal APNet2 trained without distillation and 4.00 for non-causal APNet2; distillation depth ablation shows monotonic improvement from 3.73 (no blocks) to 3.98 (all 8 blocks) on VCTK unseen speakers. (§V-A, §V-B, Table I, Table II)
-
complicates: Causal adaptation of GAN-based vocoders is not universally applicable across all architectures.
Evidence: The causal variant of BigVGAN failed to converge during training and could not produce intelligible speech under the same causal substitution procedure applied to HiFi-GAN, iSTFTNet, APNet2, and Vocos, all of which converged. (§V-A)
-
complicates: Causal constraints in neural vocoders disproportionately degrade pitch accuracy relative to spectral quality.
Evidence: Replacing standard convolutions with causal counterparts caused substantially larger relative degradations in F0-RMSE and V/UV error than in MCD and LAS-RMSE across all causal baseline vocoders on VCTK (e.g., causal HiFi-GAN: F0-RMSE 58.06 vs. 31.61 non-causal; causal iSTFTNet: F0-RMSE 54.43 vs. 32.87 non-causal). (§V-A, Table I)
-
supports: All-frame-level vocoders that predict spectral features before waveform reconstruction are more resilient to causal adaptation than waveform-upsampling vocoders.
Evidence: Under the same causal substitution procedure on VCTK, causal APNet2 (UTMOS 3.90), causal Vocos (3.87), and DLL-APNet (3.98) outperform causal HiFi-GAN (3.88) and causal iSTFTNet (3.75), and computational cost (GFLOPS) is unchanged by causality conversion in all cases. (§V-A, Table I)
Limitations and Open Questions
Warning
Evaluation is entirely objective; no human listening test is reported. All perceptual quality claims rest on UTMOS (an automated MOS predictor), which may not fully capture quality degradation under causality constraints or differences between causal and non-causal models.
Evaluation uses a single English dataset (VCTK at 16 kHz) with 8 unseen speakers, so generalisation to other languages, higher sampling rates, or diverse speaking conditions is untested. Model size (parameter count) is not reported. The comparison against causal BigVGAN is limited to reporting convergence failure, with no diagnostic on why the architecture is more fragile to causal substitution. Future work is directed at reducing model size and computational overhead.
Wiki Connections
- GAN Vocoder — DLL-APNet extends the APNet2 all-frame-level GAN vocoder with causal convolutions and distillation-based training, directly contributing to the low-latency frontier of GAN vocoding.
- Streaming TTS — the paper’s central motivation and design constraint is real-time frame-by-frame synthesis, eliminating future-context dependence through causal convolutions to enable concurrent transmission and decoding.
- Evaluation Metrics — the paper uses UTMOS as an automated perceptual metric alongside five objective acoustic metrics (SNR, LAS-RMSE, MCD, F0-RMSE, V/UV error) to profile quality-latency trade-offs across causal and non-causal vocoders.