arXiv · 2025 · Preprint
Rui-Chen Zheng et al. (University of Science and Technology of China) · → Paper · Demo: ? · Code: ?
Introduces a training strategy that simulates noise-induced quantization instability directly inside RVQ codebooks, improving neural codec robustness to unseen background noise without any paired noisy-clean training data.
Problem
Neural speech codecs built on residual vector quantization (RVQ) reconstruct clean speech at high fidelity but degrade noticeably when the input contains even mild background noise, limiting their deployment in real-world communication and as tokenizers for speech generation pipelines. The conventional remedy, fine-tuning on paired noisy-clean speech, requires collecting diverse noisy-clean pairs and tends to produce robustness that is specific to the noise types seen during training, sometimes degrading clean-speech performance in the process. The paper traces the underlying vulnerability to the RVQ quantization step itself: each encoder feature is deterministically mapped to its nearest codebook entry, so a small perturbation from noise can push the feature across a decision boundary into an adjacent Voronoi cell, producing an entirely different codeword and downstream reconstruction error.
Method
The authors first quantify this fragility: comparing the first-stage RVQ codewords chosen for 120 clean VCTK utterances against their noisy counterparts (DEMAND noise at 15 dB SNR) shows that while the noisy codeword usually matches the clean top-1 choice, there is a pronounced long tail of cases where it instead matches the clean signal’s 2nd, 3rd, or lower-ranked nearest neighbor (a “codeword shift”).
To address this without noisy training data, the method replaces the deterministic nearest-neighbor rule with probabilistic top-K sampling during training: for each quantizer stage, the K nearest codewords to the encoder feature are identified, and the quantized output is sampled from a softmax distribution over their negative distances (temperature τ), rather than always taking the closest one. At inference, quantization reverts to standard nearest-neighbor selection. Because sampling probability decays with distance, this stochastic perturbation mimics the locality of real noise-induced shifts more faithfully than uniform random codeword substitution.
Applying this perturbation to all RVQ stages at once destabilizes training, since early stages encode the coarse structural content of the signal. The paper therefore introduces a progressive (curriculum-style) training schedule: probabilistic sampling is first applied only to the final quantizer (which encodes fine residual detail), and once the model adapts, the perturbed stage is stepped back one layer at a time toward the first quantizer, which is perturbed last. Both mechanisms require only clean speech; no noisy-clean pairs are used at any point.
The method is validated on two codecs: Encodec (24 kHz, 6 kbps, 6 VQ stages of codebook size 1024), fine-tuned from a pretrained clean-speech checkpoint with the full progressive strategy, and WavTokenizer (24 kHz, 0.9 kbps, single VQ of codebook size 4096), fine-tuned with plain probabilistic sampling (progressive scheduling does not apply to a single-stage codec). Hyperparameters (K=10, τ=5) were chosen from the codeword-shift analysis, which showed that most shifts fall within the 10 nearest candidates.
Key Results
On a VCTK subset with DEMAND noise mixed at 15 dB and 10 dB SNR, the proposed strategy improves every reported metric (SI-SDR, PESQ, STOI, UTMOS) over the standard nearest-neighbor-trained baseline for both codecs, with statistical significance (paired t-test) on all comparisons. For Encodec at 15 dB SNR, UTMOS rises from 3.475 to 3.586 and PESQ from 2.399 to 2.466; gains are smaller but consistent at 10 dB SNR. Notably, clean-speech metrics also improve (Encodec clean UTMOS 3.732 to 3.854), which the authors attribute to the codeword-level perturbation acting as an implicit data augmentation.
Compared against a strong noise-exposed fine-tuning baseline (“Closest*”, trained end-to-end on the same 18 DEMAND noise types mixed with clean training speech), the paper’s perturbation-only method trails on some noisy-speech metrics (e.g., PESQ) under matched training/test noise, as expected since Closest* has direct access to the test noise distribution. However, Closest* degrades clean-speech quality relative to the proposed method, and on three held-out noise types not seen by either method’s noise exposure, the proposed method outperforms Closest* on all four metrics, despite Closest* having trained on those same noise types. An ablation replacing distance-based probabilistic sampling with uniform random sampling (“Proposed†”) underperforms the full method on every metric for both clean and noisy speech, and applying probabilistic sampling to all RVQ stages simultaneously (“Direct Top-K”) underperforms the progressive schedule.
Novelty Assessment
The contribution is a training-recipe innovation rather than a new architecture: both probabilistic top-K sampling and curriculum-style progressive training are applied to existing, unmodified codec architectures (Encodec, WavTokenizer). The core idea, replacing deterministic quantization with a locality-aware stochastic sampling during training, is a reasonably novel way to introduce a “perturbation prior” into RVQ without touching the network structure, and the diagnostic codeword-shift analysis (Figure 1) that motivates it is a useful contribution on its own. The comparisons against the noise-exposed fine-tuning baseline are honest about the trade-off: the proposed method is not competitive under matched-noise conditions but is more general and preserves clean-speech quality, which the paper demonstrates rather than merely asserts. The evaluation is limited to a VCTK subset and two codecs, and noise levels are restricted to 10-15 dB SNR (relatively mild noise), so the generality of the robustness gains to more severe or non-additive distortions is untested.
Field Significance
Moderate — the paper isolates a specific and previously under-examined failure mode of RVQ (discrete decision-boundary instability under perturbation) and proposes a self-contained, clean-data-only training fix that generalizes better than noise-specific fine-tuning on the noise types tested. It is an incremental but well-evidenced technique for the codec-robustness sub-area rather than a new architecture or paradigm; the codeword-shift diagnostic (Figure 1) is a reusable analysis tool for future robustness work on RVQ-based codecs.
Claims
- supports: Deterministic nearest-neighbor codeword selection in residual vector quantization is fragile to small input perturbations, producing codeword reassignments that compound across RVQ stages into audible reconstruction artefacts.
Evidence: Codeword-shift analysis on Encodec’s first RVQ stage (24 kHz, 6 kbps) shows that although the noisy codeword usually matches the clean top-1 choice, a pronounced long tail of shifts (k>1) occurs when 15 dB SNR DEMAND noise is added to 120 clean VCTK utterances. (§2.1, Figure 1)
- supports: Neural speech codec noise robustness can be improved without any paired noisy-clean training data by simulating perturbation-induced instability directly at the quantization step.
Evidence: Fine-tuning Encodec and WavTokenizer with distance-weighted probabilistic top-K sampling, trained exclusively on clean speech, improves SI-SDR, PESQ, STOI, and UTMOS at both 15 dB and 10 dB SNR with statistically significant paired t-tests (e.g., Encodec UTMOS 3.475 to 3.586 at 15 dB). (§3.4, Table 1)
- complicates: Training on paired noisy-clean data achieves stronger robustness than perturbation-only training under the specific noise conditions it was exposed to, but this advantage does not transfer to unseen noise types and comes at the cost of clean-speech quality.
Evidence: A noise-exposed fine-tuning baseline (Closest*) outperforms the proposed clean-data-only method on some noisy-speech metrics (e.g., PESQ) under matched training/test noise, yet degrades clean-speech scores and is outperformed by the proposed method on all four metrics across three held-out noise types not present in either method’s training. (§3.5.1, Tables 2-3)
- refines: Introducing training-time quantization perturbations in a curriculum-like schedule, from the finest residual quantizer stage toward the coarsest, stabilizes robustness training more effectively than perturbing all quantizer stages simultaneously.
Evidence: PESQ and UTMOS improve steadily as probabilistic top-K sampling is progressively rolled out from the 6th to the 1st VQ of Encodec’s RVQ, whereas applying the same sampling to all six VQs at once (“Direct Top-K”) yields inferior results due to premature perturbation of core structural features. (§3.5.3, Figure 2)
Limitations and Open Questions
The evaluation is confined to a VCTK subset (a single-domain, studio-quality English read-speech corpus) and to relatively mild noise conditions (10-15 dB SNR from DEMAND). Whether the robustness gains extend to more severe noise, non-additive distortions (e.g., reverberation, codec cascading, packet loss), or multilingual/spontaneous speech is untested. The method is validated on only two codecs (Encodec, WavTokenizer); both use RVQ-style quantization, so its applicability to non-RVQ or single-codebook designs with different codebook geometries is unconfirmed. The paper itself notes extending the framework to more streamable architectures and integrating it with large speech-language models as future directions, but does not attempt either.
Wiki Connections
- Neural Audio Codec — the paper directly targets a robustness failure mode of RVQ-based neural codecs and proposes a training-time fix that requires no architectural change.
- GAN Vocoder — both codecs modified in this work (Encodec, WavTokenizer) rely on adversarial reconstruction losses typical of GAN-based neural audio coding, and the paper’s perturbation strategy is applied on top of that training pipeline.
- Autoregressive Codec TTS — the paper motivates codec robustness by noting that RVQ-derived discrete codes serve as the intermediate token representation consumed by codec-based speech language models, making codec-level noise sensitivity a potential upstream risk for such systems.