arXiv · 2025 · Preprint
Julian et al. (Neuphonic) · → Paper · Demo: ? · Code: ?
Introduces NeuCodec, an FSQ-based neural audio codec, and demonstrates that FSQ’s fixed-grid quantization encodes inherent redundancy into code sequences, making them significantly more robust to bit-level transmission errors than RVQ-based codecs.
Problem
Most neural audio codecs depend on Residual Vector Quantization, which imposes training difficulties: gradient propagation to codeword vectors requires auxiliary loss functions, and the hierarchical code structure frequently leads to codebook collapse where only a subset of entries is utilised. The multi-codebook design also complicates downstream autoregressive modeling, requiring separate transformer stages to predict each residual level per timestep. Beyond these training challenges, virtually no attention has been paid to how codec code sequences behave when transmitted through noisy channels. A single bit-flip in an RVQ code index can select an entirely unrelated codeword, causing catastrophic reconstruction failure.
Method
NeuCodec adopts an asymmetric encoder-decoder design built primarily on XCodec2. The encoder combines a frozen pretrained Wav2Vec2-BERT-large semantic encoder (600M parameters, pretrained on 4.5 million hours of unsupervised speech) with a trainable BigCodec-derived acoustic encoder (35M parameters, stacked Residual CNNs with Snake activations), yielding a total of 635M parameters. The discrete bottleneck is an FSQ module projecting encoder outputs to an 8-dimensional bounded space, quantising each dimension to equidistant scalar levels for a total codebook size of 2^16 (65,536 entries). The decoder is a standard transformer that directly predicts magnitude and phase for a Vocos waveform synthesis head. Training combines multi-resolution mel-spectrogram reconstruction, adversarial losses from a Spectrogram Discriminator and the HiFiGAN multi-period discriminator, discriminator feature matching, and an L2 semantic reconstruction loss. The base model trains for 800k steps on roughly 167,000 hours of commercially licensed 16kHz speech (Emilia-YODAS, MLS, LibriTTS-R, Fleurs-R, Common Voice, HUI, and 1,000 hours of proprietary data). A 24kHz upsampling decoder variant trains separately for 200k steps with the encoder and quantizer weights frozen.
An encoder distillation experiment investigates whether FSQ’s fixed quantization space allows multiple encoders to share a single decoder. The BigCodec encoder is replaced with the smaller L3AC encoder (21M acoustic parameters) and Wav2Vec2-BERT-large is replaced with DistilHuBERT (21M semantic parameters), reducing total encoder size from 635M to 42M parameters and increasing encoding speed 6-fold. A distillation loss (MSE between pre-quantization encoder outputs of teacher and student) supplements the original training objectives, activated after 20k warmup steps to prevent early divergence.
Key Results
NeuCodec with the 16kHz decoder achieves WER 2.3%, CER 0.9%, STOI 0.90, and PESQ 2.06 on a 100-utterance-per-speaker subset of CMU-Arctic (18 speakers). The distilled encoder reaches WER 2.8%, CER 1.2%, STOI 0.91, and PESQ 2.11 on the same benchmark, slightly outperforming NeuCodec on the reference-based perceptual metrics despite using 15x fewer parameters and encoding 6x faster.
Code-level analysis of the distillation experiment reveals that only 2% of codes match between the original and distilled encoder outputs element-wise, while mean cosine similarity between their pre-quantization projections is 0.73. In the FSQ implicit codebooks, element-wise agreement reaches 53%, and 93% of level predictions are either correct or off by exactly one level, indicating local structure in the quantization space.
The bit-level perturbation experiment on LibriSpeech test-clean shows a large gap between FSQ and RVQ robustness. FSQ codecs (NeuCodec, Distill-NeuCodec, StableCodec) maintain stable STOI and PESQ for bit-flip probabilities up to 10%, with NeuCodec retaining intelligible reconstructions even at 10% bit-flip rates. RVQ codecs (EnCodec with 12 codebooks, DAC with 6 codebooks) experience a sharp quality collapse above 1% bit-flip probability across all four metrics (STOI, PESQ, SI-SDR, mel-spectrogram MSE).
Novelty Assessment
NeuCodec itself is an engineering integration: its architecture derives directly from XCodec2, the encoder borrows from BigCodec, the waveform head uses Vocos, and the training losses follow established codec recipes. The FSQ substitution for RVQ, while the paper’s enabling design choice, is not independently novel. The genuine contributions are the encoder distillation finding and the perturbation robustness analysis. Demonstrating that two independently trained FSQ encoders can produce radically different code sequences yet decode to perceptually equivalent audio is a substantive empirical finding about the redundancy structure that FSQ imposes. The binary symmetric channel simulation is a clean evaluation methodology for codec transmission robustness that does not yet have an established protocol in the literature. The theoretical explanation (FSQ perturbations produce bounded embedding-space steps; RVQ perturbations produce unbounded ones) is concise and testable.
Field Significance
Moderate — This paper characterises a previously unreported property of FSQ-based codecs: their code sequences tolerate substantially higher bit-flip rates than RVQ-based alternatives, making them candidate architectures for transmission-sensitive applications such as telephony or wireless audio streaming. The encoder distillation finding provides empirical evidence that FSQ’s fixed quantization grid enables flexible encoder design without retraining the decoder, which could simplify future low-latency codec deployments. Neither result is paradigm-shifting, but together they establish an empirical basis for choosing FSQ over RVQ in robustness-critical scenarios.
Claims
-
supports: FSQ-based neural audio codecs develop inherent redundancy in their code representations, enabling multiple independently trained encoders to produce radically different code sequences that nonetheless decode to perceptually equivalent audio.
Evidence: Encoder distillation experiment on NeuCodec showing only 2% element-wise code match between original (635M params) and distilled (42M params) encoders, while cosine similarity between pre-quantization projections is 0.73 and reconstruction metrics (WER, CER, STOI, PESQ) remain within 0.5% WER of each other. (§4.2, Table 2)
-
supports: FSQ codecs are substantially more robust to bit-level transmission errors than RVQ codecs of comparable codebook size, maintaining intelligibility at bit-flip rates an order of magnitude higher than the RVQ failure threshold.
Evidence: Binary symmetric channel simulation on LibriSpeech test-clean showing FSQ codecs (NeuCodec, Distill-NeuCodec, StableCodec) maintain stable STOI and PESQ up to 10% bit-flip probability, whereas RVQ codecs (EnCodec, DAC) experience sharp quality collapse above 1% bit-flip rate. (§5, Figure 3)
-
refines: The perturbation robustness advantage of FSQ over RVQ is structural: FSQ’s fixed-grid quantization maps bit-flip perturbations to bounded, predictable steps in the embedding space, whereas RVQ code perturbations produce arbitrarily large embedding-space displacements.
Evidence: Implicit codebook confusion matrices show that 93% of level predictions between original and distilled encoders are either correct or off by exactly one neighbouring level, confirming local structure in the FSQ embedding space. (§4.2, §6)
-
complicates: FSQ-based codec architectures that achieve competitive intelligibility metrics often rely on very large pretrained semantic encoders, concentrating most of the parameter budget in a component that contributes indirectly to the quantization benefit.
Evidence: Wav2Vec2-BERT-large accounts for 600M of NeuCodec’s 635M total parameters; replacing it with DistilHuBERT reduces the model to 42M parameters with only a 0.5% WER increase, suggesting the semantic representation is partially substitutable without sacrificing the FSQ robustness property. (§3, §4.1, Table 2)
Limitations and Open Questions
Warning
Evaluation uses WER, CER, STOI, and PESQ but no perceptual naturalness metric (MOS or MUSHRA), making it impossible to assess how NeuCodec compares to state-of-the-art codecs on speech generation quality. The perturbation experiment uses a binary symmetric channel that does not reflect real-world transmission protocols (e.g., structured burst errors or packet loss), so robustness claims cannot be extrapolated directly to deployment scenarios.
Training includes 1,000 hours of proprietary data, partially limiting reproducibility. The model is evaluated only at 16kHz and 24kHz; the 24kHz upsampling decoder is trained on only 2,600 hours of data, and high-fidelity (48kHz) audio is not addressed. The paper does not evaluate whether FSQ robustness properties are preserved when the codec is used as a tokenizer for downstream TTS or language model tasks, which is the stated application motivation.
Wiki Connections
- Neural Audio Codec — NeuCodec proposes a new FSQ-based codec design and provides the first systematic comparison of RVQ versus FSQ robustness under bit-level transmission noise.
- Self-Supervised Speech — NeuCodec’s architecture uses Wav2Vec2-BERT-large as a frozen semantic encoder, making self-supervised pre-training a core structural component rather than an optional initialisation.
- Autoregressive Codec TTS — NeuCodec is designed as a low-bitrate tokenizer for LLM-based TTS and downstream speech generation tasks, following the single-codebook paradigm that simplifies autoregressive modeling.
- Evaluation Metrics — The binary symmetric channel simulation for measuring bit-level perturbation robustness introduces a new evaluation methodology for assessing codec resilience to transmission noise.