arXiv · 2022 · Preprint
Défossez et al. (Meta AI) · → Paper · Demo: ? · Code: ✓
EnCodec introduces a streaming convolutional encoder-decoder with residual vector quantization (RVQ), a multi-scale STFT-only discriminator, and a novel gradient balancer, establishing a neural audio codec that outperforms Opus and SoundStream across all tested bandwidths and audio domains.
Problem
Prior neural audio codecs, notably SoundStream, relied on a combination of waveform-domain (multi-scale) and STFT-domain discriminators, and complex multi-loss training that is sensitive to the relative scale of gradient contributions from each loss term. At very low bitrates (1.5-3 kbps), traditional codecs such as Opus and EVS degrade severely on non-speech content like music. Existing neural approaches had not demonstrated convincing results across a wide range of audio domains and sample rates simultaneously, and the training instability introduced by discriminator-heavy objectives made hyperparameter tuning costly.
Method
EnCodec is a fully convolutional streaming encoder-decoder with three components: an encoder E, a residual vector quantization layer Q, and a decoder G. The encoder uses 1D convolutions followed by strided convolution blocks (strides 2, 4, 5, 8), each doubling the channel count, plus a two-layer LSTM and a final 1D projection. At 24 kHz, this yields 75 latent frames per second; at 48 kHz, 150 frames per second. The decoder mirrors the encoder using transposed convolutions in reverse stride order.
The RVQ layer uses up to 32 codebooks of 1024 entries each (10 bits per codebook), supporting multiple target bandwidths (1.5, 3, 6, 12, 24 kbps at 24 kHz) by selecting a variable number of codebooks during training. A straight-through estimator handles gradient propagation through quantization.
A key training contribution is the multi-scale STFT (MS-STFT) discriminator, replacing the MSD+Mono-STFT combination from SoundStream. Five sub-networks each operate on complex-valued STFTs at window sizes of 2048, 1024, 512, 256, and 128. The training objective combines L1 time-domain loss, multi-scale mel-spectrogram loss, adversarial loss, feature matching loss, and RVQ commitment loss.
The gradient balancer decouples loss weight selection from the typical gradient scale of each term by normalising gradient magnitudes via exponential moving average and rescaling so that each loss’s weight directly represents its fractional contribution to the total gradient. This stabilises multi-loss optimisation without per-loss scale tuning.
Optionally, a 5-layer Transformer language model trained over RVQ codes enables arithmetic entropy coding, reducing effective bitrate by 25-40% without perceptual quality degradation. The streamable variant uses causal convolutions with 13 ms initial latency and operates at 9.8x real time on a single CPU core. (§3)
Key Results
MUSHRA evaluations (Table 1, Figure 3) show that at 3 kbps, EnCodec surpasses Lyra-v2 at 6 kbps and Opus at 12 kbps for clean speech. At 6 kbps, it reaches MUSHRA 83.1 for clean speech versus 30.1 for Opus at the same bitrate. For stereophonic 48 kHz music (Table 4), EnCodec at 6 kbps matches MP3 at 64 kbps (both ~82.9 MUSHRA), achieving comparable quality at 10x lower bitrate. With entropy coding, effective bitrates drop to 1.9 kbps (from 3) and 4.1 kbps (from 6) while quality is preserved.
Discriminator ablations (Table 2) confirm that the proposed MS-STFT discriminator alone achieves MUSHRA 77.5, close to the MS-STFT+MPD combination at 79.0, and substantially better than MSD+Mono-STFT at 62.9. Streamable mode incurs a modest objective-metric cost (Table 3) but maintains strong MUSHRA performance while enabling 13 ms initial latency on CPU.
Novelty Assessment
EnCodec’s architecture draws directly from SoundStream (Zeghidour et al., 2021): the same convolutional encoder-decoder skeleton and the same RVQ scheme. The primary novelties are: (1) replacing the compound discriminator setup with a multi-scale STFT-only discriminator that performs comparably while simplifying training; (2) the gradient balancer, a training-side contribution that stabilises multi-loss optimisation without loss-specific scale tuning; (3) the optional Transformer entropy coding layer that reduces bitrate without quality loss. The empirical coverage is broad: speech, noisy speech, music, mono and stereo, across five bandwidths, with methodologically rigorous MUSHRA testing. The architectural contribution is evolutionary relative to SoundStream, but the training improvements (gradient balancer, MS-STFT discriminator), multi-domain evaluation, and open-source release distinguish it from prior work. The open-source codebase lowers the barrier for downstream research relative to the proprietary SoundStream baseline.
Field Significance
Important
Foundational — EnCodec introduces a streaming convolutional codec with RVQ quantization, a gradient balancer for stable multi-loss training, and an MS-STFT discriminator that matches or exceeds prior multi-domain discriminator setups. It outperforms Opus and SoundStream across all tested bandwidths and audio domains while supporting real-time streaming on CPU. The open-source release provides a reproducible, multi-domain baseline that is directly reusable by downstream speech and audio research.
Claims
- A multi-scale STFT discriminator alone is sufficient for perceptual quality in neural audio codecs, removing the need for waveform-domain discriminators. (§4.5.1, Table 2)
- Gradient balancers that normalise loss contributions by expected gradient magnitude substantially stabilise training when combining reconstruction, adversarial, and commitment losses with widely varying natural scales. (§3.4, Table A.4)
- Residual vector quantization supports variable-bitrate operation from a single model by varying the number of active codebooks at inference, with each additional codebook yielding diminishing quality returns. (§3.2, Table 1)
- Auxiliary Transformer language models over RVQ codes can reduce effective bitrate by 25-40% through entropy coding without perceptual quality degradation, at the cost of increased latency. (§3.3, §4.5)
- Neural audio codecs outperform traditional DSP codecs at low bitrates across both speech and music domains, with the quality gap widening as bitrate decreases. (§4.5, Table 1, Figure 3)
Limitations and Open Questions
Warning
The 48 kHz model in non-streamable configuration operates slower than real time on CPU, limiting deployment without GPU acceleration or hardware-specific optimisation. Arithmetic coding precision issues (floating-point non-determinism across architectures) required a probability rounding workaround that the authors note may be insufficient for practical deployment.
The Transformer language model is small (5 layers, 200 channels) and neglects mutual information between codebooks at the same time step, leaving compression gains on the table. Music compression at 1.5 kbps still shows significant perceptual degradation. Training datasets mix multiple licenses, which may complicate commercial use. No evaluation of robustness to codec chaining (encode-decode-re-encode) or of downstream task performance degradation from quantization artefacts is provided.
Wiki Connections
Core concept: neural-codec — EnCodec introduces the RVQ-based streaming codec design that this concept page covers.
gan-vocoder — shares adversarial training infrastructure (multi-scale discriminators, feature matching loss) with GAN vocoder work; the MS-STFT discriminator design draws on and extends prior discriminator families.
autoregressive-codec-tts — codec-based TTS systems require a discrete audio tokenizer; EnCodec’s RVQ output is the representation format papers in this concept build on.
evaluation-metrics and subjective-evaluation — MUSHRA methodology with crowd-sourced annotation and explicit outlier filtering is described in detail and provides a methodological reference for codec evaluation.
In-corpus papers cited: 1912.06670 (Common Voice), 1609.03499 (WaveNet), 2104.00355 (speech resynthesis from discrete representations).
Cited by: 2403.03100 (NaturalSpeech 3, reference codec design for FACodec comparison) · 2411.13577 (WavChat survey, EnCodec as reference neural audio codec)
2303.03926 — Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling 2305.02765 — HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec 2305.09636 — SoundStorm: Efficient Parallel Audio Generation 2306.00814 — Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis 2306.12925 — AudioPaLM: A Large Language Model That Can Speak and Listen 2310.00704 — UniAudio: An Audio Foundation Model Toward Universal Audio Generation 2402.01912 — Natural language guidance of high-fidelity text-to-speech with synthetic annotations 2402.05755 — Spirit LM: Interleaved Spoken and Written Language Model 2402.08093 — BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data 2403.16973 — VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild 2409.05377 — BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec 2411.19842 — Scaling Transformers for Low-Bitrate High-Quality Speech Coding 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction