arXiv · 2024 · Preprint
Wu et al. · → Paper · Demo: ? · Code: ?
A structured comparative review of six open-source neural audio codec models and eleven codec-based language models, examining their design choices, training strategies, and task coverage to map the emerging codec-based audio generation landscape as of early 2024.
Problem
By early 2024, a wave of neural audio codec models and codec-based language models had appeared in rapid succession, each with its own architectural choices, quantisation strategies, training data configurations, and supported tasks. The literature lacked a systematic side-by-side analysis of how these systems differ in design and where they agree, making it difficult for practitioners and researchers to orient themselves within the emerging paradigm.
Scope and Coverage
The survey covers two interconnected families: (1) six open-source neural audio codec models (SoundStream, Encodec, AudioDec, AcademiCodec, SpeechTokenizer, DAC, FunCodec), compared across discriminator design, semantic token integration, quantisation strategy, training data, sample rate, and bitrate; and (2) eleven codec-based language models (AudioLM, VALL-E, VALL-E X, VioLA, AudioPaLM, MusicLM, MusicGen, LauraGPT, UniAudio, SpeechX, AudioGen), compared by input modality, output modality, downstream tasks, and codec dependency.
On the codec side, the review identifies a convergence around Residual Vector Quantisation (RVQ) across all surveyed models, while distinguishing how each approaches discriminator design (multi-period, multi-scale, and multi-band STFT variants), whether semantic tokens from self-supervised models such as HuBERT are used to guide the first RVQ layer, and whether group-residual quantisation is employed to reduce bitrate. SpeechTokenizer, for instance, explicitly regularises its first RVQ layer to encode semantic content via HuBERT distillation, while DAC uses periodic activation functions and multi-band STFT discriminators to reduce phase artefacts across speech, music, and general audio. FunCodec proposes a frequency-domain encoding alternative, claiming comparable quality at lower parameter count.
On the language model side, the review organises systems by task coverage and codec dependency. VALL-E and its variants (VALL-E X, SpeechX) use Encodec as a tokeniser for codec language modelling conditioned on text or acoustic context. AudioPaLM and VioLA represent unified text-audio models capable of processing and generating in both modalities. UniAudio covers the broadest task set (TTS, VC, speech enhancement, target speaker extraction, music and sound generation, singing) using a multi-scale Transformer to accelerate autoregressive prediction over hierarchical codec codes. A separate branch based on self-supervised discrete units (GSLM, pGSLM, dGSLM, TWIST) is also surveyed, covering textless speech language modelling where discrete units serve as pseudo-text without acoustic codec compression.
The paper explicitly covers prompting techniques applied to speech LMs, including prompt tuning, in-context learning, and instruction tuning, noting these as an emerging direction enabled by the growing maturity of codec-based speech LMs. (§III.A)
Key Results
The survey is descriptive rather than evaluative; no new experimental results are reported. The primary outputs are Tables I, II, and III comparing codecs and language models on a structured set of dimensions. Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion).
Novelty Assessment
The contribution is organisational rather than technical. No new model, architecture, or benchmark is introduced. The value lies in providing a structured comparison table and taxonomy at a point when the literature was moving quickly, making design choices explicit that were otherwise scattered across individual papers. The scope is somewhat narrow: only six codecs and eleven LMs are covered, and the survey does not address evaluation methodology, training data diversity, or the downstream quality impact of codec design choices on generation systems. Given the speed at which this area has moved since early 2024, the snapshot has aged, but it captures a historically useful cross-section of the codec-based paradigm at its consolidation phase.
Field Significance
Moderate — this overview provides a structured comparison of the early codec-based language modelling landscape, making explicit the design axes (RVQ variants, semantic token integration, discriminator families, task coverage) that were implicit in the contemporaneous literature. It is valuable as a historical cross-section of the codec-and-LM paradigm around early 2024, but introduces no new architectural or empirical contribution.
Claims
- Residual vector quantisation is the dominant quantisation strategy across neural audio codec models, with variation concentrated in discriminator design, bitrate, and semantic token integration rather than in the core compression mechanism. (§II.A, Table II)
- Codec-based audio language models increasingly target multi-task coverage rather than single-task specialisation, with several systems spanning TTS, voice conversion, speech editing, speech enhancement, and translation in a single framework. (§III.B, Table III)
- Integrating semantic tokens from self-supervised speech representations into the codec quantisation process improves audio quality at low bitrates, with HuBERT-guided RVQ being the most common approach. (§II.B)
- Discrete units derived from self-supervised representations enable textless speech language modelling but sacrifice speaker and paralinguistic information relative to codec-based approaches. (§III.A)
Limitations and Open Questions
The survey covers only open-source codec models and does not include proprietary codecs used in industry systems. Evaluation methodology is not addressed: the paper does not compare codecs on shared benchmarks or report reproduction numbers, making it difficult to assess quality claims from the original papers in a unified way. The codec-based LM survey relies entirely on self-reported results from each system’s own paper, with no cross-system comparison on shared evaluation sets. Coverage is limited to models available as of early 2024; the field has moved substantially since, with systems such as SpeechTokenizer and more recent codec variants not included in the analysis.
Open questions the survey surfaces: What is the quality impact of codec design choices (bitrate, discriminator type, semantic integration) on downstream speech generation systems? Can a universal codec serve all audio types equally well? How should speech LMs be prompted efficiently in zero-shot settings?
Wiki Connections
- neural-codec — this survey provides a structured comparison of six open-source neural audio codec models, making it a useful reference point for the early-2024 state of the neural-codec landscape.
- autoregressive-codec-tts — the paper reviews eleven codec-based language models, several of which (VALL-E, VALL-E X, SpeechX) are the foundational systems for autoregressive codec TTS.
- spoken-language-model — the survey maps the landscape of audio language models that use codec codes as discrete tokens, including textless speech LMs and multi-task systems.
- self-supervised-speech — the paper discusses the role of HuBERT and w2v-BERT in providing semantic tokens that guide codec quantisation and enable textless speech LM training.
- zero-shot-tts — VALL-E and VALL-E X, reviewed in this survey, are the primary early examples of zero-shot TTS via codec language modelling.
- 2210.13438 (Encodec) — surveyed as the dominant codec backbone adopted by VALL-E, VALL-E X, VioLA, UniAudio, SpeechX, and MusicGen.
- 2308.16692 (SpeechTokenizer) — discussed as an example of semantic-guided RVQ codec designed specifically for speech language model applications.
- 2301.02111 (VALL-E) — reviewed as the pioneering neural codec language model for zero-shot TTS, establishing the codec-based paradigm the survey centres on.
- 2303.03926 (VALL-E X) — reviewed as an extension of VALL-E to cross-lingual zero-shot TTS via codec language modelling.
- 2310.00704 (UniAudio) — reviewed as the broadest-coverage codec-based LM surveyed, spanning speech, music, and audio generation tasks.
- 2306.12925 (AudioPaLM) — reviewed as an example of unified text-audio language model capable of both ASR and TTS.
- 2301.11325 (MusicLM) — reviewed as an application of the codec-based hierarchical LM paradigm to conditional music generation.
- 2308.11596 (SeamlessM4T) — cited as an example of applying codec-based discrete unit translation in a multilingual, multimodal setting.
- 2312.05187 (Seamless) — cited alongside SeamlessM4T as part of the discrete-unit-based speech translation line of work.