arXiv · 2025 · Preprint
Wei-Cheng Tseng et al. (University of Texas at Austin) · → Paper · Demo: ? · Code: ?
Introduces the first self-supervised speech representation learning framework that operates exclusively on discrete neural-codec units throughout pre-training, replacing continuous waveform or spectrogram input entirely (to be presented at ASRU 2025).
Problem
Self-supervised learning (SSL) has become the dominant recipe for building general-purpose speech representations, but pre-training these foundation models is expensive: the SSL phase processes raw waveforms or high-dimensional continuous features (Mel spectrograms) through convolutional feature extractors, which drives up compute, storage, and I/O cost as pre-training corpora scale. Separately, neural audio codecs compress speech into compact discrete token sequences and have proven effective as inputs for supervised tasks such as ASR and speaker verification (CodecASR, Puvvada et al., VioLA), but benchmarks such as DASB show that codec units alone, without any additional learning, underperform on tasks that require contextual understanding. No prior work had asked whether a general-purpose self-supervised representation model could be pre-trained directly and entirely on discrete codec units, closing that gap between raw codec output and a contextualized foundation model.
Method
Codec2Vec follows the standard SSL “pretrain-then-finetune” paradigm but replaces every continuous input in that pipeline with pre-computed discrete codec units. Speech is first compressed offline with an off-the-shelf DAC codec (16kHz variant, 12 codebook sequences at 50Hz). Each codebook’s discrete codes are mapped through an embedding table (initialized from the codec’s own codebook embeddings, which the authors found improves performance) and the per-codebook embeddings are summed into a single frame-level vector sequence that is fed to a BASE-sized Transformer encoder (12 layers, 768-dim). Quantizer dropout, applied by randomly dropping a subset of codebook sequences during training, is used to improve robustness. The encoder is trained with a masked-prediction objective: a random subset of frames (8% of positions, 10-frame mask spans) is replaced with a learnable mask embedding, and the encoder plus a projection layer predicts a distribution over the training target for each masked position.
The paper investigates three strategies for deriving that training target, all applied to the same discrete-input encoder: (a) reconstruction-based, where the model predicts the original (masked) codec codes themselves, analogous to DiscreteBERT-style masked language modeling; (b) iterative clustering, following the HuBERT recipe, where k-means is repeatedly applied to the model’s own intermediate representations to produce refined frame-level cluster targets across successive training rounds; and (c) online clustering, following DinoSR, where a teacher network (an exponential moving average of the student) dynamically maintains per-layer codebooks over its own representations and the student is trained to predict the teacher’s cluster assignments. After pre-training, a lightweight downstream module is appended to the frozen or fine-tuned encoder and trained on each SUPERB task using the same discretized inputs.
Key Results
Evaluated on the SUPERB benchmark (phoneme recognition, ASR, keyword spotting, intent classification, slot filling, speaker diarization, speaker verification, emotion recognition), the online-clustering variant of Codec2Vec is the strongest configuration: it matches or exceeds the waveform-input HuBERT baseline on most tasks (e.g., 96.7 keyword-spotting accuracy vs. HuBERT’s 96.3, 98.4 intent-classification accuracy vs. HuBERT’s 98.3) and closely approaches, but does not fully match, the stronger continuous-input DinoSR baseline (6.2 vs. 4.7 WER on ASR; 6.0 vs. 5.5 EER on speaker verification) (§IV-A, Table I). A controlled comparison using the identical HuBERT-iter0-layer9-kmeans500 target for both a waveform-input HuBERT and a discrete-input Codec2Vec model shows near-parity across all eight SUPERB tasks (e.g., 5.2 vs. 5.4 phone error rate, 96.7 vs. 96.3 keyword-spotting accuracy), isolating the effect of substituting the input modality (§IV-A, Table I). Among the three target-derivation strategies, reconstruction-based targets perform markedly worse than either clustering-based strategy (e.g., 19.6 vs. 4.2–5.5 phone error rate), establishing clustering-derived targets as necessary for competitive performance from discrete codec input (§III-B, §IV-A, Table I). On efficiency, replacing continuous waveform storage (.wav) with pre-extracted codec-unit storage (.npz) for the 960-hour LibriSpeech pre-training set reduces dataset size from 60.4GB to 3.6GB (16.5x) and reduces SSL pre-training wall-clock GPU hours from 830 to 356 (2.3x), including the one-time offline codec-extraction cost (§IV-B, Table II).
Novelty Assessment
Codec2Vec does not introduce a new architecture or a new SSL learning objective; masked prediction, iterative k-means clustering, and online EMA-teacher clustering are all adopted directly from HuBERT and DinoSR, and the codec itself (DAC) is used off the shelf. The paper is explicit about this: its contribution is demonstrating that this existing SSL machinery remains effective when the input modality is switched entirely to discrete codec units for the whole pre-training phase, rather than proposing a new mechanism. The genuine novelty is in being the first to run a full SSL pre-training pipeline exclusively on codec tokens (as opposed to prior work that either used codec units for supervised task-specific training, such as CodecASR and VioLA, or benchmarked raw codec units without any additional pre-training, such as DASB) and in quantifying the resulting storage and compute savings. This makes the contribution primarily an engineering-integration and empirical-feasibility study rather than an architectural one.
Field Significance
moderate — Codec2Vec provides the first concrete evidence that a full self-supervised pre-training pipeline, not just downstream fine-tuning, can operate entirely on discrete codec tokens without a convolutional waveform encoder, and it quantifies the resulting storage and compute savings. This is a useful data point for the broader move toward discrete audio representations, but the study is limited to a single 960-hour English read-speech corpus, a single primary codec (DAC), and standard SUPERB probing tasks, so its generalization to larger-scale or noisier pre-training regimes is not yet demonstrated within the paper.
Claims
- supports: Masked-prediction self-supervised objectives originally developed for continuous audio input can be applied directly to discrete neural-codec token sequences as the sole model input throughout pre-training, without a convolutional waveform feature extractor, and still yield representations competitive with continuous-input SSL models on a broad multi-task benchmark.
Evidence: Using the identical HuBERT-iter0-layer9-kmeans500 training target, the discrete-input Codec2Vec model achieves near-parity with waveform-input HuBERT across all eight SUPERB tasks (e.g., 5.2 vs. 5.4 phone error rate, 6.9 vs. 6.4 WER, 96.7 vs. 96.3 keyword-spotting accuracy). (§IV-A, Table I)
- refines: For self-supervised pre-training on discrete codec-unit input, the strategy used to derive masked-prediction training targets matters more than the choice to use discrete input itself: targets built from the model’s own clustered latent representations substantially outperform targets that simply reconstruct the (masked) input codec codes.
Evidence: Reconstruction-based Codec2Vec trails both iterative-clustering and online-clustering Codec2Vec by a wide margin on nearly every SUPERB task (e.g., 19.6 vs. 4.2–5.5 phone error rate, 78.3 vs. 97.1–98.4 intent-classification accuracy). (§III-B, §IV-A, Table I)
- complicates: Replacing continuous waveform input with pre-quantized discrete codec-unit sequences for self-supervised pre-training introduces a consistent, task-specific degradation on ASR-style content recognition, even when overall representation quality is otherwise competitive.
Evidence: Across matched target-derivation strategies, discrete-input Codec2Vec variants show higher (worse) ASR word error rate than their continuous-input counterparts (HuBERT 6.4 vs. Codec2Vec 6.9 with identical targets; DinoSR 4.7 vs. Codec2Vec 6.2 with online-clustering targets), a gap the authors attribute to quantization-induced information loss and the narrowband characteristics of the codec. (§IV-A, §V, Table I)
- supports: The specific neural audio codec used to generate discrete input tokens materially affects the quality of representations learned by a downstream self-supervised model trained on those tokens, independent of the SSL training recipe.
Evidence: With an identical training recipe and target (HuBERT-iter0-layer9-kmeans500), switching the input codec from DAC to an unofficial 16kHz Encodec variant degrades phone error rate from 5.2 to 6.0, slot-filling F1/CER from 88.6/24.5 to 87.9/26.3, and speaker-verification EER from 5.1 to 6.2. (§IV-C, Table III)
- complicates: Storage and compute efficiency gains from pre-computing discrete codec tokens for self-supervised pre-training have not been validated under noisy or acoustically mismatched conditions, leaving open whether the efficiency and performance results generalize beyond clean, read-speech pre-training corpora.
Evidence: The authors state they have not extensively evaluated performance in real-world noisy conditions and flag the interaction between noise, codec compression, and SSL pre-training as an open research question. (§V, Discussion and Limitations)
Limitations and Open Questions
All results are demonstrated on a single 960-hour, clean, read-English corpus (LibriSpeech) with one primary codec (DAC) and one model scale (BASE, 12 layers); scaling behavior, multilingual generalization, and noisy-condition robustness are not evaluated in the paper.
- Codec selection is shown to substantially affect downstream representation quality (§IV-C), but the paper does not identify which codec properties drive this effect or how to select an optimal codec for representation learning; the authors describe this as an open research question.
- Discrete-input models consistently trail continuous-input baselines on ASR specifically, even in the best-performing configuration, indicating an information bottleneck that current codec-based SSL has not closed.
- Current mainstream SSL pre-training strategies (masked prediction, iterative and online clustering) were designed for continuous input signals; the paper notes that closing remaining gaps may require SSL objectives specifically tailored to discrete input sequences, which it leaves to future work.
- Code and demo availability are not stated in the paper.
Wiki Connections
- Neural Audio Codec — investigates how the choice and properties of a neural audio codec’s discrete output tokens affect downstream self-supervised representation quality, running a direct DAC-vs-Encodec ablation under an otherwise identical training recipe.
- Self-Supervised Speech — proposes a new pre-training regime for self-supervised speech representation learning that replaces continuous waveform or spectrogram input with discrete codec tokens throughout the entire SSL phase, evaluated against HuBERT and DinoSR on SUPERB.
- High Fidelity Neural Audio Compression — used as one of the two neural codecs (an unofficial 16kHz variant) in a controlled ablation isolating the effect of codec choice on downstream SSL representation quality.
- DASB - Discrete Audio and Speech Benchmark — its finding that raw codec units underperform on contextual tasks without additional learning is cited as the direct motivating gap that Codec2Vec’s self-supervised pre-training approach is designed to close.