ACL · 2025 · Conference
Wenrui Liu et al. (Alibaba Group / Zhejiang University) · → Paper · Demo: ✓ · Code: ?
This paper formally characterizes a Discrete Representation Inconsistency (DRI) problem in RVQ-based neural audio codecs — where identical audio segments produce divergent token sequences depending on their context window — and proposes two consistency constraint losses (slice-consistency and perturbation-consistency) that increase token consistency by up to 36% and reduce downstream VALL-E WER by 1.98% absolute.
Problem
Neural audio codecs based on Residual Vector Quantization (RVQ) use convolutional encoders with large receptive fields to capture contextual information, which improves compression efficiency and reconstruction quality. However, this design choice introduces a serious side effect: the same audio segment is tokenized differently depending on what audio surrounds it. The resulting discrete token sequences are perceptually equivalent (both decode to the same audio) but are numerically divergent — a many-to-one mapping from audio to token sequences.
Text tokenizers do not have this property: the same characters always produce the same tokens regardless of context. When a neural codec language model (like VALL-E) is trained on codec-tokenized speech, it must simultaneously learn to predict many different correct next-token sequences for identical audio inputs. This increases uncertainty in the prediction distribution, raising WER and reducing speaker similarity in downstream zero-shot TTS. Prior work (LLM-Codec, Yang et al. 2024) had observed anecdotally that codec tokens can be inconsistent; this paper provides a systematic, quantitative treatment and a practical remedy.
Method
The paper first defines a consistency accuracy metric: given a full audio clip and a random slice extracted from it, consistency accuracy measures what fraction of the slice’s RVQ token positions are identical between encoding the slice in isolation and encoding the same frames within the full-length clip. A lower score means a larger DRI effect. The authors measure consistency accuracy across six popular codecs (EnCodec, HiFiCodec, SpeechTokenizer, DAC, FunCodec, and their proposed codec) at multiple codebook depths and slice lengths (0.2 s, 0.3 s, 0.4 s).
The analysis shows that: (1) all existing codecs suffer from low consistency — FunCodec 8 kbps has only 3.58% consistency across all 16 codebook layers; (2) consistency deglines substantially with deeper codebook layers in every codec examined, because shallow layers encode semantic (context-independent) information while deeper layers encode fragile fine-grained acoustic detail; and (3) the large receptive field of the RVQ encoder (up to 2718 samples at layer 18, per Table 5 in the paper) is the structural cause.
To mitigate DRI without shrinking the encoder’s receptive field (which would harm reconstruction quality), the authors add two auxiliary MSE losses during codec training:
Slice-consistency: A random 20% segment is carved out of each training audio clip and encoded in isolation to obtain Z_slice. The full-clip encoder output Z is constrained to match Z_slice at the corresponding time steps via MSE. This teaches the encoder to produce context-independent representations without changing its architecture.
Perturbation-consistency: The audio clip is slightly phase-perturbed (phase shift applied in the frequency domain, which is imperceptible to human hearing) to obtain Z_perception. MSE is applied between Z and Z_perception. This teaches the codec to ignore perceptually irrelevant signal variations.
In the final implementation, both constraints are combined into a single alignment objective that aligns Z_slice against Z_perception rather than against Z, and this combined loss is added to the standard RVQ-GAN training objective (reconstruction, adversarial, feature-matching, and commit losses). The resulting codec is a reproduced FunCodec architecture (~66M parameters) trained at 16 kHz, 4 or 8 kbps, 8 or 16 codebooks. The augmented codec is then used as the tokenizer for VALL-E, which predicts the first codebook layer autoregressively and remaining layers non-autoregressively. VALL-E training runs for 1.3M steps on LibriTTS (960 h) and on the larger MLS dataset (44,000 h).
Key Results
Codec consistency (Table 1 and 6, slice length 0.2 s): The proposed codec at 4 kbps achieves 71.03% all-layer consistency and 88.82% first-3-layer consistency, compared to EnCodec’s 47.43% / 61.49%, DAC’s 39.14% / 48.43%, SpeechTokenizer’s 14.70% / 26.91%, and FunCodec’s 6.86% / 16.39%. Per-layer detail (Table 6) shows the proposed codec’s layer-1 consistency is 96.13% vs. EnCodec’s 74.66%.
Codec quality (Table 1): ViSQOL at 4 kbps improves from 4.47 (FunCodec baseline) to 4.45 (comparable); PESQ improves from 3.26 to 3.25 (negligible change). At 8 kbps ViSQOL 4.64 is the best of all compared codecs, and PESQ 3.59 is within 0.03 of the best. Consistency gains therefore come essentially free in terms of reconstruction quality.
Downstream TTS (Table 2, LibriTTS test-clean): VALL-E trained with the proposed codec at 4 kbps and 960 h achieves WER 3.51%, SPK-SIM 60.97%, UTMOS 4.32, MOS 3.97, SMOS 3.73 — compared to the same VALL-E without consistency constraint at 8.51% WER, 55.90% SIM, UTMOS 4.08, MOS 3.73, SMOS 3.50. Scaling to 44,000 h further reduces WER to 3.13% and increases SIM to 61.72%, outperforming all baselines including VoiceCraft 830M (WER 4.72%, SIM 55.78%), XTTS v2 (WER 3.5%, SIM 60.06%), and USLM (WER 7.15%).
Cross-codec generalizability (Table 3): Applying consistency constraint to EnCodec (as input to VALL-E) reduces WER from 9.02% to 5.13% at 960 h. Applying it within UniAudio reduces WER from 5.90% to 2.39% and increases SIM from 54.20% to 59.09%, confirming the method is not architecture-specific.
Ablation (Table 4): Removing either the slice-consistency or perturbation-consistency component degrades both consistency accuracy and downstream WER. Slice percentage of 20% is optimal; larger slices (40%, 60%) carry more context and therefore do less to reduce DRI.
The reported correlation between consistency accuracy and WER (Figure 4) shows a clear monotonic relationship — improving consistency reliably reduces downstream WER.
Novelty Assessment
The primary contribution is conceptual and diagnostic: naming, formalizing, and quantifying the DRI phenomenon across a breadth of modern codecs. The proposed remedy is pragmatic engineering — two auxiliary MSE losses applied during codec fine-tuning — rather than a fundamentally new architecture. The slice-consistency idea is related to the PhaseAug data augmentation approach (Lee et al. ICASSP 2023), which the paper acknowledges, though the application to codec consistency training is novel.
The connection to the broader problem of paralinguistic variability in speech tokens (2412.17048) is not drawn explicitly in the paper, but DRI is in effect a training-time signal of the same underlying structural issue: many-to-one mappings from audio to discrete representations that inflate modeling difficulty for downstream LMs. This paper addresses one specific, measurable root cause — context dependence in RVQ encoding — and offers a fix that is applicable without architectural redesign.
The large-scale validation on 44,000 h (MLS) is a strength: most codec analysis papers are confined to LibriSpeech-scale data, and the consistent improvement at both scales supports practical applicability. The cross-codec generalizability experiments (Table 3) are unusually thorough for a paper of this type.
Field Significance
Moderate — This paper names and quantifies a systematic failure mode in RVQ-based codec tokenization that had previously been observed only anecdotally. The consistency accuracy metric it introduces provides a concrete diagnostic tool for comparing codecs beyond reconstruction quality alone. The mitigation — two MSE auxiliary losses applied at codec training time — requires no architectural redesign and demonstrates reliable downstream gains for autoregressive speech LMs across multiple codecs, architectures, and data scales.
Claims
- Context-dependent encoding in RVQ-based neural audio codecs causes perceptually equivalent audio segments to produce divergent discrete token sequences, increasing prediction uncertainty in downstream codec language models. (§2, §2.3)
- Codec token consistency and downstream autoregressive TTS quality are monotonically correlated: improvements in consistency accuracy reliably reduce word error rate and increase speaker similarity. (§5.2, Figure 4)
- Auxiliary consistency losses applied during codec training can substantially increase token consistency with negligible impact on reconstruction quality. (§5.1, Table 1)
- Consistency constraint methods applied to codec training generalize across neural codec architectures and autoregressive LM backbones, as demonstrated by cross-system experiments on both EnCodec-VALL-E and FunCodec-UniAudio pipelines. (§5.2, Table 3)
- In RVQ codecs, deeper codebook layers suffer disproportionately from context-induced inconsistency, because they encode fine-grained acoustic detail that is more sensitive to contextual perturbation than the semantic information stored in shallow layers. (§2.3, Appendix A.3, Table 6)
Limitations and Open Questions
The DRI fix is applied at codec training time; it does not address the underlying structural cause (large receptive fields in the convolutional encoder). A follow-up could explore whether directly reducing receptive field size with compensating distillation achieves similar or better consistency.
The method is validated on English speech (LibriTTS and MLS, which is multilingual but dominated by English). Whether DRI and its mitigation generalize to tonal languages (where fine-grained acoustic context is phonemically contrastive) is unstudied.
Evaluation is restricted to the VALL-E architecture for downstream TTS. The generalizability claim to “any neural codec language model” rests on two additional cross-codec experiments (Table 3), which is suggestive but not comprehensive. In particular, the paper does not evaluate on flow-matching TTS (e.g. CosyVoice or E2-TTS), where consistency constraints on the codec may have different downstream effects.
The MOS and SMOS evaluations use 50 audio samples — a small panel by modern standards. Confidence intervals are not reported for subjective metrics.
DRI may interact with audio compression artifacts differently at different bitrates; the paper evaluates 4 kbps and 8 kbps but does not explore very low bitrates where context-dependence may be even stronger.
Wiki Connections
This paper provides strong empirical grounding for neural-codec: it introduces a new measurable property (consistency accuracy) that characterizes DRI across EnCodec, HiFiCodec, SpeechTokenizer, DAC, and FunCodec, and demonstrates that improving this property directly benefits downstream autoregressive-codec-tts (VALL-E). The paper is directly complementary to 2412.17048, which identifies paralinguistic variability in speech tokens as the dominant bottleneck in speech LMs from an information-theoretic angle, while this paper identifies RVQ context-dependence as a practical, trainable root cause of a closely related phenomenon.
For zero-shot-tts, the results show that codec consistency is a meaningful lever for improving WER and SPK-SIM in zero-shot synthesis, independent of model scale — the consistency-improved codec at 960 h matches or exceeds baseline systems trained on much larger corpora.
The use of UTMOS, PESQ, ViSQOL, WER, and SPK-SIM as evaluation tools in a codec-to-TTS pipeline is a useful template for evaluation-metrics. The consistency accuracy metric itself is a novel contribution that deserves to be considered a standard diagnostic tool alongside existing codec benchmarks. A complementary direction — replacing RVQ entirely with continuous tokens — is explored by Cont-SPT, which achieves better high-frequency information retention by removing discretization rather than improving consistency within it.