arXiv · 2025 · Preprint
Ahasan et al. · → Paper · Demo: ? · Code: ✓
FuseCodec enriches neural codec training by injecting both self-supervised semantic features (HuBERT) and pre-trained language model contextual embeddings (BERT) into the codec latent space via three complementary strategies, achieving the strongest reconstruction quality and downstream task performance on LibriSpeech and CodecSUPERB among compared codecs at 4 kbps.
Problem
Standard neural codecs (EnCodec, SoundStream, DAC) are trained purely on acoustic reconstruction objectives, producing discrete tokens that capture waveform-level fidelity but lack high-level semantic content and contextual grounding. Prior work partially addressed this by aligning the first RVQ layer with HuBERT-derived semantic features via cosine similarity (SpeechTokenizer), or by adding BERT-based contextual matching (DM-Codec). However, these approaches use only one modality at a time, rely on similarity-based supervision without direct latent injection, and lack robust cross-modal alignment between text token sequences and acoustic frame sequences. No prior codec jointly unifies acoustic, semantic, and contextual information with direct integration and cross-modal alignment.
Method
FuseCodec builds on the standard encoder-RVQ-decoder codec architecture (8 quantization codebooks, 50 Hz frame rate, 1024-dim embeddings, 4 kbps) and adds three training-time strategies to enrich the discrete representations. All pre-trained guidance models are frozen throughout training.
Two pre-trained models provide multimodal guidance: HuBERT (base-ls960, 768-dim) provides frame-level semantic embeddings, and BERT (bert-base-uncased) provides contextual token embeddings derived from the speech transcript produced by a frozen wav2vec 2.0 ASR model. Three codec variants, each using a different integration strategy, are trained and evaluated independently.
FuseCodec-Fusion (Latent Representation Fusion) computes global representations from each modality (mean-pooled HuBERT frames; BERT [CLS] token), applies 8-head multi-head cross-attention between the two global vectors, projects both into the encoder’s latent dimension, and adds them to the encoder output Z via additive fusion with 10% modality dropout before quantization. This fuses information directly into the representation that the RVQ codebooks learn to quantize.
FuseCodec-Distill (Global Semantic-Contextual Supervision) broadcasts the same global modality vectors across all time steps and supervises the first RVQ layer’s output Q(1) using a joint cosine similarity loss averaged over time. This is a distillation variant that does not modify the latent but enforces global-to-local temporal coherence via a modified training objective.
FuseCodec-ContextAlign (Temporally Aligned Contextual Supervision) addresses the sequence length mismatch between contextual text token embeddings (n tokens) and acoustic frame outputs (T’ frames) using a dynamic window-based alignment algorithm (Algorithm 1). For each contextual embedding, the algorithm defines a sliding window of RVQ tokens, computes cosine similarities within the window, and assigns the contextual embedding to the highest-similarity token(s). The resulting aligned sequence C* supervises Q(1) with token-level precision.
The training objective combines standard codec losses (time-domain reconstruction, frequency-domain reconstruction, adversarial, feature matching, RVQ commitment) with the variant-specific distillation loss, weighted by fixed coefficients.
FuseCodec-TTS extends the codec to zero-shot speech synthesis following the neural codec language model paradigm. An autoregressive decoder-only 12-layer Transformer predicts first-layer tokens conditioned on phoneme input and an acoustic prompt; higher RVQ layers are predicted non-autoregressively. Both AR and NAR models use 16 attention heads, 1024-dim embeddings, and 4096-dim feed-forward layers. FuseCodec-TTS is trained on LibriTTS (570 hours).
Key Results
Speech reconstruction (Table 2, LibriSpeech test-clean): FuseCodec-Fusion achieves the best overall reconstruction across compared codecs, with WER 3.99 (vs. 4.04 for EnCodec, 4.16 for SpeechTokenizer), ViSQOL 3.47 (vs. 3.08 for SpeechTokenizer), and PESQ 3.13 (vs. 2.31 for EnCodec). FuseCodec-Distill ties for the best UTMOS (3.65) and Similarity (0.996), matching DM-Codec on UTMOS while improving WER. FuseCodec-ContextAlign delivers competitive results (UTMOS 3.65) with slightly lower content preservation than the Fusion variant.
Downstream task quality (Table 3, CodecSUPERB at 4 kbps): FuseCodec-Fusion achieves the best audio signal quality score (Audio 0.785, Speech 0.744) and the highest emotion recognition accuracy (73.96%). FuseCodec-Distill leads on audio event classification (57.25). These results hold at 4 kbps, substantially below DAC (6 kbps) and FunCodec (8 kbps) competitors.
Zero-shot TTS (Table 4, LibriSpeech and VCTK): FuseCodec-Distill-TTS attains the best WER on both benchmarks (8.55 / 3.66 on LibriSpeech / VCTK), outperforming DM-Codec-TTS (10.26 / 5.02) and USLM (16.72 / 14.79). FuseCodec-ContextAlign-TTS leads on perceptual naturalness (UTMOS 3.86 / 3.96). All FuseCodec-TTS variants outperform USLM on all metrics.
Multilingual generalization (Table 5): FuseCodec-Fusion, trained only on English (LibriSpeech), achieves the best WER and ViSQOL in most of 7 unseen languages in Multilingual LibriSpeech, with PESQ improvements of 0.3 or more over the next best model in multiple languages.
Ablations (Appendix D): Cross-modal attention applied before projection into the encoder dimension (Cross-Before) yields the best configuration for FuseCodec-Fusion. Combining both semantic and contextual distillation signals outperforms using either alone. 10% modality dropout provides the best robustness-informativeness balance.
Novelty Assessment
The contribution synthesizes and extends two existing directions: semantic distillation into RVQ layers (SpeechTokenizer) and BERT-based contextual matching (DM-Codec). The three proposed strategies are individually refinements of known techniques (cosine distillation, cross-modal attention, dynamic alignment) rather than entirely new paradigms. The specific window-based alignment algorithm for handling text-to-acoustic sequence length mismatch is a concrete novel mechanism. The key contribution is in systematically combining all three types of grounding (acoustic, semantic, contextual) in a single codec framework and demonstrating that each strategy offers a different quality-interpretability trade-off. The paper does not introduce a new architecture in the traditional sense; the encoder-RVQ-decoder backbone follows EnCodec directly. The novelty is in the training objective framework and integration strategy. Evaluation is thorough, using multiple established benchmarks (LibriSpeech, CodecSUPERB, VCTK, Multilingual LibriSpeech) with fair comparisons against official checkpoints.
Field Significance
Moderate — FuseCodec advances the line of work on semantically enriched codec representations by adding contextual language model signals and proposing direct latent integration rather than similarity-only supervision. The result is a clearer picture of what each type of guidance (latent fusion, global distillation, temporally aligned supervision) contributes to codec token quality. The work is incremental relative to SpeechTokenizer and DM-Codec but offers more complete coverage of the guidance modality space and competitive results at a lower bitrate than several strong baselines.
Claims
-
supports: Injecting multimodal guidance directly into the encoder latent space of a neural codec improves reconstruction quality beyond similarity-based supervision of the quantized layer.
Evidence: FuseCodec-Fusion, which fuses global semantic and contextual vectors into Z via additive fusion, achieves WER 3.99, ViSQOL 3.47, and PESQ 3.13 on LibriSpeech test-clean, outperforming FuseCodec-Distill (ViSQOL 3.43, PESQ 3.06) and FuseCodec-ContextAlign, both of which only supervise Q(1) without modifying the latent. (§2.3.1, §3.2.1, Table 2)
-
supports: Neural codecs trained with joint semantic and contextual supervision generalize to unseen languages without multilingual training data.
Evidence: FuseCodec, trained exclusively on English LibriSpeech train-clean-100, achieves the best WER and ViSQOL in the majority of 7 tested languages from Multilingual LibriSpeech and outperforms all baselines on PESQ by at least 0.3 in most languages. (§3.3, Table 5)
-
supports: Codec representations enriched with semantic and contextual signals support stronger downstream task generalization (emotion recognition, audio event classification) than acoustic-only codecs.
Evidence: On CodecSUPERB at 4 kbps, FuseCodec-Fusion achieves emotion recognition accuracy of 73.96% and audio signal quality 0.785, versus 66.18% and 0.697 for EnCodec at 6 kbps. All FuseCodec variants exceed SpeechTokenizer, EnCodec, and DAC on audio signal quality. (§3.2.2, Table 3)
-
complicates: Fine-grained temporal alignment between text tokens and acoustic frames improves local interpretability but is constrained relative to global supervision strategies.
Evidence: FuseCodec-ContextAlign, which aligns contextual embeddings to RVQ tokens via a windowed similarity matching algorithm, achieves WER 4.15 and ViSQOL 3.18, lagging FuseCodec-Fusion (WER 3.99, ViSQOL 3.47) and FuseCodec-Distill (ViSQOL 3.43). The paper attributes this to constrained local alignment limiting global contextual guidance. (§2.3.3, §3.2.1, Table 2)
-
supports: Distilling both semantic (self-supervised speech model) and contextual (language model) signals into codec token supervision outperforms semantic-only distillation for perceptual naturalness.
Evidence: FuseCodec-Distill achieves UTMOS 3.65 and Similarity 0.996 on LibriSpeech test-clean, while codecs using only semantic distillation (SpeechTokenizer, Mimi, X-Codec2) score below 3.55 UTMOS and fail to consistently match speaker similarity. (§3.2.1, Table 2)
Limitations and Open Questions
The codec is trained on LibriSpeech train-clean-100 (100 hours, English read speech), which limits conclusions about robustness to spontaneous speech, diverse accents, or noisy conditions. Multilingual generalization results are promising but the training data and BERT model are English-only, leaving the mechanism behind cross-lingual transfer unclear. The ASR transcription step (wav2vec 2.0) introduces an error-prone intermediate representation during training; the effect of ASR errors on contextual embedding quality is not quantified. Model size is not reported, making it difficult to assess computational cost relative to baselines. The TTS evaluation compares only against other codec-based systems trained on LibriTTS and does not benchmark against the strongest current TTS models.
Wiki Connections
- Neural Audio Codec — FuseCodec proposes three training strategies to enrich RVQ-based codec representations with semantic and contextual signals, extending the standard encoder-RVQ-decoder paradigm.
- Self-Supervised Speech — HuBERT (base-ls960) and wav2vec 2.0 are core frozen components of FuseCodec’s training pipeline, providing semantic embeddings that supervise or enrich codec latents.
- Autoregressive Codec TTS — FuseCodec-TTS demonstrates the codec’s applicability to downstream zero-shot TTS using an AR + NAR transformer LM that predicts RVQ tokens.
- Zero-Shot TTS — FuseCodec-TTS is evaluated in zero-shot settings on LibriSpeech and VCTK, using a 3-second acoustic prompt to clone speaker identity.
- Evaluation Metrics — The paper evaluates across a broad set of reconstruction and downstream metrics (WER, WIL, STOI, ViSQOL, PESQ, UTMOS, Similarity) and the CodecSUPERB benchmark, providing a comprehensive multi-dimensional codec comparison.