ACL 2026 (Findings) · 2026 · Conference
Jiacheng Shi et al. (College of William & Mary / Emory University / George Mason University) · → Paper · → Preprint · Demo: ✓ · Code: ✗
Adds emotion-aware training signals to a SpeechTokenizer-style RVQ codec (emotion and semantic cross-attention on the encoder latents, relation-preserving distillation onto the first RVQ layer, and emotion-weighted text alignment), raising emotion similarity of reconstructed speech on EmoVoiceDB from 0.88 (best baseline, FACodec) to 0.94 at 4 kbps while its LibriSpeech WER (4.15) stays slightly behind EnCodec’s 4.02.
Problem
Codec benchmarks report that emotion recognition degrades when speech passes through neural codecs, even when overall reconstruction quality is high. The authors argue that existing codecs preserve emotion only as a byproduct of acoustic reconstruction (EnCodec, SoundStream) or semantic distillation (X-Codec, Llasa’s X-Codec 2), with no objective that protects affective cues during quantization. Analysis work such as EMO-Codec measures this loss but does not change how the codec is trained.
Method
The backbone follows SpeechTokenizer (SpeechTokenizer): a convolutional encoder with strides (2, 4, 5, 8) followed by a two-layer BiLSTM, an 8-layer RVQ with 1,024-entry codebooks at 50 Hz (4 kbps), and a mirrored decoder, at 16 kHz with 44M parameters. Training uses the standard codec losses (commitment, multi-resolution mel, hinge adversarial loss and feature matching) with multi-period, multi-scale and multi-scale STFT discriminators, unchanged from prior codec recipes. Three frozen encoders provide guidance: CLAP-LAION as the emotion encoder, HuBERT base as the self-supervised semantic audio encoder, and wav2vec 2.0 ASR followed by BERT as the text-semantic encoder. Layer outputs are averaged in each case.
Three additions are the contribution:
- Emotion-semantic guided latent (EG-Latent). The acoustic latents act as queries in two cross-attention blocks, one over projected emotion embeddings and one over projected HuBERT embeddings. The two outputs pass through a shared projection, receive independent stochastic dropout, and are added to the latent before RVQ.
- Relation-preserving distillation (RP-Distill). Pairwise Euclidean distances between frames of the first RVQ layer are matched (L1) to the pairwise distances in the emotion and HuBERT spaces. This adapts a cross-modal relational distillation method from visual place recognition. Only the first RVQ layer is supervised.
- Emotion-weighted semantic alignment (EW-Align). Each frame of the first RVQ layer is softly aligned to a local window of BERT token embeddings (centre index from a monotonic frame-to-token mapping, softmax over cosine similarities), with no dynamic programming. A cosine-sigmoid loss pulls the frame toward its weighted text target. Frames where the emotion embedding changes most between adjacent frames get larger weights (softmax over the L1 difference, rescaled to unit mean).
The total loss is a weighted sum of the four reconstruction losses plus the relational and alignment terms. Training uses about 2.3k hours (LibriSpeech, VCTK, AISHELL-3, a 1,000-hour AudioSet subset, MSP-Podcast and CMU-MOSEI) for 200 epochs on four A100 GPUs.

For downstream evaluation, a VALL-E-style (VALL-E) TTS is trained on the codec tokens using LibriTTS. A decoder-only autoregressive Transformer predicts the first RVQ stream from phonemes and an acoustic prompt. A non-autoregressive Transformer then predicts streams 2 to 8. Both are 12-layer, 16-head, 1024-dimensional models.
Key Results
Reconstruction (Table 2). Emotion metrics are computed on EmoVoiceDB and content and quality metrics on LibriSpeech test-clean, averaged over three seeds. Emo SIM is emotion2vec embedding similarity and Recall comes from an emotion2vec-based SER model. The proposed codec has the highest value of the ten codecs on Emo SIM (0.94 vs. FACodec 0.88), Pros SIM (0.86 vs. WavTokenizer 0.81), SER recall (0.48 vs. X-Codec 2, listed as Llasa, 0.40), WIL (6.43 vs. DAC 6.52), LSD (0.78 vs. BigCodec 0.84), PESQ (3.04 vs. FACodec 2.85) and UTMOS (3.68 vs. TAAE 3.57). It does not lead on WER: EnCodec (4.02), DAC (4.10) and FACodec (4.14) are all slightly lower than its 4.15. Pitch error (MSEP) is 19.21, second to FACodec’s 12.16.
These comparisons are not bitrate-matched. The proposed codec runs at 4 kbps with 8 codebooks. BigCodec (1 kbps), Mimi (1.1 kbps), TAAE (0.6 kbps), WavTokenizer (0.9 kbps) and X-Codec 2 (0.8 kbps) use roughly four to seven times fewer bits, and EnCodec and DAC are configured at 6 kbps (Table 1). SpeechTokenizer, the backbone, also runs at 4 kbps with 8 codebooks. Against it, the proposed codec improves Emo SIM from 0.82 to 0.94 and UTMOS from 3.43 to 3.68, with WER 4.15 vs. 4.19.
Emotion recognition after resynthesis (Table 3, EMO-SUPERB macro-F1). The codec scores highest among all codecs on five of six sets: IEMOCAP 0.338, CREMA-D 0.629, MSP-IMPROV 0.513, MSP-Podcast 0.319 and BIIC-Podcast 0.256. On these five it is also above the original audio (0.313, 0.594, 0.491, 0.301, 0.247), although the text only remarks on IMPROV. On NNIME it scores 0.182, below DAC at 6 kbps (0.184) and the original audio (0.183). The baselines in this table are mostly older codecs (AudioDec, AcademiCodec, EnCodec, FunCodec, SoundStream) plus MP3, Opus and AAC. The low-bitrate tokenizers from Table 2 are not included.
Downstream zero-shot TTS (Table 4). On LibriSpeech, the TTS built on the codec reaches the highest SIM-O (0.80 vs. SparkTTS 0.78) and UTMOS (4.29 vs. CosyVoice 2 4.23). CosyVoice 2 has lower WER (2.45 vs. 2.51) and WIL (6.72 vs. 7.29), and Llasa also has lower WER (2.49). On EmoVoice-DB it leads on Emo_SIM (0.91 vs. 0.87) and Recall (0.41 vs. 0.37). WER (3.48) trails MaskGCT and SparkTTS (3.28), and UTMOS (4.35) trails CosyVoice 2 (4.42), although the text claims the highest UTMOS on both emotional sets. On SECAP it leads on Emo_SIM (0.84), Recall (0.49) and UTMOS (3.20), and is second on WER (8.64 vs. CosyVoice 2 8.55). The baselines are complete published systems with their own tokenizers, training data and model sizes, so Table 4 compares whole systems rather than codecs under one TTS model. The Table 4 caption calls SIM-O speaker similarity, while §4.2.3 calls it prosody similarity.
Subjective tests (Appendices D and E, 24 listeners). In reconstruction (Figure 2), MUSHRA is 90.26 vs. Llasa 87.52, EnCodec 78.96 and ground truth 91.37. MOS and Emotion-MOS are 4.02 and 4.21 vs. Llasa 3.69 and 3.50. In AB tests, listeners prefer the codec over EnCodec in 78.9% of quality judgments and 87.6% of emotion judgments. For TTS (Figure 3), MOS is 3.79 vs. CosyVoice 2 3.41 and F5-TTS 2.85, and Emotion-MOS is 4.16 vs. 3.53 and 2.98. AB preference is 74.7% against CosyVoice 2 and 85.5% against F5-TTS. Only two or three baseline systems are rated in each test.
Ablation (Table 5). Without any of the three components, the codec scores reconstruction Emo SIM 0.87, WER 4.85, UTMOS 3.34 and MUSHRA 86.27. Each component alone helps: EG-Latent gives Emo SIM 0.90, RP-Distill gives WER 4.47 and EW-Align gives UTMOS 3.49. All three together give 0.94, 4.15, 3.68 and 90.71, and TTS Emo SIM rises from 0.86 to 0.91. Appendix F adds finer ablations (Tables 6 to 10). Applying cross-attention before projection beats self-attention and direct injection. Relational distillation beats direct feature distillation (recall 0.48 vs. 0.42). Frame-wise emotion weighting beats a uniform global weight (recall 0.48 vs. 0.42). Supervising only the first RVQ layer beats averaging the first 4 or all 8 layers. CLAP-LAION is the most balanced emotion encoder, although CLEP-DG gives slightly higher Emo SIM (0.95).
Novelty Assessment
The codec backbone, quantizer, discriminators and losses are inherited from SpeechTokenizer and EnCodec. The new parts are the three training-time mechanisms, and the most specific of them is EW-Align, which weights frame-to-text alignment by local emotion change. Relational distillation is borrowed from a vision method. Distilling SSL features into the first RVQ layer comes from SpeechTokenizer. Here it is extended to emotion features and to pairwise geometry instead of per-frame features. The ablations isolate each component and several design choices, which supports the contribution.
Some of the evidence is weaker than the headline numbers suggest. The main codec comparison mixes bitrates from 0.6 to 6 kbps. The TTS comparison pits a LibriTTS-trained VALL-E-style model against large systems with different tokenizers and data. The automatic emotion scores come from emotion2vec-based models (Tables 2, 4 and 5) and EMO-SUPERB classifiers (Table 3), and the subjective tests use small panels comparing two or three systems. The text also misreports some table values. §4.3 says RP-Distill reduces WER from 5.75 to 5.18, but Table 5 lists 4.85 to 4.47. The full-model reconstruction MUSHRA is 90.71 in Table 5 and 90.26 in Appendix D, with no explanation of the difference.
Field Significance
Moderate — This paper makes emotion preservation an explicit training objective for an RVQ speech codec, and its ablations show that each objective term helps. It demonstrates that frozen emotion and SSL teachers can be distilled into the first codebook while keeping LibriSpeech WER within 0.13 of the best baseline at 4 kbps. The open questions are whether the gains hold at matched bitrates and whether codec-LM TTS benefits under a controlled downstream comparison.
Claims
- supports: Explicit emotion-aware supervision during codec training improves preservation of emotional cues in reconstructed speech compared with codecs optimized only for acoustic or semantic fidelity.
Evidence: The emotion-guided codec reaches Emo SIM 0.94, Pros SIM 0.86 and SER recall 0.48 on EmoVoiceDB, against 0.88, 0.81 and 0.40 for the best of nine baseline codecs. Removing all three emotion components drops reconstruction Emo SIM to 0.87. (§4.2.1, Table 2; §4.3, Table 5)
- refines: Supervising the first RVQ layer with external teacher structure is more effective than spreading the same supervision across deeper residual layers.
Evidence: Applying relational distillation and alignment to the averaged first 4 or all 8 RVQ layers, instead of only the first, lowers Emo SIM (0.93, 0.91 vs. 0.94) and recall (0.45, 0.41 vs. 0.48), and raises WER (4.22, 4.38 vs. 4.15). (§Appendix F.4, Table 9)
- refines: Matching pairwise relational geometry between codec tokens and teacher representations preserves emotion better than matching teacher features directly.
Evidence: Relation-preserving distillation gives recall 0.48 and PESQ 3.04, against 0.42 and 2.80 for direct feature distillation. Using the emotion teacher alone raises recall to 0.45 but leaves WER at 4.60 vs. 4.15 for both teachers combined. (§Appendix F.2, Table 7)
- complicates: Emotion-preservation gains in codecs are often reported without bitrate-matched comparisons, which makes the reconstruction trade-off hard to read.
Evidence: The proposed codec runs at 4 kbps with 8 codebooks, while the five single-codebook baselines run at 0.6 to 1.1 kbps. Its LibriSpeech WER (4.15) is still above EnCodec (4.02) and DAC (4.10) at 6 kbps, and its pitch error (MSEP 19.21) is above FACodec (12.16). (§3.4, Table 1; §4.2.1, Table 2)
- supports: Emotion-preserving codec tokens transfer to more emotionally faithful codec-language-model TTS output.
Evidence: A VALL-E-style AR+NAR model trained on the codec tokens gets the highest Emo_SIM and Recall on EmoVoice-DB (0.91, 0.41) and SECAP (0.84, 0.49) among eight systems. In the codec-component ablation, its TTS Emo SIM rises from 0.86 to 0.91. Listeners give it Emotion-MOS 4.16 vs. 3.53 for CosyVoice 2, but baselines differ in tokenizer, data and scale. (§4.2.3, Table 4; §4.3, Table 5; §Appendix E, Figure 3)
Limitations and Open Questions
The main limitation is that the comparisons are not controlled. Codec baselines run at 0.6 to 6 kbps against the proposed 4 kbps. The TTS baselines are complete systems with different tokenizers, training data and sizes. No experiment trains the same TTS model on SpeechTokenizer tokens and on AffectCodec tokens. Because of this, the downstream gains cannot be attributed to the codec alone.
- Learned emotion evaluators. Emo SIM and Recall come from emotion2vec embeddings and an emotion2vec-based SER model, and Table 3 uses EMO-SUPERB classifiers. The training teacher is CLAP-LAION, which is a different model. Even so, every automatic emotion score relies on learned emotion models, and the subjective tests compare only two or three systems.
- Reporting inconsistencies. §4.3 gives RP-Distill WER values (5.75 to 5.18) that do not appear in Table 5 (4.85 to 4.47). §4.2.3 claims the highest UTMOS on EmoVoice-DB, where CosyVoice 2 scores 4.42 vs. 4.35. Full-model MUSHRA is 90.71 in Table 5 and 90.26 in Appendix D.
- Teacher costs. Training needs four frozen encoders (CLAP-LAION, HuBERT, wav2vec 2.0 ASR and BERT), and the text branch depends on ASR transcripts. The authors’ Limitations section mentions only computational efficiency.
- Scope. Speech is 16 kHz only. The paper does not evaluate streaming, lower bitrates, or more than 8 codebooks. NNIME results fall slightly below the original audio and DAC, which the authors attribute to subtle emotional expression.
- Reproducibility. A demo page is released, but the paper does not mention a code release.
Wiki Connections
- Neural Audio Codec — Proposes an RVQ speech codec whose training objective explicitly targets emotion preservation alongside reconstruction.
- Emotion Synthesis — Evaluates emotion preserved in reconstructed and codec-LM-generated speech with emotion2vec similarity, SER recall and listener Emotion-MOS.
- Self-Supervised Speech — Uses frozen HuBERT as the semantic audio teacher for latent modulation and relational distillation into the first RVQ layer.
- Autoregressive Codec TTS — Trains a VALL-E-style AR+NAR TTS on the codec’s tokens to test whether emotion preservation carries into generation.
- Subjective Evaluation — Reports 24-listener MUSHRA, MOS, Emotion-MOS and AB preference tests for both reconstruction and TTS.
- SpeechTokenizer — Supplies the encoder, RVQ and decoder settings and the first-layer distillation idea that this codec extends, and serves as a baseline codec.
- VALL-E — Provides the AR plus NAR codec-language-model design used for the downstream TTS evaluation.
- Llasa — Its X-Codec 2 tokenizer is the strongest emotion baseline in reconstruction, and the Llasa TTS is a Table 4 baseline and a subjective-test comparison.
- NaturalSpeech 3 — Its FACodec is the closest competitor on reconstruction Emo SIM and has the lowest pitch error.
- EnCodec — Its codec losses and discriminators are reused; it is a baseline with the lowest reconstruction WER and the reference system in the subjective tests.
- CosyVoice 2 — Strongest TTS baseline, with lower WER on LibriSpeech and higher UTMOS on EmoVoice-DB, and a comparison system in the TTS listening tests.
- EmoVoice — Source of the EmoVoiceDB emotional speech set used for reconstruction and TTS emotion evaluation.
- emotion2vec — Provides the emotion embeddings and SER model behind the objective emotion similarity and recall metrics.