arXiv · 2026 · Preprint

Meng et al. (Beijing University of Posts and Telecommunications) · → Paper · Demo: ? · Code: ✗

AffectCodec is a 16 kHz speech codec whose residual FSQ quantizer uses block-diagonal projections to reserve a fixed share of every token for emotion features. The goal is to keep emotional cues intact at low bitrates while still emitting one flat token per stage for downstream speech LMs.

Problem

Neural speech codecs now serve as tokenizers for speech LMs, so any information the codec discards is out of reach for downstream models. Codec objectives (mel, STFT, adversarial losses) target broadband acoustic fidelity and only weakly constrain the pitch trajectories and energy dynamics that carry emotion. The authors show that encode-quantize-decode lowers IEMOCAP speech emotion recognition Macro-F1 from about 64% to around 53% at low bitrates, with a gap that persists at 6.0 kbps (Figure 1). They attribute this to two mechanisms. The first is reconstruction-driven bit allocation: under capacity pressure, affective cues are sacrificed. The second is cross-stream leakage: if a pretrained emotion representation is concatenated with the acoustic latent before a quantizer with fully connected projections, reconstruction gradients repurpose the dimensions meant for emotion. Attribute-factorized codecs such as FACodec avoid mixing but emit heterogeneous per-attribute tokens that need specialized downstream generators.

Method

AffectCodec has a dual-path front end. The acoustic branch is a DAC-style convolutional encoder (strides [2, 4, 5, 8], 320x downsampling, 50 Hz frames at 16 kHz). The emotion branch is a frozen emotion2vec-large encoder followed by a trainable CNN adapter. Emotion is injected at two granularities. At the coarse level, a Coarse-granularity Emotion Modulation (CEM) module attention-pools the emotion features into a global embedding and applies FiLM modulation to the acoustic features. At the fine level, frame-level emotion features enter a dedicated partition of the quantizer.

The quantizer, Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ), is built in two steps. Factorized RFSQ wraps each residual FSQ stage with learnable input and output projections between the high-dimensional latent (d = 1024) and a compact FSQ space (f = 9), so residuals stay in the expressive latent space while rounding happens where FSQ works well. BD-RFSQ then constrains both projections to be block-diagonal over the emotion partition (256 latent dims to 3 FSQ dims, 2 levels each) and the acoustic partition (768 latent dims to 6 FSQ dims, 4 levels each). A learnable per-dimension affine normalization, with softplus-parameterized scale, counters residual magnitude decay in later stages. Because it uses only fixed parameters, index-only decoding needs no side information. Appendix B proves by induction that emotion residuals depend only on emotion inputs at every stage. Each stage still emits a single composite 15-bit index (3 emotion bits plus 12 acoustic bits), so with K = 8 stages at 50 Hz the codec spans 1.5 kbps (2 stages) to 6.0 kbps (8 stages), with a fixed 20% emotion share.

Figure 2: Left: Internal structure of a single BD-RFSQ stage. The residual r k -1 is projected to the compact FSQ space via a block-diagonal input matrix (red: emotion partition; blue: acoustic partition), affine-normalized, scalar-quantized by FSQ, affine-de-normalized, and mapped back to the latent space via a block-diagonal output matrix, yielding the stage reconstruction u k. Center: BD-RFSQ chains K such stages with residual connections; each stage quantizes the current residual and subtracts its reconstruction before passing the remainder to the next stage. Right: AffectCodec architecture. A frozen emotion teacher and a DAC-style acoustic encoder form a dual-path front-end; the CEM module fuses coarse-grained emotion cues into the acoustic pathway, and the concatenated representation is discretized by BD-RFSQ. Separate acoustic and emotion decoders reconstruct the waveform and supervise the emotion partition, respectively.

Training combines the DAC reconstruction recipe (multi-scale mel, time-domain L1, multi-scale STFT adversarial and feature-matching losses) with a commitment loss and three emotion terms. An emotion decoder must reconstruct emotion2vec features from the quantized emotion partition. An emotion cycle-consistency loss penalizes the cosine distance between mean-pooled emotion2vec embeddings of the output and the input waveform. A multi-rate loss supervises the 2-stage and 4-stage cumulative reconstructions with mel and cycle terms. Biased stage dropout samples truncated stage counts for 75% of training examples, with probabilities 0.5, 0.3 and 0.2 for 2, 4 and 8 stages. The model trains on LibriSpeech (960 h) plus the IEMOCAP training split for 250K steps on 4 RTX 4090 GPUs (about 72 hours). It has 135.05M trainable parameters, most of them in the DAC encoder/decoder and the emotion CNN adapter (55.6M); the quantizer itself adds 0.1M. The frozen emotion2vec (164.05M) is needed for encoding but not for decoding from tokens.

Key Results

Evaluation covers IEMOCAP, CREMA-D (91 speakers, none seen in training) and ESD at 1.5, 3.0 and 6.0 kbps. The baselines are official checkpoints of EnCodec, DAC, SpeechTokenizer and X-Codec, truncated to matching bitrates; SpeechTokenizer and X-Codec stop at 4.0 kbps and have no 6.0 kbps entries. The primary metric is Emotion Degradation Rate (EDR), the relative drop in SER F1 after codec reconstruction. It is averaged over three SER probes on frozen HuBERT-Large, WavLM-Large and wav2vec 2.0-Large features, so that emotion2vec, the training teacher, is not also the judge. The paper also reports V/A/D MSE from an MSP-Podcast-finetuned wav2vec 2.0 regressor, ViSQOL, STOI, and Whisper-Large-v3 WER. There is no human listening test.

Emotion preservation (Table 1). AffectCodec has the lowest Macro EDR in 7 of 9 dataset-bitrate cells. At 1.5 kbps it scores 5.27% on IEMOCAP (X-Codec 9.09%, DAC 17.05%), 12.67% on CREMA-D (X-Codec 26.72%) and 20.04% on ESD (X-Codec 21.14%). At 3.0 kbps it leads on IEMOCAP (1.77% vs. X-Codec 3.08%) but trails X-Codec on CREMA-D (9.25% vs. 8.51%) and SpeechTokenizer on ESD (9.66% vs. 8.75%). Weighted EDR follows the same pattern, including the same two 3.0 kbps losses. At 6.0 kbps it leads all three datasets against EnCodec and DAC (IEMOCAP 0.85% vs. DAC 1.95%; CREMA-D 1.19% vs. 3.19%; ESD 2.20% vs. 6.84%). V/A/D MSE is lowest in 8 of 9 cells; the exception is CREMA-D at 3.0 kbps (2.64 vs. X-Codec 2.57, x10^-3). The ESD margin at 1.5 kbps is small (20.04% vs. 21.14%).

Acoustic quality and intelligibility (Table 2). At 1.5 kbps AffectCodec has the best ViSQOL and STOI on IEMOCAP (3.31 / 0.730) and CREMA-D (3.14 / 0.707) and the best STOI on ESD (0.872). X-Codec has the better ESD ViSQOL (3.92 vs. 3.82) and a much lower WER on all three sets (IEMOCAP 9.54 vs. 15.39; CREMA-D 5.09 vs. 12.62; ESD 3.46 vs. 6.04). At 3.0 kbps AffectCodec has the best ViSQOL and STOI everywhere (e.g., IEMOCAP 3.82 / 0.833 vs. DAC 3.46 / 0.775). Its WER is best on IEMOCAP (6.99 vs. 7.03) and second to X-Codec on CREMA-D (3.44 vs. 3.29) and ESD (2.77 vs. 2.64). At 6.0 kbps DAC beats AffectCodec on every acoustic metric (e.g., IEMOCAP ViSQOL 4.37 vs. 4.18, WER 4.42 vs. 4.63; CREMA-D WER 1.13 vs. 1.83).

Ablations (Table 3, IEMOCAP, 1.5 kbps). With the full model at 5.27% Macro EDR, swapping BD-RFSQ for standard RVQ raises it to 14.44%. Factorized RFSQ without the block-diagonal constraint gives 10.23%. Removing multi-rate training gives 8.37% (V/A/D MSE 5.79 vs. 2.48), and removing CEM gives 6.94%. The full model also has the best ViSQOL (3.31) and STOI (0.730) of all variants. A linear probe in Appendix C (Table 4) regresses log-mel frames from the 24 binary emotion-partition codes. It gets R² = 0.0985 for a fully connected-projection variant versus 0.0196 for BD-RFSQ, which supports the leakage explanation.

Novelty Assessment

The main contribution is the block-diagonal constraint on factorized residual FSQ projections. It is a small structural change (0.1M parameters), but it turns a soft, loss-balanced emotion reservation into a hard per-stage bit allocation while keeping a single token stream per stage, which separates it from attribute-specific quantizers such as FACodec and from semantic-distillation codecs such as SpeechTokenizer. The ablation that isolates the constraint (Factorized RFSQ vs. BD-RFSQ) and the leakage probe make the mechanism credible. The surrounding components are adaptations of known techniques: the DAC backbone, FiLM conditioning on a frozen emotion2vec teacher, quantizer dropout with a reweighted distribution, multi-rate supervision, and affine normalization in the spirit of Robust RFSQ. The emotion teacher concatenation also echoes X-Codec’s semantic-feature concatenation, applied to affect instead of phonetics.

Field Significance

Moderate. The paper shows that reserving quantizer capacity for a designated speech attribute through block-diagonal projections preserves that attribute at low bitrates better than reconstruction-only RVQ codecs, with little acoustic cost at 1.5 to 3.0 kbps. It also gives a concrete mechanistic account (cross-stream gradient leakage) of why simply concatenating attribute features before a dense quantizer is not enough. Its relevance to emotional speech generation is still indirect: no speech LM or TTS system is trained on the tokens.

Claims

  • supports: Structurally partitioning a residual quantizer so that attribute-specific and acoustic dimensions never mix preserves the protected attribute through compression better than leaving allocation to the reconstruction loss.

    Evidence: On IEMOCAP at 1.5 kbps, Macro EDR is 5.27% with block-diagonal RFSQ, 10.23% with the same factorized RFSQ using dense projections, and 14.44% with standard RVQ, with all other components unchanged. (§4.3, Table 3)

  • supports: Concatenating an auxiliary attribute representation with the acoustic latent before a densely projected quantizer lets reconstruction gradients repurpose the reserved dimensions for acoustic content.

    Evidence: A linear probe recovers log-mel frames from the 24 binary emotion-partition codes with R² of 0.0985 for a dense-projection variant versus 0.0196 for the block-diagonal codec, against a near-zero permutation baseline. (Appendix C, Table 4)

  • supports: Standard reconstruction-oriented neural codecs lose emotion-relevant information mostly at low bitrates, and the loss shrinks as bitrate grows.

    Evidence: DAC’s Macro EDR on CREMA-D falls from 40.48% at 1.5 kbps to 17.52% at 3.0 kbps and 3.19% at 6.0 kbps, and EnCodec shows the same trend on all three datasets. (§4.2, Table 1)

  • complicates: Reserving codec capacity for paralinguistic attributes can cost intelligibility relative to codecs that target semantic content, especially at the lowest bitrates.

    Evidence: At 1.5 kbps the emotion-preserving codec has a higher WER than the semantic-concatenation X-Codec on IEMOCAP (15.39 vs. 9.54), CREMA-D (12.62 vs. 5.09) and ESD (6.04 vs. 3.46), even though it leads on ViSQOL and STOI. (§4.2, Table 2)

  • complicates: The advantage of attribute-reserving quantization narrows at higher bitrates, where a strong reconstruction-only codec retains better acoustic fidelity.

    Evidence: At 6.0 kbps DAC beats AffectCodec on ViSQOL, STOI and WER on all three datasets (e.g., IEMOCAP ViSQOL 4.37 vs. 4.18), while AffectCodec keeps a smaller emotion-preservation lead. (§4.2, Tables 1 and 2)

Limitations and Open Questions

  • No downstream generation test. The paper argues that preserved emotion benefits speech LMs, but it never trains or evaluates a speech LM or TTS model on AffectCodec tokens. The authors list this as future work (§Limitations), so the downstream claim is untested.
  • Evaluation relies on model-based proxies only. EDR and V/A/D MSE come from SER and regression models; there is no human perceptual test of emotion or quality.
  • Possible training-evaluation overlap. IEMOCAP is in the training mix and is also an evaluation set; CREMA-D and ESD are the cleaner out-of-domain tests.
  • Narrow operating range. Only 16 kHz English speech, three emotion corpora, and up to 6.0 kbps are covered. Partition sizes and multi-rate targets are hand-chosen (emotion partition from a rate-distortion search in Appendix D), and the emotion2vec teacher’s biases carry into the reserved partition.
  • Encoder cost. Encoding requires the frozen 164M-parameter emotion2vec plus a 26M-parameter adapter, which roughly doubles the loaded model size relative to the codec itself (Table 10).
  • Code not yet released. The authors promise release upon acceptance.

Wiki Connections

  • Neural Audio Codec — Proposes a residual FSQ speech codec with block-diagonal projections that reserve a fixed share of each token for emotion, evaluated from 1.5 to 6.0 kbps.
  • Emotion Synthesis — Targets emotion preservation in codec-reconstructed speech, measured with SER-based degradation rates and V/A/D error across three emotional corpora.
  • Disentanglement — Separates emotion and acoustic subspaces inside the quantizer through a block-diagonal constraint, backed by an ablation against dense projections and a leakage probe.
  • Self-Supervised Speech — Uses frozen emotion2vec, a self-supervised emotion representation, as the teacher for both the emotion pathway and the emotion losses.
  • emotion2vec — Serves as the frozen emotion teacher whose features fill the reserved quantizer partition and define the emotion reconstruction and cycle-consistency losses.
  • EnCodec — Used as an RVQ baseline that shows the largest emotion degradation at low bitrates.
  • SpeechTokenizer — Compared as a semantic-distilled RVQ baseline, which beats AffectCodec on ESD emotion preservation at 3.0 kbps.
  • NaturalSpeech 3 — Its FACodec is contrasted as an attribute-factorized codec whose heterogeneous per-attribute tokens AffectCodec avoids with a flat per-stage token.
  • Finite Scalar Quantization — Provides the scalar quantizer that BD-RFSQ stacks residually and wraps with factorized block-diagonal projections.
  • HiFi-Codec — Discussed as a group-residual quantization precedent that partitions encoder features without attribute-specific supervision.
  • CosyVoice 2 — Cited as a speech LM using FSQ tokens, motivating the flat-token interface BD-RFSQ preserves.
  • Whisper — Whisper-Large-v3 is used to compute WER for intelligibility evaluation.