arXiv · 2026 · Preprint
Nikita Kuzmin et al. · → Paper · Demo: ✓ · Code: ✓
Restores emotional expressiveness lost by streaming neural-codec speaker anonymization by restructuring finetuning pairs and distilling frame-level emotion features into acoustic token hidden states, without adding inference latency.
Problem
Streaming speaker anonymization (SA) systems built on neural audio codec (NAC) language models achieve competitive privacy-intelligibility trade-offs at low latency, but they systematically degrade the emotional content of the source speech. The paper traces this to two causes: the audio-continuation training paradigm teaches the model to reproduce dominant acoustic patterns from its prompt rather than transferring the source utterance’s paralinguistic content, and the vector-quantization (VQ) bottleneck in the codec discards fine-grained acoustic detail that carries emotion. Prior streaming work addressed this only by feeding multiple emotion-diverse prompts at inference, which partially restores emotion but degrades intelligibility and requires emotion-labeled prompts that are harder to source than the neutral prompts normally used for anonymization. Offline SA systems can preserve emotion more effectively by exploiting full utterance context, but that context is unavailable under real-time streaming constraints.
Method
The system finetunes a pretrained streaming SA baseline (an NAC language model with interleaved Slow AR and Fast AR branches for semantic and acoustic tokens) rather than modifying its architecture. Training pairs are constructed from an emotional speech corpus so that each pair contains a neutral and an emotional utterance from the same speaker: the model is given a neutral prompt and must generate emotional acoustic tokens for the source content, forcing emotion information to come from the source content rather than being copied from the prompt’s acoustic style. Neutral-to-neutral pairs are also included so the model does not assume every source utterance is emotional. Separate learned separation tokens (“[SEP]”) mark the prompt/source boundary independently in the semantic and acoustic token streams, giving the model an explicit signal for when to stop reproducing the prompt and start preserving the source.
On top of this restructured supervision, the method adds frame-level emotion distillation: a small causal transformer (2 layers) is trained to regress frame-level emotion embeddings from a pretrained emotion feature extractor (Emotion2Vec+) using the acoustic hidden states of the Slow AR branch as input, supervised by an MSE loss . The total training loss combines this distillation term with the language-modeling losses of the Slow AR and Fast AR branches, weighted by a coefficient . The distillation head deliberately targets the acoustic branch rather than the semantic branch (already supervised by next-token prediction) to avoid gradient competition between content and emotion learning. At inference time, and the emotion extractor are discarded entirely, so the anonymization model runs with the same architecture, parameter count, and 180ms streaming latency as the un-finetuned baseline. Finetuning itself uses only the Slow AR, Fast AR, and distillation modules (the rest of the pipeline stays frozen), on roughly 25,000 neutral-emotion pairs built from CREMA-D, and completes in under two hours on 4 GPUs.
Key Results
On the VoicePrivacy 2024 protocol, the full method (frame-level acoustic distillation) reaches 49.2% UAR for emotion preservation on IEMOCAP with 5.77% WER and 48.98% lazy-informed EER, all evaluated on the finetuned streaming baseline. This is a +24% relative UAR improvement over the unmodified streaming baseline (39.7% → 49.2% UAR) at a modest WER cost (4.54% → 5.77%), and a +10% relative UAR improvement over an emotion-prompt variant of the same baseline that used four emotion-diverse prompts (44.6% UAR, 6.59% WER) while using only a single neutral prompt. Privacy also improves slightly alongside emotion preservation (EER-lazy 47.19% → 48.98%). Against other streaming SA methods with viable privacy (EER-lazy ≥ 40%), the method improves UAR by 32-44% relative over GenVC-small (34.2%), DarkStream (34.7%), and TVTSyn (37.3%), with comparable or better WER and privacy. A gap remains to the offline EASY system (63.8% UAR), which the paper attributes to offline methods’ access to full-utterance bidirectional context.
The ablation study isolates the source of the gains: naive finetuning on emotional data alone (Exp1) yields only +1.4 UAR over baseline, while restructuring training pairs to force generation from neutral prompts (Exp2) yields +4.1 UAR, about 3x larger, establishing that emotion degradation is primarily a training-paradigm problem rather than a data-scarcity or model-capacity problem. The explicit separation token adds a further +2.1 UAR. Comparing distillation targets, distilling to the semantic branch (Exp6) reaches 48.2% UAR but raises WER to 6.23% from gradient competition with content supervision, while distilling to the acoustic branch (Exp7, the reported configuration) reaches 49.2% UAR with better WER (5.77%). Per-emotion breakdown shows the largest gain for the “sad” class (8.0% → 42.6% recall) and a partial correction of an “happy” over-prediction bias present in the baseline (81.9% → 62.8%, interpreted as reduced default-bias rather than a genuine regression).
Novelty Assessment
The contribution is a training-recipe and training-objective refinement of an existing streaming SA architecture (the authors’ own prior StreamVoiceAnon system) rather than a new model architecture: the underlying Slow AR / Fast AR codec-LM backbone, codec, and inference procedure are unchanged, and the added distillation transformer is discarded at inference, leaving zero architectural or latency footprint. What is genuinely new is the causal identification that emotion degradation in this class of model is a training-paradigm artifact rather than a capacity or data limitation, demonstrated through a controlled ablation (Exp1 vs Exp2), and the specific design choice of applying frame-level (not utterance-level) emotion distillation to the acoustic branch rather than the semantic branch, which the paper shows outperforms the more natural-seeming semantic-branch alternative. The same-speaker neutral/emotional pairing strategy and explicit separation tokens are simple but effective mechanisms motivated by a clear failure-mode analysis. Overall this is an incremental, well-ablated engineering advance on a specific narrow problem (emotion preservation under streaming SA) rather than a broadly new direction.
Field Significance
moderate — the paper provides a targeted, well-ablated fix to a specific and previously under-addressed failure mode (emotion loss) in streaming speaker anonymization, achieving the highest reported emotion preservation among streaming methods at zero added inference cost. Its main transferable insight, that restructuring same-speaker training pairs matters more than adding emotional training data, and that distillation targets should avoid the content-supervised branch to prevent gradient competition, is a concrete, paper-demonstrated finding rather than a broad architectural or conceptual shift.
Claims
- supports: In neural-codec language-model speech generation, restructuring training pairs to force generation of the target attribute from source content (rather than copying it from the conditioning prompt) yields larger improvements than simply adding more training data with that attribute present.
Evidence: Finetuning on an emotional corpus without restructuring pairs (Exp1) improved UAR by only +1.4 over baseline, while pairing neutral prompts with emotional source utterances from the same speaker (Exp2) improved UAR by +4.1, about 3x larger, with sad-emotion recall rising from 8.0% to 21.7% (§4.3, Table 2).
- supports: When distilling an auxiliary attribute (e.g., emotion) into the hidden states of a multi-branch autoregressive codec-token model, targeting the branch without existing token-level supervision avoids gradient competition and yields better results than targeting the content-supervised branch.
Evidence: Distilling frame-level emotion features into acoustic-branch hidden states (Exp7) achieved 49.2% UAR with 5.77% WER, while distilling into the semantic (content-supervised) branch (Exp6) achieved a similar 48.2% UAR but degraded WER to 6.23% (§4.3, Table 2).
- complicates: Zero-inference-overhead finetuning fixes for attribute preservation in streaming generation can close much, but not all, of the gap to offline systems that can exploit full-utterance bidirectional context.
Evidence: The proposed streaming method reaches 49.2% UAR versus 63.8% UAR for the offline EASY system on the same VoicePrivacy protocol, a gap attributed to offline methods’ access to full utterance context for bidirectional and utterance-level emotion optimization (§4.2, §5).
- complicates: Emotion-preservation evaluation in speaker anonymization pipelines is sensitive to evaluator choice and corpus type, limiting how strongly single-corpus, single-evaluator results generalize.
Evidence: The paper reports reliance on a single SER evaluator (a wav2vec 2.0-based model trained on IEMOCAP), the absence of subjective listening tests, and evaluation restricted to acted-emotion corpora (CREMA-D for finetuning, IEMOCAP for evaluation) rather than spontaneous emotion data (§Limitations/§5).
Limitations and Open Questions
All evaluation uses a single automated SER evaluator on acted-emotion speech (CREMA-D finetuning, IEMOCAP evaluation), with no subjective listening tests reported, so emotion-preservation gains are not independently corroborated by human raters or spontaneous speech data.
The paper also notes that the causal (streaming) constraint limits the model to short-range emotional context; the authors flag extending emotional context within the causal constraint, moving beyond categorical to dimensional emotion models, and testing adversarial robustness against adaptive privacy attackers as future work. The reported gap to offline emotion-aware anonymization (EASY, 63.8% UAR vs. 49.2% UAR here) reflects a structural latency-quality trade-off rather than a limitation specific to this method.
Wiki Connections
- Voice Conversion — applies a training-time fix for attribute (emotion) preservation to a neural-codec-based streaming voice-conversion/anonymization pipeline, without altering the conversion architecture itself.
- Emotion Synthesis — directly targets emotion preservation through source speech transformation, showing that emotion loss in streaming codec-LM anonymization is a training-paradigm problem rather than a capacity limitation.
- Self-Supervised Speech — uses a self-supervised pretrained model (Emotion2Vec+, data2vec-based) as the frame-level distillation teacher supplying emotion targets during training.
- Neural Audio Codec — builds on a neural-codec language-model backbone and explicitly attributes part of the emotion-degradation problem to the codec’s VQ bottleneck discarding fine-grained acoustic detail.
- StreamVoiceAnon — is the pretrained streaming speaker-anonymization baseline this paper finetunes; all comparisons and the “neutral prompt” ablation are measured against it.
- DarkStream — is used as a streaming SA baseline for comparison in Table 1, showing lower UAR (34.7%) at comparable EER than the proposed method.
- emotion2vec — provides the frame-level emotion representations (via its Emotion2Vec+ variant) used as the distillation target for the proposed acoustic-branch distillation loss.
- ECAPA-TDNN — is the speaker-verification model used to compute privacy (EER) under the VoicePrivacy 2024 lazy- and semi-informed attacker protocols.