arXiv · 2026 · Preprint
Ge et al. (Northeastern University / NiuTrans Research) · → Paper · Demo: ? · Code: ?
Systematically tests whether Speech Emotion Recognition (SER) models and generative speech LLMs that reliably recognize emotion in human speech also generalize to synthesized speech, and finds that they largely do not.
Problem
A growing body of work treats emotion understanding as a transferable signal: SER models trained on human recordings are used to score or reward the emotional expressiveness of TTS and speech-to-speech (S2S) systems, on the assumption that a model which recognizes emotion in human speech will recognize it equally well in synthesized speech. This assumption underlies RL-based and evaluation-based pipelines for emotional TTS, but it had not been rigorously tested. Separately, generative Speech Language Models (SLMs) are increasingly deployed as emotion-aware components in voice interaction, yet whether they actually attend to paralinguistic acoustic cues, as opposed to inferring emotion from transcribed text content, is unclear. The paper asks directly: can current SER models and SLMs reliably understand emotion in synthesized speech, and if not, why?
Method
The paper formulates SER as a function mapping a speech dataset (human-recorded, TTS-synthesized, or S2S-LLM-synthesized) to a discrete emotion label set. It evaluates a discriminative SER model (Emotion2vec, plus the stronger open-source C2SER) and generative SLMs (Qwen3-Omni, GPT-4o Audio) on human speech (TESS, CREMA-D, IEMOCAP, RAVDESS, MELD) and on speech synthesized from the same or matched text by open-source TTS models (CosyVoice 2, IndexTTS 2), open-source S2S LLMs (Kimi-Audio, GLM-4-Voice), and commercial systems (GPT-4o mini TTS, GPT-4o Audio). To isolate whether measured degradation stems from the synthesis process itself rather than confounds, the authors run a series of controlled ablations: (1) manual annotation by four human raters to exclude synthesized clips with indistinct emotional expression; (2) an identical-text-distribution control, where speech is resynthesized from ASR transcripts of real recordings rather than LLM-generated text, isolating synthesis artifacts from text-distribution shift; and (3) a three-stage decomposition of the TTS pipeline (autoregressive speech-token generation, flow matching to mel-spectrogram, vocoder waveform synthesis) using CosyVoice 2 and IndexTTS 2, in which each stage is independently substituted with its ground-truth counterpart to isolate its individual contribution to the SER accuracy drop. To probe why the gap persists, the authors also apply t-SNE visualization and linear probing to the frozen Emotion2vec representation space, and attempt to close the gap via supervised fine-tuning and domain-adversarial fine-tuning (a frozen Emotion2vec encoder feeding a trainable MLP feature extractor and emotion classifier, with a gradient-reversal-layer domain classifier trained to make the shared representation invariant to synthesis, vocoder, or model identity) evaluated on in-domain and out-of-domain (unseen synthesis-model) test splits.
Key Results
Emotion2vec achieves near-perfect diagonal confusion matrices on human speech but collapses on TTS-synthesized speech, even after manual filtering confirms the target emotion is clearly perceptible (§4.1, Fig. 2). The identical-text control shows the same collapse when synthesizing from real ASR transcripts, ruling out text-distribution shift as the cause (§4.1.3, Fig. 4). Fine-tuning Emotion2vec separately on human vs. synthesized data shows the models transfer poorly in both directions (e.g. weighted accuracy of 71.28% and 15.31% for a human-trained model on human-TESS and synthesized-TESS respectively; the pattern reverses for a synthesized-trained model), confirming a genuine distribution gap rather than a labeling artifact (§4.1.4, Table 1). The three-stage decomposition finds the vocoder and flow-matching stages contribute only marginal error (real speech: 83.93% SER accuracy; mel-spectrogram reconstruction: 82.43%; ground-truth-token resynthesis: 84.07%), while driving synthesis from AR-predicted tokens instead of ground-truth tokens drops accuracy to roughly 30-50% depending on reference conditions, identifying token-generation error as the dominant source of the gap (§5.3, Fig. 9). Even leading commercial systems do not close the gap: Emotion2vec scores only 32.94%-64.16% accuracy on GPT-4o mini TTS and GPT-4o Audio outputs across CREMA-D and TESS (§4.2), and top-tier Gemini 2.5 Pro likewise shows a persistent human-synthesis gap when scoring GPT-4o-generated speech (§4.4, Table 2, Fig. 6). Linear probing of the Emotion2vec representation space shows near-perfect discrimination of synthetic-vs-human identity, vocoder, and synthesis-model identity (99.81%, 96.25%, 75.01% balanced accuracy respectively) but comparatively weak discrimination of emotion (59.86%), indicating the embedding space is dominated by synthesis artifacts rather than affective content (§5.1, Table 3). Domain-adversarial fine-tuning raises in-domain accuracy (73.14% human, 88.73% synthesized on average) but fails to improve generalization to out-of-domain synthesis models or held-out speaker-emotion combinations (§5.2, Table 4). Generative SLMs (Qwen3-Omni, GPT-4o Audio) reach 73.3% average accuracy on emotion benchmarks where the transcript itself is informative of emotion (StepEval-Audio-Paralinguistic, Multi-Speaker Emotional Speech Dataset) but drop to 19.6% average on TESS/CREMA-D, where the same sentence is read with different emotions and paralinguistic cues are required; prompt engineering explicitly instructing the model to ignore text content does not change this behavior (§4.3, Fig. 5).
Novelty Assessment
The contribution is empirical and diagnostic rather than architectural: no new SER model, TTS system, or SLM is proposed. Its novelty lies in the systematic ablation design that isolates the synthesized-speech SER gap to a specific pipeline stage (autoregressive token generation) and in the representation-space probing that traces the failure to synthesis-artifact features rather than missing emotional information. The domain-adversarial fine-tuning experiment is a straightforward application of existing gradient-reversal-layer techniques used as a diagnostic probe (to test whether the gap is a solvable representation-invariance problem) rather than as a proposed new method, and its negative result (failure to generalize out-of-domain) is itself part of the paper’s evidence.
Field Significance
Tip
High — this paper provides direct evidence against a load-bearing assumption in recent emotional-TTS and S2S work, namely that SER accuracy (from discriminative models or SLM judges) is a valid reward or evaluation signal for synthesized emotional expressiveness. By decomposing the TTS pipeline and showing the gap concentrates in autoregressive token generation rather than the vocoder or flow-matching stages, it gives a concrete mechanistic target for future work on closing the synthesized-speech understanding gap, and its persistence across open-source and commercial, discriminative and generative systems suggests the finding is not an artifact of any single weak component.
Claims
- supports: SER models trained exclusively on human speech exhibit a substantial and consistent accuracy drop when applied to synthesized speech, even when the target emotion is confirmed by human listeners to be clearly expressed in the synthesized audio.
Evidence: Emotion2vec confusion matrices lose their diagonal structure on TTS-synthesized TESS/CREMA-D after manual filtering for perceptible emotion, and weighted accuracy falls from 71.28%/99.64% (human IEMOCAP/TESS) to 49.58%/15.31% (synthesized IEMOCAP/TESS) for a human-trained model. (§4.1, §4.1.4, Table 1, Fig. 2)
- refines: Within a multi-stage neural TTS pipeline (autoregressive token generation, flow matching, vocoder), the dominant source of downstream understanding-model domain shift is the autoregressive token-generation stage, not the acoustic-synthesis stages that follow it.
Evidence: Substituting ground-truth mel-spectrograms or ground-truth speech tokens for real audio degrades SER accuracy only marginally (83.93% real to 82.43%-84.07% reconstructed), while driving synthesis from AR-predicted tokens drops accuracy to approximately 30-50%. (§5.3, Fig. 9)
- complicates: Fine-tuning an emotion-recognition model on a mixture of human and synthesized speech, including with a domain-adversarial objective intended to remove synthesis-identity information from the representation, improves in-domain accuracy but does not transfer to unseen synthesis models or held-out attribute combinations, indicating the model exploits synthesis-specific shortcuts rather than learning emotion features that are invariant to how the speech was produced.
Evidence: Mixed-domain MLP training reaches 73.14%/88.73% in-domain accuracy on human/synthesized speech but drops sharply on OOD CREMA-D partitions; gradient-reversal-layer domain-adversarial variants (DANN_syn/vocoder/model) do not meaningfully improve OOD generalization despite a rising domain loss confirming the adversarial objective is active. (§5.2, Table 4)
- complicates: Generative speech language models infer emotion primarily from the semantic content of the transcribed speech rather than from paralinguistic acoustic cues, and this text-dominant behavior is not corrected by prompt instructions to ignore textual content.
Evidence: Qwen3-Omni and GPT-4o Audio average 73.3% accuracy on benchmarks where the transcript is informative of emotion but drop to 19.6% average accuracy on TESS/CREMA-D, where the same sentence is read under different emotions; adding explicit “ignore textual content, judge emotion only from acoustic characteristics” instructions does not change this result. (§4.3, Fig. 5, Appendix B)
- supports: The synthesized-speech emotion-understanding gap is not confined to weaker or open-source components; it persists when both the synthesis model and the recognition model are top-tier commercial systems.
Evidence: Gemini 2.5 Pro, evaluated on speech synthesized by GPT-4o mini TTS and GPT-4o Audio, still produces confusion matrices without a clear diagonal structure, and Emotion2vec accuracy on GPT-4o mini TTS/GPT-4o Audio outputs remains at 32.94%-64.16% across CREMA-D and TESS. (§4.2, §4.4, Table 2, Fig. 6)
Limitations and Open Questions
The evaluation depends on manual human filtering to establish ground truth for "clearly expressed" target emotion in synthesized clips, using only four in-house annotators; this introduces potential subjectivity and caps the scale of the synthetic test sets (only ~4k synthetic clips survive filtering versus ~40k human clips used for fine-tuning), which the authors themselves flag as a limitation.
The attempted remedy (domain-adversarial fine-tuning with a gradient-reversal layer) fails to generalize to unseen synthesis models, so the paper leaves open how the synthesized-speech representation gap could actually be closed; the authors suggest pretraining better representation models as future work rather than proposing a validated fix. The three-stage pipeline decomposition is demonstrated only on CosyVoice 2 and IndexTTS 2 (both autoregressive-token-then-flow-matching architectures), so it is not established whether the same token-generation-dominant failure mode holds for TTS architectures built on different generative mechanisms (e.g., pure diffusion or GAN-based systems without a discrete token-prediction stage).
Wiki Connections
- Emotion Synthesis — directly challenges the practical use of SER accuracy as a proxy signal for evaluating or rewarding emotional expressiveness in synthesized speech, since the paper shows SER models cannot reliably score synthesized emotion in the first place.
- Spoken Language Model — evaluates the paralinguistic understanding of generative Speech LLMs (Qwen3-Omni, GPT-4o Audio) and finds a persistent text-dominance bias that prompt engineering does not remove.
- Speech-to-Speech — includes S2S LLMs (Kimi-Audio, GLM-4-Voice) as one of the synthesis paradigms tested for emotion-recognition domain shift, alongside TTS.
- Evaluation Metrics — questions the validity of SER-based automatic metrics for emotional TTS/S2S evaluation, providing systematic evidence of a human-synthesis distribution gap that undermines their use as ground truth.
- Subjective Evaluation — relies on four human annotators to verify that synthesized target emotions are perceptually salient before including clips in the evaluation and fine-tuning sets.
- CosyVoice 2 — used as one of the two open-source TTS systems whose autoregressive-token-to-flow-matching-to-vocoder pipeline is decomposed to isolate the source of the synthesized-speech SER gap.
- IndexTTS2 — used as the second open-source TTS system for both the main SER evaluation and the three-stage pipeline decomposition, corroborating the CosyVoice 2 findings.
- Qwen3-Omni Technical Report — evaluated as one of two generative SLMs whose emotion recognition is shown to rely on textual semantics rather than paralinguistic cues.
- StyleBench — companion work from the same research group evaluating a related but distinct capability (speech LMs following instructions to control conversational speaking style) rather than emotion recognition accuracy.