Interspeech · 2026 · Conference
Tsai et al. (National Taiwan University / University of Southern California / Gilbert AI Lab) · → Paper · Demo: ✗ · Code: ✗
Stress-tests the emotion similarity metric (EMO-SIM, cosine similarity of emotion2vec-style embeddings between a reference and a generated utterance) used across expressive TTS and emotional voice conversion, and finds it is dominated by speaker and linguistic content, insensitive to valence and arousal shifts, and only weakly aligned with human judgments of emotional resemblance.
Problem
Expressive TTS and emotional voice conversion papers increasingly report emotion similarity as cosine similarity between emotion2vec embeddings of the reference and the generated sample, typically following the VERSA toolkit recipe (last layer, mean pooled). The implicit assumption is that an encoder with high speech emotion recognition accuracy also yields a latent space in which proximity tracks emotional resemblance, independent of who is speaking and what is said. That assumption had not been tested. If it fails, the metric can reward acoustic copying of the reference speaker and text rather than genuine emotion transfer, and it can steer model selection before any listening test is run.
Method
The paper evaluates the metric itself, not any synthesis system, against three criteria: categorical emotion robustness, dimensional emotion sensitivity, and alignment with human perception.
- Calibration. Raw cosine similarities on 1,000 random pairs per dataset fall between 0.92 and 0.98, indicating an anisotropic embedding space. All scores are therefore computed after mean centering (subtracting the dataset mean embedding) before cosine similarity. A footnote states that uncalibrated results are consistently inferior or comparable, so the reported numbers are the more favourable setting for the metric.
- Categorical triplets. For a reference, positive (same emotion) and negative (different emotion), the metric is correct if it scores the positive higher. Four sampling conditions control acoustic confounds: unconstrained; speaker-linguistic match (all three share speaker and text); speaker distractor (only the negative shares the reference speaker); linguistic distractor (only the negative shares the reference text).
- Dimensional tests. Trend monotonicity is Spearman’s rho between similarity and the absolute valence or arousal difference (the other attribute held fixed; a valid metric should be negative). Shift discriminability uses triplets where the positive exactly matches the reference score and the negative differs by at least 1.0 on a 1 to 5 scale.
- Human alignment. Reference utterances from CREMA-D (acted) and MSP-Improv (spontaneous) are cloned by pairs of systems drawn from CosyVoice, Spark-TTS, F5-TTS, E2 TTS, Qwen3-TTS, MaskGCT, Diff-HierVC and FreeVC, with the two candidates sharing the reference voice and text. Five researchers pick the candidate whose emotion better matches the reference (or “hard to distinguish”); Fleiss’ kappa over the pool is 0.7349, and 400 triplets with at least 4 of 5 agreement (200 per source) are kept. Accuracy is how often the metric picks the human-preferred candidate.
- Encoders. Base emotion2vec, the fine-tuned emotion2vec+ seed/base/large, and general self-supervised models HuBERT, wav2vec 2.0 and TERA. Data are restricted to four shared categories (neutral, happy, sad, angry) across six corpora in English, Chinese and Russian, and any corpus present in an encoder’s pre-training data is excluded for that encoder (so emotion2vec+ is not scored on MSP-Podcast or CREMA-D). Each condition uses five sampling runs of 1,000 instances (500 for the speaker-linguistic match condition).
- Layer-wise probing. The eight Transformer blocks (L0 to L7) of base emotion2vec are probed on MSP-Improv across the same tasks.
Key Results
Categorical robustness (Table 1). Under unconstrained sampling, emotion2vec family accuracies range from about 51% to 66%, with the general self-supervised models at 52% to 67% (TERA reaches 67.30% on CREMA-D, above every emotion2vec variant on that set). Even when speaker and text are matched, base emotion2vec is at chance on MSP-Improv (49.84%) and emotion2vec+ reaches 60.00% to 69.32%. Distractors expose the core failure: base emotion2vec drops to 37.86% (MSP-Improv) and 43.94% (CREMA-D) with a speaker distractor, and to 20.14% (MSP-Improv) and 3.38% (CREMA-D) with a linguistic distractor, meaning it systematically prefers the wrong-emotion sample that shares the reference text. The fine-tuned emotion2vec+ variants are more resilient: they stay at 51.92% to 57.82% under the speaker distractor and 50.10% to 53.48% under the linguistic distractor on MSP-Improv, which is near chance rather than sub-random. HuBERT, wav2vec 2.0 and TERA also fall below chance under the speaker distractor on MSP-Improv (29.86% to 31.88%).
Dimensional sensitivity (Tables 2 and 3). Shift discriminability for the emotion2vec family sits between 51.48% and 61.92% across MSP-Improv, MSP-Podcast, BIIC-Podcast and NNIME, close to the 50% chance level. Spearman’s rho is negative in the expected direction but small everywhere, with the strongest value -0.20 (emotion2vec+ seed, MSP-Improv valence) and most values between -0.01 and -0.11.
Human alignment (Figure 1, §4.3). emotion2vec+ variants agree with the human-preferred candidate 52.25% to 65.00% of the time. Per the binomial tests annotated in Figure 1, emotion2vec+ seed and emotion2vec+ large, as well as wav2vec 2.0, are significantly above chance, while base emotion2vec (p = 0.051), emotion2vec+ base and HuBERT are not, and TERA falls significantly below 50% (p = 0.031). So the best fine-tuned variant does carry a real but modest signal.
Layer-wise (Figure 2, §4.4). No emotion2vec layer rescues the metric: the linguistic distractor condition stays roughly between 14% and 32% at every layer, and the speaker distractor condition around 35% to 38%. Human alignment is 58.0% at L0 and 45.0% at L7; Figure 2 shows it roughly flat in the mid-50s through L6, with most of the drop occurring at the final layer, which is the layer the VERSA recipe uses.
All comparisons are internally fair (same triplets per encoder, five runs with standard deviations), but no candidate replacement metric is benchmarked, so the paper establishes a failure without a quantified alternative.
Novelty Assessment
The contribution is diagnostic. Mean centering for anisotropic embeddings and triplet-based probing are established techniques; what is new is applying controlled speaker and linguistic distractors to the specific emotion similarity metric used in expressive TTS and emotional voice conversion evaluation, and pairing it with a human preference set built from real TTS and VC outputs that share voice and text. The finding that a classifier-trained or self-supervised embedding is dominated by speaker and content is not surprising in itself, but the quantification (sub-random accuracy under a linguistic distractor) is a concrete and actionable result. The human set is small (400 triplets), annotated by researchers rather than naive listeners, and filtered to high-consensus items, and the paper offers only a speculative remedy (contrastive training) without testing it.
Field Significance
Moderate. The paper provides a direct negative result for an emotion similarity metric that expressive TTS and emotional voice conversion work report as evidence of emotion transfer, showing it can rank a wrong-emotion sample above a right-emotion one whenever the former shares the reference’s text or speaker. It serves as a cautionary data point for reading EMO-SIM gains in voice-cloning settings, where generated speech inherently copies reference timbre, and supplies a reusable distractor protocol for vetting any future emotion similarity metric.
Claims
- contradicts: High speech emotion recognition accuracy does not imply that cosine similarity in the same embedding space measures emotional resemblance between utterances.
Evidence: Base emotion2vec scores below chance in categorical triplets once acoustic confounds are introduced (20.14% on MSP-Improv and 3.38% on CREMA-D with a linguistic distractor; 37.86% and 43.94% with a speaker distractor), and is at chance (49.84%) on MSP-Improv even with speaker and text matched. (§4.1, Table 1)
- complicates: Embedding-based emotion similarity between a reference and a generated sample can reward copying of speaker identity and linguistic content rather than emotion transfer.
Evidence: In the speaker- and linguistic-distractor conditions the wrong-emotion sample that shares the reference speaker or text is preferred, and HuBERT, wav2vec 2.0 and TERA also drop to 29.86% to 31.88% under the speaker distractor on MSP-Improv; fine-tuned emotion2vec+ variants are more resilient but only reach near chance (50.10% to 57.82%). (§4.1, Table 1)
- complicates: Utterance-level emotion embedding similarity lacks the resolution to track continuous valence and arousal differences.
Evidence: Shift discriminability for emotion2vec variants stays between 51.48% and 61.92% across four corpora, and Spearman’s rho between similarity and attribute difference is no stronger than -0.20. (§4.2, Tables 2 and 3)
- complicates: Automatic emotion similarity scores agree only weakly with human judgments of which generated utterance better matches a reference emotion.
Evidence: On 400 high-consensus TTS/VC preference triplets (Fleiss’ kappa 0.7349), emotion2vec+ variants reach 52.25% to 65.00% agreement; seed and large are significantly above chance, base emotion2vec is not, and TERA is significantly below. (§3.3, §4.3, Figure 1)
- refines: The final layer of an emotion representation model, commonly used for similarity scoring, can be less aligned with human emotion perception than earlier layers.
Evidence: Layer-wise probing of base emotion2vec shows human alignment of 58.0% at L0 versus 45.0% at L7, with distractor sensitivity present at every layer. (§4.4, Figure 2)
Limitations and Open Questions
- The human alignment set is annotated by five researchers rather than a listener panel, and only triplets with at least 4 of 5 agreement are kept, which removes ambiguous cases where an automatic metric might matter most; it is also limited to 400 triplets from two reference corpora.
- Only four emotion categories are used, and dataset label harmonisation (e.g., mapping Dusha “positive” and NNIME “joy/excited” to happy) may blur category boundaries.
- The fine-tuned emotion2vec+ variants are excluded from MSP-Podcast and CREMA-D because of pre-training overlap, so their robustness is measured on fewer corpora than the base and general self-supervised models.
- Only the cosine-on-mean-pooled-embeddings recipe is tested; frame-level alignment, probe-based classifiers, or categorical agreement metrics are not compared, and the proposed remedy (contrastive training of the encoder) is untested.
- The paper’s framing that deeper layers progressively suppress affective features is only partly reflected in Figure 2, where human alignment is roughly flat through L6 and drops mainly at L7.
- No code or released triplet sets are reported, limiting direct reuse of the protocol.
Wiki Connections
- Evaluation Metrics — Provides a negative validation of emotion embedding similarity as an objective metric for expressive speech generation.
- Subjective Evaluation — Builds a human preference set of TTS/VC outputs to measure how well the automatic metric predicts human emotion-matching judgments.
- Self-Supervised Speech — Tests emotion2vec (data2vec-based), HuBERT, wav2vec 2.0 and TERA embeddings as the basis of the metric and probes emotion2vec layer by layer.
- Emotion Synthesis — Questions the main automatic measure of emotion transfer used to evaluate emotional TTS and emotional voice conversion.
- emotion2vec — The encoder whose embedding similarity is the metric under examination; its base and emotion2vec+ variants fail under acoustic distractors.
- IndexTTS2 — Cited as an expressive zero-shot TTS system that reports emotion similarity of this kind.
- Maestro-EVC — Cited as an emotional voice conversion framework relying on the emotion similarity metric.
- F5-TTS — One of the TTS systems used to synthesize candidates for the human alignment triplets.
- MaskGCT — Both cited as an EMO-SIM user and used to generate candidates for the human alignment triplets.
- CosyVoice — Used to generate candidate utterances for the human alignment set.
- Qwen3-TTS — Used to generate candidate utterances for the human alignment set.
- Spark-TTS — Used to generate candidate utterances for the human alignment set.