arXiv · 2026 · Preprint

Siyi Wang et al. (The University of Melbourne / Monash University) · → Paper · Demo: ✗ · Code: ✗

Compares the speech language model and the flow-matching decoder of a frozen CosyVoice 2 as activation-steering sites for mixed-emotion TTS, and links each site’s representation geometry (probe accuracy, local intrinsic dimensionality) to how well steering there controls emotion proportions without harming speaker identity.

Problem

Hybrid TTS systems pair an autoregressive speech language model (SLM) over discrete speech tokens with a conditional flow-matching (CFM) decoder that renders mel-spectrograms. Activation steering has been applied to each half separately: the authors’ own CoCoEmo steers the SLM to compose mixed emotions, and EmoSteer-TTS steers the flow-matching decoder for continuous single-emotion intensity. Nobody had compared the two sites directly, asked whether their representation geometry predicts how well they steer, or tested whether steering both at once helps or interferes. This short paper, presented at the ICML 2026 Workshop on Machine Learning for Audio, addresses those three questions.

Method

The backbone is CosyVoice 2, kept frozen throughout. Its SLM is a 24-layer transformer built on Qwen2.5 (hidden size 896), and its CFM is a 56-layer DiT (hidden size 256) run for 10 denoising steps. No model weights are trained; the only learned components are per-layer linear probes used for analysis.

Geometry analysis. Two measurements characterize how emotions are organized at each site, using ESD, CREMA-D and RAVDESS over five emotions (angry, happy, neutral, sad, surprise).

  • Linear probing: a linear classifier per layer (and per denoising step for the CFM) predicts emotion. Thirty percent of speakers are held out (4,530 utterances) to measure cross-speaker accuracy; the rest give 11,311 training and 4,850 within-speaker test utterances. A large within/cross gap signals that emotion is entangled with speaker identity.
  • Local intrinsic dimensionality (LID): a Levina-Bickel maximum-likelihood estimate over k=50 nearest neighbours, on 4,000 sampled utterances averaged over 10 resamplings. The paper defines ΔLID as pooled LID (all emotions together) minus the mean per-emotion LID. A positive ΔLID means that pooling emotions adds independent directions, which the authors argue favours composing emotion vectors.

Steering. For each emotion, a steering vector is the mean activation difference between emotional and neutral samples at a layer. SLM vectors follow CoCoEmo: attention-output activations at the last token, applied at layers 14 and 17. CFM vectors follow EmoSteer-TTS: residual-stream activations, L2-normalized, restricted to the top-k emotion-relevant frames chosen by an emotion classifier, applied at every fifth layer (12 layers) across all 10 steps. A mixed-emotion vector is the proportion-weighted sum of single-emotion vectors. At inference the scaled vector (strength α) is added to the activation and the result is renormalized to the original scale (activation addition, after Turner et al.). Joint steering applies both sites together.

Evaluation. Steering vectors come from 50% of speakers. Tests run on CREMA-D (in-distribution) and IEMOCAP (out-of-distribution), with mixed-emotion targets derived from disagreement among multiple human annotators. All metrics are automatic:

  • E-SIM: cosine similarity of emotion2vec embeddings between synthesized and ground-truth speech.
  • TEP: mean emotion2vec probability on the target emotions.
  • ρ: Spearman correlation between the ranking of emotion-probability increases and the ground-truth emotion ranking.
  • H-Rt: fraction of samples where the dominant ground-truth emotion gains the most probability.
  • Speaker similarity (S-SIM, WavLM embeddings) and WER from Whisper Large-v3.

Key Results

Geometry (Table 1, Figure 1). The two sites differ sharply.

PropertySLMCFM
Probe accuracy, best layer (within / cross speaker)0.80 / 0.710.89 / 0.62
Mean within-cross gap (all layers and steps)0.080.32
Manifold dimension (LID)about 28about 13
Mean ΔLID+0.84 (positive from layer 6 on)−1.48 (negative at all 56 layers and 10 steps)
Discriminability peakMid-to-late layers (10-17)Uniform across layers and steps

The CFM separates emotions well within a speaker but generalizes poorly across speakers, which the paper reads as speaker-emotion entanglement. Its LID also falls across denoising steps as the representation converges toward the output.

Steering (Table 2). Docling flattened the Table 2 header; the values below are taken from the PDF.

Test setConfigE-SIM ↑TEP ↑ρ ↑H-Rt ↑S-SIM ↑WER ↓
CREMA-DNo steering.743.065––.8711.07
CREMA-DCFM α=1.0.767.097.098.691.8580.76
CREMA-DCFM α=2.0.786.160.193.717.8070.79
CREMA-DSLM α=3.0.762.100.166.709.8721.01
CREMA-DSLM α=5.0.779.149.209.724.8700.78
CREMA-DJoint α=1.0.767.131.112.695.8591.02
CREMA-DJoint α=2.0.787.163.176.711.8081.06
IEMOCAPNo steering.903.197––.8886.70
IEMOCAPCFM α=1.0.910.218.138.729.8856.08
IEMOCAPCFM α=2.0.909.272.117.721.8446.15
IEMOCAPSLM α=3.0.911.228.186.744.8915.86
IEMOCAPSLM α=5.0.915.253.215.755.8906.27
IEMOCAPJoint α=1.0.912.237.193.746.8846.05
IEMOCAPJoint α=2.0.911.274.170.737.8456.29
  • Proportional control: SLM α=5.0 has the highest ρ and H-Rt on both test sets (CREMA-D .209 / .724; IEMOCAP .215 / .755). Comparing the lower-strength settings with each other and the higher-strength settings with each other, SLM beats CFM on both measures on both sets.
  • Intensity: joint α=2.0 has the highest TEP on both sets, but only narrowly ahead of CFM α=2.0 (.163 vs .160 on CREMA-D; .274 vs .272 on IEMOCAP). E-SIM is close across all steered settings.
  • Speaker identity: CFM steering and joint steering lower speaker similarity. At α=2.0, S-SIM falls from .871 to .807 (CFM) and .808 (joint) on CREMA-D, and from .888 to .844 and .845 on IEMOCAP. SLM steering keeps S-SIM at the unsteered level (.870 to .872 on CREMA-D; .890 to .891 on IEMOCAP).
  • Joint steering on CREMA-D: proportional control is worse than SLM-only at both strength levels (ρ .112 vs .166 for joint α=1.0 vs SLM α=3.0; .176 vs .209 for joint α=2.0 vs SLM α=5.0). On IEMOCAP, joint α=1.0 is slightly above SLM α=3.0 (ρ .193 vs .186) but below SLM α=5.0.
  • WER: every steered setting has lower WER than the unsteered baseline on both sets. Joint steering has higher WER than the single-site settings on CREMA-D. The paper does not explain why steering lowers WER.

The reported α values were chosen as the best configurations that keep S-SIM within 10% of the baseline and raise WER by less than 0.5. All comparisons use one backbone, the same test sets, and only automatic emotion scorers.

Novelty Assessment

The steering methods are not new: the SLM-only and CFM-only conditions reimplement CoCoEmo and EmoSteer-TTS, and joint steering simply applies both. The new parts are the side-by-side comparison of the two sites within one hybrid model and the use of linear-probe cross-speaker gaps and ΔLID as geometric predictors of steerability. The link from geometry to steering is made at the module level only (two data points: one SLM, one CFM), so it is suggestive rather than tested. This paper is a follow-up analysis to CoCoEmo, whose author list includes all three authors here; it is not an extended version of it. It is a short workshop paper (ICML 2026 Workshop on Machine Learning for Audio) with no human listening test.

Field Significance

Low — The paper gives a concrete, measured account of where emotion steering works in a hybrid SLM-plus-flow-matching TTS model: the token LM carries speaker-invariant, composable emotion directions, while the flow-matching decoder entangles emotion with speaker identity. This is useful guidance for steering design, and ΔLID is a cheap diagnostic others could reuse. However, the evidence comes from one backbone, automatic scorers only, and small margins on several metrics, so it is mainly confirmatory evidence for SLM-site steering, not a new method.

Claims

  • supports: In hybrid TTS, the autoregressive token LM encodes emotion in speaker-invariant, low-dimensional subspaces, while the flow-matching acoustic decoder entangles emotion with speaker identity.

    Evidence: On frozen CosyVoice 2, best-layer probe accuracy is 0.80 within-speaker and 0.71 cross-speaker for the SLM, against 0.89 and 0.62 for the CFM; the mean within-cross gap is 0.08 for the SLM and 0.32 for the CFM. Mean ΔLID is +0.84 in the SLM and −1.48 in the CFM. (§4.1, Table 1, Figure 1)

  • supports: Steering a TTS model’s token language model gives more precise proportional control of mixed emotions than steering its flow-matching decoder, with less loss of speaker identity.

    Evidence: SLM α=5.0 has the highest ρ and H-Rt on CREMA-D (.209 / .724) and IEMOCAP (.215 / .755). Speaker similarity stays at the unsteered level (.870 vs .871; .890 vs .888), while CFM α=2.0 drops it to .807 and .844. (§4.2, Table 2)

  • complicates: Applying independent steering vectors at several modules of a TTS pipeline raises emotion intensity but can interfere with compositional control instead of adding up.

    Evidence: Joint α=2.0 has the highest TEP on both sets (.163; .274), but on CREMA-D its ρ (.176) is below SLM-only α=5.0 (.209) and CFM-only α=2.0 (.193). Its speaker similarity falls to .808. The authors attribute this to distribution shift from SLM steering, speaker entanglement in the CFM, and accumulating uncoordinated perturbations. (§4.2, Table 2, “Steering analysis”)

  • refines: Activation steering on decoder modules conditioned on speaker embeddings and reference speech tends to perturb speaker identity, because emotion directions there are not separable from speaker directions.

    Evidence: The CosyVoice 2 flow-matching module is conditioned on speaker embeddings and reference speech, while the SLM is not. CFM steering lowers S-SIM on both test sets, and its best-layer cross-speaker probe accuracy is lower than the SLM’s (0.62 vs 0.71). (§4.1, §4.2, Tables 1-2)

Limitations and Open Questions

Warning

All findings come from a single backbone (CosyVoice 2), and every emotion and quality measure is automatic (an emotion2vec classifier, WavLM speaker similarity, Whisper WER). No human listening test checks whether the measured gains in emotion proportion or intensity are audible.

  • The geometry-to-steering link rests on two modules. The paper itself proposes per-layer and per-step analysis, and testing other architectures such as IndexTTS2, to test whether the link holds.
  • Several “best” results are separated by very small margins (TEP .163 vs .160; E-SIM .787 vs .786). No variance or significance tests are reported.
  • Steering strength is shared across sites in joint steering. Per-site α tuning, CFM vectors extracted from SLM-steered outputs, and orthogonalizing CFM vectors against speaker directions are proposed but not tested.
  • The unexplained WER drop under every steering setting is reported but not analyzed.
  • Mixed-emotion ground truth comes from annotator disagreement, which is an indirect proxy for genuinely mixed emotional expression.

Wiki Connections

  • Emotion Synthesis — Evaluates training-free mixed-emotion control by activation steering and compares which module of a hybrid TTS model is the better steering site.
  • Autoregressive Codec TTS — Steers the frozen CosyVoice 2 speech LM over discrete speech tokens and shows it holds speaker-invariant, composable emotion directions.
  • Flow Matching — Steers the frozen flow-matching DiT decoder and finds emotion entangled with speaker identity there, with LID falling across denoising steps.
  • CosyVoice 2 — The frozen backbone whose speech LM and flow-matching decoder are the two steering sites compared.
  • CoCoEmo — Earlier work by a team including all three authors on SLM activation steering for composable emotion; this paper reuses its SLM steering recipe and layers and compares it against decoder and joint steering.
  • EmoSteer-TTS — Provides the flow-matching-decoder steering recipe used for the CFM-only condition, which this paper finds weaker for proportional control and worse for speaker identity.
  • emotion2vec — Supplies the emotion embeddings and classifier behind every emotion-control metric (E-SIM, TEP, ρ, H-Rt).
  • Steering Language Models With Activation Engineering — The activation-addition method with renormalization that both steering sites apply at inference.
  • IndexTTS2 — Named as an architecturally different hybrid TTS system for testing whether the findings generalize.
  • Whisper — Whisper Large-v3 provides the WER measurement used to track intelligibility under steering.