arXiv · 2026 · Preprint
Daniel Oliveira de Brito et al. (Universidade Estadual Paulista (UNESP)) · → Paper · Demo: ? · Code: ✓
An elimination study on Qwen3-TTS finds that emotion is not controllable through weight-space task vectors, codec embeddings, or codec tokens, but is carried by the co-trained ECAPA-TDNN speaker embedding, where training-free centroid arithmetic transfers emotion to unseen English and Brazilian Portuguese speakers.
Problem
Task-vector arithmetic (subtracting a neutral fine-tuned model’s weights from an emotional one and scaling the difference) has been used for cross-speaker emotion-intensity control in modular TTS such as VITS and Conformer-FastSpeech2, where emotion has an identifiable functional locus (for example the text encoder’s convolutional layers or explicit pitch/energy/duration predictors). Codec language-model TTS with in-context learning (LM-TTS) has no such dedicated prosody modules: prosody emerges from conditioned autoregressive continuation. The paper asks whether weight arithmetic still works in this setting and, if not, which component (backbone weights, codec embeddings, discrete codec tokens, or speaker embedding) carries emotional information. A secondary motivation is that the purely in-context-learning Base model of Qwen3-TTS exposes no emotion parameters, unlike its instruction-trained variants.
Method
All experiments use Qwen3-TTS-12Hz-1.7B-Base (Qwen3-TTS): a 28-layer, 2048-dimensional backbone Transformer that autoregressively predicts the first (semantic) codebook of a 12Hz, 16-codebook tokenizer, a code predictor that fills codebooks 1 to 15, and a vocoder. Speaker identity enters through an x-vector produced by an ECAPA-TDNN encoder (ECAPA-TDNN) that, unlike a frozen speaker-verification model, is co-trained with the backbone under the synthesis objective and is injected directly into the codec-embedding sequence alongside the in-context reference codes.
The elimination study tests four progressively narrower operands, using English ESD data:
- Weights. Full fine-tuning and LoRA (attention-only, about 29M trainable parameters; or attention plus the codec head and the 15 code-predictor heads, about 60M) on about 30 minutes of a single speaker’s angry or neutral speech, sweeping learning rates and epochs, then forming the task vector as the difference between the angry and neutral fine-tuned weights.
- Codec embeddings. Per-codebook centroid differences between angry and neutral utterances in the talker’s 16 embedding tables, injected either as a single summed vector or per layer.
- Discrete tokens. Substitution experiments on parallel ESD pairs, the key condition (full_swap) combining all reference codec tokens of an angry utterance with the neutral x-vector, plus partial codebook-subset swaps.
- x-vectors. An emotion direction computed as the difference between mean emotional and mean neutral x-vectors, averaged either over four ESD speakers (avg4spk, gender-balanced, 50 utterances per speaker per emotion) or over one speaker (single0017), then added to an unseen target’s neutral x-vector with a scale α (0 to 2.5) while keeping the target’s own reference audio for in-context learning.
For Step 4 the TTS model is kept frozen and nothing is trained; intensity is controlled only through α at inference. The authors note that this arithmetic is operationally the same as EmoKnob’s speaker-embedding emotion direction and as the centroid arithmetic of Suni et al. on StyleTTS2 style embeddings (mean rather than geometric-median centroids); the difference is the operand and model class.
Evaluation is fully automatic: emotion embedding cosine similarity (EECS, emotion2vec_plus_large cosine to the paired emotional human recording), speaker similarity with an external WavLM speaker-verification encoder (SECS_W, chosen to avoid circularity with the ECAPA operand), normalized WER with Whisper-large-v3 (Whisper), and UTMOSv2 for naturalness. Every table includes a natural-ceiling row computed on human recordings. For the cross-lingual emoUERJ test, a further metric (xvec_cos_GT) measures ECAPA cosine between the synthesized x-vector and the same speaker’s real emotional recording.
Key Results
Steps 1 to 3 (qualitative, perceptual). Table 1 reports outcomes only as descriptions, with no metric values. Full fine-tuning yields either noise (high learning rate) or calm speech indistinguishable from the base model (low learning rate); LoRA that includes the codec and code-predictor heads produces generic expressivity, including off-task laughter, with no directional emotion control. Codec-embedding sweeps jump from no effect directly to degenerate noise. In the full_swap condition, angry codec tokens with a neutral x-vector produce calm, coherent speech, while the angry baseline (angry tokens with angry x-vector) sounds angry; the authors read this dissociation as the most direct evidence that the x-vector is the dominant emotion carrier. Partial codebook swaps degrade output (§4.1, Table 1).
English held-out speakers (ESD 0013, 0019; n = 30 per combination; Table 2). At the per-emotion best α, EECS rises over the unmanipulated Base (α = 0) for every emotion: angry 0.539 to 0.925 (avg4spk) or 0.869 (single0017); happy 0.425 to 0.687 or 0.686; sad 0.540 to 0.761 or 0.816. The mean gain is +0.288 (avg4spk) and +0.291 (single0017), with all six target-by-emotion gains between +0.20 and +0.39. The two variants are effectively tied on emotion transfer (single0017 is higher on sad, avg4spk higher on angry, near-identical on happy), but avg4spk preserves identity and naturalness better: mean SECS_W 0.912 versus 0.810 and mean UTMOSv2 3.23 versus 3.03. The full comparison also includes the Base row, which scores higher than both proposed variants on SECS_W (0.945 to 0.951) and UTMOSv2 (3.378 to 3.397) in every emotion, so the emotion gain is bought with some identity and naturalness loss; avg4spk’s SECS_W nonetheless stays above the human-recording ceiling row for all three emotions (0.907 vs. 0.898, 0.902 vs. 0.853, 0.926 vs. 0.914), a ceiling the authors note reflects the human speaker’s own identity shift under emotion. Normalized WER stays between 0.032 and 0.086 across proposed variants, comparable to the human-recording ceiling (0.053 to 0.069) (§4.2.1, Table 2, Figure 3).
Cross-lingual EN to PT-BR (emoUERJ m03, m04, w04; n of about 6 to 14 per combination; Table 3). With the emotion direction extracted from English ESD only, EECS rises from Base 0.724 / 0.786 / 0.855 (angry / happy / sad) to 0.877 / 0.867 / 0.902 (avg4spk) and 0.844 / 0.865 / 0.923 (single0017), a mean gain of +0.092. The smaller gain than in English is attributed to a higher starting point: the PT-BR Base already scores 0.70 to 0.91 because emoUERJ’s neutral references are less prototypically neutral and emotion2vec is less discriminative in Portuguese. avg4spk keeps SECS_W at or above about 0.88 across all nine combinations, whereas single0017 drops to 0.70 for m03/happy at α = 2.0; WER stays near zero (mean 0.006). On the xvec_cos_GT falsification test, avg4spk stays on the natural within-speaker reference line across the α grid while single0017 falls below it at high α, which the authors interpret as source-speaker timbre leakage (§4.2.2, Table 3, Figures 4 and 5).
Embedding geometry. The emotion direction is a small perturbation (about 15% of the x-vector norm; cosine between speaker 0017’s neutral and angry x-vectors 0.988) with a negligible projection onto the identity axis. Multi-speaker averaging reduces the direction norm by 36 to 60% and lowers pairwise cosines between emotion directions from [0.35, 0.72] to [0.21, 0.61] (§4.2, §5.2).
No human listening test is reported in either language, and no external emotion-control system is evaluated as a baseline; comparisons are only against the same model at α = 0 and against human recordings.
Novelty Assessment
The arithmetic itself is not new: the paper states it is operationally the same as EmoKnob’s speaker-embedding emotion direction and conceptually equivalent to the centroid arithmetic of Suni et al. on StyleTTS2. The contribution is localization and transfer. The elimination study is a useful diagnostic that weight-space task vectors, which worked in modular TTS (Task Vector in TTS, Accent Vector), did not produce controllable emotion in this LM-TTS regime, and the full_swap dissociation is a clean probe of where the model reads emotion from. Applying the known embedding arithmetic to the co-trained speaker encoder of a token-based LM-TTS answers the incompatibility with token-based architectures that Suni et al. had flagged, and the EN to PT-BR test with paired ground truth is a modest extension. The negative weight-space result is qualitative and restricted to single-speaker, single-emotion fine-tuning, which the authors themselves caveat; the evidence base (one model, automatic metrics, small PT-BR sample) is narrow.
Field Significance
Low. The paper provides diagnostic evidence that, in one codec-LM TTS with a speaker encoder co-trained for synthesis, emotional prosody is readable from and steerable through the speaker embedding rather than the backbone weights or codec tokens, and it shows that a training-free emotion direction extracted from English transfers to Portuguese target speakers. The arithmetic is borrowed, the backbone is a single model, and all evidence is automatic metrics without a listening test, so it adds a confirmatory, narrowly scoped data point rather than a new method.
Claims
- complicates: Weight-space task-vector arithmetic, which yields controllable emotion in modular TTS with explicit prosody modules, does not transfer straightforwardly to codec language-model TTS, where prosody emerges from conditioned autoregressive continuation.
Evidence: Full fine-tuning and two LoRA configurations of Qwen3-TTS on about 30 minutes of single-speaker ESD data produced only noise, base-like calm speech, or undirected expressivity with off-task laughter across learning-rate and epoch sweeps; the authors limit the conclusion to this low-variability single-speaker regime. (§4.1, Table 1; §5.1; §5.5)
- supports: In codec language-model TTS whose speaker encoder is trained jointly with the synthesis objective, the speaker embedding carries recoverable emotional prosody, not only identity.
Evidence: Combining all codec tokens of an angry reference with the neutral x-vector produced calm speech, while the angry x-vector produced angry speech; the PT-BR Base already reaches EECS 0.70 to 0.91 from reference prompts alone. (§4.1, Table 1; §5.2)
- supports: Training-free arithmetic on conditioning embeddings can provide continuous, inference-time emotion-intensity control for unseen speakers, including across languages.
Evidence: An English-derived x-vector emotion direction raises EECS over the unmanipulated model by a mean of +0.29 on held-out ESD speakers (all six combinations between +0.20 and +0.39) and +0.09 on Brazilian Portuguese emoUERJ speakers, with intensity set by a single scale α. (§4.2.1, Table 2; §4.2.2, Table 3)
- complicates: Emotion transfer by embedding steering trades off against speaker identity and naturalness, and directions estimated from a single source speaker leak that speaker’s timbre into the target.
Evidence: The unmanipulated Base scores higher SECS_W and UTMOSv2 than both steered variants in every English emotion; the single-speaker direction lowers mean SECS_W to 0.810 versus 0.912 for the four-speaker average with equal EECS gain, and drops to 0.70 on one PT-BR combination, with xvec_cos_GT falling below the within-speaker reference at high α. (§4.2.1, Table 2, Figure 3; §4.2.2, Figure 5; §5.3)
- complicates: Embedding-based emotion similarity metrics saturate when neutral references are already expressive or the emotion encoder is weak for the language, compressing measured gains.
Evidence: emotion2vec EECS saturates around 0.85 to 0.97 in PT-BR, where the Base already starts about 0.20 higher than in English, so the steered output reaches a similar absolute range as English while the measured gain falls from +0.29 to +0.09. (§4.2.2; §5.3; §5.5)
Limitations and Open Questions
Warning
All evidence is automatic (emotion2vec, WavLM speaker similarity, UTMOSv2, Whisper WER) on a single backbone, Qwen3-TTS-12Hz-1.7B-Base; there is no human listening test in either language, and Steps 1 to 3 of the elimination study are reported only as qualitative perceptual descriptions without metric values.
- Generalization to other LM-TTS systems with learnable speaker encoders is argued architecturally but not tested (§5.5).
- The negative weight-space result rests on single-speaker, single-emotion fine-tuning; larger multi-speaker, multi-emotion fine-tuning might open a usable window, which the authors explicitly do not rule out (§5.5).
- emotion2vec_plus_large is trained mostly on Chinese and English and UTMOSv2 is not trained on Portuguese, so PT-BR scores are relative within-experiment proxies; the PT-BR sample is small (about 6 to 14 utterances per combination, one speaker evaluated cross-text).
- The best α is selected per emotion, and reported results are at that selected operating point.
- No comparison against other emotion-control systems (for example instruction-trained Qwen3-TTS variants or EmoKnob) is reported.
- Encoders trained only for speaker verification, or with explicit identity-emotion disentanglement, may leave a narrower usable range of α (§5.5).
Wiki Connections
- Autoregressive Codec TTS — Localizes where emotion is carried inside a frozen codec-token autoregressive TTS and steers it without touching the backbone.
- Emotion Synthesis — Proposes training-free emotion-intensity control for neutral-only target speakers via an emotion direction in speaker-embedding space.
- Fine-Tuning — Reports that full and LoRA fine-tuning task vectors fail to yield controllable emotion in this LM-TTS under a single-speaker, single-emotion regime.
- Qwen3-TTS — The backbone whose Base model, co-trained ECAPA-TDNN speaker encoder, and 12Hz multi-codebook tokenizer are probed and steered.
- Task Vector in TTS — Prior weight-space task-vector work for dialect-emotion composition in F5-TTS that motivates the transfer question.
- Accent Vector — Prior LoRA task-vector work for accent control in XTTS-v2, part of the weight-arithmetic line this paper tests on LM-TTS.
- When Fine-Tuning Fails — Cited to explain the loss-quality divergence seen when fine-tuning LM-TTS on low-variability data in Step 1.
- SelfTTS — Contrasting approach that engineers identity-emotion separation with explicit disentanglement losses in VITS, whereas this paper exploits separation that emerges from co-training.
- IndexTTS2 — Fine-tuning-based emotion control in an AR LM-TTS, positioned as a costlier alternative with richer control.
- MiniMax-Speech — LM-TTS with a learnable speaker encoder and per-emotion LoRA, cited as the fine-tuning route to emotion control.
- ECAPA-TDNN — The speaker-encoder architecture whose x-vector is identified as the dominant emotion carrier and used as the arithmetic operand.