arXiv · 2026 · Preprint

Hongfei Du et al. (William & Mary) · → Paper · Demo: ✓ · Code: ?

Trains a Top-k sparse autoencoder on the residual stream of a frozen autoregressive TTS backbone (IndexTTS2) and shows that adding or subtracting a small, selectivity-ranked set of its latent features induces or suppresses a target emotion at inference time, with no change to backbone weights.

Problem

Emotion control in LLM-based TTS is usually exercised through external conditioning: discrete labels, natural-language prompts, or a reference clip. These interfaces are opaque, fix the expression once the inputs are chosen, and give no view of how emotion is represented inside the model. Training-free activation steering, as in EmoSteer-TTS, modifies hidden states directly, but it steers along a single dense mean-difference direction in a diffusion-transformer acoustic stage, which captures a global shift rather than separable feature-level factors. The paper asks whether emotion-related variation can instead be decomposed into sparse, interpretable features upstream, in the autoregressive semantic backbone, before acoustic synthesis.

Method

The host system is IndexTTS2, used entirely frozen: a GPT-style autoregressive transformer that generates discrete semantic tokens from text plus an emotion style reference and a speaker timbre reference, followed by a frozen conditional flow-matching (CFM) module and vocoder. The authors intercept the layer-16 pre-LayerNorm residual stream (d = 1280) at each decode-phase semantic-token position and train a k-sparse (Top-k) autoencoder on these activations: a 1280 → 4096 → 1280 network (about 10.5M parameters, n = 4096 latents, k = 32 active per token), with a linear decoder whose unit-norm columns act as residual-space directions. The objective is reconstruction MSE plus an auxiliary loss that has selected inactive latents model the residual error (λ_aux = 0.1), and a usage-based sparsity regulariser (weight 0.01) to prevent dead latents (Appendix B). SAE training data are activations from 56,000 IndexTTS2 generations: 400 English texts × 7 emotions × 20 IEMOCAP timbre references, each using an emotion-specific IEMOCAP style reference. Training runs 30,000 steps on one H100 and is a one-time offline step.

Figure 2. Overview of SAE-based emotion modulation in an LLM-based TTS model. The semantic backbone generates token-level hidden representations conditioned on text and optional audio references. A sparse autoencoder (SAE) maps residual-stream activations to sparse latent activations. Selected emotion-related latent features are modulated through the intervention path and decoded back into the residual stream. The resulting representations are passed to the CFM module and vocoder for acoustic synthesis.

Emotion-related latents are identified with a paired, sentence-level selectivity score: for matched text and speaker, a latent counts as active in an utterance if it fires at any generated token, and its score is the mean difference in this indicator between the target-emotion sample and the matched neutral sample (computed over a separate 43,408-generation analysis set covering happiness, anger, sadness, and neutral). The top m = 6 latents per emotion form the steering set. At inference, the residual state is encoded, a scalar α_e is added to the activations of the selected latents (positive to induce, negative to suppress), and the result is decoded back, which approximates adding α_e times the sum of the selected decoder directions to the residual. The same α_e is applied at every token position. Two steering baselines share this formulation: Global Steering (dense mean-difference vector in residual space) and Random SAE (six uniformly random latents).

Key Results

Evaluation uses a paired set of 100 matched text-speaker cases per emotion (anger, happiness, sadness). Emo-SIM is cosine similarity in emotion2vec space to an IEMOCAP emotion prototype (to the neutral prototype for suppression), WER uses Whisper-Large V3, and Spk-SIM uses ERes2Net between the neutral and emotion-controlled generations.

Induction (neutral → target), Table 1. SAE-Emotion has the highest Emo-SIM for anger (0.912 vs. 0.910 Global Steering) and happiness (0.885 vs. 0.879), margins of under 0.01. For sadness it is second (0.880), behind Spark-TTS (0.907). WER is lowest among all seven systems for happiness (2.2 vs. 2.9 CosyVoice) and sadness (1.5 vs. 1.7 Random SAE), and second for anger (0.3 vs. 0.1 Global Steering). Speaker similarity is where the method trails: for anger it ties CosyVoice at 0.569 behind Random SAE (0.628); for happiness it is second (0.515, behind CosyVoice 0.597); for sadness it is fifth of seven (0.481), behind Random SAE (0.637), CosyVoice (0.605), Spark-TTS (0.523), and Global Steering (0.516).

Suppression (target → neutral), Table 1. Only the three steering methods are compared here, because the TTS baselines have no suppression mechanism. SAE-Emotion has the highest neutral Emo-SIM for all three emotions (0.939 vs. 0.915; 0.924 vs. 0.920; 0.941 vs. 0.939), again by small margins for happiness and sadness. It has the highest WER of the three for anger (2.8 vs. 2.6 and 0.8) and happiness (2.31 vs. 1.48 and 2.14), and the lowest Spk-SIM for happiness (0.301 vs. 0.379 and 0.343); its Spk-SIM is highest for sadness (0.441 vs. 0.436 and 0.427). The paper’s summary that WER and speaker similarity are “generally comparable” holds loosely; in suppression the method is the weakest of the three on content preservation for two of three emotions.

Human evaluation, Table 2. In a blind test with 20 raters (0-5 scale) comparing only the three steering methods, SAE-Emotion scores EMOS 3.22 and NMOS 3.49, against 3.10 / 3.38 for Global Steering and 1.82 / 3.22 for Random SAE. No confidence intervals or significance tests are reported, and the TTS baselines are not included.

Analyses. Selectivity scores are concentrated near zero with a sparse positive tail (Figure 3); each of the top-6 anger latents has selectivity 1.0. Steering one latent raises mean F0 by 23.11 Hz (p = 1.07 × 10⁻⁴) and RMS energy by 0.00435 (p = 0.00769) with no significant duration change (p = 0.687) (Table 4), and shifts spectral centroid monotonically with α over -60 to +60 (Figure 5). The sentence-level selectivity criterion gives higher induction Emo-SIM than magnitude-based or token-level selection for all three emotions (e.g., anger 0.912 vs. 0.822 / 0.825; Table 3). Under a high steering scale, SAE steering keeps mean WER at 0.57% with zero deletions versus 2.86% WER and up to 14.89% deletions for Global Steering (Table 5). A cross-backbone check on LLaSA shows a monotonic shift toward the anger prototype with steering scale (Figure 13, qualitative only).

Fairness. The TTS baselines (VALL-E X, Spark-TTS, EmoVoice, CosyVoice) are run with their own emotion-instruction interfaces and default settings, so the comparison mixes different backbones and control mechanisms; the unsteered IndexTTS2 backbone driven by its own emotion reference, the most direct reference point, is not reported in Table 1.

Novelty Assessment

Sparse autoencoders and SAE feature steering are established in LLM interpretability, and residual-space activation steering for emotional TTS already exists in EmoSteer-TTS. What is new is moving both to the autoregressive semantic backbone of an LLM-based TTS system, and the paired sentence-level selectivity score for picking emotion latents, which the paper ablates against two alternatives. The intervention itself is a standard SAE clamp-and-decode operation. The interpretability results (sparse heavy-tailed selectivity, individual latents with measurable F0, energy, and brightness effects, non-overlapping top-6 sets across emotions) are the more distinctive contribution, though they come from one backbone, three emotions, and a single detailed latent. Quantitatively, the gain over dense Global Steering is small in Emo-SIM; the clearer advantages are in human EMOS/NMOS and in WER under strong steering.

Field Significance

Moderate. The paper demonstrates that emotion-related variation in an autoregressive TTS semantic backbone can be isolated in a handful of sparse autoencoder features and used for bidirectional, training-free emotion control, giving a feature-level and inspectable alternative to dense activation steering. Its evidence is concentrated on one frozen backbone and three emotions, so it is best read as a proof of concept for SAE-based interpretability in speech generation rather than a new state of emotion control.

Claims

  • supports: Emotion-related variation in the hidden states of autoregressive TTS language models is concentrated in a small subset of sparse dictionary features rather than spread evenly across the representation.

    Evidence: Paired emotion-vs-neutral selectivity scores over 4,096 SAE latents are centred near zero with a sparse positive tail for anger, happiness, and sadness; each of the top-6 anger latents fires in every anger sample and no matched neutral sample, and the top-6 sets do not overlap across emotions. (§4.2.1, Figure 3; Appendix D, Figure 8; §D.3)

  • supports: Inference-time intervention on internal representations of a frozen TTS model can both induce and suppress a target emotion without retraining the backbone.

    Evidence: Adding or subtracting α_e on six selectivity-ranked SAE latents in frozen IndexTTS2 yields induction Emo-SIM of 0.912 / 0.885 / 0.880 and suppression (neutral) Emo-SIM of 0.939 / 0.924 / 0.941 for anger / happiness / sadness, and EMOS 3.22 vs. 1.82 for random latents in a 20-rater test. (§4.3, Tables 1-2)

  • refines: Sparse-feature steering degrades content less than dense mean-difference steering at high intervention strength, though the two are close in emotion strength at moderate strength.

    Evidence: At a high steering scale, SAE steering gives 0.57% mean WER and 0% deletions vs. 2.86% WER and 14.89% maximum deletion for Global Steering; at the main operating point, induction Emo-SIM differs by under 0.01 for anger and happiness, and in suppression SAE steering has higher WER than Global Steering for anger and happiness. (§F, Table 5; §4.3, Table 1)

  • complicates: Representation steering for emotion trades off against speaker-identity preservation, and a feature set selected for emotion does not necessarily preserve speaker similarity better than random or dense interventions.

    Evidence: In induction, SAE-Emotion’s Spk-SIM is 0.481 for sadness vs. 0.637 for Random SAE and 0.516 for Global Steering, and in suppression it is lowest for happiness (0.301 vs. 0.379 and 0.343); the paper notes ERes2Net similarity is itself sensitive to pitch and energy changes. (§4.1; §4.3, Table 1)

  • refines: Individual internal features of a TTS language model can map onto specific acoustic correlates of emotion, such as pitch, energy, and spectral brightness, while leaving timing unchanged.

    Evidence: Steering one selected latent raises mean F0 by 23.11 Hz (p = 1.07 × 10⁻⁴) and RMS energy by 0.00435 (p = 0.00769) with no significant duration change (p = 0.687), and shifts spectral centroid monotonically across scales from -60 to +60. (§4.2.2, Figure 5; Appendix E, Table 4)

Limitations and Open Questions

All quantitative results come from a single frozen backbone (IndexTTS2) with three target emotions, evaluated on generations whose reference audio is drawn from IEMOCAP. The cross-backbone check on LLaSA is a single qualitative figure for anger only, so generality across TTS architectures is not established.

  • The comparison against existing TTS systems is uneven: baselines use their own instruction interfaces and default inference settings, only the induction direction is compared, the unsteered IndexTTS2 with an emotion reference is not reported, and the human evaluation includes only the three steering variants.
  • Margins over Global Steering in Emo-SIM are small (under 0.01 in several cells), and no variance, confidence intervals, or significance tests are reported for Table 1 or Table 2.
  • Emo-SIM uses emotion2vec embeddings and the same IEMOCAP-derived prototypes that shape the analysis data, which may favour interventions aligned with that emotion space.
  • The acoustic analyses (F0, energy, spectral centroid, prototype alignment) characterise one or two individually chosen latents; the paper notes that a systematic characterisation of the latent space is left open.
  • A 10,240-latent SAE produced more fragmented emotion features; this sensitivity to dictionary size is reported only qualitatively, and the sparsity level k is not ablated.
  • The SAE, activation extraction, and analysis set require large-scale generation (about 100k utterances), which the authors identify as a computational barrier to extending the method to other backbones.

Wiki Connections

  • Emotion Synthesis — Provides a training-free, feature-level method for both inducing and suppressing emotion in an LLM-based TTS system, evaluated with emotion2vec similarity and human EMOS.
  • Autoregressive Codec TTS — The steered component is the frozen autoregressive semantic-token transformer of IndexTTS2 (with a LLaSA codec-LM check), and the paper analyses its internal representations rather than modifying it.
  • Flow Matching — The evaluated system renders steered semantic tokens through IndexTTS2’s frozen conditional flow-matching module and vocoder; the paper deliberately intervenes upstream of this stage.
  • Subjective Evaluation — Reports a 20-rater blind listening test with emotion (EMOS) and naturalness (NMOS) scores for the three steering methods.
  • IndexTTS2 — Frozen backbone whose layer-16 residual stream the SAE is trained on and steered through.
  • EmoSteer-TTS — Prior training-free activation steering for emotional TTS in a diffusion transformer; this paper contrasts dense mean-difference steering with sparse SAE features and moves the intervention upstream.
  • LLaSA — Second backbone used for a cross-architecture check of scale-dependent emotion alignment.
  • Spark-TTS — Instruction-driven baseline that has the highest induction Emo-SIM for sadness, ahead of SAE-Emotion.
  • CosyVoice — Baseline with the strongest speaker similarity among non-steering systems in the induction comparison.
  • EmoVoice — Freestyle-prompt emotional TTS baseline in the induction comparison.
  • VALL-E X — Codec-language-model baseline in the induction comparison.
  • emotion2vec — Embedding space used to compute the Emo-SIM metric against IEMOCAP emotion prototypes.