arXiv · 2026 · Preprint
Kun Zhou et al. (Alibaba Group) · → Paper · Demo: ✓ · Code: ?
Introduces an LM-based TTS framework that conditions synthesis on a continuous, psychologically grounded pleasure-arousal-dominance (PAD) emotion vector, learned from categorical emotion labels via a separately trained predictor, so the TTS backbone itself never needs emotion-labeled training data.
Problem
Emotional TTS systems are constrained by the narrow, fixed set of categorical emotion labels available in existing speech datasets, even though psychological research suggests human emotional experience spans a far broader and more continuous range. Prior work follows two paradigms — supervised training on categorical labels, or style transfer from reference speech — both bounded by the limited diversity of labeled emotional corpora, which yields synthesized speech that sounds average or stereotypical. More recent hierarchical or text-driven emotion control methods improve fine-grained expressiveness but remain fundamentally dependent on labeled emotion data. LM-based TTS systems such as CosyVoice and EmoCtrl-TTS address expressiveness by training on thousands of hours of emotional speech, which is costly to collect and annotate, making it hard to train a robust LM-based system from limited or unlabeled emotional data.
Method
The system adopts an LM-based TTS backbone similar to CosyVoice: an autoregressive LM decoder generates discrete acoustic tokens from text, an optimal-transport conditional flow-matching (OT-CFM) module converts these tokens to Mel spectrograms, and a HiFi-GAN vocoder synthesizes the waveform. The speech tokenizer is an ESPnet Conformer ASR model with a vector quantizer (codebook size 4096) inserted after the first 6 encoder layers, following the CosyVoice-style supervised semantic token design. The text encoder has 6 transformer layers (8-head, 512-d attention) and the LM has 12 transformer layers (8-head, 512-d attention).
The core contribution is an Emotional Dimension (ED) predictor that maps categorical emotion labels into a continuous pleasure-arousal-dominance (PAD) space, grounded in Russell’s three-factor emotion theory. Each of the 151 emotion categories from Russell’s psychological study is assigned a PAD anchor from prior human-rated measurements. The ED predictor extracts features from emotional speech with a pre-trained WavLM encoder, passes them through a linear layer and classification layer, derives a 128-d emotional feature vector, and fine-tunes ED vectors from their anchor initialization using a k-nearest-neighbor graph plus UMAP dimensionality reduction that minimizes cross-entropy loss between high-dimensional features and the ED vectors.
During TTS training, the ED predictor is frozen and used only to extract PAD vectors from prompt speech; the TTS LM itself is trained with a teacher-forcing cross-entropy objective on speech token prediction and never sees explicit emotion labels. At inference, the ED vector can either be inferred from a prompt utterance (“emotion cloning,” for zero-shot speaker and emotion cloning) or directly specified by the user (“emotion control,” for synthesizing named PAD-anchored emotions such as those in Table 1, including ones absent from the TTS training data). The PAD vector is concatenated with the speaker embedding (x-vector) from a pre-trained voiceprint model and fed to the LM alongside the text encoding.
Key Results
The ED predictor reaches 81% emotion classification accuracy on the ESD English test set, comparable to reported SER baselines on that dataset (§3.1). For zero-shot emotion cloning naturalness, a MOS listening test on LibriTTS test-clean gives ground truth 4.80±0.08, the proposed framework 4.54±0.18, and a CosyVoice baseline trained from scratch on the same data 4.36±0.13 (Table 2) — the proposed system outperforms the CosyVoice baseline but trails ground truth. On emotional intelligibility (E-MOS), the proposed framework consistently outperforms CosyVoice; a third system, MixedEmotion, which is trained directly on emotional data, attains the best score among the three synthesis systems but still falls short of ground truth (Figure 3, values not tabulated in text). Objective acoustic analysis shows pitch and spectral flux patterns for 9 PAD-controlled emotions align with expected emotion-arousal correlates from established theory (§3.3, Figure 2). In XAB tests on four dimension-matched emotion pairs (e.g., “Angry” vs. “Anxious,” distinguished mainly by Dominance), correct-match accuracy exceeds 50% for all pairs, reaching approximately 84% for “Angry vs. Anxious” (§3.4, Figure 4).
Novelty Assessment
The backbone architecture (AR LM + flow matching + HiFi-GAN, supervised semantic speech tokens) is adapted directly from CosyVoice-style systems rather than newly proposed. The genuinely new contribution is the ED predictor: an anchored-dimensionality-reduction mechanism that converts categorical emotion labels into a continuous, psychologically grounded PAD embedding space, decoupled from the TTS training pipeline so the TTS backbone itself trains without any emotion labels. This addresses a real practical constraint (scarcity of labeled emotional speech) with a lightweight, modular mechanism rather than a new end-to-end architecture. The evaluation is comparatively small in scale (single training corpus for the TTS backbone, one baseline for MOS, two baselines for E-MOS), so the contribution is best read as a focused conditioning-mechanism paper rather than a new TTS architecture.
Field Significance
Moderate — this paper contributes a modular, label-efficient mechanism for continuous emotion control in LM-based TTS, built on an existing CosyVoice-style backbone rather than a new architecture. It demonstrates that emotional expressiveness can be decoupled from explicit emotion labels in the main TTS training data, and grounds emotion control in an established psychological dimensional model (PAD) rather than an ad hoc learned space.
Claims
- supports: A predictor that maps categorical emotion labels into a continuous, psychologically grounded dimensional space can supply conditioning signals for an LM-based TTS system that itself trains without any explicit emotion labels.
Evidence: The ED predictor is trained separately on the ESD English subset (~10h, 5 emotions) via WavLM features and anchored dimensionality reduction, reaching 81% classification accuracy on the ESD test set, while the TTS LM trains only on LibriTTS (~600h), which contains no emotion labels (§2.2, §3.1).
- supports: Conditioning zero-shot LM-based TTS on continuous dimensional emotion embeddings, rather than categorical labels, allows synthesis of emotions not present as discrete categories in the TTS training data.
Evidence: In the “Emotion Control” setting, PAD values for 9 named emotions (Table 1) absent from LibriTTS training labels are manually specified, and the resulting synthesized speech shows pitch and spectral-flux patterns consistent with established emotion-arousal correlates (§3.3, Figure 2).
- complicates: A zero-shot, label-free emotion-conditioned TTS system trained on unlabeled expressive speech can trail a system trained directly on labeled emotional data in emotional intelligibility, even while surpassing a same-setting zero-shot baseline.
Evidence: In the E-MOS listening test, MixedEmotion (trained on emotional data) attains the highest score among the three compared systems, ahead of both the proposed zero-shot framework and the CosyVoice zero-shot baseline, though all remain below ground truth (§3.4, Figure 3).
- supports: Confusability between synthesized emotion pairs correlates with how many of the three PAD dimensions the pairs share, with pairs differing on only one dimension harder to disambiguate than pairs differing on more.
Evidence: XAB test accuracy across four dimension-matched pairs (e.g., “Angry” vs. “Anxious,” differing mainly in Dominance) stays above 50% for all pairs but is lowest for the most dimensionally similar pairs, consistent with Russell’s three-factor theory (§3.4, Figure 4).
Limitations and Open Questions
The subjective evaluation is modest in scale: 10 native English listeners rated 260 synthesized samples in total, and the paper itself characterizes this as standard practice rather than a large-scale study. E-MOS results for emotional intelligibility are reported only as a figure without exact numeric values in the text, limiting the precision of any downstream comparison. The TTS backbone is trained on a single English corpus (LibriTTS), so cross-lingual or cross-domain generalization of the ED conditioning mechanism is untested. The paper compares against only two baselines (CosyVoice, MixedEmotion) rather than other explicit dimensional-emotion TTS systems it cites in related work (e.g., EmoCtrl-TTS, EmoSphere-TTS). The authors note dynamic within-utterance emotion control and multilingual adaptation as open future directions, not addressed in this work (§4).
Wiki Connections
- Emotion Synthesis — introduces a continuous, psychologically grounded PAD conditioning mechanism for LM-based TTS that avoids reliance on categorical emotion labels during TTS training.
- Zero-Shot TTS — performs zero-shot speaker and emotion cloning from prompt speech, extending zero-shot conditioning to the emotional dimension alongside speaker identity.
- Autoregressive Codec TTS — uses an autoregressive LM decoder over discrete supervised speech tokens as the core text-to-token generation mechanism.
- Flow Matching — uses an optimal-transport conditional flow-matching module to convert generated speech tokens into Mel spectrograms.
- Neural Audio Codec — relies on a supervised VQ-based speech tokenizer (ESPnet Conformer encoder with codebook 4096) to discretize speech for the LM.
- CosyVoice — the TTS backbone (AR LM + flow matching + HiFi-GAN, supervised semantic tokens) is adapted directly from this system, which also serves as the zero-shot cloning baseline trained from scratch on the same data.
- VALL-E — cited as a foundational LM-based TTS framework demonstrating in-context learning and zero-shot speaker cloning that this paper builds its LM-decoder paradigm on.
- VALL-X — cited alongside VALL-E as evidence of strong in-context learning capabilities in LM-based TTS that motivate the LM-decoder design choice.
- BASE TTS — cited as an example of scaling LM-based TTS to large data volumes, illustrating the data cost this paper’s label-efficient ED conditioning aims to reduce.
- LibriTTS — used as the TTS training and evaluation corpus (train-clean + train-other for training, test-clean for evaluation).