EMORL-TTS: Reinforcement Learning for Fine-Grained Emotion Control in LLM-based TTS

ICASSP · 2026 · Conference

Haoxun Li et al. (Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences) · → Paper · Demo: ✓ · Code: ?

EMORL-TTS applies reinforcement learning (GRPO) on top of a frozen LLM-based TTS model to achieve fine-grained, continuously controllable emotion intensity and prosodic emphasis in a discrete-token speech generation framework.

Problem

LLM-based TTS systems (such as SparkTTS and CosyVoice2) achieve strong naturalness and zero-shot generalization, but operate over discrete speech tokens rather than continuous representations. This discretization makes it structurally difficult to model continuous emotion intensity or fine-grained prosodic prominence — signals that are inherently graded and subtle. Prior approaches to emotional TTS either confined emotion to categorical labels (happy, sad, angry) or relied on non-LLM architectures that exploit continuous latent spaces, and therefore could not be directly adapted to the autoregressive-LM paradigm. The gap is: how to endow a frozen, token-predicting LLM with controllable intensity gradations and local emphasis, without redesigning its token representation.

Method

The framework, EMORL-TTS, is built on top of SparkTTS, an LLM-based TTS model whose BiCodec jointly encodes global acoustic identity and semantic content into discrete tokens. The BiCodec weights are frozen throughout post-training; only the LLM parameters are updated.

Training proceeds in two stages. In Stage I (Supervised Fine-Tuning), the LLM is conditioned on two additional control tokens prepended to the text: an emotion-category token (angry, happy, sad, surprise, neutral) and a discretized intensity token (weak / medium / strong). Intensity labels are derived offline from a pretrained VAD estimator: each utterance’s Euclidean distance from a neutral centroid in Valence-Arousal-Dominance space is computed and binned per emotion category. The model is trained with standard token-level cross-entropy, establishing basic emotion-category control and calibrating the intensity interface.

In Stage II (GRPO), the problem is cast as a sequential decision process: the LLM policy samples K=16 candidate token sequences per prompt, and three task-aligned reward terms are computed per candidate.

• Emotion Classification Reward (R_ser): an emotion2vec-based speech emotion recognizer checks whether the generated speech matches the intended category. Correct category yields +5, incorrect −1, with a higher relative weight to anchor category control learned during SFT.
• Global Emotion Intensity Reward (R_int): the pretrained VAD predictor measures the intensity of generated speech and computes both a hard bin-match term and a soft Gaussian penalty centered on the target bin midpoint.
• Local Emphasis Control Reward (R_emp): NeMo Forced Aligner extracts word boundaries; pitch (max log-F0) and energy (mean STFT norm) features are computed per word. Hard indicators check whether emphasized words are pitch/energy maxima; soft clipped z-score terms provide gradient signal. The composite reward R = R_ser + R_int + R_emp is optimized via GRPO with a KL penalty anchoring the policy to the SFT checkpoint (β=0.1).

Global prosody control targets overall emotional intensity in the three-dimensional VAD space; local prosody control targets emphasis placement through prosodic prominence (pitch and energy peaks at marked word positions). The model is trained on 8 NVIDIA RTX 4090 GPUs. No model size is reported. Synthesized demo samples are available online.

Key Results

Emotion accuracy (Table 1, objective via emotion2vec on 500 samples): EMORL-TTS achieves a mean accuracy of 0.88 versus 0.85 for EmoSphere++ and 0.63 for CosyVoice2. The ablation without GRPO (SFT only) scores 0.81, confirming that the RL stage is responsible for the accuracy gains. Subjective emotion accuracy (Table 2, 30 listeners): EMORL-TTS scores 0.89 mean, compared to 0.74 for EmoSphere++ and 0.55 for CosyVoice2, again with the RL stage contributing substantially.

Emotion intensity distinguishability (Table 3, pairwise forced-choice): EMORL-TTS leads EmoSphere++ and Relative Attribute baselines across four emotion categories and three intensity comparisons, with an overall average of 0.71/0.65/0.72 for weak<medium, medium<strong, weak<strong.

Emphasis recognition accuracy (Table 4): EMORL-TTS achieves 0.75 mean versus 0.73 for EME-TTS and 0.35 for CosyVoice2, with particularly strong gains on the Angry category (0.92 vs 0.70).

Speech naturalness (Table 5): MOS of 4.94 for EMORL-TTS, matching SparkTTS (4.96) and CosyVoice2 (4.96) and surpassing EmoSphere++ (4.24). NISQA scores follow the same pattern (4.11 vs 4.15/4.14 for baselines and 3.78 for EmoSphere++). The RL stage does not degrade naturalness compared to the SFT-only ablation.

Part-of-speech emphasis analysis (POSET task): emphasis on adverbs produces the strongest perceived emotional enhancement, followed by adjectives; verbs, nouns, and other categories have weaker effects.

Novelty Assessment

The primary contributions are methodological: applying GRPO (a variant of PPO popularised for LLM reasoning by DeepSeek-R1) to the speech synthesis token-prediction task, and constructing a multi-objective reward that jointly covers emotion category, VAD-based global intensity, and prosodic prominence. The use of VAD-space intensity control in an LLM-based TTS system is presented as a first. The architectural choice — building on SparkTTS with a frozen BiCodec — is incremental relative to SparkTTS itself; the novelty is in the post-training recipe. The emphasis reward design (NFA-derived forced alignment + pitch/energy prominence) is a reasonable engineering choice but not deeply novel in isolation. The integration of all three reward terms and the demonstration that RL does not damage naturalness (MOS parity with strong LLM baselines) is the paper’s strongest empirical claim.

Field Significance

Moderate — EMORL-TTS demonstrates that reinforcement learning with task-specific rewards can close the gap between categorical emotion labels and continuously graded intensity control in LLM-based TTS, without degrading naturalness. The post-training recipe (SFT followed by GRPO with multi-objective rewards) provides a template for adding fine-grained prosodic control to frozen LLM-TTS backbones. The contribution is primarily a novel training procedure applied to an existing architecture, with practical value for expressive synthesis research.

Claims

• Reinforcement learning with task-specific rewards can improve fine-grained emotion controllability in LLM-based TTS without degrading speech naturalness. (§2.3, §3, Table 5)
• VAD-space intensity modeling enables continuously graded emotion intensity control in discrete-token speech generation. (§2.2, §2.3, Table 3)
• Prosodic emphasis (pitch and energy prominence at marked word positions) can be reliably controlled in autoregressive LLM-based TTS via reward-guided post-training. (§2.3, Table 4)
• Emphasis placement on adverbs and adjectives produces stronger perceived emotional intensity than emphasis on verbs, nouns, or other categories. (§3.2, Figure 2)

Limitations and Open Questions

The system is English-only; cross-lingual extension is noted as future work. The intensity discretization (weak/medium/strong) reduces continuous VAD distances to three bins, which may limit expressiveness at fine granularity. Training data volume is modest (ESD: ~12 hours across 10 speakers; Expresso: ~4,717 utterances), so generalization to diverse speaker identities and unseen emotion categories is uncertain. The GRPO text corpus (1,000 internet sentences with randomly assigned emphasis) is synthetic, and whether random emphasis placement covers realistic prosodic patterns is an open question. The paper does not evaluate speaker similarity or intelligibility (WER/CER), so trade-offs along those dimensions are unknown. Instruction-based controllability (as opposed to structured category+intensity tokens) is listed as future work, which would bring the system closer to natural-language-directed expressive synthesis.

Wiki Connections

• emotion-synthesis — core target capability; the paper advances fine-grained intensity control beyond categorical labels
• prosody-control — the local emphasis mechanism directly addresses word-level prosodic prominence
• rlhf-speech — first reported application of GRPO-based RL to emotion and emphasis control in LLM-based TTS
• autoregressive-codec-tts — EMORL-TTS is built on SparkTTS, an autoregressive LLM operating over BiCodec discrete tokens
• neural-codec — BiCodec (SparkTTS’s joint global/semantic codec) is the speech representation backbone
• EmoSteerTTS — in-corpus citation; activation-steering-based approach to emotion control, contrasted as a training-free alternative