IJCAI · 2026 · Conference
Yihang Lin et al. (The Chinese University of Hong Kong, Shenzhen / Agency for Science, Technology and Research / National University of Singapore / Shenzhen Research Institute of Big Data / Shenzhen Loop Area Institute) · → Paper · → Preprint · Demo: ✗ · Code: ✓
Recasts emotion intensity control in an instruction-prompted LLM-based TTS model as a learning-to-rank problem, fine-tuning it with listwise preference optimization over rule-ordered lists of same-transcript samples, and releases ESD-plus, a 13-label emotion-intensity corpus built to train and test it.
Problem
Prompt-conditioned LLM-based TTS systems follow natural-language emotion categories (“happy”, “sad”) well, but a prompt that asks for “slightly” versus “extremely” happy speech often produces outputs whose intensities are hard to tell apart. The authors attribute this to a semantic-acoustic gap: intensity is realized only implicitly at the acoustic level. Earlier intensity-control methods relied on acoustic features or dedicated architectures (relative-attribute ranking, intensity embeddings, diffusion soft-label guidance), and preference-aligned emotional TTS (Emo-DPO, EMORL-TTS and related work) optimizes categorical emotion with pairwise or reward-based signals. None of these approaches tells the model that intensity levels form an ordered scale.
Method
The task: given a transcript t and a natural-language emotion prompt P naming a category c and, for non-neutral emotions, one of K ordered intensity levels, generate speech that keeps the content, matches the category, and is perceived as strictly stronger at higher levels. The weakest level must still be distinct from neutral speech.
The backbone is CosyVoice-300M-Instruct (CosyVoice 1), whose LLM autoregressively generates supervised semantic speech tokens from the prompt, text, and a speaker embedding. A flow-matching module then decodes those tokens. Training has two stages, following the Emo-DPO recipe. First, supervised fine-tuning (SFT) on prompt-speech pairs gives a reference policy π_ref. Second, the LLM is optimized with LiPO (Listwise Preference Optimization, borrowed from the text-LLM alignment literature), starting from π_ref. Figure 2 marks only the LLM as trainable. The page treats the flow-matching decoder as inherited and untouched, because the paper never describes changing it.
For each prompt, a list of K+2 candidates with the same transcript is built by rule. In order of preference, the list holds the target sample at the requested intensity, the other same-emotion intensities ranked by absolute distance from the target level (ties broken at random), one neutral sample, and one sample from a random non-target emotion. Each position i gets the label ψ(i) = 1 − (i−1)/(K+2). Each candidate’s score is the policy-to-reference log-likelihood ratio scaled by β, as in DPO. The loss sums pairwise score differences over all ordered pairs, each weighted by a λ term built from NDCG-style gain and discount functions of the list positions. Pairs that sit further apart in the intended ordering therefore get a larger penalty when they are misordered. Section 3.1 calls the list members speech samples generated by the TTS model, and Figure 2 shows both the SFT targets and the lists coming from the same dataset-construction step. The main text does not say exactly how list samples are obtained.
ESD-plus reuses 350 parallel English sentences from ESD. It keeps four non-neutral emotions (happy, surprise, sad, angry) at three intensity levels each, plus neutral, for 13 labels. All speech is synthesized with OpenAI’s gpt-4o-mini-tts in 10 voices (4 male, 6 female), with prompts that pair a description with a deliberately spaced scalar indicator (for example 1/5, 3/5, 5/5). The corpus has 45,500 samples (about 36.89 h, mean 2.92 s), split by sentence 300/20/30 for train/dev/test. Human annotators ran pairwise intensity-ordering checks on the dev and test sets only and regenerated samples that failed; 91.23% passed. The training set was not verified.
Key Results
All evaluation uses the ESD-plus test split, with the following baselines:
- CosyVoice and EmoVoice (EmoVoice), both trained with supervision only.
- Three Emo-DPO variants trained on random pairs (R), cross-emotion pairs at the same intensity (E), or cross-intensity pairs within an emotion (I).
The metrics are Whisper-Large-v3 WER, NISQA, DNSMOS and UTMOS, plus three emotion2vec-based measures: EmoSIM (embedding cosine similarity to ground truth), Recall (classification accuracy), and Recall-ft (accuracy after fine-tuning emotion2vec on ESD-plus) (§5.1).
- Emotion relevance (Table 1): among the six systems, Emo-LiPO has the highest EmoSIM (91.93, vs 91.85 for Emo-DPO (I)), Recall (27.56, vs 26.87), and Recall-ft (39.54, vs 37.21). The margins over Emo-DPO (I) are small on EmoSIM and Recall. The gains over the supervised baselines are larger (Recall-ft 29.90 for CosyVoice, 28.51 for EmoVoice). The ablation without λ has a higher EmoSIM (92.30) but lower Recall (26.21) and Recall-ft (37.59).
- Speech quality (Table 1): among the six systems, Emo-LiPO has the lowest WER (4.26, vs 4.47 for CosyVoice and 6.79 for Emo-DPO (I)). Its NISQA ties EmoVoice for the highest value (4.79). It is not best on the other quality metrics: EmoVoice has a higher DNSMOS (3.29 vs 3.26) and CosyVoice a higher UTMOS (4.30 vs 4.18). The DPO variants lose quality more clearly: Emo-DPO (R) has a WER of 12.78 and a UTMOS of 3.90. Quality is maintained rather than improved overall.
- Intensity ordering (Figure 3): Emo-LiPO has the highest Recall-ft at the low and high intensity levels and rises steadily from low to high. Emo-DPO (I) is slightly higher at mid intensity, but it peaks at mid intensity and drops at high. CosyVoice scores lowest at mid. EmoVoice and Emo-DPO (E) also rise from low to high, but from lower levels and with nearly flat low-to-mid steps. The figure is unlabelled, so no exact per-level values are given here.
- Human Arena test (Table 2): five annotators compared pairs of adjacent-intensity samples from two systems, with a tie option. Emo-LiPO’s win rates against CosyVoice, Emo-DPO (E) and Emo-DPO (I) are 94.29/79.37/89.28% for speech quality, 90.34/80.65/78.24% for emotion expression, and 86.08/66.44/58.33% for intensity control. The narrowest margin is intensity control against Emo-DPO (I). The paper does not say how ties enter the win rate.
- Ablations (Figure 5, Recall/Recall-ft):
- Number of negatives: one negative gives 27.56/39.54. Zero gives 21.15/31.23, two give 23.05/35.44 and three give 24.95/36.64.
- Ranking rule: the rule-based order beats ranking by distance in energy (24.64/37.90) or pitch (20.64/30.03).
- Weighting: index-based λ (27.56/39.54) and score-based λ (27.56/39.49) are almost equal, and both beat a fixed λ = 1 (26.21/37.59).
- Negative sampling: drawing negatives at random intensities beats restricting them to low (25.69/37.54), mid (25.74/38.79) or high (23.00/33.87) intensity.
The comparisons are fair in one sense: the DPO variants and Emo-LiPO were trained by the authors on the same data. The baselines are limited to CosyVoice-family and Emo-DPO-style systems. No other intensity-control method (EmoSphere-TTS, EmoDiff, EMORL-TTS) is evaluated, and all reported numbers come from one in-domain synthetic test set.
Novelty Assessment
The new element is the problem formulation and preference-data design. Emo-LiPO treats prompt-specified intensity as an ordinal ranking target. Its preference lists mix same-emotion intensity neighbours, a neutral anchor, and a cross-emotion negative, ordered by a fixed rule. The optimizer itself is LiPO with its standard NDCG-derived λ weighting, carried over from text-LLM alignment. The paper’s “intensity-aware” weighting is that existing λ computed from list positions; the paper does not present a new loss form. The backbone and the SFT-then-preference recipe come from CosyVoice and Emo-DPO. The contribution is therefore a new preference-data recipe for emotional TTS (listwise in place of pairwise preferences, with ordinal lists) on an existing objective and an unchanged architecture. The ablations show that the specific design choices matter: one negative, rule-based ordering, and λ weighting each help.
ESD-plus is a second contribution, but it is fully synthetic, generated by one commercial TTS system. Its intensity labels come from the prompts given to that system, so Emo-LiPO learns to reproduce how gpt-4o-mini-tts renders intensity, and is evaluated against the same renderings.
Field Significance
Moderate — This paper shows that listwise preference optimization over ordinal intensity lists improves category recognition and the low-to-high ordering of prompt-specified emotion intensity in an LLM-based TTS model, compared with pairwise DPO variants trained on the same data. It also provides a released 13-label emotion-intensity corpus. The evidence comes from a single synthetic, in-domain dataset and one 300M backbone, so it is an early data point for ranking-based preference objectives in expressive TTS rather than a general result.
Claims
- supports: Listwise preference optimization over ordered candidate lists improves prompt-specified emotion control in LLM-based TTS beyond pairwise preference optimization.
Evidence: On the ESD-plus test set, Emo-LiPO reaches Recall-ft 39.54 and Recall 27.56, against 37.21 and 26.87 for the strongest pairwise variant, Emo-DPO (I), with lower WER (4.26 vs 6.79). In the human Arena test, Emo-LiPO’s win rates against Emo-DPO (I) are 78.24% for emotion expression and 58.33% for intensity control. (§5.2, Table 1; §5.3, Table 2)
- supports: Preference data that encodes ordinal intensity structure within an emotion strengthens category-level emotion control, not only intensity control.
Evidence: Emo-DPO trained on cross-intensity pairs within an emotion (I) reaches a higher Recall-ft than the variant trained on cross-emotion pairs at a fixed intensity (E): 37.21 vs 34.92. The full listwise version raises it to 39.54. (§5.2, Table 1)
- refines: In preference optimization for emotional TTS, the way preference lists are ordered and which negatives are sampled matters as much as the loss itself, and more negatives are not better.
Evidence: One cross-emotion negative gives Recall-ft 39.54. Zero negatives give 31.23, and two or three give 35.44 and 36.64. Rule-based ordering beats ordering by pitch distance (30.03) or energy distance (37.90). Drawing negatives at random intensities beats restricting them to one level, with the high-only setting lowest at 33.87. (§5.4, Figure 5)
- complicates: Pairwise preference optimization for emotional TTS can trade intelligibility and naturalness for emotion gains, depending on how pairs are built.
Evidence: Emo-DPO trained on random pairs reaches a WER of 12.78 and a UTMOS of 3.90, against 4.47 and 4.30 for the supervised CosyVoice baseline. The cross-intensity variant also degrades, to a WER of 6.79 and a UTMOS of 4.00. Emo-LiPO keeps a WER of 4.26 but still has a lower UTMOS than CosyVoice (4.18). (§5.2, Table 1)
- complicates: Weighting preference pairs by ordinal distance helps classification-based emotion measures but not embedding-similarity measures, so emotion metrics can disagree on the same change.
Evidence: Removing λ raises EmoSIM from 91.93 to 92.30, while Recall drops from 27.56 to 26.21 and Recall-ft from 39.54 to 37.59. (§5.2, Table 1)
Limitations and Open Questions
Warning
All training and evaluation data are synthesized by one commercial TTS system (gpt-4o-mini-tts) in 10 voices. Intensity labels come from the generation prompts, and the training split was never human-verified. All automatic and human results are on the in-domain ESD-plus test set, and Recall-ft uses an emotion classifier fine-tuned on the same corpus. Nothing in the paper shows that the learned intensity control transfers to human emotional speech or to unseen speakers.
- The human Arena study uses five annotators, does not say how ties count toward win rates, and leaves out EmoVoice. The scale, platform, and guidelines are deferred to a supplementary appendix that is not part of the proceedings text.
- Training details (learning rates, number of LiPO steps, the value of β, and how candidate lists are sourced) are not given in the main paper.
- Speakers are shared between the training and test splits (the split is by sentence), so the results do not test zero-shot speaker generalization.
- Only one backbone (CosyVoice-300M-Instruct) is tested. Whether the listwise gains hold for larger LLM-based TTS models or for CosyVoice 2’s architecture is open.
- No other intensity-control approach (spherical emotion vectors, diffusion soft-label guidance, RL with emotion rewards) is compared directly.
- Three discrete intensity levels are modelled. Continuous intensity, or intensities finer than the prompt’s three steps, are not evaluated.
Wiki Connections
- RLHF Speech — Applies LiPO, a listwise learning-to-rank preference objective, to an LLM-based TTS model and compares it with three pairwise DPO variants trained on the same data.
- Emotion Synthesis — Targets fine-grained, prompt-specified emotion intensity and measures both category recognition and low-to-high intensity ordering of the generated speech.
- Instruction-Conditioned TTS — The evaluated control signal is a natural-language prompt naming an emotion and an intensity level (“Extremely happy”) fed to an instruction-tuned TTS backbone.
- Autoregressive Codec TTS — The fine-tuned component is CosyVoice’s LLM, which autoregressively generates supervised semantic speech tokens; the flow-matching decoder is inherited unchanged.
- Subjective Evaluation — Uses a five-annotator Arena-style pairwise test on adjacent-intensity sample pairs across speech quality, emotion expression, and intensity control.
- CosyVoice — Supplies the CosyVoice-300M-Instruct backbone that Emo-LiPO fine-tunes, and serves as the supervised baseline.
- EmoVoice — Supervised freestyle-prompt emotional TTS baseline; it has strong DNSMOS and NISQA but the lowest emotion Recall in Table 1.
- EMORL-TTS — Reinforcement-learning approach to fine-grained emotion control in LLM-based TTS that the paper cites as related preference-based work but does not evaluate against.
- ARDM-DPO — Pairwise DPO for speech autoregressive diffusion models, cited as prior pairwise preference work in emotional TTS.
- emotion2vec — Provides the speech emotion recognizer behind the EmoSIM, Recall, and Recall-ft metrics.
- UTMOS — One of three automatic naturalness predictors used to show that quality is maintained.
- Whisper — Whisper-Large-v3 computes the WER intelligibility metric.