ACL 2026 (Findings) · 2026 · Conference

Yue Zhao et al. (Geely AI Lab) · → Paper · → Preprint · Demo: ✓ · Code: ✓

Trains a small LLM with value-based reinforcement learning to pick the agent’s emotion before the text response is written, so that an off-the-shelf emotional TTS can start synthesizing in streaming mode with the emotion already fixed.

Problem

Cascaded voice agents stream LLM text into TTS token by token, and an emotional TTS needs its emotion label before synthesis starts. Emotion recognition in conversation (ERC) labels a response only after it is fully written, so it cannot drive streaming emotional TTS. Emotion prediction in conversation (EPC) and prompting methods that decode the emotion first can produce the label early, but the authors argue that prompting plans poorly without parameter updates and that EPC only imitates dataset labels without optimizing the conversation over several turns.

Method

Self-EmoQ treats the agent’s emotion as an action in an utterance-level Markov decision process. The state is the dialogue description, the history (with past emotions) and the current user query; the action is one emotion from a fixed set (7 classes on DailyDialog, EmoryNLP and MELD; 10 on IEMOCAP). A frozen Llama3.1-8B-Instruct generator then writes the response conditioned on that emotion.

The planner is a 1B Llama model (named Llama3.1-1B-Instruct in the paper) prompted with a multiple-choice template that lists the candidate emotions. Following the authors’ earlier StraQ* recipe, the Q-value of each emotion is the average log-probability of the output tokens for that option. The model is trained as a Deep Q-Network: epsilon-greedy exploration, a replay buffer, a target network synchronized every C steps, and a Bellman (temporal-difference) loss with discount factor 0.8.

The reward mixes two terms with weight w: an imitation reward (1 if the chosen emotion matches the dataset label) and a Plutchik score. GPT-4o computes the Plutchik score by rating the response from 0 to 5 on emotion alignment, plausibility of the emotion transition on Plutchik’s wheel (adjacent emotions plausible, opposite ones not), and consistency with the emotion’s typical behavioural function. The Plutchik score is the average of the three ratings.

At inference the planner takes the argmax-Q emotion and injects it into the generator’s prompt. The same label conditions an emotional TTS. In the downstream experiment this is CosyVoice 2 in its “instruct inference” mode, used without training (the paper calls it “Emo-TTS”). Because the label exists before any text is decoded, the TTS can stream partial text with the emotion fixed (Setting C in the paper’s Figure 1). The paper describes this streaming design but does not measure latency.

Key Results

Emotion determination is scored by ranking the candidate emotions by Q-value against the reward signal (Reward, Recall@3/5, NDCG, MRR). Response quality is scored with BLEU-2, ROUGE-L, Distinct-2 and CIDEr against the dataset response. The baselines are prompting methods (0-shot, ECoT, Plan-and-Solve, Metacognitive Prompting), supervised ones (SFT, FSM), and tabular Q-learning (EMDP).

  • Emotion determination (Table 3). Self-EmoQ has the highest Reward on all four datasets: 0.57 vs. 0.55 for SFT on DailyDialog, 0.71 vs. 0.70 for FSM on EmoryNLP, 0.86 vs. 0.84 for FSM on MELD, and 0.81 vs. 0.73 for FSM on IEMOCAP. It also has the highest R@5 on every dataset. It loses on MRR on three datasets: EmoryNLP (0.50 vs. 0.51 for SFT and FSM), MELD (0.54 vs. 0.59 for SFT and FSM) and IEMOCAP (0.45 vs. 0.54 for FSM, 0.53 for SFT, 0.48 for PS and 0.47 for 0-shot). It also loses on DailyDialog R@3 (0.82 vs. 0.83 for EMDP) and ties SFT on EmoryNLP NDCG (0.83).
  • Response generation (Table 4). Self-EmoQ has the highest BLEU-2 and ROUGE-L on all four datasets, and the highest CIDEr on DailyDialog, EmoryNLP and IEMOCAP. On MELD its CIDEr (9.32) is below FSM (13.38) and SFT (10.48). Its Distinct-2 is below the prompting baselines on EmoryNLP (16.82 vs. 60.62 for 0-shot) and MELD (9.86 vs. 52.17 for 0-shot, and below SFT and FSM). This contradicts the paper’s statement that Self-EmoQ scores better on all four metrics.
  • Human evaluation of text responses (Table 5). Ten interns rated responses from 50 sampled sessions per test set on a 1-5 scale. Self-EmoQ has the highest mean on Fluency (3.58), Emotion (3.74), Acceptance (3.11), Sensitivity (3.21), Alignment (3.22) and Satisfaction (3.31). On Effectiveness it scores 3.01, below FSM (3.05). Cohen’s kappa between annotators on Fluency is 0.76-0.85 (Table 10).
  • Downstream speech (Table 7). Speech from CosyVoice 2 is scored with SpeechBERTScore and PESQ against a reference utterance. Self-EmoQ’s Setting B and Setting C rows are identical (e.g. DailyDialog 0.54 / 1.28), which is consistent with its emotion being fixed before decoding in both settings; the paper does not comment on this. Every baseline’s SpeechBERTScore drops in Setting C on all four datasets (FSM on DailyDialog falls from 0.53 to 0.41), and their PESQ drops on DailyDialog, MELD and IEMOCAP; on EmoryNLP, baseline PESQ stays flat or rises slightly (SFT 1.23 to 1.26). In Setting B, Self-EmoQ has the highest SpeechBERTScore on DailyDialog (0.54) and IEMOCAP (0.52), ties FSM on EmoryNLP (0.50), and is below FSM on MELD (0.53 vs. 0.55). Its PESQ is the highest on DailyDialog, MELD and IEMOCAP but below MP (1.31) and ECoT (1.29) on EmoryNLP, where MP also stays higher in Setting C (1.31 vs. 1.28). All PESQ values lie between 0.97 and 1.31, so the spread between systems is small.
  • Ablations (Table 6). Replacing the logprob Q-value with an MLP head trained from scratch lowers every metric (e.g. DailyDialog R@3 0.82 to 0.23). Adding an SFT stage before RL, or removing the history or the dialogue description, lowers Reward, but some of these variants beat the full model on individual ranking metrics (w/ SFT: EmoryNLP R@3 0.66 and MRR 0.54; w/o history: MELD R@3 0.76).
  • Reward validation (Table 11). On 100 samples, GPT-4o and human Plutchik ratings correlate at rho 0.87-0.96 with kappa 0.58-0.78.

Two inconsistencies appear in the source tables. First, the full-model row in the ablation table (Table 6) differs from Table 3 in three cells: EmoryNLP Reward is 0.81 in Table 6 vs. 0.71 in Table 3, MELD R@5 is 0.88 vs. 0.89, and IEMOCAP Reward is 0.75 vs. 0.81. Second, in Table 5, FSM’s Effectiveness (3.05) is higher than Self-EmoQ’s (3.01), yet the p-value is listed as <0.01 under the stated null hypothesis that the baseline scores higher than Self-EmoQ.

Novelty Assessment

The contribution is engineering integration. Estimating Q-values from LLM token log-probabilities and training them with a DQN loss is reused from the same group’s StraQ* work. The new parts are the action space (emotion labels instead of dialogue strategies) and the reward, which adds GPT-4o ratings grounded in Plutchik’s wheel to an imitation term. The speech side is an unmodified CosyVoice 2 driven by the planned label. Its benefit for streaming TTS follows from the order of operations (emotion before text), not from any change to synthesis. Evidence for emotional speech quality is thin: no human listening test on speech, no emotion recognition on the synthesized audio, and no latency measurement. The text-level comparisons are broader, but Self-EmoQ does not win every metric.

Field Significance

Low — The paper shows that an emotion label planned before response generation can drive an off-the-shelf instruct-capable TTS without waiting for the full text, and that a reinforcement-learned planner improves label choice over prompting and supervised baselines on text dialogue datasets. Its evidence about emotional speech itself is limited to two reference-based objective metrics, so it mainly provides a pipeline design for streaming emotional voice agents rather than evidence about emotional speech synthesis.

Claims

  • supports: In streaming cascaded voice agents, the emotion that conditions TTS must be decided before the response text is decoded, which rules out post-hoc emotion recognition as the driver.

    Evidence: Baselines whose emotion becomes available only after text generation (Setting B) degrade when forced into the streaming order (Setting C), e.g. FSM’s SpeechBERTScore on DailyDialog falls from 0.53 to 0.41; Self-EmoQ, which plans the emotion first, has identical scores in both settings. (§4.5, Table 7)

  • supports: Optimizing emotion choice over multiple turns with reinforcement learning selects emotion labels that score higher on reward than prompting or supervised imitation.

    Evidence: The DQN-trained planner has the highest Reward on all four datasets (e.g. IEMOCAP 0.81 vs. 0.73 for FSM), but it trails the supervised baselines on MRR on EmoryNLP, MELD and IEMOCAP. (§4.4, Table 3)

  • complicates: Reference-based objective metrics such as PESQ and SpeechBERTScore provide weak evidence about the quality of emotional speech in conversational synthesis.

    Evidence: All systems drive the same CosyVoice 2 model; PESQ ranges only 0.97-1.31 across systems, the reference utterances for the text-only DailyDialog and EmoryNLP datasets are not described, and no human listening test or speech emotion recognition is reported. (§4.5, Table 7)

  • refines: Theory-guided LLM ratings can serve as a reward for emotion selection when they are checked against human ratings.

    Evidence: On 100 samples, GPT-4o Plutchik ratings correlate with human ratings at rho 0.87-0.96 (kappa 0.58-0.78), and the learned policy’s transitions concentrate on adjacent emotions of Plutchik’s wheel. (§4.6, §Appendix B.2, Table 11, Figure 6)

Limitations and Open Questions

Warning

The emotional TTS is an off-the-shelf CosyVoice 2 that is never trained or analysed. Speech is evaluated only with SpeechBERTScore and PESQ against unspecified reference utterances. There is no human listening test on speech, no check that the synthesized audio carries the planned emotion, and no latency measurement for the streaming pipeline.

  • Human evaluation covers text responses only. The raters are 10 interns, and the paper gives no detail on blinding or rater recruitment beyond that.
  • The Plutchik reward depends on GPT-4o, which adds cost and possible bias, as the authors acknowledge. Table 11 labels its three dimensions Alignment, Emotion and Effectiveness, while the text names them Alignment, Transition and Function.
  • Emotion is restricted to a discrete label set, so mixed or graded emotions cannot be planned (§Limitations).
  • The best reward weight w and discount factor differ by dataset (Table 12, Figure 5). For example, w = 1 (Plutchik score only) gives the highest average reward on DailyDialog, while w = 0 (imitation only) does so on EmoryNLP and IEMOCAP, so the preferred balance between the two reward terms varies by dataset.
  • The baselines’ emotion-labelling protocol in Settings B and C is not described in detail, so the size of their drop in Setting C may depend on implementation choices.
  • The table inconsistencies noted under Key Results (Table 3 vs. Table 6; FSM Effectiveness p-value in Table 5) are unresolved in the paper.

Wiki Connections

  • Emotion Synthesis — Plans a per-turn emotion label with reinforcement learning and uses it to condition an off-the-shelf emotional TTS, evaluated on synthesized speech with SpeechBERTScore and PESQ.
  • CosyVoice 2 — The TTS used, untrained, in instruct-inference mode to synthesize speech from each method’s planned emotion in the downstream experiment.
  • Word-Level Emotional Expression Control — Cited as an emotional TTS that treats emotion as a static input, in contrast to Self-EmoQ’s turn-by-turn planning.