ACL 2026 (Findings) · 2026 · Conference

Haiyang Sun et al. (Shanghai Jiao Tong University / VUI Labs) · → Paper · Demo: ? · Code: ?

Builds a 1,335-hour Mandarin audio-drama corpus annotated by an LLM with the speaker’s personal experience, the dialogue context, paralinguistic descriptions and open-vocabulary emotion labels, and shows that a CosyVoice2 LLM fine-tuned to take these descriptions as text input produces emotional speech that human listeners rate as more natural than sentence-level emotion prompting.

Problem

Emotional TTS systems are usually conditioned on sentence-level signals: a discrete emotion category or a short natural-language style instruction. The authors argue, drawing on cognitive appraisal theory, that such labels describe the emotion without its cause. A model told only “angry” has to guess how that anger should sound, which tends to produce averaged or stereotyped delivery. Existing emotional speech corpora (acted lab recordings, in-the-wild media, conversational sets such as IEMOCAP and MELD) also lack explicit annotations of the speaker’s history, so there is no training data for modelling why an utterance is expressed the way it is.

Method

Emotional-Context-Speech dataset. The corpus is built from publicly available multi-speaker Mandarin audio dramas paired with their source novels. Each episode is transcribed with Whisper and force-aligned with the Montreal Forced Aligner. Character dialogue is extracted from the novel text by quotation marks (utterances shorter than 5 characters are dropped), and the ASR transcript is matched to the novel dialogue with a sliding buffer at the Pinyin level using difflib’s SequenceMatcher, which tolerates ASR homophone errors.

Annotation uses DeepSeek-V3.2 with chain-of-thought prompting. Because a text-only LLM cannot hear the audio, the pipeline quantizes acoustic features (F0, speech rate, pause count and duration, spectral centroid, RMS, MFCC variance, HNR) into integer tokens: the range observed on 1,000 random samples is split into 10 bins plus 5 outlier bins, and time-series features are downsampled to 10 Hz. The LLM reads the novel context and these acoustic tokens, then outputs a context summary, the character’s personal experience, a paralinguistic description, an emotion-cause rationale and open-vocabulary (OV) emotion labels. As an alternative paralinguistic source, the authors also caption the audio with Qwen3-Omni-Captioner and summarize the captions with Qwen3-8B. After filtering misaligned samples with Paraformer-zh, the training set holds about 730k samples (1,335 hours) spanning 3,359 emotion categories. One audio drama series is held out for validation and testing.

Emotional-Context-TTS. The TTS model is CosyVoice2 with only its 0.5B LLM fine-tuned. The augmented condition set (personal experience, context, paralinguistic description and OV emotion labels) is concatenated with the target text, and the LLM autoregressively predicts CosyVoice2’s supervised semantic speech tokens (the authors’ “semantic tokens”) conditioned on both. The CosyVoice2 conditional flow-matching decoder, which maps the tokens plus speaker embedding and reference mel-spectrogram to a mel-spectrogram, and the HiFi-GAN vocoder are used frozen without further training. Training runs for 20 epochs on 8 H100 GPUs with Adam at a constant 1e-5 learning rate, and the final model averages the top-5 validation checkpoints.

Key Results

Zero-shot quality (Table 1, Seed-TTS test-zh, inference without instructions). Fine-tuning costs a little intelligibility and speaker similarity relative to CosyVoice 2: CER rises from 1.4 to 1.6-1.8 and SIM falls from 75.2 to 74.5-74.8, depending on which annotations were used in training. UTMOS (3.5) and UVMOS (4.7) are unchanged across all variants. Ground-truth recordings score lower on both predicted-MOS metrics (UTMOS 2.8, UVMOS 4.5) than any synthesized system.

Emotional expression (Table 2, ETS-C and ETS-P, 25 utterances each). Ten native Mandarin listeners (five general and five speech-AI graduate students) scored each sample on a 1-100 scale for emotional accuracy and naturalness, conditioned on the character background, context and target emotion, and judged whether it was human (HA rate). The full model with experience, context, Qwen3-Omni paralinguistic description and emotion prompts reaches 63.9 average subjective score and 51.8% HA. The strongest baselines are CosyVoice 3 (58.8, 42.2%), CosyVoice 2 (58.2, 44.7%), Gemini 2.5 Pro TTS (52.8, 33.1%) and IndexTTS2 given the same experience and context prompts (49.8, 39.6%). The confidence intervals overlap with the CosyVoice baselines (63.9 ± 3.3 vs 58.8 ± 4.1), and ground truth remains far ahead at 79.4 and 79.3%.

The full model also has the highest Gemini 3 Pro judge average (75.9), but the margin is small and not consistent across variants. On ETS-P it ties CosyVoice 3 at 78.0. The DeepSeek-paralinguistic variant averages 73.6, below CosyVoice 2 and CosyVoice 3 (74.6) and Step-Audio 2 Mini (74.0). The experience-plus-context variant ties both CosyVoice systems at 74.6. Giving open-source baselines the same experience and context text raises the judge score for Higgs Audio v2 (69.1 to 71.3) and IndexTTS2 (68.9 to 70.3) but lowers it for Step-Audio 2 Mini (74.0 to 73.4), and the judge gains do not transfer to human ratings: Higgs Audio v2 with those prompts scores only 17.1 subjective. GPT-4o Audio Preview leads the commercial systems on the judge score (73.4) yet is rated 24.6 by humans, which the authors use to argue that the LLM judge is not a substitute for listening tests.

Ablation of inputs. Within the proposed model, subjective scores rise as inputs are added: emotion labels only 60.1 (50.0% HA), plus experience and context 61.8 (51.0%), plus DeepSeek paralinguistic description 62.6 (51.3%), plus Qwen3-Omni paralinguistic description 63.9 (51.8%) (Table 2). The emotion-label-only variant scores below CosyVoice 3 on the ETS-C subjective score (58.9 vs 59.0). Subjective prosodic similarity to the ground-truth recording follows the same order (70.3, 71.1, 73.1, 73.8), while SpeechBERTScore similarities computed on WavLM and HuBERT barely move (76.3-76.9 and 77.4-77.9) and the emotion2vec variant is non-monotonic (88.1, 89.0, 87.2, 88.4) (Table 3).

Simulated continuations (Table 4). On GPT-4.1-generated follow-up lines, the experience-plus-context model scores 76.2 (53.4% HA) against 74.4 (47.2%) for emotion labels only, 73.3 (48.2%) for CosyVoice 3 and 71.5 (39.8%) for CosyVoice 2. With emotion labels alone, the proposed model’s HA rate is slightly below CosyVoice 3’s.

Annotation pipeline checks. On IEMOCAP Session 5 four-class emotion classification with DeepSeek-reasoner, adding quantized acoustic tokens to the transcript raises mean accuracy from about 65.5% to 67.8% over five runs (Table 5). Five human raters scored 60 generated annotations at 3.68-3.91 out of 5 without context and 3.87-4.08 with context (Table 6).

Novelty Assessment

The model side is engineering integration. It fine-tunes the CosyVoice2 LLM on longer text prompts and leaves the decoder untouched; there is no new architecture or training objective. The contributions are the data and the framing. The corpus pairs expressive speech with narrative-derived personal-experience and context descriptions, and the acoustic-feature quantization lets a text-only LLM produce audio-aware paralinguistic captions. The ablation is the most informative evidence: the same backbone and data gain in human ratings as experience and context descriptions are added. The size of the gain is modest, though. The comparison against open-source baselines also mixes models that were never trained on such descriptions with one that was trained on in-domain audio-drama data, so part of the advantage over baselines may come from domain match rather than from the contextual inputs.

Field Significance

Moderate — The paper provides a training corpus and controlled ablation showing that narrative context and speaker-history descriptions, not only emotion labels, can be used as TTS conditioning and are preferred by human listeners in Mandarin audio-drama speech. It also adds a cautionary data point that an LLM judge and SSL-embedding similarity both miss differences that human raters detect in complex emotional speech.

Claims

  • supports: Conditioning emotional TTS on descriptions of the dialogue context and the speaker’s background improves perceived emotional naturalness over conditioning on emotion labels alone.

    Evidence: With the same fine-tuned CosyVoice2 backbone, adding experience and context to open-vocabulary emotion prompts raises the average subjective score from 60.1 to 61.8 on ETS-C/ETS-P and from 74.4 to 76.2 on GPT-4.1 continuation texts, with HA rate rising from 47.2% to 53.4% on the continuations. (§6.2.2, §6.4, Table 2, Table 4)

  • complicates: LLM-as-judge scores for emotional speech can diverge sharply from human ratings, compressing differences between systems and misranking some.

    Evidence: Gemini 3 Pro averages for all systems fall between 68.9 and 75.9, while human averages span 17.1 to 63.9. GPT-4o Audio Preview scores 73.4 on the judge but 24.6 from listeners, and Higgs Audio v2 with context prompts scores 71.3 on the judge but 17.1 from listeners. (§6.2.1, Table 2)

  • complicates: Reference-aware similarity on self-supervised speech embeddings is insensitive to fine-grained differences in emotional prosody that human listeners detect.

    Evidence: Across four input configurations, subjective prosodic-similarity scores rise monotonically from 70.3 to 73.8, while WavLM and HuBERT SpeechBERTScore move by at most 0.6 points and the emotion2vec variant fluctuates non-monotonically (88.1, 89.0, 87.2, 88.4). (§6.3, Table 3)

  • complicates: Fine-tuning a zero-shot TTS model for rich expressive conditioning can slightly degrade intelligibility and speaker similarity in standard prompt-based cloning.

    Evidence: On Seed-TTS test-zh, every fine-tuned variant has higher CER (1.6-1.8 vs 1.4) and lower SIM (74.5-74.8 vs 75.2) than the CosyVoice 2 base model, with UTMOS and UVMOS unchanged. (§6.1, Table 1)

  • supports: Discretizing prosodic acoustic features into tokens lets a text-only LLM use audio cues when annotating emotion.

    Evidence: On IEMOCAP Session 5 four-class emotion classification, adding quantized acoustic tokens to a dialogue-context transcript prompt raises DeepSeek-reasoner mean accuracy from about 65.5% to 67.8% over five runs. (§3.2.1, §Appendix B, Table 5)

Limitations and Open Questions

Warning

The emotional-expression evaluation rests on two 25-utterance test sets from a single held-out audio drama. The paralinguistic descriptions are produced by captioning recorded audio, and the paper does not say that test-time descriptions come from anywhere other than the target recordings; if they do come from the targets, the best-scoring configuration receives information about the reference delivery that a deployed system would not have.

  • Most subjective-score confidence intervals overlap between the proposed model and the CosyVoice baselines, and no statistical significance test is reported.
  • The baselines were not trained on context or experience descriptions, and none were fine-tuned on the in-domain audio-drama data, so the comparison does not isolate the effect of contextual conditioning from domain adaptation. The within-model ablation is the cleaner evidence.
  • The training data is performative audio-drama speech in Mandarin only. The authors note that transfer to spontaneous conversation and to other languages is untested beyond the IEMOCAP annotation ablation (§Limitations).
  • All annotations are LLM-generated. Human validation covers 60 samples, and the in-house DeepSeek paralinguistic descriptions score below the Qwen3-Omni-Captioner alternative in every downstream subjective measure.
  • The raw audio cannot be redistributed because of copyright (§Ethics Statement), which limits reproducibility of the dataset contribution. Code and demo availability are not stated.

Wiki Connections

  • Emotion Synthesis — Trains and evaluates a TTS model whose emotional delivery is conditioned on context, personal experience and open-vocabulary emotion labels.
  • Instruction-Conditioned TTS — Extends free-text style conditioning from short instructions to narrative context and speaker-history descriptions, with human-evaluated effect on expression.
  • Autoregressive Codec TTS — The fine-tuned component is the CosyVoice2 autoregressive LLM over discrete speech tokens; the flow-matching decoder stays frozen.
  • Subjective Evaluation — Uses context-aware listening tests with 10 raters and a human-or-AI judgment, and shows that an LLM judge diverges from those ratings.
  • CosyVoice 2 — Base model whose 0.5B LLM is fine-tuned and whose decoder is reused frozen; also the main baseline in zero-shot and emotional evaluations.
  • CosyVoice 3 — Strongest open-source baseline on human emotional-naturalness scores and on continuation texts.
  • IndexTTS2 — Emotion-controllable baseline evaluated with and without the experience and context prompts.
  • Step-Audio 2 — Its Mini model is an open-source baseline given the same context prompts.
  • EmoVoice — Free-text emotional TTS cited as an example of sentence-level natural-language emotion control that this work moves beyond.
  • Qwen3-Omni — Its captioner provides the alternative paralinguistic annotations that yield the best-scoring configuration.
  • Seed-TTS — Source of the test-zh set used to check zero-shot intelligibility and speaker similarity after fine-tuning.
  • emotion2vec — Embedding used in one SpeechBERTScore variant that fails to track human prosodic-similarity judgments.