arXiv · 2026 · Preprint

Nie et al. (South China University of Technology / Huya Inc. / Alibaba Group / Foshan University) · → Paper · Demo: ✓ · Code: ✓

HPRO extends differentiable reward optimization for LLM-based emotional TTS by routing generated speech tokens through a reward model that splits them into separate content and style token streams, then applying frame-, word-, and sentence-level rewards in three progressive stages, which lowers WER and raises emotion-similarity scores over a CosyVoice 2 SFT baseline.

Problem

Supervised fine-tuning of LLM-based TTS with a cross-entropy objective tends to regress towards averaged prosody, which flattens emotional expression. Preference and reward optimization (policy-gradient RL with emotion-recognition rewards, DPO-style pair training, GRPO, and the differentiable reward optimization of DiffRO) can push emotional intensity up, but the authors argue that in emotional TTS these methods are prone to reward hacking that damages intelligibility. They attribute this to two mismatches. The first is information conflict: content and emotion share one speech-token representation, so maximising an emotion reward also perturbs the acoustic structure that carries the words. The second is a scale gap: a sparse sentence-level emotion reward gives weak credit assignment for dense frame-level token generation. RRPO addresses reward-model robustness but, in the authors’ framing, not these two structural issues.

Method

HPRO has two parts: a differentiable reward model called the HD-Emo codec, and a staged optimization schedule for the TTS LLM.

HD-Emo codec (reward model). Despite the name, HD-Emo is a token-to-token model, not a waveform codec. It takes discrete speech tokens from the CosyVoice 2 tokenizer and passes them through two preference-token extractors with identical architecture but separate parameters (8-layer Conformers). Each stream is quantised with finite scalar quantization (FSQ, from Finite Scalar Quantization), giving content-preference tokens (codebook size 1296) and style-preference tokens (codebook size 64). The content stream’s pre-quantization latents feed a content adapter and an ASR decoder initialised from the Whisper-medium decoder, trained with an autoregressive ASR loss. A stop-gradient detaches the content extractor from the reconstruction path, so only the ASR loss updates it; the authors’ goal is to keep prosodic detail from leaking into the content tokens. The style stream feeds an emotion decoder with two heads: an SER head trained with cross-entropy against soft emotion distributions produced by a pre-trained emotion2vec model (sentence level), and a word-level valence-arousal-dominance (wVAD) head trained with a concordance-correlation-coefficient (CCC) loss against wVAD trajectories from a fine-tuned wav2vec 2.0 model, using Montreal Forced Aligner word boundaries and a one-word context window on each side. The two streams are fused by FiLM-style modulation (the content representation is scaled and shifted by factors projected from the style tokens, following Emo-FiLM), and an 8-layer autoregressive transformer reconstructs the original speech tokens under a cross-entropy loss. The content branch is pre-trained on LibriSpeech with ASR supervision, continued on the emotional datasets, then frozen while the rest of the codec trains on LSSED and EmoVoice-DB (Adam, 100 epochs, 8 RTX 4090 GPUs). The design adapts the token codec from the authors’ earlier HD-PPT system.

Fig. 2. Overview of the HD-Emo Codec. Monotonic speech tokens are processed by dual preference extractors with FSQ bottlenecks to obtain content and style preference tokens. The two streams are respectively supervised by ASR and hierarchical emotional objectives (SER and wVAD), and subsequently fused via dynamic feature modulation for speech token reconstruction.

Hierarchical rewards. The TTS model is CosyVoice 2’s autoregressive speech-token LM (Qwen2.5-0.5B architecture). Its hidden states are projected to the speech-token vocabulary and sampled with straight-through Gumbel-Softmax, giving a differentiable token sequence that the frozen HD-Emo codec maps into its preference spaces, so gradients from the reward losses flow back into the LLM without RL rollouts. Three levels of reward losses are defined against ground-truth targets:

  • Frame level: L1 regression of the generated pre-quantization content and style latents onto the ground-truth FSQ content and style tokens.
  • Word level: the wVAD CCC loss plus the ASR cross-entropy loss (to penalise lexical drift).
  • Sentence level: cross-entropy between the predicted emotion distribution and the target soft labels.

A token-wise categorical KL term on the original speech-token distribution regularises the LLM.

Progressive schedule. Stage I trains only the frame-level losses and KL (λ_KL = 0.05, λ_sp = 2, λ_cp = 1, Gumbel τ = 2). Stage II adds the word-level ASR and wVAD losses (λ_KL = 0.02, λ_ASR = 5, λ_wVAD = 1, τ = 1). Stage III adds the sentence-level SER loss (λ_SER = 0.5, τ = 0.8). The LLM is trained with Adam at learning rate 1e-5. Only LLM optimization is described; the paper does not discuss CosyVoice 2’s flow-matching token-to-mel decoder or vocoder, which it presumably inherits unchanged for waveform generation. How the target emotion is specified at inference (beyond the speech prompt) is not stated.

Fig. 3. Overview of the HPRO framework. The LLM generates differentiable speech tokens via Gumbel-Softmax, which are mapped by HD-Emo codec into preference spaces. Hierarchical frame-, word- and sentence-level rewards are progressively applied to update the LLM.

Key Results

All evaluation uses prompt-based synthesis with a randomly selected reference utterance from the same speaker, on the LSSED and EmoVoice-DB test sets. Baselines are CosyVoice 2, CosyVoice 3, IndexTTS2, and HD-PPT (adapted from its instruction setting to prompt-based synthesis). Subjective tests use 90 emotionally balanced utterances rated by 18 listeners on a 5-point scale for naturalness (MOS-N) and emotion-content consistency (MOS-E). Objective metrics are WER (Whisper-large-v3), wVAD-CCC, EMO-SIM, and DNSMOS P.835, all computed on synthesized waveforms with external models (Table I):

SystemMOS-NMOS-EWERwVAD-CCCEMO-SIMDNSMOS
TokenRecon (ground-truth tokens)--7.34%0.570.7753.58
CosyVoice 24.094 ± 0.2573.530 ± 0.4025.45%0.3070.6133.76
CosyVoice 34.137 ± 0.2853.538 ± 0.3434.90%0.2750.6113.72
IndexTTS24.026 ± 0.2383.692 ± 0.2136.74%0.2930.5263.53
HD-PPT4.068 ± 0.2483.547 ± 0.3284.92%0.3230.6463.75
HPRO4.171 ± 0.3183.650 ± 0.3474.02%0.3390.6723.73

Among the five synthesis systems, HPRO has the lowest WER and the highest wVAD-CCC and EMO-SIM. TokenRecon (resynthesis from ground-truth tokens) scores higher on wVAD-CCC and EMO-SIM but has a higher WER. HPRO has the highest MOS-N mean, but the margins are small relative to the reported standard deviations (4.171 ± 0.318 against 4.137 ± 0.285 for CosyVoice 3) and no significance test is given. IndexTTS2 has the highest MOS-E (3.692 against 3.650 for HPRO), which the authors attribute to stereotyped, over-intense emotion from IndexTTS2’s text-predicted emotion embeddings. DNSMOS is not HPRO’s strongest metric: CosyVoice 2 (3.76) and HD-PPT (3.75) both score above HPRO (3.73), even though the text describes HPRO as superior “across all objective metrics”.

Progressive schedule ablation (Table II, from a CosyVoice 2 SFT baseline). Adding frame-level rewards moves WER from 5.42% to 4.85%, wVAD-CCC from 0.297 to 0.332, and EMO-SIM from 0.641 to 0.650. Adding word-level rewards gives the best WER (3.99%) and wVAD-CCC (0.350) of the four configurations. Adding the sentence-level reward (the full HPRO) raises EMO-SIM to 0.672 and DNSMOS to 3.73 (both the highest in the table) but slightly worsens WER (4.02%) and wVAD-CCC (0.339) relative to the word-level stage.

Reward-component ablation (Table III, all rewards applied from the start, not progressively). Removing the content rewards raises WER to 13.61% (wVAD-CCC 0.285, EMO-SIM 0.584). Removing the emotion rewards gives the lowest WER in the table (3.80%) and the highest DNSMOS (3.78), but lower wVAD-CCC (0.295) and EMO-SIM (0.637). Removing frame-level rewards gives WER 4.97% and EMO-SIM 0.608. Removing wVAD gives wVAD-CCC 0.310 and EMO-SIM 0.659. Removing both frame and wVAD rewards, which the authors use as a simulation of DiffRO’s single-scale sentence reward because no official implementation exists, gives WER 4.35%, wVAD-CCC 0.315, and EMO-SIM 0.662. Full HPRO gives 4.02%, 0.339, and 0.672. Table III reports the same HPRO row as Tables I and II, although its caption says the ablation variants are trained non-progressively, so it is unclear whether the full-model row was retrained in that setting.

Novelty Assessment

The main contribution is to the training objective. Differentiable reward optimization through Gumbel-Softmax-relaxed speech tokens comes from DiffRO. HPRO’s new elements are (a) a reward model that factorises speech tokens into separately supervised content and style FSQ streams, so lexical and emotional rewards are computed in different subspaces, and (b) a multi-granularity reward with a curriculum (frame, then word, then sentence). The HD-Emo codec reuses the authors’ HD-PPT token codec, with emotion-specific heads added, so the reward-model architecture is an adaptation rather than a new design. Several components are borrowed: FSQ, Whisper-initialised ASR decoding, emotion2vec soft labels, wav2vec 2.0 VAD regression, and FiLM modulation.

The evidence has gaps. The DiffRO comparison is a re-implementation inside HPRO’s own framework, using the HD-Emo reward model, so it isolates the effect of reward granularity but does not test the content/style factorisation against a monolithic reward model. No ablation replaces HD-Emo with a single-stream reward model. EMO-SIM and wVAD-CCC are computed with external models, but they follow the same emotion and VAD definitions the rewards optimise, so some alignment between training signal and metric remains. The subjective study is small (18 raters, 90 utterances) and has no significance testing. The gains over the SFT baseline on WER and emotion metrics are consistent across tables.

Field Significance

Moderate. The paper provides an ablated example of splitting a TTS reward model into content and style token subspaces and scheduling rewards from frame to sentence granularity in differentiable reward optimization, with a clear account of the trade-off between sentence-level emotion rewards and word-level prosodic tracking. It offers useful evidence for preference-based emotional TTS, but the factorisation itself is not directly ablated and the evaluation is limited to two emotional corpora and a single backbone.

Claims

  • supports: Reward optimization for emotional TTS is prone to intelligibility collapse when lexical-content constraints are removed from the reward.

    Evidence: Training without the content-related rewards (ASR loss and content-token alignment) raises WER from 4.02% to 13.61%, and also lowers wVAD-CCC and EMO-SIM. (§III-D, Table III)

  • complicates: In multi-scale reward optimization, a sentence-level emotion objective improves global emotion similarity at a small cost to word-level prosodic tracking and intelligibility.

    Evidence: Adding the sentence-level SER reward after the word-level stage raises EMO-SIM from 0.653 to 0.672 but lowers wVAD-CCC from 0.350 to 0.339 and moves WER from 3.99% to 4.02%. (§III-C, Table II)

  • supports: Dense frame-level reward terms help stabilise differentiable reward optimization of autoregressive speech-token LMs, compared with sparse utterance-level rewards alone.

    Evidence: Dropping the frame-level rewards raises WER to 4.97% and lowers EMO-SIM to 0.608. A DiffRO-style configuration with only sentence-level and ASR rewards (no frame or wVAD terms) reaches WER 4.35% and wVAD-CCC 0.315, against 4.02% and 0.339 for full HPRO. (§III-D, Table III)

  • complicates: Gains in emotional expressiveness from preference-style optimization trade off against intelligibility, and removing emotion objectives gives the best WER.

    Evidence: The variant without emotion rewards has the lowest WER (3.80%) and highest DNSMOS (3.78) in the reward ablation, but lower wVAD-CCC (0.295) and EMO-SIM (0.637) than full HPRO. (§III-D, Table III)

  • refines: Explicit emotion-embedding conditioning can raise perceived emotional intensity while lowering naturalness and objective emotion alignment to the reference.

    Evidence: IndexTTS2 has the highest MOS-E (3.692) but the lowest MOS-N (4.026), the lowest EMO-SIM (0.526), and a WER of 6.74%; the authors attribute this to stereotyped emotion from text-predicted emotion embeddings. (§III-B, Table I)

Limitations and Open Questions

Warning

The content/style factorisation of the HD-Emo reward model, which the paper presents as its fix for information conflict, is never ablated against a single-stream reward model. The DiffRO comparison is a simulation inside HPRO’s own framework, not the published system.

  • The subjective study (18 raters, 90 utterances) reports standard deviations of about 0.2 to 0.4 MOS and no significance tests. HPRO’s MOS-N lead (4.171 against 4.137 for CosyVoice 3) is well within that spread.
  • HPRO does not have the highest DNSMOS in Table I: CosyVoice 2 and HD-PPT both score higher.
  • The reward ablation (Table III) is described as non-progressive, yet its HPRO row matches the progressive full model in Tables I and II, so the comparison settings are not fully clear.
  • Evaluation uses the test sets of the same two corpora (LSSED, EmoVoice-DB) used for training, with references from the same speaker; the paper does not state whether test speakers are unseen, and out-of-domain text is not tested.
  • Only one backbone (CosyVoice 2) is optimised, and only two emotional corpora are used.
  • The paper does not explain how the target emotion is specified at inference, beyond the speech prompt.

Wiki Connections

  • RLHF Speech — Applies a hierarchical, curriculum-scheduled differentiable reward optimization stage (via Gumbel-Softmax-relaxed tokens) on top of SFT, with a content/style factorised reward model.
  • Emotion Synthesis — Targets emotional expressiveness through sentence-level emotion and word-level valence-arousal-dominance rewards, evaluated with MOS-E, EMO-SIM, and wVAD-CCC.
  • Autoregressive Codec TTS — The optimised model is CosyVoice 2’s autoregressive speech-token LM, trained here through differentiable token sampling.
  • Self-Supervised Speech — The reward model’s emotion heads are trained on targets from emotion2vec (sentence-level soft labels) and a fine-tuned wav2vec 2.0 model (word-level VAD), so SSL models shape the training reward.
  • Subjective Evaluation — Reports MOS-N and MOS-E from 18 human raters on 90 utterances, against four baselines.
  • DiffRO — HPRO extends DiffRO’s differentiable reward optimization and compares against a simulated DiffRO-style single-scale reward configuration.
  • CosyVoice 2 — Provides the backbone LLM and speech tokenizer that HPRO fine-tunes, and serves as a baseline.
  • CosyVoice 3 — Compared as a baseline; it has a higher WER and lower emotion scores than HPRO, with a similar MOS-N.
  • IndexTTS2 — Compared as a baseline; it has the highest MOS-E but the lowest MOS-N and EMO-SIM.
  • RRPO — Cited as prior emotional-TTS reward optimization that improves reward robustness, which HPRO argues does not resolve the content/emotion conflict.
  • emotion2vec — Supplies the soft emotion labels that supervise the reward model’s sentence-level SER head.
  • EmoVoice — Source of the EmoVoice-DB emotional corpus used for training and evaluation.
  • Emo-FiLM — Inspires the FiLM-style modulation that fuses content and style tokens in the HD-Emo codec.