arXiv · 2025 · Preprint
Seungyoun Shin et al. · → Paper · Demo: ✓ · Code: ?
Shows that GRPO-based reinforcement learning on transcription- and likelihood-derived rewards collapses TTS prosody into monotone speech, and recovers naturalness with an iterative Direct Preference Optimization scheme using only a few hundred human preference pairs per round.
Problem
Recent reinforcement-learning post-training for TTS uses Group Relative Policy Optimization (GRPO) with automatically verifiable rewards such as ASR-derived character error rate (CER), token likelihood (NLL), or speaker-similarity scores. These rewards are attractive because they require no human labeling, but prosody, the pitch movement and phrasing that make speech sound conversationally natural, has no reliable automatic proxy. The paper asks what happens to prosody when an RL objective optimizes only what can be measured automatically, and whether a small amount of human preference feedback can close that gap without abandoning RL-style post-training altogether.
Method
The base model, channel-base, is built on Llasa, a LLaMA-initialized Transformer that autoregressively generates discrete speech tokens decoded into waveform by XCodec2. Starting from the Llasa-1B checkpoint, the authors continually train on approximately 36k hours of Korean (text, audio) pairs from AIHUB to instill language competence, then fine-tune on 18 hours of proprietary single-speaker manager-customer call-center dialogue (manager channel only, extracted with pyannote.audio and transcribed with Whisper-large-v3) to adapt prosody toward conversational style.
On top of this base, the paper trains two competing post-training regimes. The first is GRPO with a harmonic-mean reward combining a CER-derived utility and an NLL-derived utility (weights 0.6/0.4), and a variant that adds a speaker-similarity utility from cosine similarity of speaker embeddings (weights 0.5/0.3/0.2). The harmonic mean is chosen specifically because it penalizes any single low-scoring component, creating pressure against high error while still crediting acoustic likelihood.
The second regime is iterative Direct Preference Optimization (DPO), run for three rounds. At each round, the policy from the previous round serves as both the initialization and the (moving) DPO reference; the model generates candidate utterances, roughly 200 fresh human preference pairs are collected via blind A/B comparison, and the DPO objective is optimized without replaying preference data from earlier rounds. This produces channel-base-dpo-v1 through -v3. Because the reference moves each round, the regularization target itself shifts toward prosodic naturalness rather than staying anchored to the original base model.
Evaluation uses a newly curated Korean call-center benchmark, KoCC-TTS (50 human-curated samples from real manager-user conversations), scored on CER (via Whisper-large-v3 transcription) and human preference aggregated into Chatbot-Arena-style ELO ratings from 596 blind pairwise votes across 27 raters aged 20-60.
Key Results
Starting from channel-base (CER 2.90%, ELO 1150.1), GRPO with the CER/NLL reward drives CER down to 2.20%, the lowest of any system tested, but collapses ELO to 753.7, the lowest of any system, with log-F0 pitch distributions showing markedly reduced variability relative to the baseline. Adding the speaker-similarity term does not rescue prosody: CER instead inflates to 42.63% and the model exhibits degenerate non-terminating generations (missing end-of-sequence tokens), which the authors interpret as reward hacking rather than a fixable RL instability.
Iterative DPO reverses the CER/ELO trade-off without any automatic prosody reward. Round 1 raises ELO to 1096.5 at the cost of CER rising to 5.80% as the policy explores more varied prosody; round 2 peaks at ELO 1190.1 with CER recovering to 3.60%, exceeding all three commercial baselines tested (ElevenLabs Multilingual v2, Supertone, GPT-4o-mini-tts) in human preference while remaining CER-competitive; round 3 further reduces CER to 3.30% but ELO settles back to 1064.2. The authors attribute the round-2 peak to diminishing informativeness of preference pairs as the policy-reference gap narrows across rounds. Open-source TTS baselines (Llasa-1B/3B/8B) are reported for CER only, without ELO, and off-the-shelf open-source systems in general were excluded from the human-preference comparison on the grounds that they produced floor-effect prosody in Korean.
Novelty Assessment
The paper’s architecture is not novel: it fine-tunes an existing autoregressive codec-token TTS model (Llasa) with existing post-training algorithms (GRPO, DPO). The contribution is diagnostic and methodological rather than architectural. Diagnostically, it isolates and demonstrates a specific failure mode, that CER/NLL-driven GRPO reward for TTS systematically trades prosodic variation for transcription accuracy, and that adding a speaker-similarity term does not fix this but instead destabilizes training. Methodologically, it shows that a small, round-based, moving-reference DPO loop, using roughly 200 human preference pairs per round rather than a large static preference dataset, can recover the lost prosodic naturalness while keeping CER close to the RL-optimized baseline. The KoCC-TTS release is a genuine, if small, data contribution: a domain-specific (call-center), human-curated Korean test set intended to jointly stress transcription robustness and conversational prosody.
Field Significance
moderate — provides a concrete negative result for GRPO-based TTS reinforcement learning: optimizing transcription- and likelihood-derived rewards alone drives prosody toward near-monotone collapse, and naively extending the reward with speaker similarity destabilizes training rather than correcting it. The paper demonstrates iterative, small-batch human preference optimization as a practical countermeasure when no automatic prosody reward exists, and releases a Korean call-center evaluation set for testing transcription robustness and conversational prosody jointly. The evidence is confined to a single base architecture, a single language, and a small human evaluation panel, which limits how far the specific findings generalize.
Claims
- complicates: Reinforcement learning with automatically verifiable rewards can optimize the measured objective while degrading unmeasured perceptual qualities such as prosodic naturalness.
Evidence: GRPO trained on a harmonic-mean reward of CER and token likelihood reduced CER to the lowest value among all tested systems (2.20%) but produced the lowest human-preference ELO (753.7), with log-F0 pitch distributions showing collapse toward monotone speech relative to the baseline. (§4.3, Table 1, Figure 2)
- complicates: Adding an auxiliary reward term intended to fix a known RL side effect can destabilize training instead of correcting the targeted attribute.
Evidence: Extending the GRPO reward with a speaker-similarity utility increased similarity scores but inflated CER to 42.63% and produced degenerate non-terminating outputs (missing end-of-sequence tokens), consistent with reward hacking rather than genuine prosody improvement. (§4.4)
- supports: Small-scale iterative human preference optimization can recover perceptual speech qualities lost to metric-driven RL training without requiring a large static preference dataset.
Evidence: Using approximately 200 human preference pairs per round with a moving DPO reference, three iterative rounds raised ELO from 753.7 (GRPO) to a peak of 1190.1 at round 2, exceeding three commercial TTS baselines in human preference, while CER recovered to 3.30-3.60% by rounds 2-3. (§4.5, Table 1)
- refines: The benefit of successive rounds of iterative preference optimization is not monotonic and can saturate or reverse as the policy converges toward its own reference.
Evidence: ELO peaked at round 2 (1190.1) and declined at round 3 (1064.2); the authors attribute this to a narrowing policy-reference gap reducing the informativeness of newly collected preference pairs in later rounds. (§4.5)
Limitations and Open Questions
The evaluation is narrow in scope: a single base architecture (Llasa-1B derived), a single language and domain (Korean call-center dialogue), a single fine-tuned speaker voice, a 50-utterance test set, and 596 votes from 27 raters. Open-source TTS systems were deliberately excluded from the human-preference comparison after a preliminary screening judged them to produce floor-effect prosody, which removes a category of baseline from the headline ELO comparison.
Beyond the evaluation scope, the paper does not report an ablation over the DPO sharpness parameter (beta) or over preference-pair budget per round, so it is unclear how sensitive the round-2 peak is to these choices. The mechanism behind the round-3 ELO decline is offered as a hypothesis (narrowing policy-reference gap) rather than directly verified. The GRPO speaker-similarity failure is also reported as a single run rather than across multiple reward-weighting configurations, leaving open whether a different weighting could avoid the observed instability.
Wiki Connections
- RLHF Speech — directly compares GRPO-based and DPO-based preference/reward optimization for TTS post-training and documents a specific reward-design failure mode for prosody.
- Prosody Control — uses iterative human preference optimization as an explicit mechanism to restore prosodic naturalness after RL-driven collapse, without any automatic prosody metric.
- Autoregressive Codec TTS — builds its base system on Llasa, a LLaMA-initialized autoregressive model generating discrete codec tokens decoded via XCodec2.
- Subjective Evaluation — relies on blind pairwise human preference votes aggregated into Chatbot-Arena-style ELO ratings as the primary quality signal, rather than an automatic prosody proxy.
- Llasa — adopts this architecture as its base autoregressive codec-token TTS model, continually trained and fine-tuned on Korean data before RL/DPO post-training.
- F5R-TTS — cited as a comparable GRPO-based TTS system that couples WER with speaker similarity, contrasted against this paper’s finding that such composite GRPO rewards omit explicit prosody sensitivity.
- DMOSpeech 2 — cited as a GRPO-based duration-prediction TTS system optimizing SIM+WER, used as a related-work example of reward designs that do not target prosody directly.
- TTS-1 Technical Report — cited as an industrial GRPO system with a composite WER/SIM/DNSMOS reward, contrasted with this paper’s DPO-based approach to prosody.
- MPO — cited as a concurrent multi-criteria preference-screening approach for speech synthesis, situating this paper’s single-criterion iterative DPO within the broader preference-optimization landscape.