arXiv · 2026 · Preprint
Zhicheng Ouyang et al. (Meta AI) · → Paper · Demo: ✓ · Code: ✗
A cascaded ASR-LLM-TTS system in which an LLM emits textual style tokens that are mapped to single human-curated audio prompts, used as in-context prompts for a two-stage AR-plus-diffusion TTS, with the AR prosody model further tuned by online RL on an aesthetic reward regularized by a CTC term to suppress hallucination.
Problem
Fine-grained control of speaking style and emotion in conversational TTS usually requires large amounts of annotated expressive speech, and end-to-end conversational audio LLMs struggle to control expressivity because expressive conversational audio and reliable reward models are scarce. Multi-turn conversations add a second problem: when the reference prompt changes from turn to turn, the synthesized voice can drift in timbre, loudness, and room characteristics. Finally, posterior-sampling approaches that pick the best of several candidates at inference time improve quality but are computationally expensive.
Method
The system is a text-native cascade: an LLM (LLaMA 3 70B) generates the response together with a textual style token chosen from conversational context, and that token is forwarded to the TTS module. The TTS is an in-house system following a TortoiseTTS-like design: an autoregressive (AR) prosody model, pretrained on a large human speech corpus, generates discrete speech tokens; a diffusion-based acoustic model renders them; and a BigVGAN vocoder produces the waveform. Model sizes are not reported.
Each style token is bound to an audio prompt, and the prompt acts as in-context learning (ICL): style is adapted at inference without weight updates. Prompting operates at two levels. For the AR prosody model, a pool of candidate voice-actor clips is curated per fine-grained style; at least ten samples are generated per candidate and scored with the Aesthetic Quality Content Enjoyment predictor (AES-CE), a Monte Carlo lower-bound score is computed, the best candidate is chosen, and a final human listening pass removes candidates that cause hallucinations or unnatural breathing. For the acoustic model, fine-grained styles are grouped into coarser categories that share one acoustic prompt, which the authors find reduces speaker drift across turns. They also observe that the AR prompt speaker need not match the acoustic prompt speaker: timbre is mostly set by the acoustic model and prosody by the AR model, so the two can be selected independently.
To improve generation quality without inference-time re-ranking, the AR prosody model is trained with online RL while conditioned on the same audio prompts used at inference (hence “ICL-based” RL). The reward is AES-CE computed on the waveform from the frozen acoustic model and vocoder, minus a weighted CTC loss between the generated token sequence and the ground-truth transcript; the objective maximizes expected reward with a KL penalty toward the SFT reference policy. The expectation is estimated with six Monte Carlo samples per transcript, with α_AES = 1.0, α_CTC = 3.0, β = 50.0, and Adam at a learning rate of 5.0 × 10⁻⁷. Training data are pairs of audio prompts and transcripts whose content is independent of the prompt audio.
Expressivity is rated with a protocol over four paralinguistic dimensions (Clarity, Valence, Arousal, Dominance; CVAD) on five-point Likert scales, from which comparative MOS (CMOS) net win rates are derived. Multi-turn speaker consistency is checked with ECAPA-TDNN cosine similarity to an averaged target embedding, flagging drift below 0.7.
Key Results
All evaluations are human listening tests on in-house data (Table 1, Table 2):
- Against a zero-shot baseline (no prompt in the AR decoder, default acoustic prompt), cascaded prompting gains +7.5% net win rate in naturalness CMOS (400 samples for each of 10 speakers, 50 crowd raters).
- On CVAD expressivity CMOS (50 samples from an emotional wellness set, five expert raters per sample), it gains +79.6% over the zero-shot baseline and +5.6% over the GPT-4o API.
- The RL-AES-CTC model after SFT beats SFT only by +7.1% average CMOS win rate (95% CI 3.97% to 10.23%).
- Training curves (Figure 3) show AES-CE rising while CTC loss stays controlled; without the CTC term, CTC loss grows, consistent with the reported reward hacking via text hallucination when optimizing AES-CE alone.
- Figure 2 shows speaker similarity varying with the granularity of acoustic prompt styles, motivating coarse-grained acoustic prompts; exact values are not tabulated.
The comparisons are against internal baselines and one commercial API; no standard benchmark, no objective intelligibility metric (WER/CER), and no automatic speaker-similarity numbers are tabulated. The authors acknowledge that much of the expressivity gain comes from human-in-the-loop prompt selection, so the +79.6% partly measures curation effort rather than a model property.
Novelty Assessment
The cascaded prompting component is engineering integration: mapping LLM-generated style tokens to curated reference prompts for a prompt-conditioned TTS uses existing capabilities, and the prompt selection is a manual curation procedure supported by an automatic aesthetic score. The online RL component combines familiar pieces (policy-gradient optimization with a KL anchor to the SFT model, an automatic quality predictor as reward) with one practically useful addition: a CTC alignment penalty in the reward to counter reward hacking. Conditioning the RL policy on the inference-time prompts is a sensible but modest design choice. The observation that AR-stage and acoustic-stage prompts can come from different speakers, decoupling prosody from timbre, is interesting but reported qualitatively. Everything (models, data, prompts) is in-house and code is not released, so the contribution is mainly an industrial recipe with limited reproducibility.
Field Significance
Moderate — The paper provides industrial evidence that, in a two-stage token-plus-acoustic TTS, single curated audio prompts can deliver large perceived expressivity gains without collecting emotional training data, and that aesthetic-reward RL on the AR stage needs an explicit intelligibility constraint to avoid hallucination. Its value is as a practical data point for prompt-driven conversational TTS and reward design rather than as a new architecture.
Claims
- supports: Curated reference prompts used as in-context conditioning can substantially improve perceived expressivity and naturalness of prompt-conditioned TTS without training on additional annotated emotional speech.
Evidence: Cascaded prompting with one human-selected prompt per fine-grained style beats a no-prompt zero-shot baseline by +7.5% naturalness CMOS and +79.6% CVAD expressivity CMOS, and beats the GPT-4o API by +5.6% CVAD CMOS. (§4.1, Table 1)
- complicates: Optimizing an autoregressive speech-token TTS model against an automatic perceptual quality reward alone leads to reward hacking in the form of text hallucination, so an alignment or intelligibility constraint is needed.
Evidence: Using AES-CE alone as reward produces severe text hallucinations; adding a CTC loss against the transcript keeps CTC loss controlled while AES-CE rises, whereas CTC loss grows when the term is disabled. (§3.3, §4.2, Figure 3)
- supports: Online reinforcement learning with a KL anchor to the supervised model improves human-rated quality of autoregressive TTS over supervised fine-tuning alone.
Evidence: The RL-AES-CTC model trained after SFT achieves a +7.1% average CMOS win rate over the SFT-only model (95% CI 3.97% to 10.23%). (§4.2, Table 2)
- refines: In two-stage TTS with a token-level AR model and an acoustic model, prosody is governed mainly by the AR stage and timbre mainly by the acoustic stage, so their reference prompts can be chosen independently.
Evidence: The AR prompt speaker need not match the acoustic prompt speaker; voice timbre is predominantly set by the acoustic model while the AR model primarily controls prosody. (§3.1.2)
- complicates: Varying reference prompts across turns at fine style granularity causes speaker drift in multi-turn synthesis, even when prompts come from the same speaker.
Evidence: Prompts from one speaker vary in volume, spatial characteristics, and timbre across sessions; grouping fine-grained styles into coarser acoustic-prompt categories reduces speaker drift as measured by ECAPA-TDNN similarity. (§3.1.2, Figure 2)
Limitations and Open Questions
Warning
All models, prompt data, and evaluation sets are in-house and no code is released; evaluations use internal protocols rather than a standardized benchmark, so results cannot be reproduced or compared with other systems.
- The expressivity gains depend heavily on human-in-the-loop prompt selection, which the authors themselves credit for most of the improvement; the cost and scalability of that curation as the number of styles grows are not quantified.
- No objective intelligibility (WER/CER) or speaker similarity numbers are tabulated; the hallucination suppression is shown only through CTC loss curves.
- The zero-shot baseline removes the AR prompt entirely, which is a weak comparison point for a prompt-based method.
- The AES-CE reward measures content enjoyment and acoustic quality, not style appropriateness; whether RL preserves the prompted style is not separately evaluated.
- Model sizes, pretraining data scale, and RL training budget are not reported.
Wiki Connections
- Emotion Synthesis — Controls fine-grained emotional speaking styles through LLM-generated style tokens bound to curated audio prompts, evaluated with a valence/arousal/dominance rating protocol.
- Zero-Shot TTS — Relies on reference-prompt in-context adaptation at inference for style and voice, and documents multi-turn speaker drift when prompts change.
- Autoregressive Codec TTS — The trained component is an AR model over discrete speech tokens, both prompted and RL-optimized.
- RLHF Speech — Applies online RL with an aesthetic-quality reward, a CTC penalty against reward hacking, and a KL anchor to the SFT policy.
- Subjective Evaluation — All headline results are human CMOS tests with crowd and expert raters, including a CVAD expressivity protocol.
- Tortoise TTS — The in-house TTS follows a TortoiseTTS-like AR-plus-diffusion design.
- GPT-4o — Used as an external expressivity baseline in the CVAD CMOS comparison.
- MoonCast — Cited as a representative two-stage AR-then-NAR conversational speech generator, the design family this system belongs to.
- ZipVoice-Dialog — Cited as the contrasting end-to-end flow-matching approach to dialogue generation.
- EmoVoice — Cited as prior work using free-form natural language descriptions for emotional TTS control, the alternative to curated audio prompts.