arXiv · 2025 · Preprint
Min Liu et al. (Ximalaya Inc.) · → Paper · Demo: ✓ · Code: ✗
Proposes a context-aware, emotion-controllable TTS framework built on CosyVoice2 that models cross-sentence context and decouples style control from the speaker prompt, targeting coherent multi-character audiobook narration and dialogue.
Problem
Standard TTS systems synthesize speech one sentence at a time, which is inadequate for multicast audiobooks: long-form fiction requires consistent narrative prosody across hundreds or thousands of consecutive sentences, distinct and stable character personas across a chapter, and fine-grained emotional/instruction control that does not collide with the voice identity carried by the reference (prompt) audio. Prior automated audiobook pipelines either wrap existing TTS backbones in multi-agent orchestration without modifying the underlying synthesis model, or introduce context modeling in a coarse way that lacks controllability, or rely so heavily on extended context that they introduce redundancy and persona drift. Existing instructed TTS systems such as CosyVoice2 improve controllability but do not adapt well to audiobook-specific narrative structure.
Method
Audiobook-CC extends CosyVoice2’s architecture: it keeps the text/speech tokenizer and flow-matching decoder but replaces the HiFi-GAN vocoder with BigVGAN for improved fidelity and robustness. The core is an autoregressive speech language model trained to predict shifted speech tokens conditioned on a Cam++-derived speaker embedding, a structured contextual sequence, a structured control sequence, and the input text tokens. The contextual sequence explicitly demarcates pre-context and post-context sentence spans with boundary tokens (preS/preE, poS/poE), while the control sequence demarcates emotion/volume/rate attribute tokens with EM/EM boundary markers, giving the model an explicit sequence-level representation of both narrative context and instruction control rather than relying on an implicit memory mechanism.
Three training-side mechanisms operationalize this design. First, a decoupled training strategy separates the audio used as the speech prompt from the target audio (rather than the conventional coupled prompt-target pairing), reducing cross-audio prosodic interference so that synthesized prosody tracks the current text’s semantics rather than the prompt’s prosody; a multi-constraint prompt-selection procedure (same-chapter sampling, a tuned voiceprint-similarity threshold, and supplementary multi-emotional data) is used to prevent this decoupling from causing timbre drift. Second, natural-language instructions (e.g., “shouting angrily”) are decomposed at training time into discrete attribute labels (e.g., emotion intensity, volume, speed) so the model learns an explicit attribute-to-acoustic mapping; at inference, a separate module converts free-form instructions into the same attribute space. Third, a self-distillation procedure addresses the scarcity of high-intensity emotional training data: a pretrained emotional TTS model generates samples across intensity levels, these are filtered by phoneme error rate, speaker similarity, and pitch, and the resulting filtered set is used to rebalance the intensity distribution via targeted augmentation. Training proceeds in three stages on top of a CosyVoice2 checkpoint: domain adaptation on 1M hours of audiobook audio, context/instruction fine-tuning on a 100K-hour context-annotated corpus plus 500 hours of emotion-labeled recordings, and a final controllability/expressiveness stage on a 5K-hour self-distilled augmentation set, using AdamW with staged learning rates (1e-5, 1e-5, 1e-6) across 720K/300K/10K steps on 64 A800 GPUs.
Key Results
Against CosyVoice2 and MOSS-TTSD baselines on three internally constructed test sets (Test-NAR, Test-DIA, Test-CHAP), the full model (context + instruction inference, “Infer-ctx&inst”) reaches Dialogue S-MOS 4.11±0.06 and Chapter M-MOS 4.25±0.11, versus 3.84±0.09 and 3.88±0.07 for CosyVoice2, and 3.67±0.07 and 3.72±0.09 for MOSS-TTSD (Table 1); dialogue-test gains over both baselines are consistently larger than narration-test gains, and ABX preference tests corroborate the MOS ranking (Fig. 2). Ablations isolate each mechanism: the decoupled prompt/target training strategy raises S-MOS from 3.45 to 3.82-3.93 relative to the non-decoupled variant while reducing an inflated speaker-similarity score (SS 0.87 to 0.69-0.8) that the authors interpret as timbre/prosody over-similarity rather than genuine quality (Table 2); adding sentence-level pre/post-context text raises S-MOS further and is illustrated with a qualitative example where the model inserts context-appropriate laughter absent an explicit emotion label. On the CV3-Eval emotional test set, the proposed instructed-inference model shows larger high- vs. low-intensity emotion discriminability (ΔF1) than a CosyVoice2-instruct baseline, particularly on text-unrelated prompts (e.g., angry ΔF1 0.31 vs. 0.07), and higher mixed-emotion S-MOS (3.87 vs. 3.35 on text-unrelated prompts; Tables 3-4). All comparisons are against the paper’s own baselines run on internal test sets; no external or public benchmark is used.
Novelty Assessment
The contribution is best characterized as a targeted architectural and training-recipe extension of CosyVoice2 for a specific downstream use case (long-form multicast audiobook narration), not a new synthesis paradigm. The AR speech-LM backbone, flow-matching decoder, and speaker-embedding conditioning are inherited wholesale; the vocoder swap to BigVGAN is a component substitution. What is genuinely new is the explicit sequence-level encoding of contextual and control boundaries (structuring context and instruction as demarcated token spans rather than an opaque memory module), the decoupled prompt/target training strategy for separating style from voice identity, and the self-distillation pipeline for expanding high-intensity emotional data. Each is a modest but concrete training-side innovation with supporting ablations; none individually would constitute a strong standalone contribution, but together they form a coherent, well-validated adaptation of an existing backbone to a narrow but commercially relevant task. The evaluation, while thorough in ablation coverage, relies entirely on proprietary datasets and internally constructed test sets, and the only baselines are two prior systems (CosyVoice2, MOSS-TTSD) rather than a broader field comparison.
Field Significance
Moderate — this paper demonstrates that explicit sequence-level context and control tokens, combined with decoupled prompt/target training, can measurably improve narrative coherence and instruction controllability for long-form, multi-character TTS built on an existing AR speech-LM backbone. Its contribution is a domain-specific engineering and training-recipe advance rather than a new architecture, and its evaluation is confined to proprietary Chinese-language audiobook data.
Claims
- supports: Decoupling the training pairing between the speech prompt and the synthesis target (rather than using identical prompt/target audio) improves text-prosody alignment in style-controllable TTS, at the cost of an inflated speaker-similarity score under the coupled setup.
Evidence: Switching from non-decoupled to decoupled prompt/target training raises S-MOS from 3.45±0.09 to 3.82-3.93±0.06-0.07 while reducing speaker-similarity (SS) from 0.87 (interpreted as over-similar timbre/prosody) to 0.69-0.8. (§3.3, Table 2)
- supports: Providing an autoregressive speech language model with explicit surrounding-sentence text context improves narrative coherence in long-form speech generation beyond what sentence-isolated synthesis achieves.
Evidence: Adding pre/post-context text sequences raises Chapter M-MOS from 3.88±0.07 (CosyVoice2 baseline, sentence-level synthesis) to 4.13±0.09 (Infer-ctx), and a qualitative example shows the model generating contextually appropriate laughter for a sentence with no explicit emotional markers when given the preceding sentence’s context. (§2.1, §3.3, Table 1)
- complicates: The strength of the prompt/target decoupling threshold used during training trades off speaker-identity stability against naturalness, rather than improving both simultaneously.
Evidence: Sweeping the voiceprint-similarity clustering threshold shows a lower threshold reduces speaker-similarity (with occasional timbre discontinuities) but yields slightly higher S-MOS, while a higher threshold approaches the non-decoupled model’s inflated similarity at the cost of lower S-MOS. (§3.3, Table 2)
- supports: Self-distillation from a pretrained emotional TTS model, filtered by objective quality metrics, can mitigate the scarcity of high-intensity emotional training data and improve controllable emotional expressiveness.
Evidence: On the CV3-Eval text-unrelated test set, the model trained with 500 hours of emotional data plus 5,300 hours of self-distilled augmentation achieves larger high- vs. low-intensity F1 discriminability than a CosyVoice2-instruct baseline (e.g. angry ΔF1 0.31 vs. 0.07) and higher mixed-emotion S-MOS (3.87±0.09 vs. 3.35±0.07). (§2.2, §3.3, Table 3, Table 4)
Limitations and Open Questions
Warning
All training data (1M-hour audiobook corpus, 100K-hour context-aware corpus, 500-hour emotional corpus, 5K-hour augmentation set) and all three test sets (Test-NAR, Test-DIA, Test-CHAP) are proprietary and internally constructed; no code, model weights, or evaluation data are released, which limits independent verification of the reported gains.
The evaluation is confined to Chinese-language audiobook content with a single subjective rater pool (50 native Chinese speakers); generalization to other languages or narration styles is untested. The decoupling-threshold ablation reports trends across tested values without identifying a clearly optimal setting, and the paper notes the risk of persona instability at high thresholds is hypothesized rather than directly measured across the full range. The authors flag future work on scaling chapter-level context data and applying reinforcement learning for further quality improvement.
Wiki Connections
- Instruction-Conditioned TTS — decomposes free-form style instructions (e.g. “shouting angrily”) into discrete attribute labels for emotion, volume, and speed, with a dedicated inference-time module for parsing user instructions.
- Emotion Synthesis — introduces a self-distillation pipeline that filters and rebalances high-intensity emotional training data to improve controllable emotional expressiveness.
- Zero-Shot TTS — retains prompt-speech-based voice specification inherited from its CosyVoice2 backbone for character voice casting in the audiobook pipeline.
- Prosody Control — decouples style/prosody control from the speaker prompt via a dedicated training strategy so synthesized prosody follows the text semantics rather than the prompt audio.
- Disentanglement — proposes an explicit training paradigm that separates speech-prompt identity from style/prosody control to reduce cross-audio prosodic interference.
- Autoregressive Codec TTS — builds its text-speech language model directly on CosyVoice2’s autoregressive speech-token prediction architecture, extended with structured context and control token sequences.
- CosyVoice 2 — the architectural backbone this paper extends, reusing its text/speech tokenizer and flow-matching module while swapping the vocoder and adding context/control mechanisms.
- MoonCast — cited as a long-context podcast-generation system that models context in a coarser way than this paper’s fine-grained, controllable approach.
- CosyVoice 3 — its CV3-Eval Chinese emotional speech test set is used to evaluate this paper’s fine-grained emotional controllability.
- emotion2vec — used as the emotion classification tool underlying the F1-score metric for single-emotion controllability evaluation.