arXiv · 2025 · Preprint
Dai et al. · → Paper · Demo: ✓ · Code: ?
DeepDubbing is an end-to-end system for multi-participant audiobook synthesis that combines a flow-matching Text-to-Timbre (TTT) module — which generates character-specific speaker embeddings from natural language descriptions — with a context-aware instruction-conditioned TTS model that uses LLM-derived emotion-scene instructions to produce expressively appropriate speech.
Problem
Automating audiobook production at scale faces two compounding problems: assigning appropriate voice timbres to characters without manual selection from a fixed speaker pool, and synthesising speech with prosodic delivery that reflects the surrounding narrative context rather than treating each sentence in isolation. Existing TTS systems address neither well. Speaker selection methods rely on predefined timbre lists that cannot cover the diversity of characters in long-form fiction, while systems that synthesise sentence by sentence lack the broader context that voice actors use to calibrate emotional tone, resulting in flat and contextually inappropriate narration.
Method
DeepDubbing integrates two purpose-built modules into a single automated pipeline. First, an LLM parses the raw book text to identify all characters and generate a structured timbre description for each (specifying age, gender, identity, and personality), then derives emotion-scene instructions for each dialogue segment. The TTT module and CA-Instruct-TTS module then handle synthesis.
The Text-to-Timbre (TTT) model generates a speaker embedding from a natural language character description using Optimal-Transport Conditional Flow Matching (OT-CFM). The vector field estimator uses a Diffusion Transformer (DiT) backbone with four attention heads and 392 hidden dimensions. Text conditions are encoded with Qwen3-Embedding-0.6B (projected to 192 dimensions) and injected into each DiT block via Style-Adaptive Layer Normalisation (SALN); gender labels are incorporated through Feature-wise Linear Modulation (FiLM) and concatenation. Classifier-free guidance (scale 3.0) is applied at inference, and an Euler ODE solver integrates from Gaussian noise to the target speaker embedding. The training objective is an MSE loss regressing the true velocity field under the OT-CFM framework.
The Context-Aware Instruct-TTS (CA-Instruct-TTS) model follows a CosyVoice-style architecture: a 12-layer Transformer LM (continually trained from a closed-source QinYu-based speech model) autoregressively generates speech token sequences from four concatenated inputs: the speaker embedding, tokenised emotion-scene instruction, target text, and prior speech token context. A DiT-based conditional flow matching decoder then converts the predicted speech tokens together with the speaker embedding and masked acoustic features into a mel spectrogram, which an NSF-BigVGAN vocoder renders to a waveform. The emotion-scene instructions follow a structured template: [emotion] | [contextual scenario] | [utterance text], covering more than 44 fine-grained emotion categories.
The system also releases BookVoice-50h, a synthetic audiobook dataset with annotated timbre profiles and emotion-scene instructions generated by the DeepDubbing models themselves, intended to support further research on text-to-timbre mapping and expressive TTS.
Key Results
For the TTT module, three text encoders are compared: T5-Large, RoBERTa-Large, and Qwen3-Embedding-0.6B. On a balanced test set of 80 samples spanning four age groups:
- Sex Accuracy (SA): TTT-Qwen3-0.6B achieves 96.25% (child), 100% (youth), 100% (middle), 100% (elder).
- Age Accuracy (AA): TTT-Qwen3-0.6B achieves 74.38% (child), 74.38% (youth), 90% (middle), 73.13% (elder).
- Character Matching Score (CMS): TTT-Qwen3-0.6B scores 2.866 ± 0.036 (on a 0–4 scale), substantially above T5-Large (2.375) and RoBERTa-Large (2.359).
Child voice gender accuracy is notably lower across all encoders. The t-SNE analysis of generated embeddings shows that child voices cluster less distinctly by gender, consistent with known acoustic similarity in pre-pubertal speech.
For the CA-Instruct-TTS module, evaluated on 195 utterances covering 44+ emotion categories:
| System | WER | MOS-N | MOS-E |
|---|---|---|---|
| CA-TTS (no instruction) | 2.39% | 3.10 | 3.67 |
| CA-Instruct-TTS | 2.54% | 3.33 | 4.15 |
Adding emotion-scene instruction conditioning improves both naturalness (MOS-N: +0.23) and emotional expressiveness (MOS-E: +0.48) at the cost of a minor WER increase (0.15pp). All subjective tests used rigorously trained expert listeners.
Novelty Assessment
The TTT module represents a genuine architectural contribution: using conditional flow matching to generate continuous speaker embedding spaces from natural language descriptions is a distinct approach, differing from both classifier-based voice selection and diffusion-based speaker generation (DreamVoice). The multi-scale conditioning via SALN and FiLM within a DiT backbone for speaker embedding generation has not been previously combined in this form.
The CA-Instruct-TTS is primarily an engineering integration of the CosyVoice architecture with explicit emotion-scene instruction conditioning. The replacement of the original flow-matching decoder with a DiT-based variant and of the vocoder with NSF-BigVGAN are incremental refinements rather than structural innovations. The context-aware instruction mechanism itself is the meaningful addition.
The evaluation is limited to proprietary internal data, small test sets (80 and 195 samples), and lacks comparisons to strong expressive TTS baselines or published audiobook synthesis systems such as those based on context-aware prosody modeling.
Field Significance
Moderate — DeepDubbing introduces a practically motivated pairing: automated character voice assignment via text-to-timbre flow matching alongside context-aware instruction-conditioned TTS. The TTT module’s approach of generating speaker embeddings directly from natural language descriptions (rather than selecting from a fixed roster or conditioning on reference audio) is a useful direction for scalable multi-character speech synthesis. The demonstration is somewhat self-contained given the reliance on internal data, but the release of BookVoice-50h and the open demo provide a research foothold for others working on expressive audiobook TTS.
Claims
-
supports: Conditioning a flow matching model to generate speaker embeddings from natural language descriptions enables automatic voice assignment for characters without requiring reference audio.
Evidence: The TTT model, trained with OT-CFM to map text descriptions (age, gender, personality, identity) to speaker embeddings, achieves CMS of 2.866 and sex accuracy above 96% for adult demographics; Qwen3-Embedding-0.6B outperforms T5-Large and RoBERTa-Large on all character attribute metrics. (§2.2, §3.3, Table 2)
-
supports: Providing LLM-derived emotion-scene instructions as an additional conditioning signal to TTS improves emotional expressiveness without substantially degrading intelligibility.
Evidence: CA-Instruct-TTS achieves MOS-E of 4.15 vs. 3.67 and MOS-N of 3.33 vs. 3.10 compared to the instruction-free CA-TTS baseline, with WER increasing only from 2.39% to 2.54%. (§3.3, Table 3)
-
complicates: Embedding-based speaker generation from text degrades for child voices due to acoustic similarity and training data artifacts.
Evidence: The TTT model shows notably lower age accuracy for the child category (74.38% with TTT-Qwen3-0.6B) compared to adult categories (90%+). The paper attributes this to pre-pubertal voice similarity and to adult speakers imitating child voices in available training data. (§3.3, Figure 2)
-
supports: Larger, semantically richer text encoders improve text-conditioned speaker embedding generation across demographic categories.
Evidence: TTT-Qwen3-0.6B achieves substantially higher CMS (2.866) and age accuracy across most categories compared to TTT-T5-Large (CMS 2.375) and TTT-RoBERTa-Large (CMS 2.359), suggesting that language understanding capacity from pre-trained LLMs transfers to voice attribute generation. (§3.3, Table 2)
-
refines: The flow matching framework extends naturally from speech waveform generation to speaker embedding synthesis.
Evidence: The OT-CFM framework is applied directly to the generation of continuous speaker embedding vectors from noise, using the same DiT backbone and conditioning architecture as the speech-side flow matching decoder. The speaker embedding space is treated as the target data distribution. (§2.2)
Limitations and Open Questions
Warning
All evaluations use proprietary internal data. No comparisons to published expressive audiobook TTS systems are included, limiting the ability to independently verify the reported gains in emotional expressiveness.
The TTT model struggles with child voice generation, an acknowledged limitation tied to data scarcity and label noise in child speech corpora. Future work is noted toward collecting authentic child speech.
The CA-Instruct-TTS relies on a closed-source base model (QinYu), which prevents reproducibility of the LM component despite the open demo and dataset release.
The system has not been evaluated in multilingual settings, and the emotion-scene instruction templates were designed for Chinese-language audiobooks, raising open questions about cross-lingual or cross-cultural generalisation.
Wiki Connections
- Flow Matching — the TTT module uses OT-CFM to generate speaker embeddings from noise conditioned on text descriptions, extending flow matching beyond waveform synthesis.
- Instruction-Conditioned TTS — the CA-Instruct-TTS model accepts emotion-scene natural language instructions in a structured template as a conditioning signal for expressive speech synthesis.
- Emotion Synthesis — the CA-Instruct-TTS mechanism explicitly conditions on over 44 fine-grained emotion categories derived from narrative context, improving MOS-E by 0.48 over the instruction-free baseline.
- Autoregressive Codec TTS — the CA-Instruct-TTS follows a CosyVoice-style LM-plus-flow-matching architecture, autoregressively generating speech tokens before decoding to mel spectrogram.
- Zero-Shot TTS — the TTT module generates novel speaker identities for previously unseen characters from text descriptions alone, without requiring reference audio at inference time.