ACL · 2025 · Conference
Chuanqi Cheng et al. (Renmin University of China) · → Paper · Demo: ? · Code: ✓
DNASpeech introduces a movie-derived TTS dataset annotated with three types of situated prompts — Dialogues, Narratives, and Actions (DNA) — along with a leaderboard for the new Contextualized and Situated TTS (CS-TTS) task.
Problem
Prompt-based TTS methods have progressed toward controllable speech generation, but existing datasets provide only direct, attribute-style prompts (e.g., “angry, high pitch, slow speed”) that explicitly describe speech properties. These prompts require manual curation and do not reflect how contextual information naturally occurs around speech in practice. Two specific gaps are identified: (1) no dataset simultaneously provides conversational dialogue history, environmental scene descriptions, and speaker action/expression cues aligned to individual utterances; (2) the speaker diversity in contextual TTS datasets is limited, constraining exploration of varied acoustic characteristics.
Method
The paper proposes CS-TTS as a new task formulation and constructs DNASpeech to support it. The dataset is built from 126 movies spanning genres released between 1940 and 2023, collected from licensed sources via the MovieClip channel and direct purchases.
Dataset construction pipeline has five stages:
- Data collection — movie videos and scripts (from IMSDb) are gathered, with scripts parsed into four structural elements: Dialogues, Narratives, Actions, and Characters.
- Information extraction — subtitle SRT timestamps are used to extract speech clips at 16 kHz; scripts are parsed into structured units for each of the DNA prompt types.
- Cross-modal alignment — a two-stage process first applies Longest Common Subsequence (LCS) coarse-grained textual filtering (similarity ≥ 0.9), then fine-grained semantic similarity matching using the all-mpnet-base-v2 sentence transformer (threshold ≥ 0.7), yielding 22,975 aligned (speech, text, DNA) tuples totalling 18.37 hours.
- Speech denoising — Resemble Enhance (a denoiser and enhancer) is applied to remove background noise and restore audio quality.
- ASR quality filter — whisper-large-v3 is used to transcribe and verify alignment between speech and subtitle text; mismatched samples are discarded.
The result is a dataset with 2,395 distinct characters, 4,452 scenes, and a mean human quality assessment score of 2.57 out of 3 across clarity, emotional richness, speech speed, and contextual relevance.
Baseline model (DNA-TTS) is designed specifically for the CS-TTS task. It is a FastSpeech2-style (transformer-enc-dec) architecture with five modules: a BERT-based Phoneme Encoder, a BERT-based Context Encoder (with auxiliary classification tasks for emotion, pitch, energy, and speed), a Style Fusion module (cross-attention for aligning context with phonemes), a Variance Adaptor (duration, pitch, loudness prediction inspired by FastSpeech2), and a pre-trained HiFi-GAN vocoder for waveform generation. Speaker embeddings are inserted after style fusion to separate speaker identity from contextual style.
Leaderboard defines two subtasks with distinct evaluation metrics: CS-TTS with Narratives (MOS-E for environmental alignment, MOS-C for generation consistency) and CS-TTS with Dialogues (MOS-D for dialogue coherence, MOS-S for action-state alignment). Objective metrics include PESQ, MCD, F0 distance, and WER.
Key Results
Results are from Table 2, covering 11 baselines across three categories. Key findings:
- Codec TTS models (VALL-E, NaturalSpeech2, VoiceCraft) achieve the best objective scores overall (PESQ up to 4.38, WER down to 6.03 for VoiceCraft).
- VoiceCraft leads on MOS-D (3.88) and MOS-S (3.89), marginally above DNA-TTS (3.85 / 3.83).
- DNA-TTS achieves the best PESQ (4.10) and WER (6.36) among non-codec models, and edges out prompt-based baselines on MOS-E (3.96 vs. 3.94 for InstructTTS/VoiceLDM).
- None-prompt TTS methods score consistently lower on MOS-D and MOS-S (3.65–3.78), confirming that contextual prompts carry measurable benefit.
- Data quality is validated on general TTS: models trained on DNASpeech achieve MOS comparable to LJSpeech and DailyTalk (Table 1; FastSpeech2 on DNASpeech: 4.01 ± 0.07 vs. 3.97 ± 0.08 on LJSpeech).
Ablation on DNA-TTS (Appendix D, Table 3 in paper) shows that removing the context encoder’s classification task reduces MOS-D from 3.85 to 3.80 and PESQ from 4.10 to 3.86; further removing Style Fusion drops MOS-D to 3.74 and PESQ to 3.59, roughly matching none-prompt TTS performance.
Analysis of dialogue turn count shows performance peaks at 4–6 turns for both DNA-TTS and VALL-E, then degrades as context grows beyond 8 turns.
Novelty Assessment
The primary contribution is a dataset and task formulation rather than a new architectural advance. The CS-TTS framing — requiring TTS systems to infer speech style from narrative and dialogue context rather than direct acoustic attribute labels — is a meaningful reformulation of instruction-conditioned TTS that is better grounded in how contextual information naturally exists in the world. The automatic annotation pipeline combining LCS + semantic similarity for cross-modal movie alignment is pragmatic and replicable. The DNA-TTS baseline is incremental over FastSpeech2/PromptTTS, adding a context encoder and cross-attention fusion; it does not claim to be state of the art in architecture. The leaderboard and its custom MOS variants (MOS-E, MOS-C, MOS-D, MOS-S) are practical evaluation contributions, though the evaluator pool (8 people) is small. The core limitation is that codec TTS models, which were not designed for CS-TTS, already outperform the proposed baseline on most metrics, suggesting the baseline design has room to grow.
Field Significance
Moderate — DNASpeech provides the field with a movie-sourced dataset and leaderboard for contextually situated TTS, filling a gap left by attribute-label prompt datasets. Its primary value is enabling future work to benchmark systems on dialogue and narrative conditioning, not in advancing generation architecture. The evaluation framework (four task-specific MOS variants) can serve as a template for similar contextual TTS benchmarks.
Claims
- Incorporating conversational and environmental context into TTS prompts produces measurably more contextually appropriate speech than attribute-label prompts, even when the model architecture is otherwise unchanged. (§4.3, Table 2)
- Codec-based TTS systems, despite not being designed for contextual conditioning, outperform purpose-built prompt-based baselines on most metrics when evaluated on a situated TTS benchmark. (§4.4, Table 2)
- The benefit of dialogue context in speech generation is bounded: performance peaks at 4–6 dialogue turns and degrades significantly beyond 8 turns, suggesting that long-context integration remains an open problem. (Appendix F, Table 3)
- Movie scripts provide a viable automated source for constructing contextually rich TTS datasets, with speech quality comparable to purpose-built corpora on standard naturalness metrics. (§3.4, Table 1)
Limitations and Open Questions
- Movie-sourced speech may not generalize to real-world acoustic conditions or spontaneous conversational speech. The authors note plans to diversify to real-world contexts.
- DNA prompt types, while more comprehensive than prior work, do not cover all possible contextual cues (e.g., prosodic history, listener affect, cultural context).
- Performance degrades with more than 8 dialogue turns; long-context integration remains an open problem.
- The dataset is 18 hours with English speech only, limiting multilingual and scale applications.
- Human evaluator pool is small (8 individuals, Kendall’s W ≥ 0.5 threshold), which may reduce reliability of MOS estimates.
- No zero-shot generalization experiments are reported; it is unclear whether CS-TTS benefits transfer to unseen speaker-scene combinations.
Wiki Connections
- instruction-conditioned-tts — the CS-TTS task extends instruction-conditioned TTS by replacing attribute-label prompts with situated narrative and dialogue context
- evaluation-metrics — the leaderboard introduces four task-specific MOS variants (MOS-E, MOS-C, MOS-D, MOS-S) tailored to contextual and situated evaluation
- subjective-evaluation — the evaluation design and inter-rater agreement methodology (Kendall’s W) are directly relevant to subjective evaluation practice in TTS
- autoregressive-codec-tts — codec TTS systems (VALL-E, NaturalSpeech2, VoiceCraft) are evaluated as leaderboard baselines and consistently outperform prompt-based and non-prompt models