arXiv · 2025 · Preprint
Michel et al. (Deezer Research / Loria / IDIAP) · → Paper · Demo: ✓ · Code: ✓
LibriQuote is a 18K-hour audiobook dataset segmented around fictional character quotations rather than arbitrary sentence breaks, paired with LLM-extracted speech-verb and adverb pseudo-labels, and used to benchmark and fine-tune expressive TTS — revealing that flow-matching models benefit substantially from this targeted data while autoregressive models gain primarily in intelligibility.
Problem
Standard large-scale audiobook corpora (LibriSpeech, LibriTTS, LibriHeavy) are constructed with arbitrary sentence-level segmentation that mixes neutral narration and expressive character dialogue in the same audio segments. This mixing produces complex multi-modal pitch distributions within single training examples, which hinders TTS models from learning expressive speech patterns. At the same time, no publicly available benchmark specifically targets contextual expressivity — the ability to synthesize speech in a style consistent with the surrounding narrative — leaving the field without a way to measure progress on this dimension. Discrete emotion label datasets (IEMOCAP, RAVDESS) are small and constrained, while in-the-wild corpora lack the structured narrative context needed to evaluate intent-appropriate delivery.
Method
LibriQuote is constructed from LibriVox English fiction recordings using a pipeline that aligns audio against Project Gutenberg book texts, detects character quotation boundaries with BookNLP, and segments audio accordingly. The result is a training set of 5,359 hours of character quotations and 12,723 hours of neutral narration across 3,314 speakers and 2,991 books. A filtered high-expressivity subset (Q_f, 379 hours) is built by retaining quotations where Phi-4 (with self-reported confidence scoring) extracts a speech adverb or an explicitly expressive speech verb from the narrative context — producing pseudo-labels such as “he whispered softly” that characterise intended delivery.
The test set (7.4 hours, 15 unseen speakers) is paired such that each quotation is matched with a nearby narration utterance as a neutral reference, enabling cross-sentence evaluation of expressivity. Evaluation uses WER (Whisper-Large v3), speaker similarity (WavLM-large cosine similarity), emotion similarity (Emotion2Vec cosine similarity), and two novel contextual metrics: ContextMOS (a 1–5 rating by Gemini-2.5 Pro of how well the synthesized speech intent matches the book context) and Win-Rate (a pairwise comparison against ground truth).
Training experiments fine-tune SparkTTS (autoregressive, Qwen2.5-based) and F5-TTS (flow-matching, DiT-based) on LibriQuote subsets, and also train SparkTTS from scratch on full narration+quotation data. A contextual conditioning variant replaces SparkTTS’s text input with the surrounding narrative paragraph as additional context. The benchmark section evaluates four open-source systems — SparkTTS, F5-TTS, MaskGCT, and 2506.21619 (IndexTTS2) — on LibriQuotetest with both objective and subjective metrics.
Key Results
Fine-tuning F5-TTS on Q_f yields the largest expressivity gains among all variants: ContextMOS increases from 2.95 to 3.33 (0.38 points), while WER remains comparable (6.6 vs. 6.9). Fine-tuning SparkTTS improves WER substantially (2.0–2.1 on LibriSpeech-PC vs. 3.06 baseline) but does not meaningfully improve ContextMOS (2.89–2.97 vs. 2.94 baseline), suggesting the autoregressive training objective is less responsive to expressive data than the flow-matching objective.
Training SparkTTS from scratch on Q alone raises ContextMOS to 3.09 (from 2.94), and adding contextual narrative conditioning pushes it to 3.15 — at the cost of substantially degraded intelligibility (WER 9.5 vs. 4.8). Using the full dataset (N∪Q) during scratch training recovers intelligibility and reaches ContextMOS 3.30, the strongest result across SparkTTS variants.
In the zero-shot benchmarking (Table 4), IndexTTS2 achieves the highest ContextMOS (3.33) and Win-Rate (46%), with IndexTTS2-Context (emotion conditioned on narrative rather than text) reaching 3.45 ContextMOS and 54% Win-Rate — matching human ground-truth performance. Human evaluation shows IndexTTS2 achieves positive CMOS (+0.25) relative to ground truth, meaning it is rated more expressive than amateur audiobook readers on average, while SparkTTS, F5-TTS, and MaskGCT all score negative CMOS (~−0.97–0.98).
Note
ContextMOS and Win-Rate are computed with Gemini-2.5 Pro as judge on a subset of 201 quotations matched with adverb pseudo-labels, not the full test set. MOS and CMOS subjective scores use only 30 samples (2 per speaker), limiting statistical precision.
Novelty Assessment
The primary contribution is the dataset and its narrative-aware construction methodology: LibriQuote is the first large-scale corpus explicitly segmented around character quotations with LLM-extracted delivery pseudo-labels. The contextual metrics (ContextMOS, Win-Rate) adapt the LALM-as-a-judge framework from EmergentTTS-Eval to audiobook expressivity, which is a useful methodological extension but not a substantial departure from existing automated evaluation practice. The finding that flow-matching models benefit more from fine-tuning on expressive data than autoregressive models is an interesting empirical observation, though the mechanism is not rigorously isolated — differences in model scale, training data, and architecture could all contribute. The contextual conditioning experiment is exploratory and the gains are modest.
Field Significance
Moderate — LibriQuote fills a genuine gap in the available training and evaluation resources for expressive TTS, and its narrative-aware segmentation principle is a useful design contribution for future corpus construction. The benchmark results reveal a meaningful separation between current systems on contextual expressivity, establishing LibriQuotetest as a challenging test set. However, the finding that amateur LibriVox readers themselves achieve only mediocre ContextMOS scores sets a ceiling on what constitutes “human-level” expressivity in this benchmark, and the practical impact of fine-tuning on the full corpus (as opposed to the current 379-hour expressive subset) remains unexplored.
Claims
- Narrative-aware segmentation of audiobook data into character quotation and narration subsets yields training material with measurably higher emotional diversity than standard sentence-level audiobook splits. (§3.2, Table 5)
- Flow-matching TTS models show larger expressivity gains from fine-tuning on targeted expressive speech data than autoregressive models with equivalent training setups. (§4.3, Table 2)
- Conditioning TTS synthesis on surrounding narrative context rather than only the target utterance text improves contextual appropriateness of synthesized speech at the cost of modest intelligibility degradation. (§4.3, Table 2)
- Current open-source TTS systems are substantially less expressive than human audiobook narrators on contextual benchmarks, even when naturalness scores (MOS) are comparable. (§5.2, Table 4)
- LLM-extracted speech-delivery pseudo-labels (verbs and adverbs) from narrative prose are reliable enough to serve as training signals for expressive TTS, achieving high precision when confidence-filtered. (§3.3, Figure 2)
Limitations and Open Questions
Warning
The LibriQuotetest ground-truth is recorded by amateur LibriVox volunteers, who themselves show insufficient expressivity for many quotations. ContextMOS scores for ground-truth (3.55 average) are only marginally above the best TTS systems, raising questions about whether the benchmark captures professional audiobook narration standards or amateur reading behaviour.
The training set is not WER-filtered, meaning a small proportion of transcription errors may remain. The gender distribution of LibriVox speakers is unknown, introducing potential bias. Experiments use only English fiction; cross-lingual and non-fiction applicability is untested. The contextual conditioning results are encouraging but use only one-paragraph context windows — longer narrative context might yield larger gains. LibriQuote’s utility for neural audio codec training (noted as future work) remains to be demonstrated.
Wiki Connections
Concepts: prosody-control · emotion-synthesis · evaluation-metrics · subjective-evaluation · zero-shot-tts · self-supervised-speech
Related papers (in-corpus): 2506.21619 (IndexTTS2 — the strongest benchmark system evaluated on LibriQuotetest)