arXiv · 2019 · Preprint
Zen et al. (Google AI) · → Paper · Demo: ? · Code: ?
LibriTTS reprocesses the LibriSpeech audiobook materials into a TTS-ready corpus at 24 kHz with sentence-level segmentation, preserved punctuation and capitalisation, SNR-based noise filtering, and contextual sentence access, yielding 585 hours from 2,456 speakers.
Problem
LibriSpeech, while popular for ASR, carries several properties that make it unsuitable as a TTS training corpus. Its audio is at 16 kHz, which is too low for production-quality synthesis. Speech is segmented at silence boundaries rather than sentence breaks, producing utterances that cut across sentence structure and obscure prosodic cues. All text is uppercased with punctuation stripped, removing information about sentence stress and pause boundaries. Background noise is present even in the designated clean subsets, because the LibriSpeech clean/other split was defined by ASR word error rate rather than acoustic quality. Neighbouring sentence context is unavailable, preventing models from learning inter-sentence prosody.
Method
LibriTTS is produced from the same source materials as LibriSpeech (MP3 audio from LibriVox and texts from Project Gutenberg) through a four-stage processing pipeline. First, book-level texts are split into paragraphs and sentences using a proprietary sentence splitter, then normalised with a weighted finite-state transducer that handles abbreviations, numbers, and semiotic classes. Second, chapter-level audio is transcribed with Google Cloud Speech-to-Text and the resulting transcript is aligned to the full book text to extract the corresponding chapter passage. Third, the chapter audio is force-aligned to the normalised sentences using YouTube’s auto-sync engine, a bidirectional LSTM acoustic model, and a sentence-specific language model; only sentences where every word aligns with zero edit distance are retained. Fourth, sentences with more than 71 words, unusually long average word durations, or WADA-SNR below 20 dB (clean subsets) or 0 dB (other subsets) are discarded; silences at utterance boundaries are trimmed.
The resulting corpus contains 585.8 hours across seven subsets that mirror LibriSpeech’s train/dev/test and clean/other splits, with 2,456 speakers (1,185 female, 1,271 male). Both original and normalised text variants are provided. Each utterance record includes pointers to neighbouring sentences, enabling models to condition on surrounding context. Audio is retained at 24 kHz after excluding any source file with a sampling rate below 24 kHz.
To validate the corpus, GMVAE-Tacotron models (with speaker embedding conditioning) are trained on train-clean from both LibriSpeech (460 h, 16 kHz) and LibriTTS (245 h, 16/24 kHz), and evaluated by crowd-sourced MOS on six held-out speakers.
Key Results
GMVAE-Tacotron trained on LibriTTS at 24 kHz achieves MOS above 4.0 for five of the six evaluation speakers (Table 4). The 24 kHz models outperform the 16 kHz LibriTTS models by an average of 0.175 MOS for female speakers and 0.133 MOS for male speakers, demonstrating a clear benefit from higher sampling rate. Comparing LibriSpeech (16 kHz) with LibriTTS (16 kHz) at matched sampling rate, female speakers improve by 0.15 MOS on average while male speakers show no consistent gain (-0.03), though the corpora differ in size (460 h vs. 245 h), so the text processing improvements cannot be isolated cleanly. A meaningful naturalness gap relative to natural speech remains, particularly for male voices (0.61 MOS gap vs. 0.16 for female), suggesting model configuration issues beyond the corpus itself.
Note
The LibriSpeech and LibriTTS 16 kHz conditions differ both in text processing and corpus size (460 h vs. 245 h), making it impossible to attribute observed MOS differences solely to text quality improvements. The authors acknowledge this confound directly in Section 5.2.
Novelty Assessment
The contribution is entirely in corpus construction rather than modelling. No new architecture, training objective, or inference method is introduced; the GMVAE-Tacotron baseline used for validation is a pre-existing model. The novelty lies in the pipeline choices: sentence-level segmentation (rather than silence-based), retention of punctuation and capitalisation, SNR-based quality filtering, and the provision of contextual sentence access. Each of these addresses a documented limitation of applying an ASR corpus to TTS. The corpus is notable for its scale (largest TTS-specific corpus at publication), speaker diversity (2,456 speakers), and permissive license (CC-BY 4.0), but the individual processing components are applications of established engineering tools rather than research contributions.
Field Significance
Tip
High — LibriTTS fills a gap that existed since neural TTS began scaling to multi-speaker settings: a large, permissively licensed, multi-speaker corpus with native 24 kHz audio and proper sentence-level segmentation. Its release removes the need for TTS researchers to derive their own cleaned versions of LibriSpeech, standardising data conditions across the community. The corpus enables work on multi-speaker TTS, zero-shot speaker adaptation, and prosody learning that would otherwise require proprietary studio recordings.
Claims
- Training TTS models at 24 kHz rather than 16 kHz produces measurably higher subjective naturalness scores even when the underlying model architecture is held constant. (§5.2, Table 4)
- Sentence-boundary segmentation and preserved punctuation are theoretically beneficial for TTS training, but their effect is difficult to isolate empirically because corpus size and acoustic quality covary when comparing corpus versions. (§5.2)
- SNR-based filtering of audiobook recordings improves corpus quality at the cost of significantly reducing yield: approximately 25% of clean-subset sentences are removed by a 20 dB threshold. (§4, Table 3)
- Gender disparities in TTS naturalness persist under multi-speaker training on large open corpora, with male voices showing larger gaps to natural speech than female voices. (§5.2, Table 4)
Limitations and Open Questions
Warning
The experimental validation uses a single model architecture (GMVAE-Tacotron) and evaluates only six speakers from the train-clean subsets. The corpus size is approximately half that of LibriSpeech due to stricter filtering, which means some model comparisons are confounded by training data volume.
The stricter filtering pipeline produces an imbalanced per-speaker duration distribution with higher variance than LibriSpeech, which may affect the quality of low-duration speakers disproportionately. The text normalisation and sentence splitting rely on proprietary Google tools, creating a dependency that cannot be reproduced externally if regeneration is needed. The paper does not evaluate the impact of contextual sentence information, leaving that as an open research question. The significant naturalness gap for male speakers is noted but not resolved, and is attributed tentatively to model configuration rather than corpus issues.
Wiki Connections
This paper directly informs evaluation-metrics and subjective-evaluation through its MOS evaluation methodology and the observation that naturalness scores differ substantially by gender. It provides foundational training and evaluation data for zero-shot-tts and speaker-adaptation research, where multi-speaker diversity is essential. The sentence-level segmentation and context access motivate work on prosody-control.
In-corpus reference: 1412.6980 (Adam optimiser, used to train the GMVAE-Tacotron baseline).