arXiv · 2026 · Preprint
Raymond Chung (Logistics and Supply Chain MultiTech R&D Centre) · → Paper · Demo: ✓ · Code: ?
Augments a small children’s-audiobook TTS dataset by concatenating emotionally matched sentences and adds a self-supervised contrastive loss to the style-token reference encoder, improving multi-sentence prosody and speaking-style prediction for kids’ story narration.
Problem
Long-form audiobook narration requires a TTS system that can produce multiple consecutive sentences with natural inter-sentence pausing and a speaking style that stays coherent across a page of text. Standard TTS training data (e.g. LJSpeech, LibriTTS) is segmented into short, single-sentence clips, and autoregressive TTS models are known to struggle when asked to generate longer outputs than they were trained on. Prior long-form work either concatenates raw consecutive sentences from a corpus (ignoring whether adjacent sentences are emotionally coherent) or relies on datasets far larger than the 6.5-hour Blizzard 2017 children’s-audiobook corpus used here. Emotional consistency between concatenated sentences matters for style-token-based expressive TTS: mixing sentences with very different emotional content in one training utterance can degrade what the style-token module learns to represent, and consecutive sentences in a story (e.g. character dialogue) frequently do not share the same emotion.
Method
The system is built on Tacotron2 with stepwise monotonic attention (for training stability with a reduction factor of 2 to extend synthesizable length) and a Text-Predicted Global Style Token (TP-GST) module that predicts a speaking-style embedding from text so style can be generated for unseen text at inference; a pre-trained WaveGlow vocoder converts the predicted mel-spectrogram to waveform. The codebase is a modification of Mellotron.
Two training-side contributions target the expressiveness module. First, emotion-coherent data augmentation: individual-sentence training audio is segmented, and a T5-based text emotion classifier (fine-tuned for sentence emotion recognition, 93% test accuracy) labels each sentence into one of seven emotion categories. Consecutive multi-sentence training utterances are then formed by concatenating same-emotion sentences rather than simply consecutive sentences from the source text, on the reasoning that mixed-emotion concatenation would corrupt what the global style token module learns. A silent pause, drawn from a normal distribution fit to real inter-sentence gap durations in the target corpus (mean 509 ms, std 223 ms), is inserted between concatenated sentences so the model learns natural pause timing rather than concatenating raw audio directly.
Second, self-supervised contrastive style training: two augmented views of each training utterance are created via random 500 ms segment masking (pitch shifting and other perturbations are avoided because they would alter speaking style itself), and a SimCLR-style contrastive loss is applied directly to the GST reference encoder, added to the total TTS training loss with a scaling factor of 0.1. The goal is a reference encoder that extracts a more consistent style embedding across masked views of the same utterance. For multi-sentence training samples, the concatenated text is passed through the encoder as a single input, then the encoder output is split per sentence and TP-GST predicts a separate style embedding for each sentence, which is frozen and passed to the decoder to simulate inference-time behavior. The model is pre-trained on LJSpeech (300 epochs), then LibriTTS (200 epochs, following Mellotron’s filtering) to develop the GST/TP-GST modules, then fine-tuned on the target speaker, a female child storyteller from the Blizzard Challenge 2017 dataset (100 epochs). Model size and vocoder architecture details beyond WaveGlow are not reported.
Key Results
Four model variants are compared: M1 (1-sentence utterances only), M2 (1- and 2-sentence utterances from raw consecutive audio, the long-form baseline), M3 (1-sentence plus emotion-coherent augmented 2-sentence audio), and M4 (M3 plus the self-supervised contrastive loss). On the held-out test set (Table 4), L1 loss between the TP-GST-predicted and audio-extracted style embedding drops from 0.212 (M1) to 0.155 (M2) to 0.119 (M3) to 0.075 (M4), and emotion-classification accuracy of the extracted style embedding (evaluated via an SVM on an unseen speaker from the Emotional Speech Dataset) rises from 71.5% (M1) to 75.3% (M4), with M2 slightly below M1 (70.7%) despite using more data. For inter-sentence pause timing, a two-sample Kolmogorov-Smirnov test against ground-truth pause durations gives M3 a K-S statistic of 0.247 (p = 0.630, not significantly different from the ground-truth distribution) versus M1’s K-S statistic of 0.490 (p = 0.0271, significantly different), indicating the pause-modeling augmentation produces more natural pause timing than a model never trained with multi-sentence pauses at all. In a crowdsourced MOS study with 8 UK-based raters recruited via Prolific (Table 5), M4 scored higher than the M2 baseline on both naturalness (3.25±0.56 vs. 3.19±0.56) and appropriateness (3.42±0.52 vs. 3.36±0.45); both intervals overlap substantially given the small rater pool.
Novelty Assessment
The architectural components (Tacotron2, stepwise monotonic attention, TP-GST, WaveGlow, the Mellotron codebase, and SimCLR-style contrastive loss) are all pre-existing; the paper’s contribution is a training-recipe and data-augmentation combination rather than a new model structure. The emotion-coherent sentence-concatenation strategy is a straightforward but previously untested variant on existing long-form-audio concatenation approaches (which typically use raw consecutive sentences), and applying a SimCLR-style contrastive objective to a GST reference encoder specifically (as opposed to speech representation learning generally, where such objectives are established) is the paper’s other incremental contribution. The evaluation is narrow: a single speaker, a single small dataset (Blizzard 2017), and an 8-rater subjective study, which limits how far the reported gains can be expected to generalize.
Field Significance
low — This paper demonstrates that emotion-aware sentence concatenation and a contrastive auxiliary loss on the style-embedding encoder can improve style-prediction accuracy and pause naturalness on a small, single-speaker long-form TTS task, using an older (Tacotron2-era) backbone. The techniques are architecture-agnostic in principle but are validated only in this narrow setting.
Claims
- supports: Concatenating training utterances from emotionally consistent sentences, rather than merely consecutive sentences, improves the quality of a text-predicted style embedding for expressive long-form speech synthesis.
Evidence: Style-embedding prediction L1 loss drops from 0.155 (raw-consecutive-sentence training, M2) to 0.119 (emotion-matched concatenation, M3), and unseen-speaker emotion-classification accuracy of the extracted embedding rises from 70.7% to 75.1% (§IV-C, Table 4).
- supports: Modeling the duration of inter-sentence pauses during training, using a distribution fit to real pause statistics, produces synthesized pause timing closer to ground truth than a model trained only on isolated single-sentence utterances.
Evidence: A two-sample Kolmogorov-Smirnov test against ground-truth pause durations gives the pause-augmented model a K-S statistic of 0.247 (p = 0.630) versus 0.490 (p = 0.0271) for the single-sentence-only model, indicating the augmented model’s pause distribution is not statistically distinguishable from ground truth while the baseline’s is (§IV-C).
- supports: Applying a contrastive self-supervised loss to a style-token reference encoder, using masked-segment views of the same utterance as positive pairs, can further sharpen the extracted speaking-style representation beyond what supervised TTS training alone produces.
Evidence: Adding the SimCLR-style contrastive loss (M4) reduces L1 style-prediction loss to 0.075 from 0.119 (M3, without the contrastive loss) and raises emotion-classification accuracy from 75.1% to 75.3% (§IV-C, Table 4).
- complicates: Simply extending training data with longer concatenated utterances does not by itself guarantee better style-embedding quality if the concatenated sentences are not emotionally coherent.
Evidence: The model trained on raw consecutive-sentence audio (M2) achieves slightly lower emotion-classification accuracy (70.7%) than the single-sentence-only model (M1, 71.5%), despite using strictly more training data (§IV-C, Table 4).
Limitations and Open Questions
Evaluation is confined to a single speaker, a single 6.5-hour dataset, and 8 subjective raters, which limits confidence in how far the reported naturalness and appropriateness gains generalize.
The MOS naturalness and appropriateness improvements over the baseline are numerically modest and the reported confidence intervals overlap substantially, so the subjective evidence for a perceptible quality gain is weaker than the objective (L1 loss, K-S statistic) evidence. The paper notes that residual F0-range mismatch with the WaveGlow vocoder’s training data likely caps achievable naturalness regardless of the style-embedding improvements. The authors flag their own text emotion classifier (fine-tuned T5) as a bottleneck and suggest a stronger text model could improve the quality of emotion-coherent augmented data; they also note the method has not been tested across languages or with autoregressive-decoding pause insertion at inference time (rather than at training-data construction time only).
Wiki Connections
- Emotion Synthesis — introduces an emotion-coherent data augmentation strategy that concatenates training sentences by matched emotion label to improve the reliability of an expressive TTS style embedding.
- Prosody Control — models inter-sentence pause duration explicitly during data augmentation and evaluates it with a distributional (Kolmogorov-Smirnov) comparison against ground-truth pause timing.
- Speaker Adaptation — fine-tunes a multi-stage pre-trained (LJSpeech, then LibriTTS) TP-GST/Tacotron2 model onto a single target storyteller voice using a small (6.5-hour) target-speaker corpus.
- Subjective Evaluation — reports a crowdsourced MOS study with human raters scoring naturalness and contextual appropriateness of synthesized story narration.
- Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — uses Tacotron2 with stepwise monotonic attention as the base acoustic model, explicitly preferring it over FastSpeech2 for expressive long-form synthesis.
- FastSpeech 2 — considered and rejected as a backbone because its non-autoregressive, pre-decoupled acoustic-feature design was judged to limit expressive long-form speech synthesis capability.
- LibriTTS — used as an intermediate pre-training corpus (200 epochs) to develop the global style token and TP-GST modules before fine-tuning on the target storyteller voice.