EMNLP · 2025 · Conference
Seung-Bin Kim et al. (Korea University / Samsung Research) · → Paper · Demo: ✓ · Code: ?
FillerSpeech is a flow-matching TTS framework that synthesizes speech with controllable non-lexical fillers (e.g., “uh”, “um”) by tokenizing filler pitch and duration as discrete style labels and using an LLM fine-tuned with LoRA to predict filler position, type, and prosodic attributes from text input alone.
Problem
Synthesized speech from standard TTS systems is unnaturally filler-free: it lacks the hesitation markers (“uh”, “um”, “well”, etc.) that characterize natural human conversation. Prior work on filler synthesis was limited to narrow filler vocabularies and offered no mechanism to control the prosodic style (pitch, duration) of individual filler instances. Existing sampling-based filler insertion methods use statistical distributions rather than contextual understanding. No prior system jointly addressed filler insertion, style control, and natural contextual prediction from text.
Method
Dataset. FillerSpeech extracts a filler-inclusive subset from LibriHeavy (50,000 hours) using a fixed vocabulary of 12 non-lexical fillers. Montreal Forced Aligner locates filler boundaries; Parselmouth extracts F0. Each filler is annotated with: pitch label (high/medium/low, gender-stratified, using a ±4 semitone threshold relative to per-filler-type median) and duration label (top/bottom 25% → long/short). Result: 2,116 hours, 4,460 speakers.
TTS model (60.11M params). Built on a flow-matching (OT-CFM) encoder-decoder architecture:
- Text encoder uses phoneme-level tokenization. Filler pitch and duration are embedded as discrete tokens and concatenated with phoneme embeddings. Cross-attention between phoneme-level text representations and word-level pitch embeddings enables fine-grained pitch conditioning.
- A dedicated pitch predictor (structure from FastSpeech 2) is integrated into the encoder so the model explicitly models pitch during training.
- Duration predictor takes text + duration tokens. Prior loss is computed over all encoder outputs (not a sampled subset) to avoid excluding rare filler tokens.
- Decoder: Transformer-based U-Net with OT-CFM flow matching. Vocoder: BigVGAN. Speaker style: Meta-StyleSpeech encoder.
- Alignment: Super Monotonic Alignment Search (MAS). Training: 1M steps, 2× NVIDIA RTX A6000, batch 32/GPU.
LLM filler prediction module. Vicuna-7B fine-tuned with LoRA (4.20M trainable params) to predict filler position, type, pitch, and duration from input text. Four prompt templates cover: type-only specified, type+position specified, candidate set provided, and candidate set + position. Comparison across LLaMA-1B/3B/8B, Qwen-1.5B/3B/7B, and Vicuna-7B shows Vicuna-7B achieves the best position accuracy (82.56%) and type accuracy (78.44%).
At inference, the LLM output is fed to the TTS model, which synthesizes the filler with the predicted style attributes, or users can manually override style labels.
Key Results
On the LibriHeavy filler test set (Table 1):
| System | MOS | fMOS | cMOS | UTMOS | WER | SIM |
|---|---|---|---|---|---|---|
| Ground Truth | 4.02 | 4.04 | 3.91 | 3.604 | 5.64% | — |
| Matcha-TTS baseline | 3.62 | 3.86 | 3.69 | 3.354 | 4.56% | 0.9269 |
| FillerSpeech (full) | 3.84 | 3.96 | 3.82 | 3.878 | 6.33% | 0.9309 |
The full model (tokenization + pitch predictor + cross-attention) improves MOS and UTMOS over baseline. Filler MOS (fMOS) and controllability MOS (cMOS) are both improved. The pitch predictor causes a WER increase (6.33% vs 4.56%) relative to the non-filler baseline, suggesting a trade-off between prosodic expressiveness and pronunciation accuracy that cross-attention partially mitigates.
For filler prediction, Vicuna-7B with LoRA achieves 82.56% position accuracy and 78.44% type accuracy vs. a random baseline of 4.32%/28.87% — confirming contextual rather than statistical prediction.
Novelty Assessment
The idea of synthesizing fillers with controllable prosodic style within a modern flow-matching TTS framework is novel. The combination of discrete pitch/duration tokenization, cross-attention pitch conditioning, and LLM-based filler prediction is a coherent and original pipeline. The dataset contribution (2,116-hour filler-labelled subset of LibriHeavy) is practically useful. The main limitation is that pitch controllability introduces a pronunciation accuracy cost, suggesting the style conditioning and text representations are not fully decoupled. The paper is primarily engineering and dataset contribution; the architectural novelty is incremental relative to Matcha-TTS.
Field Significance
Moderate — FillerSpeech addresses a gap that most expressive TTS work ignores: the presence of non-lexical fillers in natural conversational speech. The filler-labelled dataset extracted from LibriHeavy (2,116 hours, 4,460 speakers) provides a reusable resource for training and evaluating filler-inclusive synthesis. The custom fMOS and cMOS evaluations can serve as templates for future work assessing filler naturalness and style controllability.
Claims
- Adding pitch and duration style tokens to a flow-matching TTS encoder improves filler naturalness scores but increases word error rate, indicating that prosodic conditioning and phoneme representation compete for capacity in the text encoder. (§6.1, Table 1)
- Cross-attention between phoneme-level and word-level pitch representations partially recovers pronunciation accuracy lost when a pitch predictor is integrated into the encoder. (§6.2, Table 1)
- LLM-based filler prediction substantially outperforms sampling-based insertion baselines on both position accuracy and contextual naturalness, even when the LLM is fine-tuned from only 4.20M trainable LoRA parameters. (§6.3, Table 2)
- Dialogue-oriented LLM pretraining (Vicuna) provides a stronger prior for filler insertion than general instruction tuning (LLaMA, Qwen), suggesting that conversational context modeling is a key factor in filler placement quality. (§6.3.1, Table 3)
- Duration controllability in filler synthesis is harder to achieve than pitch controllability; pitch conditioning has negligible effect on predicted duration, pointing to a need for dedicated duration control mechanisms. (§6.2)
Limitations and Open Questions
- WER increases with filler insertion and pitch conditioning; the prosody-intelligibility trade-off is not fully resolved.
- Only non-lexical fillers are covered; lexical fillers (“like”, “you know”) are explicitly excluded.
- Evaluation is English-only (LibriHeavy).
- The pitch predictor improves pitch control but reduces pronunciation accuracy — future work could decouple these representations.
- The Vicuna-7B LLM is large (7B params) relative to the 60M TTS model; a smaller filler predictor would be more deployment-practical.
- Duration controllability is less effective than pitch controllability; the mechanism needs refinement.
Wiki Connections
- flow-matching — FillerSpeech extends OT-CFM flow matching with filler-specific style token conditioning and a modified prior loss that preserves rare filler tokens
- prosody-control — the paper demonstrates pitch and duration control at sub-word (filler) granularity using discrete token labels and cross-attention conditioning
- spoken-language-model — the LLM filler prediction module shows that dialogue-oriented language model pretraining transfers to conversational speech structuring tasks
- subjective-evaluation — the paper introduces fMOS (filler naturalness) and cMOS (style controllability) as domain-specific listening test variants for evaluating filler synthesis quality