arXiv · 2025 · Preprint
Julio Cesar Galdino et al. · → Paper · Demo: ✓ · Code: ✓
Evaluates whether training a non-autoregressive TTS model on manually annotated prosodic boundaries, rather than automatically detected segment boundaries, improves the intelligibility and naturalness of synthesized Brazilian Portuguese spontaneous speech.
Problem
Spontaneous speech synthesis must reproduce turn-taking, pauses, and disfluencies that scripted-speech corpora rarely contain, and most TTS architectures address this only implicitly, by letting pitch, intensity, and duration predictors learn whatever regularities appear in the training data. The construction of the training corpus itself, and in particular how utterances are segmented into prosodic units before transcription, has received little attention. Automatic speech recognition tools such as WhisperX are increasingly used to build spontaneous-speech datasets for low-resource languages, but ASR segmentation is optimized for transcription accuracy and fixed-length chunking rather than for marking genuine prosodic boundaries. The paper asks whether this substitution loses prosodic information that matters for synthesis quality, using Brazilian Portuguese as a test case where a manually annotated spontaneous speech corpus already exists for comparison.
Method
The study is built on the NURC-SP Minimal Corpus (NURC-SP MC), a Brazilian Portuguese spontaneous speech corpus manually annotated by trained linguists (inter-rater kappa above 0.8) with terminal and non-terminal prosodic boundaries following the C-ORAL-BRASIL intonation-unit model. The authors constructed two parallel versions of the same 20 recordings: a “prosodic” subset that keeps only the manually annotated terminal intonation units, and an “automatic” subset produced by re-segmenting and re-transcribing the same audio with WhisperX, with no human revision. Both subsets were supplemented with the Portuguese portion of CML-TTS (59 hours, 30 speakers), which was necessary to reach convergence given the size of the NURC-SP recordings alone.
The synthesis model is FastSpeech 2 (mapped to the controlled-vocabulary term transformer-enc-dec), a non-autoregressive transformer TTS model chosen specifically because it explicitly conditions on duration, pitch, and energy extracted from the training waveform, with the corresponding predictors used at inference. The same open-source FastSpeech 2 training pipeline (grapheme-to-phoneme conversion and forced alignment via Montreal Forced Aligner, energy/pitch/duration extraction, then 720k training steps) was run twice under otherwise identical conditions: once on CML-TTS plus the prosodic subset, and once on CML-TTS plus the automatic subset. This isolates segmentation strategy as the single independent variable between the two trained models.
Two evaluations were then run. First, an alignment analysis treated the manual prosodic boundaries as ground truth and WhisperX boundaries as predictions, scoring precision, recall, and F1 for boundary detection as a binary classification problem. Second, intelligibility of synthesized speech was measured via WER/CER against human-annotated reference transcriptions for 361 held-out test segments from one speaker, and acoustic naturalness was assessed by comparing F0 contours: four canonical points (onset, pre-nuclear, nuclear, post-nuclear) were manually annotated on 30 neutral declarative test utterances in both natural and synthesized speech, and RMSE and per-point ANOVA/Tukey tests compared the two trained models against the natural reference.
Key Results
WhisperX’s automatic boundaries reached 60.95% precision and 79.40% recall against the manual prosodic boundaries (F1 = 68.96%), showing that WhisperX tends to over-segment relative to human annotators: automatic segmentation produced about 10,182 segments (12.79 tokens/segment on average) versus about 7,816 segments for the prosodic subset (14.33 tokens/segment), i.e. manual segments are longer and more variable (Table 1, §4.2).
On the held-out test speaker, FastSpeech 2 trained on the prosodic subset was slightly more intelligible than the automatic-subset model (WER 0.43 vs. 0.50, CER 0.31 vs. 0.35, both worse than the natural-speech transcription baseline of WER 0.16 / CER 0.09). A paired t-test found the WER gap significant (t = 2.589, p < 0.01) but the CER gap only marginal (t = 1.796, p = 0.07) (§5.1). On F0 contour matching, the prosodic model was closer to natural speech overall (RMSE ≈ 39.07 Hz vs. ≈ 44.05 Hz for the automatic model), and ANOVA/Tukey analysis showed the prosodic model did not differ significantly from natural speech at the pre-nuclear point (p = 0.08) while the automatic model did (p = 0.04); neither model differed significantly from natural speech at the nuclear point, indicating both reproduced the expected Brazilian Portuguese nuclear accent (§5.2). Visual inspection showed the prosodic model’s F0 curve preserved the natural low pre-nuclear dip before the nuclear rise, while the automatic model’s curve was a simple downward slope. Neither model matched natural speech’s F0 variation in semitones (both p < 0.01 vs. natural), so the naturalness gap was not fully closed by either segmentation strategy.
Novelty Assessment
The contribution is empirical rather than architectural: the paper uses an existing off-the-shelf TTS model (FastSpeech 2) and an existing ASR segmentation tool (WhisperX) without modification, and instead isolates a single training-data variable, prosodic versus automatic segmentation, under a controlled same-corpus, same-pipeline comparison. Its value lies in providing a direct, paired benchmark of a common low-resource practice (using ASR tools to build spontaneous-speech TTS corpora) against a costlier gold-standard alternative (linguist-annotated prosodic boundaries), for a language and speech style (spontaneous Brazilian Portuguese) where this trade-off had not previously been quantified. The public release of both the automatically segmented dataset variant and the trained checkpoints is a secondary, incremental dataset contribution rather than the paper’s main claim.
Field Significance
low — the study provides a small, carefully controlled data point on a specific and previously unquantified question (does prosodic-boundary training data help a non-autoregressive TTS model reproduce natural prosody in spontaneous speech), but it is limited to a single low-resource language, a single non-autoregressive architecture, and a small held-out test set, with effect sizes that are only partially statistically significant. It offers useful evidence for corpus-construction decisions in spontaneous-speech TTS but does not establish a new method or a broadly generalizable result on its own.
Claims
- supports: Training a TTS model on data segmented according to manually annotated prosodic boundaries, rather than automatically detected ASR segment boundaries, can yield modestly more intelligible and acoustically natural synthesized spontaneous speech.
Evidence: FastSpeech 2 trained on the manually segmented (“prosodic”) NURC-SP MC subset scored WER 0.43 / CER 0.31 versus WER 0.50 / CER 0.35 for the model trained on the WhisperX-segmented (“automatic”) subset, with the WER difference statistically significant (t = 2.589, p < 0.01) (§5.1).
- complicates: Automatic ASR-based segmentation tools can recover a majority of true prosodic boundaries but systematically over-segment relative to human prosodic annotation, which limits their reliability as a substitute for manual prosodic annotation in spontaneous-speech corpus construction.
Evidence: Treating manual boundaries as ground truth, WhisperX segmentation achieved 79.40% recall but only 60.95% precision (F1 = 68.96%), and produced ~2,366 more total segments than the manual annotation on the same 19 training interviews (§4.2, Table 1).
- refines: Explicit prosodic feature prediction in non-autoregressive TTS architectures (e.g., duration, pitch, energy predictors) can reproduce a language’s canonical accent pattern regardless of training-segmentation strategy, but reproducing the finer shape of the surrounding intonation contour (e.g., pre-nuclear dip) is more sensitive to how prosodically informative the training segmentation is.
Evidence: Neither the prosodic-segmentation nor the automatic-segmentation FastSpeech 2 model differed significantly from natural speech at the nuclear F0 point (ANOVA p = 0.86 across all three groups), but only the prosodic-segmentation model was statistically indistinguishable from natural speech at the pre-nuclear point (p = 0.08 vs. p = 0.04 for the automatic model) (§5.2).
- complicates: Improvements from better training-data segmentation do not necessarily close the full naturalness gap between synthesized and natural spontaneous speech, particularly for coarser measures of pitch range.
Evidence: Measuring average F0 variation in semitones, both the prosodic-segmentation and automatic-segmentation models differed significantly from natural speech (p < 0.01 for each), and the two training strategies did not differ significantly from each other (p = 0.99) (§5.2).
Limitations and Open Questions
The comparison is restricted to a single, small spontaneous-speech corpus (20 recordings, ~12-16 hours per subset before augmentation with CML-TTS) in one language and one non-autoregressive architecture, which limits how far the intelligibility and naturalness findings generalize.
The authors themselves note that WhisperX was not designed or trained to detect prosodic boundaries, so its underperformance against manual annotation may partly reflect a mismatched tool rather than a fundamental limit on automatic segmentation. The acoustic (F0 contour) analysis is based on only 30 manually selected “neutral declarative” utterances from a single test speaker, which constrains the statistical power of the prosodic comparison. The study also does not include a subjective listening evaluation (e.g., MOS), relying instead on objective WER/CER and F0 metrics. Finally, both the prosodic and automatic training subsets required supplementation with an unrelated read-speech corpus (CML-TTS) to reach convergence, meaning neither trained model is trained purely on spontaneous speech, which could interact with the segmentation comparison in ways the paper does not isolate. The authors plan to extend this comparison to other genres, languages, and autoregressive or flow-matching TTS architectures in future work.
Wiki Connections
- Transformer Encoder-Decoder TTS — uses FastSpeech 2, a non-autoregressive transformer TTS model, as the fixed backbone for isolating the effect of training-data segmentation strategy.
- Evaluation Metrics — applies WER/CER intelligibility metrics and F0-RMSE/ANOVA acoustic analysis to compare two training strategies under a controlled paired design.
- FastSpeech 2 — adopts FastSpeech 2 unmodified as the non-autoregressive TTS backbone specifically for its explicit duration, pitch, and energy predictors, using it as the shared architecture across both experimental conditions.