CoNLL 2025 · 2025 · Workshop
Pouw et al. (University of Amsterdam / Tilburg University) · → Paper · Demo: ? · Code: ✓
Using psycholinguistically inspired controlled stimuli, this paper reveals that TTS systems accurately place intonational phrase boundaries at simple syntactic boundaries but fail in syntactically ambiguous sentences, defaulting to the most probable intonation pattern rather than the structure-appropriate one — and that punctuation cues dominate over syntactic encoding in all tested systems.
Problem
Standard TTS evaluation measures naturalness and speaker similarity, but not whether synthesised speech reflects the linguistic structure of the input text. Human speakers reliably use intonational phrase boundaries to signal syntactic structure and to disambiguate ambiguous sentences; whether TTS systems learn this mapping from training data or merely learn to respond to surface markers like commas is an open question. A lack of syntactic sensitivity produces prosodically incorrect speech in complex sentence structures, potentially impeding comprehension.
Method
Three publicly available TTS systems with distinct architectures are evaluated: Tacotron2 (LSTM-based encoder-decoder with WaveNet vocoder), SpeechT5 (Transformer encoder-decoder with HiFi-GAN vocoder), and Parler-TTS Mini v0.1 (decoder-only Transformer with the Descript Audio Codec, trained on 10k hours from Multilingual LibriSpeech plus LibriTTS-R).
The evaluation uses two experiments drawing on controlled psycholinguistic stimuli. Experiment 1 targets syntactic ambiguity, using 45 garden path sentences and 2592 attachment ambiguity sentences (generated from templates). Each sentence is synthesised with and without a comma at the intended syntactic boundary position, to compare boundary placement with and without explicit punctuation cues. Experiment 2 targets simple clause boundaries, using Simple Wikipedia sentences with commas placed at syntactically natural positions, at syntactically unnatural positions, or absent.
Intonational boundaries are detected via two durational measures extracted by the Montreal Forced Aligner (MFA): pre-boundary word lengthening and silence insertion at the boundary position. From these, a Syntactic Sensitivity Score (precision, recall, and F1) is derived, measuring how accurately each system places pauses at syntactic boundaries vs. non-boundary positions.
A LASSO regression analysis then identifies which features (punctuation presence, constituency labels, dependency features, POS tags, length measures) best predict each system’s pause duration and pre-boundary word duration.
Finally, two fine-tuning experiments on Parler-TTS test whether redistributing training exposure improves syntactic sensitivity: one uses approximately 6 hours of prepositional phrase data from the Jenny corpus (commas removed from transcriptions), the other uses approximately 6 hours of synthetic high/low attachment sentences generated by Tacotron2 (also with commas removed). Both are diagnostic experiments rather than proposed training recipes.
Key Results
In Experiment 1, all three systems show a near-complete dependence on comma cues. Without commas, systems default to the statistically most probable intonation (no boundary in early-closure garden path sentences; no boundary in high-attachment ambiguity sentences). With commas, systems reliably produce both pre-boundary lengthening and pause insertion at the correct position. Semantic biases in attachment ambiguity sentences have no measurable effect on boundary placement in the absence of punctuation.
In Experiment 2, the Syntactic Sensitivity Score shows that all three systems can partially encode simple clause boundaries: the combined comma-plus-syntactic-cue condition produces the strongest boundaries, the syntactic-cue-alone condition produces weaker but non-zero boundaries, and the comma-at-unnatural-position condition produces boundaries at incorrect locations. Precision follows the reported MOS ranking (Tacotron2 < SpeechT5 < Parler-TTS), while recall diverges (SpeechT5 > Parler-TTS), demonstrating that spurious pauses at syntactically wrong positions are penalised more by human listeners than missed boundaries. (§5.3, Figure 3)
LASSO regression confirms comma presence as the strongest predictor across all three systems. SpeechT5 and Parler-TTS additionally select is_clause_boundary as a significant predictor; Tacotron2 does not, relying instead on token position and length features. Explained variance for pause duration is low for all systems (R² = 0.14–0.44, §6.1, Table 2), indicating that much of each system’s behaviour is not captured by surface features alone.
Fine-tuning on Jenny corpus data produces more differentiated pause durations that track syntactic function of ambiguous words (as, for, to, with). Fine-tuning on synthetic data increases overall pause duration but does not teach the model to distinguish high from low attachment based on semantic cues. (§7.3, Figure 5)
Novelty Assessment
The primary contribution is methodological: importing psycholinguistic experimental designs (garden path sentences, attachment ambiguity, minimal-pair stimuli) into TTS evaluation. The Syntactic Sensitivity Score is a simple but concrete metric for prosodic syntactic adequacy that complements MOS. No new model architecture is proposed; the fine-tuning experiments are diagnostic rather than a proposed training strategy.
The approach fills a genuine gap in TTS evaluation practice. Standard benchmarks do not assess whether prosody reflects syntactic structure. The limitation is scope: boundary detection relies on duration proxies only, omitting pitch and intensity; the study covers English only; and the evaluation systems are limited to three, all trained on audiobook speech which under-represents prosodically disambiguating speech patterns.
Field Significance
Moderate — This paper provides concrete evidence that current TTS systems function as punctuation-following pattern matchers for intonational phrase boundary placement in syntactically complex contexts, regardless of the underlying syntactic structure. It introduces a diagnostic evaluation framework grounded in psycholinguistics that demonstrates how naturalness scores can diverge from prosodic linguistic adequacy. The contribution is primarily methodological: the paper can serve as a template for more linguistically informed TTS evaluation beyond naturalness ratings.
Claims
-
complicates: Naturalness ratings do not fully reflect whether TTS prosody is linguistically appropriate.
Evidence: Syntactic Sensitivity Score precision and recall diverge from the MOS ranking across Tacotron2, SpeechT5, and Parler-TTS; specifically, recall does not follow MOS order, revealing that missed boundary errors have less impact on perceived naturalness than spurious boundary errors — a dimension MOS does not capture independently. (§5.3, Figure 3)
-
supports: Punctuation marks are the dominant surface cue for intonational boundary placement in TTS systems, overriding syntactic and semantic information in ambiguous contexts.
Evidence: All three systems place boundaries reliably only when commas are present; without comma cues in garden path and attachment ambiguity sentences, systems default to the statistically most frequent pattern regardless of the intended structure, and semantic bias in prepositional phrase attachment has no measurable effect on boundary placement. (§4.4, §5.3)
-
supports: Transformer-based TTS architectures encode partial clause boundary information for prosody that LSTM-based systems do not.
Evidence: LASSO regression selects
is_clause_boundaryas a significant predictor of pause duration for SpeechT5 and Parler-TTS but not for Tacotron2, which relies more on token position and length features. (§6.1, Figure 4) -
refines: Increasing TTS training exposure to syntactically diverse speech can improve prosodic alignment with syntactic structure for structural cues, but not for semantic disambiguation cues.
Evidence: Fine-tuning Parler-TTS on Jenny corpus prepositional phrase data produces more differentiated pause patterns correlated with syntactic function of ambiguous words; fine-tuning on synthetic semantic-bias data increases pause duration generally but does not teach the model to distinguish high from low attachment based on semantic context. (§7.3, Figure 5)
Limitations and Open Questions
The study measures intonational boundaries through duration proxies only (pause insertion and pre-boundary lengthening), omitting pitch contour and intensity changes that also contribute to boundary perception. The ToBI framework identifies multiple boundary strength levels that this methodology cannot distinguish.
Evaluation is limited to English and to three specific TTS systems trained on audiobook speech, which under-represents prosodically disambiguating speech patterns relative to conversational speech.
The Parler-TTS model supports natural-language voice conditioning, but only a single fixed voice description was used. How voice conditioning interacts with syntactic sensitivity is unexplored.
The fine-tuning results are diagnostic rather than prescriptive: datasets are small (~6 hours), and the synthetic data experiment suggests that semantic disambiguation remains out of reach even with targeted exposure.
Wiki Connections
- Evaluation Metrics — introduces the Syntactic Sensitivity Score (precision, recall, F1 at syntactic boundary positions) as a prosodic adequacy metric complementing MOS.
- Subjective Evaluation — demonstrates that MOS captures false-positive boundary errors (pauses at syntactically unnatural positions) more than false-negative ones (missed boundaries), revealing a structural blind spot in standard naturalness evaluation.
- Autoregressive Codec TTS — evaluates and diagnostically fine-tunes Parler-TTS (decoder-only Transformer with DAC), finding it achieves the highest syntactic sensitivity precision among tested systems while also showing the least predictable pause placement under LASSO regression.
- Transformer Encoder-Decoder TTS — evaluates SpeechT5, finding that its Transformer architecture encodes partial clause boundary information that the LSTM-based Tacotron2 baseline does not.