arXiv · 2026 · Preprint
Tianle Yang et al. (University at Buffalo / Australian National University) · → Paper · Demo: ? · Code: ?
Introduces a segmental-level prosodic probe, consonant-induced f0 perturbation, to test whether neural TTS systems generalize fine-grained articulatory-acoustic patterns beyond lexically memorized items.
Problem
Neural TTS evaluation has historically focused on higher-level prosodic features (overall pitch contour, naturalness, expressiveness) while leaving fine-grained segmental-level phonetic effects largely unexamined. One such effect, consonant-induced f0 perturbation, is a well-documented phenomenon in phonetics where the fundamental frequency of a vowel is systematically shaped by the voicing and articulatory characteristics of the immediately preceding consonant. This effect arises from short-range dependencies between local segmental context and continuous acoustic output that are never explicitly supervised during TTS training. Whether neural TTS models reproduce this effect at all, and whether they do so through genuine phonetic abstraction rather than memorized lexical associations, was previously untested. This gap matters because the presence or absence of such fine-grained cues has direct implications for interpretability, authenticity assessment, and deepfake detection.
Method
The authors propose a segmental-level prosodic probing framework built around consonant-induced f0 perturbation as a diagnostic signal. The framework compares f0 trajectories following three onset categories (voiced obstruents, voiceless obstruents, and sonorants, used as the articulatorily neutral reference condition) across natural and synthetic speech, using generalized additive mixed models (GAMMs) to capture the time-varying, non-linear shape of f0 over the vowel.
Experiment 1 is a controlled single-speaker study: Tacotron 2 (autoregressive) and FastSpeech 2 (non-autoregressive, with an explicit duration and pitch predictor) are both trained on LJSpeech, and 4,210 sentences sampled from COCA are synthesized and force-aligned with the Montreal Forced Aligner. Lexical frequency (from the SUBTLEX-US frequency list) is used as a proxy for training-corpus exposure, splitting a 14,387-word list into high- and low-frequency halves, with 1,000 tokens sampled per onset category per frequency band per speech source. A pooled GAMM (bam() in the R package mgcv, AR1 residual structure) models z-scored f0 as a function of time, onset type, vowel height, and word, letting the analysis isolate onset-conditioned differences from confounds like vowel-intrinsic f0. A follow-up analysis directly contrasts words verified as seen versus unseen in the LJSpeech training transcripts, to validate that the frequency proxy tracks genuine training exposure rather than an artifact.
Experiment 2 extends the probe to a large-scale, multi-speaker setting using the In-the-Wild deepfake detection corpus (58 public figures, bona-fide and deepfake audio), adding a per-speaker factor smooth to the GAMM and z-scoring f0 within speaker to control for cross-speaker pitch-range variation.
Key Results
In the high-frequency condition, both Tacotron 2 and FastSpeech 2 reproduce the expected onset-conditioned f0 pattern: voiceless obstruents are followed by a sharp f0 peak (240-270 Hz) while voiced obstruents and sonorants show flatter, lower contours (190-220 Hz), matching natural LJSpeech recordings. For low-frequency words, this pattern collapses: FastSpeech 2 shows no systematic f0 difference across onset types, and Tacotron 2 partially captures the voiceless-obstruent elevation but is inaccurate for sonorant and voiced-obstruent contours. The seen-versus-unseen follow-up confirms this is a genuine training-exposure effect rather than a frequency-proxy artifact: seen tokens show robust, stable onset separation while unseen tokens show near-zero difference smooths for both models. The large-scale In-the-Wild evaluation (Experiment 2) replicates the same qualitative pattern (strong separation for high-frequency items, weak separation for low-frequency items) across a more diverse multi-speaker, multi-system setting, though the effect is attenuated relative to the single-speaker study, which the authors attribute to inter-speaker variability and to the larger, more diverse training corpora used by the systems represented in that dataset.
Novelty Assessment
The contribution is a new diagnostic evaluation methodology, not a new TTS architecture, training method, or dataset. Tacotron 2 and FastSpeech 2 are used exactly as published; the paper’s novelty lies in applying a phonetically motivated, segmental-level probe (grounded in the f0-perturbation literature) to distinguish lexical memorization from generalized segmental-prosodic encoding in TTS output, and in showing this distinction transfers to a large-scale multi-speaker deepfake-detection setting. The GAMM-based comparison design and the seen/unseen validation check are a methodologically careful contribution to TTS evaluation and interpretability, complementing coarser metrics like MOS or WER.
Field Significance
moderate — the paper contributes a linguistically grounded diagnostic that goes beyond standard naturalness or intelligibility metrics, exposing a specific generalization failure (reliance on lexical memorization for segmental-prosodic detail) that is invisible to metrics like MOS. Its result is a negative finding about two established architectures rather than an advance in synthesis quality, and its immediate scope is limited to the two model families and probing method examined, though the authors argue the underlying limitation is architecture-agnostic.
Claims
- supports: A segmental-level phonetic probe based on articulatorily grounded acoustic effects can reveal whether a TTS system encodes abstract phonological structure or has instead memorized lexically specific acoustic-textual associations.
Evidence: High-frequency (likely training-seen) words show f0 perturbation patterns matching natural speech in both Tacotron 2 and FastSpeech 2, while low-frequency (likely unseen) words show degraded or absent separation between onset categories for both models. (§4, Figures 1-2)
- complicates: Fine-grained segmental-prosodic generalization is not resolved by architectural choice alone; both autoregressive and non-autoregressive TTS paradigms fail to generalize a local phonetic effect to unseen lexical items despite differing internal mechanisms for temporal and pitch modeling.
Evidence: FastSpeech 2 (parallel decoding with an explicit pitch predictor) shows no systematic onset-conditioned f0 difference on low-frequency items, while Tacotron 2 (autoregressive, implicit propagation) shows only partial success, capturing the voiceless-obstruent effect but not the voiced-obstruent/sonorant contrast. (§4, §6)
- complicates: Standard overall naturalness metrics such as MOS do not directly surface segmental-level generalization failures in synthetic speech, since such metrics aggregate perceptual judgments without isolating local articulatory-acoustic cues.
Evidence: The paper notes that TTS MOS has improved substantially even as small-scale models such as those tested here plausibly retain the segmental generalization gap identified by the probe, and argues establishing a causal link to perceptual naturalness would require dedicated listening experiments beyond the scope of this study. (§6, Discussion)
- supports: Lexical-frequency-based generalization failures observed in a controlled single-speaker setting can transfer, in attenuated form, to large-scale, multi-speaker, multi-system evaluation settings such as deepfake-detection corpora.
Evidence: The In-the-Wild dataset (58 speakers, bona-fide and deepfake audio) reproduces the same qualitative high-frequency-versus-low-frequency contrast in onset-conditioned f0 separation seen in the LJSpeech-based Experiment 1, albeit with weaker group-level effects attributed to inter-speaker variability. (§5, Figures 4-5)
Limitations and Open Questions
The study evaluates only two specific TTS systems (Tacotron 2 and FastSpeech 2), both trained on a single small corpus (LJSpeech), so conclusions about “TTS architectures” broadly are extrapolated from a narrow architectural sample; the authors themselves note diffusion- and flow-based models remain untested. Lexical frequency is used as a proxy for training-set membership rather than ground-truth confirmation for most of the analysis (though a seen/unseen validation check is run as a secondary confirmation). The In-the-Wild multi-speaker analysis cannot control for the exact training data or architecture of the systems that produced the deepfake audio, since these are not publicly documented, limiting causal attribution of the observed attenuation. The paper does not run a perceptual listening study to establish whether the identified segmental generalization gap actually affects listener-perceived naturalness or authenticity judgments; this connection remains speculative.
Wiki Connections
- Evaluation Metrics — introduces a segmental-level, linguistically grounded probing methodology as a complement to standard aggregate TTS metrics like MOS and WER.
- Transformer Encoder-Decoder TTS — evaluates FastSpeech 2, a non-autoregressive parallel-decoding architecture with an explicit pitch predictor, as one of its two test systems.
- Prosody Control — probes whether TTS systems implicitly encode fine-grained segmental-prosodic (f0) structure rather than only coarse prosodic contours.