arXiv · 2025 · Preprint
Tuttösí et al. · → Paper · Demo: ✓ · Code: ✓
Adds a phoneme-selective duration multiplier (“clarity mode”) to Matcha-TTS that improves English tense/lax vowel comprehension for French-L1 English-L2 listeners, demonstrating that global speech slowing reduces perceived naturalness and social appropriateness and that Whisper ASR does not serve as a valid proxy for L2 human intelligibility evaluation.
Problem
Standard TTS systems are designed and evaluated for first-language (L1) listeners and do not adapt to second-language (L2) speakers who may rely on different phonetic cues for word identification. Prior work on “clear TTS” addressed speech-in-noise scenarios rather than listener language background. French-L1 English-L2 speakers in particular struggle with the English tense/lax vowel distinction (e.g., /i/ “peel” vs. /I/ “pill”) because French lacks this contrast. Unlike L1 listeners who rely on formant differences, L2 speakers from French, Mandarin, and Japanese backgrounds depend primarily on vowel duration to discriminate these pairs. Generic speech-slowing strategies common in L2-directed speech can inadvertently cause lax vowels to be perceived as tense (because they become too long), increasing transcription errors rather than reducing them.
Method
The system modifies Matcha-TTS (a conditional flow matching TTS architecture) at inference time by adding a clarity flag and a phoneme-level duration multiplier. No model retraining is performed. When clarity mode is enabled, the user or an LLM driving a dialogue system marks target words with exclamation points (e.g., “!peel!”). The system parses each marked word to identify whether it contains a tense vowel, a lax vowel, or both with primary stress considerations. Words containing tense vowels receive a 1.6x duration stretch applied across the whole word, with a gradual ramp-up and ramp-down spanning the six adjacent phonemized items on either side (approximately 200-300 ms). Simultaneously, lax-vowel-containing words receive the base 0.75x speech rate multiplier to prevent inadvertent stretching from surrounding context. The clarity multiplier is applied via a Hadamard product on the duration predictor’s output after the base speech-rate array, meaning the modification requires only an inference-time flag and a phoneme-classification lookup.
Four TTS conditions were contrasted: Base (0.75x speech rate throughout), Stretch (1.2x applied to full phrase), Emphasis (1.6x on all target words regardless of vowel category), and Clarity (1.6x on tense-containing target words only). Stimuli consisted of 16 carefully constructed English phrases with controlled minimal-pair target words across all English tense/lax vowel pairs (/i/-/I/, /u/-/U/, /A/-/2/), with phrases designed to be semantically neutral between minimal pair members.
Key Results
In the single-word experiment (N=56 French-L1 participants), clarity mode reduced tense-vowel WER from 60.23% (base) to 29.48% (target stretch), while maintaining 16.86% WER for lax vowels. Global stretching of the entire phrase produced higher WER (37.57% tense, 29.65% lax) and scored significantly lower on naturalness (nMOS ~4.2 vs. ~7.1-7.7 for other conditions), prosody, encouragement, and respect (Table 2, §3.3.1).
In the double-word experiment, clarity mode achieved 15.15% total WER vs. 24.30% for base, 19.82% for stretch, and 24.44% for emphasis, with balanced performance across tense (14.38%) and lax (15.92%) targets (Table 4, §3.3.2). The ability to contrast tense and lax vowels within a single phrase appears to provide an additional perceptual cue beyond the duration manipulation of either word alone.
For Whisper ASR (§3.3.3, Table 7), clarity mode showed no benefit over baseline (17.68% vs. 17.10% target WER), while global slowing was most beneficial (15.98%). More strikingly, Whisper’s substitution errors for the base condition were predominantly tense-for-lax (71.42% of target-word errors were minimal-pair substitutions), a pattern inconsistent with L2 listener error distributions. The overall WER differences across all conditions were within 3% for ASR, masking qualitatively different error types.
A persistent subjective perception paradox was observed: in both experiments, L2 participants rated the emphasis condition (all target words stretched) as most intelligible subjectively, despite it having higher objective WER than clarity mode. Participants were unaware of the duration mechanism driving their improved comprehension.
Novelty Assessment
The contribution is primarily engineering integration: a phoneme-category-specific duration multiplier is appended to Matcha-TTS’s inference pipeline with no architectural changes or retraining. The underlying linguistic hypothesis (L2 speakers rely on duration rather than formants for tense/lax discrimination) comes from the authors’ prior perception studies. What is new here is the application of that hypothesis within a TTS system and its validation in a controlled human study. The evaluation design is careful: N=56 with French-L1 background, controlled phrase stimuli across all English tense/lax pairs, forced-choice identification, and multiple MOS dimensions including social perception (respect, encouragement). Two genuinely useful negative results emerge: global speech slowing reduces perceived respect and encouragement even when it reduces WER, and Whisper ASR cannot substitute for human evaluation when assessing L2-targeted TTS.
Field Significance
Moderate - the paper provides concrete empirical evidence that ASR-based WER is an insufficient evaluation metric for L2-directed TTS, a finding directly actionable for evaluation practice. The negative result on global speech slowing (it degrades perceived social appropriateness) is a useful cautionary data point for developers of accessible TTS systems. The scope is narrow (French L1, American English tense/lax pairs, high-proficiency L2 speakers, N=56), which limits immediate generalizability, but the methodology provides a template for human-centered accessible TTS evaluation beyond standard L1 benchmarks.
Claims
-
supports: Targeted duration manipulation of specific phonemic contrasts is more effective for L2 speech comprehension than undifferentiated global speech rate reduction.
Evidence: Clarity mode (1.6x stretch on tense vowels only, 0.75x elsewhere) achieved 15.15% total WER vs. 19.82% for global 1.2x stretch and 24.30% for base in the double-word experiment; tense-vowel WER improved from 60.23% (base) to 29.48% with targeted stretch vs. 37.57% with full stretch in the single-word experiment. (§3.3.1, §3.3.2, Tables 2, 4)
-
contradicts: ASR-based word error rate can serve as a reliable proxy for L2 listener intelligibility in TTS evaluation.
Evidence: Whisper ASR showed no benefit from clarity mode (17.68% WER vs. 17.10% for base) while L2 listeners achieved their lowest WER with clarity mode (15.15%); ASR’s tense/lax substitution pattern (71.42% minimal-pair substitutions on base TTS) differed fundamentally from L2 human error distributions; overall ASR WER varied by less than 3% across conditions that produced large differences in human WER. (§3.3.3, Table 7)
-
complicates: Global speech rate reduction is an effective clarity strategy for L2 listener experience.
Evidence: Full-stretch TTS scored significantly lower on naturalness (nMOS ~4.2 vs. ~7.0-7.9 for other conditions), prosody, encouragement, and respect in both single- and double-word experiments; L2 participants rated both “too fast” (base) and “too slow” (stretch) as less respectful and encouraging. (§3.3.1, §3.3.2, Tables 2, 3, 5, 6)
-
complicates: L2 listeners’ subjective perceived intelligibility judgments reflect their actual comprehension accuracy for TTS systems.
Evidence: Despite clarity mode producing lower objective WER than emphasis in both experiments, L2 participants consistently rated emphasis (stretching all target words) as most intelligible subjectively; participants believed that overall phrase slowing was most intelligible even when it was not, suggesting they were unaware of the duration mechanism improving their comprehension. (§3.3.1, §3.3.2, §5)
Limitations and Open Questions
Warning
The study is restricted to French-L1 English-L2 speakers at relatively high proficiency (mode: level 5/5), covering only three English tense/lax vowel pairs. Generalizability to other L1 backgrounds, lower proficiency levels, and vowel distinctions beyond duration contrast is untested.
The 1.6x duration stretch factor was calibrated empirically and may not be optimal across different base speech rates, speaker styles, or emotional expressivity contexts. Inter-individual differences in duration cue weighting were noted in the authors’ prior work, suggesting the mechanism may be ineffective for a subset of L2 listeners. The clarity mode addresses duration cues only; formant-based spectral cues and consonantal clarity mechanisms remain unexplored. The word-marking interface (exclamation points surrounding target words) requires either manual annotation or a large language model to identify difficult words, introducing a practical dependency for real-world deployment.
Wiki Connections
- Flow Matching - the paper builds on Matcha-TTS, a conditional flow matching TTS system, adding inference-time duration control without retraining the flow matching backbone.
- Prosody Control - introduces a phoneme-category-selective duration multiplier as a linguistically motivated prosody control mechanism targeting tense/lax vowel intelligibility for L2 listeners.
- Evaluation Metrics - demonstrates that standard ASR-based WER and self-rated intelligibility MOS are insufficient for evaluating TTS systems intended for L2 speakers, advocating for objective human listener WER as the gold standard.
- Subjective Evaluation - employs multi-dimensional MOS evaluation (naturalness, intelligibility, effort, prosody) augmented with social perception scales (respect, encouragement) from L2-directed speech research, revealing a systematic divergence between perceived and actual intelligibility.