LREC 2026 · 2026 · Conference

Pasindu Udawatta et al. (University of Auckland) · → Paper · Demo: ✓ · Code: ✓

Packages vowel-space analysis (synthesis, forced alignment, formant extraction, interactive static and dynamic F1-F2 plots) into a model-agnostic tool that runs at chosen training steps, so developers can see which vowels a TTS model has learned rather than relying on the loss curve alone.

Problem

Loss-curve saturation tells a TTS developer when a model has fit its training data, but not which sounds it has or has not learned. Vowel formant spaces are a standard phonetic tool for characterising accents, and earlier proof-of-concept studies from the same group tracked monophthong and diphthong evolution during TTS training, but only by synthesising from checkpoints after training finished. There was no reusable, automated way to produce vowel-space views at training time, and no direct check that movements in these plots correspond to what listeners perceive.

Method

TTSVowelViz is a Python package (published on PyPI) that wraps four stages around any TTS model that exposes a synthesis function. At user-selected training steps it (1) synthesises speech from evaluation texts with the intermediate model, (2) segments the output into phonemes with WebMAUS to produce TextGrids, (3) extracts F1 and F2 with Parselmouth (Praat), and (4) renders interactive Plotly plots on the Bark scale. For the static vowel space, one (F1, F2) pair per vowel category is computed from a single time point or an average of several points per token, then averaged across all occurrences; for the dynamic vowel space, F1 and F2 are sampled at an arbitrary number of user-chosen time points and averaged per time point, producing spline-smoothed trajectories. The first frame of each plot shows the ground-truth vowel space computed from the training dataset, and a slider steps through training checkpoints. Averaging across occurrences is the paper’s mitigation for errors in automatic segmentation and formant tracking.

The validation experiment fine-tunes a Tacotron 2 model (Tacotron 2), pre-trained for 120000 steps on LJSpeech (General American English, GAE, one female speaker), on the Mansfield corpus (New Zealand English, NZE, about 1 hour from one female speaker) for 28000 steps at learning rate 0.001, stopping on the loss curve. The tool is configured with eight steps (0 to 28000), NZE point vowels (FLEECE, THOUGHT, START), all eleven NZE monophthongs at the vowel midpoint for the static plot, and six diphthongs at 20%, 50%, and 80% for the dynamic plot (Table 2). Synthesis uses 25 shuffled word lists of hVd-style words per vowel class to limit coarticulation and position effects (Table 1).

A perception test then asks 23 listeners (13 first-language English speakers, all resident in New Zealand for more than a year) to rate synthesised carrier sentences (“Say the word … again”) on sliders for how NZE-like and how GAE-like they sound. Nine monophthongs are tested, two sentences per vowel per step, 144 sentences in total.

Key Results

Visually, the static vowel space moves from the GAE configuration before fine-tuning toward the triangular NZE shape of the ground truth by step 28000, though synthesised vowels remain offset in height from the ground truth, which the authors attribute to the carefully articulated recording style of the reference corpus (§4.1, Figure 3).

The vowel-by-vowel comparison with perception falls into four categories (§4.1, Figure 4):

CategoryVowelsExample (mainly/completely NZE responses)
Moved toward NZE, perception increasedFLEECE, NURSE, START, THOUGHT, TRAPTRAP: 12.5% to 37.5% (step 0 to 28000)
Did not reach NZE region, perception tracked positionGOOSE25% to 33.4% at step 3000 (closest to NZE), back to 25% at step 28000
Moved toward NZE, perception decreasedDRESS37.5% to 25%
Did not move toward NZE, perception increasedKIT, LOTKIT: 0% to 37.5%

So for six of the nine tested vowels the plotted movement and the listener responses agree in direction, and for three (DRESS, KIT, LOT) they do not. The authors explain DRESS as a mismatch between the 2014 reference recordings (raised DRESS) and listeners’ expectation of more recent DRESS lowering, and KIT and LOT as short vowels with less perceptual exposure or affected by overall voice quality; these explanations are not tested. Linear mixed models with training step as fixed effect show significant formant shifts (p < .001) within the first 10000 steps for every monophthong except FOOT, which the authors suggest as a possible early-stopping point while noting it needs further verification. In the dynamic plots, FACE and NEAR end close to the ground truth, whereas MOUTH and PRICE are closest at step 3000 and drift away with further fine-tuning (§4.2, Figure 3); no perception test covers diphthongs.

No statistical test of agreement between formant movement and listener ratings is reported; the alignment argument is qualitative, read from stacked bar charts. Ratings are given for whole carrier sentences, so a per-vowel rating also reflects the rest of the sentence.

Novelty Assessment

The underlying analysis (tracking F1-F2 vowel spaces across TTS training checkpoints) was introduced in the group’s earlier Interspeech 2022 and SST 2024 studies; the contribution here is engineering it into a model-agnostic, configurable, publicly released tool that runs during training, adding a variable number of formant sampling points, plus a first perception test relating plotted movements to accent judgements. The components (WebMAUS, Praat formant tracking, Bark-scale plotting) are standard phonetics tooling. Validation is narrow: one TTS architecture (Tacotron 2), one source and one target accent of English, one speaker each, and a perception test whose agreement with the plots is mixed and unquantified. The “first tool of its kind” framing is plausible for training-time vowel visualisation but the paper does not compare against alternative training diagnostics such as ASR-based phoneme error tracking.

Field Significance

Low — The paper provides a practical, linguistically grounded diagnostic for accent and dialect adaptation of TTS models, showing that per-vowel formant movement during fine-tuning partly tracks listener accent judgements. Its evidence base (a single Tacotron 2 GAE-to-NZE fine-tune, 23 listeners, six of nine vowels agreeing) makes it a useful tool description and pilot validation rather than a demonstration that vowel spaces are a reliable training signal.

Claims

  • supports: Acoustic vowel-space trajectories computed from synthesised speech during accent fine-tuning can expose which vowels a TTS model has and has not adapted to, information that the training loss does not provide.

    Evidence: Fine-tuning Tacotron 2 from GAE to NZE moves most monophthongs toward the NZE ground-truth triangle (e.g. TRAP from low-front to mid-front), while GOOSE stays central and KIT stays high-front, despite training being stopped on loss saturation. (§4.1, Figure 3)

  • complicates: Agreement between formant-space movement and listener accent perception is vowel-dependent, so vowel-space proximity to a target accent is not a reliable stand-alone proxy for perceived accent.

    Evidence: Six of nine tested monophthongs show plot movement and NZE ratings changing in the same direction, but DRESS moves toward NZE while NZE responses fall from 37.5% to 25%, and KIT does not move toward NZE while NZE responses rise from 0% to 37.5%; no statistical agreement measure is reported. (§4.1, Figure 4)

  • refines: During low-resource accent fine-tuning, most acoustic change in vowel formants occurs early, and some vowels reach their best match to the target before training ends.

    Evidence: Linear mixed models show significant formant shifts within the first 10000 of 28000 steps for all monophthongs except FOOT (p < .001); MOUTH and PRICE trajectories match the ground truth best at step 3000 and diverge afterwards. (§4.1, §4.2, Figure 3)

  • complicates: Vowel-space evaluation against a reference corpus inherits that corpus’s speaking style and its sociophonetic vintage, which can differ from listeners’ expectations of the target accent.

    Evidence: Synthesised vowels remain offset in height from the carefully articulated Mansfield ground truth, and the 2014 recordings’ raised DRESS conflicts with more recent DRESS lowering that the authors suggest listeners expected. (§4.1)

Limitations and Open Questions

Warning

Validation covers one model (Tacotron 2), one accent pair (GAE to NZE), one speaker per side, and a 23-listener test; agreement with perception holds for six of nine vowels and is not quantified statistically, so the tool’s reliability as a training diagnostic for modern TTS architectures is untested.

  • Only vowels are covered; consonants, rhythm, and intonation, which also carry accent, are out of scope (§7).
  • Formant analysis depends on automatic segmentation (WebMAUS) and formant tracking on synthetic speech; averaging mitigates but does not measure this error.
  • The perception test rates whole carrier sentences rather than isolated vowels, so per-vowel attribution is indirect; diphthong trajectories have no perceptual validation.
  • Running synthesis, alignment, and formant extraction at many checkpoints adds cost; the paper does not report the tool’s runtime overhead.
  • The proposed uses (early stopping from vowel-space convergence, vowel-space-informed losses or attention) are future work, not demonstrated.
  • Whether the tool generalises to languages other than English and to non-autoregressive or codec-based TTS is asserted by design but not tested.

Wiki Connections

  • Evaluation Metrics — Proposes vowel-space (F1-F2, Bark scale) visualisation as an interpretable, training-time diagnostic for accent and pronunciation learning in TTS.
  • Subjective Evaluation — Validates the acoustic diagnostic with a 23-listener Likert-slider accent rating test, finding partial agreement across vowels.
  • Transformer Encoder-Decoder TTS — Tracks how a fine-tuned Tacotron 2 acoustic model’s vowel formants move during low-resource accent adaptation, with significant shifts within the first 10000 steps.
  • Tacotron 2 — The TTS model whose GAE-to-NZE fine-tuning is visualised and evaluated with the tool.