arXiv · 2026 · Preprint
Thanathai Lertpetchpun et al. (University of Southern California) · → Paper · Demo: ✓ · Code: ✓
Introduces the phoneme shift rate (PSR), a novel metric quantifying how strongly speaker embeddings in a TTS system preserve or override explicit, linguistically motivated phonological substitution rules (flapping, rhoticity, vowel correspondences) applied to convert American English phoneme sequences toward British English targets, revealing measurable entanglement between accent and speaker identity in embedding-conditioned synthesis.
Problem
Modern TTS systems typically control accent by conditioning on speaker embeddings associated with particular accents, but these embeddings simultaneously encode voice timbre, emotion, and other speaker traits unrelated to accent, making accent representation opaque and hard to control precisely. It is unclear how much of an embedding’s apparent accent behavior comes from genuine phonological structure versus other entangled factors, and there is no direct way to measure whether an accent-control intervention (such as a rule-based phoneme transformation) actually takes effect in the synthesized output or gets silently overridden by the embedding’s own accent bias.
Method
The paper defines three phonological substitution rules that convert American English phoneme sequences to British English targets, chosen as well-documented, salient, big-stroke contrasts rather than an attempt at exhaustive dialectal modeling: flapping (American intervocalic /t/ realized as [ɾ] maps to British [t]), rhoticity (American post-vocalic /r/, retained, maps to British deletion/vocalization), and vowel correspondences (systematic lexical-set shifts, e.g., TRAP/BATH/GOAT). Rules are applied as strict one-to-one phoneme substitutions preserving phoneme count, so that any observed accent difference reflects only the segmental mapping and the speaker embedding, not confounds from timing. The pipeline obtains an American phoneme sequence via Misaki G2P from normalized text, applies the rules to derive a British target sequence, and synthesizes both variants with the pretrained Kokoro-82M TTS model, using a fixed speaker embedding and fixed per-phoneme durations (copied from the American reference voice) across all conditions, so observed accent shifts arise only from the phoneme substitutions and the embedding, not text normalization or timing.
To quantify accent strength, the paper uses a Whisper-based accent classifier (Vox-Profile) to compute both classifier accent probabilities and cosine similarity between synthesized-speech accent embeddings and group-level reference accent embeddings. To directly measure whether rule-driven transformations survive synthesis, the paper introduces the phoneme shift rate (PSR): after synthesizing speech from a rule-transformed target phoneme sequence, the output audio’s phonemes are re-recognized (via Wav2Vec2Phoneme), and the same rules are re-applied to the recognized transcript; PSR is the ratio of substitutions still needed (N2) to the substitutions originally specified (N1). PSR=0 means the output perfectly realized every rule-driven substitution; PSR=1 means the speaker embedding completely overrode the rule input, reproducing the original American-like phoneme realization despite the rule-transformed target. Naturalness is tracked throughout via UTMOS to confirm accent manipulation does not degrade perceived audio quality. Experiments manipulate both rule configuration (embeddings alone; embeddings plus one rule; embeddings plus the full rule set, with per-rule leave-one-out ablations) and speaker embedding identity (using Kokoro’s 28 preset English speaker embeddings, 20 American-derived and 8 British-derived) across 33k synthesized LibriTTS-R utterances (55.4 hours).
Key Results
Applying all three phonological rules on top of a British speaker embedding raises the accent classifier’s British-accent probability from 67.8% to 78.4% and accent-embedding similarity to the British reference from 0.67 to 0.85, while applying the same rules to a North American speaker embedding target drops North American probability from 86.5% to 58.8% and raises British probability from 3.79% to 17.3%; naturalness (UTMOS) stays essentially flat across all conditions (≈4.4 for North American settings, ≈3.7 for British settings, with the lower British scores attributed to classifier training-data bias rather than actual quality degradation). Rules are not uniformly influential: per-rule ablation shows vowel correspondences drive the largest single-rule gain (British probability to 77.8%, PSR down to 0.693), rhoticity boosts embedding-space similarity even when classifier probability shifts are modest, and flapping alone has minimal individual effect but contributes additively in combination. Despite this, PSR never reaches its theoretical floor: with a British embedding and the full rule set, PSR is 0.628, not 0, showing the speaker embedding continues to override a substantial fraction of rule-driven phoneme substitutions rather than passing them through cleanly, and phoneme-level analysis (Table 2) confirms that speaker embeddings can simultaneously reinforce some rule categories while suppressing others, depending on phoneme context. The degree of rule-embedding interaction varies by individual voice: across four British-associated preset embeddings, adding rules produces PSR reductions of 14.7-16.6 percentage points, but embeddings that already exhibit strong accent probability without rules (e.g., Daniel at 89.8%) benefit less from rule guidance than embeddings that rely more heavily on it (e.g., Fable), indicating individual embeddings encode accent with differing degrees of entanglement even within the same nominal accent category.
Novelty Assessment
The paper’s central contribution, the phoneme shift rate (PSR), is a genuinely new diagnostic instrument: unlike accent classifier probabilities or embedding similarities, which measure overall perceived accent strength, PSR isolates specifically how much of an explicit, known phoneme-level intervention survives synthesis versus gets overridden, providing a mechanistic rather than purely outcome-based probe of accent-speaker entanglement. The phonological rules themselves are drawn from well-established linguistic descriptions rather than newly discovered, but their deliberate design as coarse, salient, one-to-one substitutions (rather than an attempt at comprehensive dialectal modeling) is a considered methodological choice that makes the resulting analysis interpretable and isolates rule-versus-embedding interaction cleanly. The paper does not modify the underlying TTS architecture (Kokoro is used as a frozen, off-the-shelf model), so the contribution sits entirely in the evaluation/interpretability layer, using explicit linguistic structure as a controlled lens on what is otherwise an opaque, embedding-driven accent-control mechanism.
Field Significance
high — this paper introduces a genuinely new, mechanistically grounded metric (phoneme shift rate) for measuring how completely a known, explicit phonetic intervention survives TTS synthesis versus being overridden by speaker-embedding bias, providing a reusable diagnostic for accent-speaker disentanglement research and directly demonstrating, with per-rule and per-embedding granularity, that current speaker embeddings entangle accent with speaker identity to a measurable and unevenly distributed degree.
Claims
- supports: Applying explicit, linguistically motivated phoneme-level substitution rules on top of speaker-embedding conditioning in a pretrained TTS system measurably strengthens the perceived accent of synthesized speech beyond what the speaker embedding alone produces, without degrading naturalness.
Evidence: Adding all three phonological rules to a British speaker embedding raises British-accent classifier probability from 67.8% to 78.4% and accent-embedding similarity from 0.67 to 0.85, while UTMOS naturalness remains essentially unchanged (3.74 → 3.72). (§6.1, Table 1)
- complicates: Speaker embeddings in a pretrained TTS system partially override explicit rule-driven phoneme transformations rather than passing them through unchanged, revealing that accent and speaker identity remain entangled in the embedding space even when linguistic rules force a specific accent realization.
Evidence: The phoneme shift rate never reaches its theoretical floor of 0 even under full rule application (0.628 with a British embedding and all rules applied), showing the speaker embedding continues to pull a meaningful fraction of phoneme realizations back toward its own accent bias. (§6.1-6.2, Tables 1-2)
- supports: Among a set of documented phonological differences between two accent varieties, some contrasts contribute substantially more to perceived accent strength in synthesized speech than others, and this relative contribution can be measured directly through per-rule ablation.
Evidence: In single-rule ablations with a British speaker embedding, vowel correspondences alone raise British accent probability to 77.8% and reduce PSR to 0.693, while flapping alone has minimal measurable impact, and removing vowel correspondences from the full rule set causes the largest drop among all single-rule removals. (§6.1, Table 1)
- complicates: The degree to which a speaker embedding either reinforces or overrides explicit accent-control rules varies substantially across individual preset speaker identities, indicating embeddings encode accent characteristics with differing degrees of entanglement even within the same nominal accent category.
Evidence: Applying identical phonological rules to four British-associated preset embeddings produces PSR reductions ranging from 14.7 to 16.6 percentage points, with some embeddings (Daniel) already producing high accent probability without rules (89.8%) while others (Fable) depend much more heavily on rule guidance to reach comparable accent strength. (§6.3, Table 3)
Limitations and Open Questions
All findings rely on automated metrics (an accent classifier and a phoneme recognition model) rather than human listening evaluation, and the authors explicitly note results are subject to the noise and biases of the specific phoneme recognition architecture used; no human evaluation is reported in this study.
The study is scoped to a single accent pair (American-to-British English) and a single frozen pretrained TTS backbone (Kokoro-82M); the authors state the same pipeline could support additional accents by defining corresponding rule sets, but this generalization is not empirically tested. The rules themselves are deliberately coarse, targeting salient cross-accent contrasts rather than comprehensive dialectal micro-variation, so the framework measures interaction with big-stroke phonological differences rather than finer-grained accent phenomena.
Wiki Connections
- Disentanglement — introduces a novel diagnostic (phoneme shift rate) to directly measure the degree of entanglement between accent and speaker identity in TTS speaker embeddings, validated through systematic per-rule and per-embedding ablations.
- Evaluation Metrics — proposes the phoneme shift rate (PSR), a new metric quantifying how completely explicit phoneme-level transformations survive TTS synthesis versus being overridden by speaker-embedding bias.
- MaskGCT — cited as an example of modern zero-shot TTS systems whose accent behavior is typically controlled via speaker-embedding conditioning, the opaque mechanism this paper’s rule-based probes are designed to interrogate.
- AccentBox — cited as related prior work on high-fidelity zero-shot accent generation from the same research group studying accent control in TTS.
- Pairwise Evaluation of Accent Similarity in Speech Synthesis — cited as related work establishing accent-similarity evaluation methodology that this paper’s accent-probability and embedding-similarity metrics build on.
- UTMOS — used to verify that phonological rule application and accent manipulation do not degrade the naturalness of synthesized speech.