arXiv · 2026 · Preprint
Thanathai Lertpetchpun et al. (University of Southern California) · → Paper · Demo: ✓ · Code: ?
Extends rule-based phonological phoneme substitution from within-language accent control to cross-lingual L2-accented English synthesis, generating Spanish- and Indian-accented speech from a frozen pretrained multilingual TTS model with no accented training data.
Problem
Accented TTS systems typically require either large-scale accented speech corpora for fine-tuning or coarse text-transliteration tricks that yield a fixed accent style without phoneme-level controllability. A prior phonological rule-based approach demonstrated fine-grained, controllable accent manipulation, but only within a single language family (American-to-British English) and had not been shown to generalize to cross-lingual L2 accent transfer, where the target accent arises from a speaker’s distinct native-language (L1) phonological system rather than a regional L1 variety of English.
Method
The system pairs a hand-authored phonological rule set with an off-the-shelf pretrained multilingual TTS model (Kokoro-82M v0.19), requiring no additional model training. Rules for Spanish-accented and Indian-accented English are derived from linguistically documented phonotactic and phonemic properties of Spanish and Hindi, covering consonant substitutions, rhoticity, epenthesis, devoicing, vowel simplification, and monophthongization; each rule maps an American English IPA phoneme to its accented counterpart (§2.1, Table 1). These rules are applied deterministically to convert American English phoneme sequences into accented variants before synthesis.
At inference, the multilingual TTS model is conditioned on two inputs: a speaker embedding drawn from the model’s pretrained set (selected to correspond to the target L1, e.g. a Spanish or Indian speaker embedding) and the accent-transformed phoneme sequence. Because the model’s multilingual speaker embeddings implicitly encode language and accent characteristics, combining a target-language embedding with rule-transformed English phonemes lets the pretrained cross-lingual prior contribute both segmental pronunciation shifts and suprasegmental characteristics such as rhythm and intonation, without retraining (§2.2).
The framework additionally exposes explicit control over phoneme-level duration alignment: durations for the accented phoneme sequence can either follow the accent’s native timing or be force-aligned to match American English duration patterns, isolating the contribution of rhythmic/timing transfer to perceived accentedness (§2.3).
Key Results
Applying the Spanish and Indian phonological rules on top of the corresponding speaker embeddings substantially shifted accent classification away from American English and toward the target accent: Spanish target-accent probability rose from 23.7% (speaker embedding alone) to 51.59% (with rules), and Indian target-accent probability rose from 58.86% to 86.4%, with corresponding increases in target-accent embedding similarity (§4.1, Table 4). A per-rule ablation showed vowel simplification was the single most influential rule for the Spanish shift, while retroflexion of stops and /r/ was most influential for the Indian shift (§4.3, Table 5). UTMOS naturalness scores stayed essentially flat across conditions (4.38 baseline vs. 4.39 Spanish-rules vs. 4.16 Indian-rules), while WER and CER rose substantially under rule application (e.g. WER 3.42% baseline to 24.91% with Spanish rules), a shift the authors attribute partly to ASR systems penalizing intentional accented pronunciations rather than to genuine quality loss (§4.1). Removing phoneme-level duration alignment (letting accent-native timing pass through unaligned) further increased target-accent probability for the UK and Indian conditions (e.g. Indian accent probability rose from 86.4% aligned to 93.1% unaligned), indicating rhythm contributes to perceived accent independently of segmental substitution (§4.2, Table 3). In a 14-listener subjective study over 70 samples, adding Spanish rules to the Spanish speaker embedding raised accent-identification accuracy from 0.071 to 0.757, and adding Indian rules to the Indian speaker embedding held accuracy near 0.757-0.786 while increasing perceived accent strength; naturalness ratings stayed around 3/5 (“moderately natural”) across all conditions (§4.4, Table 6).
Novelty Assessment
The core contribution is an extension, not a new mechanism: the underlying phonological rule-substitution technique for controllable accent strength was introduced in the authors’ own prior work on American-to-British English transformation. What is new here is the generalization of that technique to genuine cross-lingual L2 accent transfer (Spanish- and Indian-accented English), the specific rule sets derived for those two L1 backgrounds, and the added analysis isolating rhythmic/duration-alignment effects on perceived accentedness. The system itself performs no learning; the contribution is a combination of linguistically motivated rule design and an existing pretrained multilingual TTS model, making this primarily an engineering and linguistic-methodology contribution rather than an architectural one.
Field Significance
moderate — This paper demonstrates that a learning-free, rule-based approach to accent control can be extended beyond a single accent pair to genuine cross-lingual L2 accent synthesis, and it isolates rhythmic timing as a controllable factor in perceived accentedness distinct from segmental phoneme substitution. It also provides a cautionary methodological note for the accented-TTS field: WER/CER degradation under accent transformation should not be read as a straightforward quality signal, since ASR systems trained predominantly on American English penalize intended accented pronunciations as errors.
Claims
- supports: Rule-based phoneme-level transformation applied to a frozen pretrained multilingual TTS model can shift perceived accent toward a target L1-influenced variety without any accented training data or fine-tuning.
Evidence: Applying Spanish and Indian phonological rule sets to speaker-embedding-conditioned synthesis with Kokoro-82M raised target-accent classification probability from 23.7% to 51.59% (Spanish) and 58.86% to 86.4% (Indian), relative to speaker embedding alone. (§4.1, Table 4)
- complicates: Word error rate is an unreliable standalone proxy for intelligibility loss when evaluating accent-transformed synthetic speech, because ASR systems penalize intended phonological substitutions as recognition errors.
Evidence: Applying Spanish phonological rules increased WER from 3.42% to 24.91% partly because of systematic substitutions such as /θ/→/s/ (“think”→“sink”), which the ASR system marks as errors despite being the intended accented outcome. (§4.1)
- supports: Speaker embeddings alone in a multilingual TTS model are insufficient to produce a strong, listener-perceptible non-native accent; explicit phoneme-level substitution is needed for robust accent identification.
Evidence: With only the Spanish speaker embedding, human listener accent-identification accuracy was 0.071 (most listeners perceived American accent); adding Spanish phonological rules raised accuracy to 0.757. (§4.4, Table 6)
- refines: Perceived accentedness in synthesized speech depends on phoneme-level duration/rhythm patterns in addition to segmental phoneme substitution, and the two can be controlled independently.
Evidence: Removing forced duration alignment to American English timing (allowing accent-native rhythm) increased Indian target-accent probability from 86.4% to 93.1% and UK target-accent probability from 80.28% to 81.12%, holding segmental rules fixed. (§4.2, Table 3)
Limitations and Open Questions
The subjective validation is small-scale: 14 listeners rating 70 total samples, which limits statistical confidence in the accent-identification-accuracy and naturalness numbers reported in Table 6, and the listener pool's own L1 backgrounds (a mix of Asian and European languages) may bias which target accents are correctly identified.
Accent classification relies on Vox-Profile’s general accent categories used as proxies (Romance for Spanish, South Asia for Indian) since no explicit Spanish- or Indian-accented English labels exist in that classifier, which may introduce measurement noise independent of the phonological rules themselves. The rule sets cover only two L1 backgrounds (Spanish and Indian); generalization to other L1-influenced English accents is not evaluated. Adding Indian rules also introduced some listener confusion between Spanish- and Indian-accent categories (§4.4), suggesting the rule sets are not fully perceptually orthogonal.
Wiki Connections
- Multilingual TTS — repurposes a pretrained multilingual TTS model’s cross-lingual speaker embeddings and synthesis prior to generate L2-accented English without any accent-specific training.
- Prosody Control — introduces explicit control over phoneme-level duration alignment as a mechanism to isolate and manipulate rhythmic contributions to perceived accent, independent of segmental phoneme content.
- Subjective Evaluation — validates accent identification, accent strength, and naturalness through a 14-listener human study alongside automatic accent-classifier and UTMOS metrics.
- Quantifying Speaker Embedding Phonological Rule Interactions — directly builds on this prior work’s phonological rule-substitution and evaluation pipeline, extending it from within-language (US-to-UK) accent control to cross-lingual (Spanish- and Indian-accented) English synthesis.