arXiv · 2026 · Preprint
Thanathai Lertpetchpun et al. (University of Southern California) · → Paper · Demo: ✓ · Code: ?
Introduces Accent Vector, a controllable parameter-space representation that induces accent characteristics in multilingual TTS by fine-tuning on native speech of a different language and computing the resulting weight shift, avoiding the need for accented training data entirely.
Problem
Most TTS systems are trained predominantly on mainstream (typically American) English because large-scale, well-annotated accented speech corpora are scarce. This data imbalance causes synthesis quality and coverage to disproportionately favor the dominant accent while underrepresented English varieties (both L1 regional accents and L2 non-native accents) are poorly modeled. Prior accent-control methods reduce reliance on accented English corpora through text transliteration, phonetic rule substitution, or duration-predictor resynthesis, but each provides only coarse or single-dimension control: transliteration produces a fixed accent realization, rule-based phoneme substitution requires manual linguistic engineering per accent pair, and duration-based resynthesis only captures durational differences while leaving segmental and other suprasegmental variation unaddressed. No prior approach combines continuous accent-strength control, composition of multiple accents, and generalization across base languages within a single mechanism.
Method
The paper builds on XTTS2, a pretrained multilingual zero-shot TTS backbone that generates discrete acoustic codes conditioned on a language ID token, a text transcript, and reference speech, trained via cross-entropy prediction of VQ-VAE codes plus a mel-spectrogram reconstruction loss, adversarial GAN loss, and speaker-consistency loss. To obtain an Accent Vector, the backbone is fine-tuned with Low-Rank Adaptation (LoRA, rank 16, applied to all linear layers in the encoder) while the language ID token is fixed to a base language (e.g., English) but the transcript and reference speech come from native speakers of a different target-accent language (e.g., Spanish). Because LoRA parameterizes the fine-tuned weights as the pretrained weights plus a low-rank update, the Accent Vector is defined exactly as that LoRA weight delta, following the task-vector formulation of prior work on parameter-space task arithmetic. At inference, the Accent Vector is added back to the pretrained parameters scaled by a coefficient α, giving continuous control over accent strength; multiple Accent Vectors can be linearly combined with individual weighting coefficients to synthesize mixed-accent speech (e.g., a mix of Spanish- and British-accented English). The same fine-tuning and extraction procedure generalizes to non-English base languages: swapping the base language to Spanish, German, or Mandarin while fine-tuning on English (British) reference speech yields English-accented speech in those languages. Fine-tuning used the Adam optimizer, learning rate 3e-5, for 60,000 steps on one A40 GPU (8 GPU hours).
Key Results
Across six English accents (British, Spanish, Hindi, German, French, Mandarin), fine-tuning consistently shifted synthesized speech toward the intended accent relative to the unmodified pretrained model: VoxProfile accent-classification probability and accent-embedding cosine similarity both increased for every accent (e.g., British accent probability rose from 23.3% to 56.7%, Hindi from 2.2% to 24.2%), while speaker similarity remained high (around 0.9), indicating accent manipulation did not degrade speaker identity. Accented speech showed higher WER/CER than standard English synthesis, attributed to Whisper’s American-English training bias; Mandarin-accented English showed the weakest accent-probability gain and highest WER/CER, attributed to greater prosodic and phonological distance from English. In the reverse direction (English accent transferred onto Spanish, German, and Mandarin base speech), English accent probability and LID-based English-language probability both increased substantially after fine-tuning, again with WER/CER increases attributed to the rarity of these cross-lingual accent-language combinations in ASR training data. Scaling the Accent Vector coefficient α from 0 to 1 produced a monotonic increase in accent strength alongside a monotonic WER increase and UTMOS decrease, demonstrating a controllable but real accentedness/intelligibility trade-off. Linearly combining two Accent Vectors generally raised both target accent probabilities simultaneously, except when British was one of the two accents, where British probability dominated at the expense of the other accent (attributed to VoxProfile’s classifier bias toward North American/British accents in its training data). A 16-listener subjective study (70 samples) found accent-identification accuracy well above chance (14%) for all six accents (53.85%-80%), with perceived accent strength and naturalness ratings both in the moderate-to-quite range.
Novelty Assessment
The individual components (task vectors, LoRA fine-tuning, an XTTS2 backbone) are all established techniques, and task-vector-style parameter arithmetic has already been applied to other controllable speech attributes such as emotion and dialectal expressiveness in prior work. This paper’s contribution is applying that mechanism specifically to accent, and showing that it uniquely combines properties that no prior accent-control method offers together: no accented training data requirement, continuous accent-strength control, composability of multiple accents, and generalization to non-English base languages. The contribution is primarily an engineering integration of known parameter-space arithmetic techniques into a new application domain, evaluated with a reasonably thorough objective-plus-subjective protocol, rather than a new architecture or training objective.
Field Significance
moderate — This paper extends parameter-space task-vector arithmetic, already demonstrated in the corpus for controllable emotion and dialectal expressiveness, to accent control in multilingual TTS, and it is the first accent-control method to combine continuous strength control, multi-accent composition, and cross-lingual generalization without accented training data in a single framework. Its contribution is a useful, well-evaluated application rather than a new underlying mechanism.
Claims
- supports: Parameter-space task vectors extracted from fine-tuning on native speech of a target language can be added back to a pretrained multilingual TTS model to induce non-native accent characteristics in synthesized speech, without requiring any accented training data.
Evidence: Fine-tuning XTTS2 with LoRA on native reference speech from a different language, while keeping the language ID fixed to the base language, increased target-accent classification probability and accent-embedding similarity across all six tested accents relative to the unmodified pretrained model, while speaker similarity remained around 0.9 (§6.1.1, Table 3).
- supports: The strength of a parameter-space accent representation can be continuously and monotonically controlled by linearly scaling the vector before adding it to the base model, producing a smooth accent-strength/intelligibility trade-off rather than a binary on/off effect.
Evidence: Scaling the Accent Vector coefficient α from 0 to 1 in increments of 0.2 for British- and Hindi-accented English produced progressively stronger accent classification scores alongside rising WER and declining UTMOS (§6.3.1-6.3.2, Figure 3).
- supports: Multiple parameter-space accent vectors can be linearly composed to synthesize mixed-accent speech reflecting influence from more than one target accent simultaneously.
Evidence: Combining Spanish and Hindi Accent Vectors (and other accent pairs) with weighted coefficients raised both target-accent probabilities in most combinations, and per-accent strength within the mix could be tuned via the individual coefficients (§6.4.1-6.4.2, Table 5, Figure 4). Evidence: One pairing failed to combine cleanly: mixing British with Spanish or Mandarin caused British accent probability to dominate at the expense of the other accent, attributed to the accent classifier’s own training-data bias toward North American and British varieties (§6.4.1, Table 5).
- complicates: Automatic accent-strength control via ASR-based intelligibility metrics conflates genuine accent-induced pronunciation shift with the ASR model’s own accent bias, so WER/CER increases under accent manipulation should not be read purely as quality degradation.
Evidence: Whisper-based WER/CER rose for all accented conditions relative to standard American English synthesis, and the authors attribute part of this rise to Whisper’s own training bias toward standard American English rather than to synthesis quality loss, noting ASR errors can partially reflect stronger accentedness rather than lower speech quality (§6.1.2).
- complicates: The effectiveness of a parameter-space accent shift depends on the phonological and prosodic distance between the base language and the target accent language, with larger distances (e.g., tonal vs. stress-timed prosody) yielding weaker accent transfer.
Evidence: Mandarin-accented English showed the smallest relative gain in accent classification probability among the six tested accents, attributed to fundamental prosodic differences between tonal, syllable-timed Mandarin and stress-timed English (§6.1.3).
Limitations and Open Questions
The paper's own objective accent evaluation relies entirely on pretrained proxy models (VoxProfile accent classifier, a VoxLingua107 LID model, Whisper ASR, and UTMOS) that are predominantly trained on well-represented English varieties and do not adequately cover many cross-lingual accent combinations; the authors caution that absolute automatic scores, especially in the cross-lingual accent-transfer setting, must be interpreted with extra caution (§6.6).
Beyond the evaluation-proxy caveat, the paper notes that Accent Vector effectiveness depends on both fine-tuning data quality and linguistic distance between languages, with weaker gains observed for Mandarin due to lower recording quality and substantial prosodic/phonological differences from English. The authors also note that representing accent adaptation as an approximately linear parameter shift may be insufficient for complex suprasegmental phenomena, particularly tonal variation (§6.6). UTMOS scores for both training and evaluation sets remained only moderate even after quality filtering (DNSMOS threshold), which the authors acknowledge may limit overall synthesis naturalness independent of the accent manipulation itself (§4.2). The training corpora used per language are restricted to a single representative dialect (e.g., Peninsular Spanish, Standard Hindi, Beijing/Northeastern Mandarin), so results may not generalize to other dialects of the same language.
Wiki Connections
- Multilingual TTS — demonstrates that a parameter-space accent-control mechanism generalizes across base languages (English, Spanish, German, Mandarin) using a single multilingual TTS backbone fine-tuned per accent.
- Zero-Shot TTS — builds on XTTS2, a zero-shot multilingual TTS backbone conditioned on reference speech, and preserves its speaker-similarity behavior under accent manipulation.
- Subjective Evaluation — conducts a 16-listener human study measuring accent identification accuracy, perceived accent strength, and naturalness to validate the objective accent-classifier results.
- Neural codec language models are zero-shot TTS synthesizers — cited as part of the broader neural-codec zero-shot TTS lineage that motivates why accented speech coverage matters for modern TTS systems.
- Task Vector in TTS: emotionally expressive dialectal speech synthesis — closely related prior application of task-vector arithmetic to controllable paralinguistic speech attributes, here applied to dialectal/emotional expressiveness rather than accent.
- Scalable Controllable Accented TTS — a labeled-conditioning baseline that this paper contrasts against in its accented-TTS comparison table, requiring accented data and geographic metadata that Accent Vector avoids.
- Quantifying speaker embedding phonological rule interactions in accented speech synthesis — prior rule-based phonetic-transformation approach to accent control from an overlapping author group, providing coarse control via manually designed phoneme substitution rules that this paper’s method supersedes with continuous parameter-space control.
- MeanVC: streaming zero-shot voice conversion via mean flows — cited as related work in the broader accented speech/voice conversion space, though it addresses voice conversion rather than accent-controllable TTS.