ASRU · 2025 · Conference
Henry Li Xinyuan et al. (Johns Hopkins University) · → Paper · Demo: ✓ · Code: ?
A pipeline for scaling accented English TTS to large, noisy in-the-wild datasets by using a speech geolocation model to automatically infer or filter accent labels, combined with kNN voice conversion for timbre augmentation to improve speaker-accent disentanglement.
Problem
Controllable accented TTS has been limited by the scarcity of large, cleanly annotated accent datasets. Available resources such as VCTK, L2-Arctic, and CMU Arctic are small (10–50 hours) and cover few speakers per accent, causing models to entangle speaker timbre with accent characteristics. Large in-the-wild corpora like CommonVoice contain diverse accented speech but suffer from noisy self-reported labels — L2 speakers frequently mislabel themselves as native accent speakers — and many accents lack any label at all. Prior work on accented TTS either relied on curated, small-scale datasets or used label-noise-tolerant methods that still depended on the same limited label sources.
Method
The approach operates on CommonVoice v20 and fine-tunes XTTS-v2, a multilingual zero-shot TTS model comprising a discrete VAE encoder, a discrete token sequence-to-sequence transformer, and a HiFi-GAN vocoder. The language token in XTTS-v2 is replaced with an accent token to condition synthesis on accent rather than language.
Accent labels are derived through two strategies. The first uses a speech geolocation model originally trained to predict the broadcast origin of radio speech from around the world. Geographic bounding boxes are defined for each target accent (US, England, India, Australia, etc.), and utterances whose predicted location falls within a box are accepted as belonging to that accent. This enables both filtering of noisy self-reported labels and discovery of accent-labeled subsets from entirely unlabeled data. The second strategy applies kNN-VC (k-nearest-neighbours voice conversion) to diversify speaker timbre within each accent class: each training utterance is converted to a randomly sampled LibriTTS speaker while preserving accent characteristics, effectively multiplying the number of distinct speaker timbres available for low-resource accents. During fine-tuning, accent classes are upsampled to maintain balanced per-accent batch representation.
Three data configurations are compared: unfiltered (raw CommonVoice self-reported labels), filtered (geolocation-based filtering of self-reported labels), and unlabeled (geolocation-based label discovery from scratch). Ablation on data augmentation compares no augmentation, pitch-shift, and kNN-VC.
Key Results
On objective accent similarity, the filtered system (DCF = 0.571 averaged across all accents) outperforms the unfiltered system (DCF = 0.735) and substantially outperforms both XTTS-v2 baselines. CosyVoice2, using zero-shot style copying, achieves DCF = 0.748 on average but is competitive or stronger on specific accents. AccentBox, trained only on British Isles accents, scores best for English, Australian, Scottish, and Irish accents on the DCF metric.
Note
AccentBox was conditioned on GenAID embeddings during training, making direct comparison to the proposed system on metric 1 (cosine similarity to ground-truth utterance) potentially unfair. The paper uses a second metric (DCF via Gaussian backend) that aggregates over the full accent pool to control for speaker and content confounds.
Human evaluation covers Irish, Indian, Australian, and US accents (15 evaluators per accent from the relevant country or region). CosyVoice2 ranks highest in all four sets, including above real utterances in the US accent test — likely due to its clean acoustics versus CommonVoice’s variable recording quality. The filtered system scores similarly to the unfiltered system in most accent evaluations, but shows clear improvements for Australian and Filipino accents. kNN-VC augmentation improves objective accent similarity but shows unclear or mixed effects in human evaluation. Speaker similarity (Resemblyzer cosine) is comparable across the proposed systems (0.843–0.858) and CosyVoice2 (0.855). WER is 13.1% for the filtered system versus 7.1% for CosyVoice2, reflecting the inherent challenge ASR faces with accented speech.
Novelty Assessment
The contribution is engineering integration: the geolocation model and kNN-VC are both existing systems applied here to a new problem domain (accent label discovery and timbre augmentation for accented TTS). The XTTS-v2 backbone is unchanged in architecture; the modification is the replacement of the language token with an accent token. The evaluation is notably broad — 11–12 accents including underrepresented varieties from Africa, Southeast Asia, and the British Isles — and the comparison across three labeling strategies (unfiltered, filtered, unlabeled) provides useful evidence about data quality trade-offs. The application of a geolocation model for unsupervised accent discovery is an original idea within the accented TTS context, though the model itself is not new.
Field Significance
Moderate — This paper provides practical evidence that accent label quality matters more than raw data quantity for accented TTS, and that geolocation-based label discovery can approach the precision of dedicated accent identification systems on many accent varieties. Its value is primarily empirical and methodological rather than architectural: the finding that geolocation filtering consistently improves over raw self-reported labels, while unlabeled discovery underperforms filtered data, gives practitioners a calibrated sense of what to expect when scaling to in-the-wild corpora. The paper also surfaces an underappreciated gap between objective and human evaluation of accented speech.
Claims
- Accent label quality exerts a larger effect on accent synthesis fidelity than raw data volume, and filtering noisy self-reported labels consistently outperforms using them directly. (§V-C, Table VI)
- Speaker-accent disentanglement in low-resource settings benefits from voice conversion-based timbre augmentation, even when the augmented data introduces acoustic degradation. (§V-D, Table VIII)
- Automatic accent identification metrics and human perceptual evaluations of accent plausibility frequently diverge, limiting the reliability of either metric alone as a benchmark. (§V-B, §V-E)
- Large zero-shot TTS systems can produce competitive accented speech through style copying without explicit accent conditioning, often matching or exceeding dedicated accent-controllable models on naturalness. (§V-B, Table VII)
- Geolocation-based accent label discovery is extensible to languages and accent varieties without existing labeled speech, but precision varies substantially across accent regions depending on broadcast training data coverage. (§III-A, Table III)
Limitations and Open Questions
Warning
The paper evaluates English accent synthesis only, despite XTTS-v2 supporting multiple languages. The claim that geolocation-based label discovery generalises to any accent in any language is theoretical; precision on Canada (13.3%), Scotland (9.4%), Germany (15.9%), Ireland (20.4%), and Malaysia (7.1%) is poor, limiting practical applicability to a subset of the 11 accents studied.
Human evaluation is expensive and was conducted for only 4 of the 11 or more accents included in training, leaving uncertain whether objective metric rankings carry over to human preference for the remaining accents. The kNN-VC augmentation results are inconsistent: human evaluators found no clear benefit for augmented versus non-augmented systems in Australian and Scottish evaluations, contradicting objective metric improvements. The WER gap between the proposed systems and CosyVoice2 (13.1% vs. 7.1%) is notable and unexplained in depth. The geolocation model was trained on radio broadcasts, which may introduce domain mismatch when applied to crowd-sourced microphone speech.
Wiki Connections
This paper is most relevant to zero-shot-tts, multilingual-tts, speaker-adaptation, disentanglement, and prosody-control. It builds on the CosyVoice line of zero-shot TTS work (CosyVoice) as a baseline, and includes CosyVoice 2 as a primary comparison system. Accent evaluation methodology connects to evaluation-metrics and subjective-evaluation.