arXiv · 2025 · Preprint
Poon et al. · → Paper · Demo: ✓ · Code: ✓
Proposes CLARITY, a backbone-agnostic framework that jointly mitigates two coupled sources of bias in instruction-guided zero-shot TTS: user-side linguistic bias (standardized input text that fails to reflect the target dialect) and system-side accent bias (models defaulting to majority accents despite explicit instructions).
Problem
Instruction-guided zero-shot TTS systems accept a transcript plus a natural-language description of the desired speaking style, but two interdependent biases undermine the authenticity of the output. Linguistic bias arises because users often supply text drawn from a “standard” dialect even when a different target dialect is requested, since their own linguistic and cultural background shapes how they write. Accent bias arises on the system side: because training corpora over-represent dominant accents (e.g., American English), TTS backbones default to majority-accent acoustics even when instructed to produce a minority accent. Prior work has examined gender bias in instruction-guided TTS, and separate studies have documented accent bias in voice cloning and broader audio foundation models, but no prior system had jointly addressed both the text-side and acoustic-side sources of bias, despite the paper’s argument that authentic accent generation requires both correctly localized text and accent-consistent acoustic conditioning.
Method
CLARITY frames dual-bias mitigation as a two-signal optimization problem layered on top of any zero-shot, instruction-guided TTS backbone: it searches for an adapted text and an accent-consistent speech prompt that jointly maximize linguistic and accent fidelity under the constraint that synthesis is performed by an unmodified backbone TTS model .
A free-form user instruction is first parsed by an LLM (Gemini-2.5-flash-lite) into structured metadata covering accent, gender, and age, falling back to defaults when an attribute cannot be inferred. Contextual linguistic text adaptation then rewrites the standard input text into a form aligned with the target dialect: candidate rewrites are generated independently by two different LLMs (GPT-4o-mini and LLaMA-3.1-8B), and an LLM-as-a-judge (GPT-5-mini) scores each candidate, along with the original standard text, on how well it aligns with the target dialect; the highest-scoring candidate is selected as the final text.
Retrieval-Augmented Accent Prompting (RAAP) addresses the acoustic side. From a curated pool of (accent, transcript, speech) triples, candidates matching the requested accent metadata are filtered, then scored using a fine-tuned ECAPA-TDNN accent classifier for accent confidence, combined with the TF-IDF cosine similarity between the candidate’s transcript and the user’s input text; the prompt with the highest combined score is retrieved and passed to the backbone TTS model as the conditioning speech prompt alongside the adapted text. The ECAPA-TDNN classifier (pretrained on CommonAccent, ~6.2M-parameter backbone frozen with only a 12-class linear head fine-tuned) is trained with AAM loss on the AESRC and SEAME accent pools for 12-way accent classification, and is reused both inside RAAP for prompt scoring and as the evaluation tool for accent accuracy.
Because the two backbone systems evaluated in the paper (CosyVoice2 and ParlerTTS) already accept instruction and prompt-speech inputs, CLARITY’s contribution is entirely at the framework level: it supplies better inputs (localized text, accent-consistent prompts) to an unmodified backbone rather than training a new generative model.
Key Results
On the ablation of RAAP’s own components (Table I), moving from no retrieval signal to accent-confidence-guided prompt selection alone raises accent accuracy from 45.6% to 61.74%, while adding GPT-adapted text on top of retrieval raises it further to 63.36%, the best configuration tested; NISQA quality remains stable across all variants (4.21-4.35 on a five-point scale). Compared against the two baselines, CLARITY with GPT-adapted text achieves substantially higher accent accuracy (63.36%) than CosyVoice2 (8.75%) and ParlerTTS (6.14%), while its NISQA score (4.31) sits between CosyVoice2 (3.16) and ParlerTTS (4.67). Fairness Discrepancy Rate (FDR) analysis shows CLARITY achieves higher fairness than both baselines, and a binomial test against a 1/12 random-chance baseline confirms that CosyVoice2 exhibits extreme, statistically significant bias toward US/CA accents across every one of the ten tested non-US/CA accents (p as low as 3.6e-67), whereas CLARITY substantially reduces (but for several accents does not eliminate) this bias. Human listening tests on four accents (Chinese, Singaporean, Indian, British) corroborate the objective results, with CLARITY rated best for Chinese- and British-accented English and comparable to or better than the baselines elsewhere.
Novelty Assessment
The contribution is primarily an inference-time engineering framework rather than a new model architecture or training objective: CLARITY combines an existing LLM-as-judge text-adaptation pattern with a retrieval mechanism built from off-the-shelf components (a fine-tuned accent classifier and TF-IDF similarity), and it deliberately avoids modifying the underlying TTS backbone. The genuinely new element is the joint framing itself: treating linguistic (text-side) and accent (acoustic-side) bias as a coupled dual-signal optimization problem, and demonstrating with a rigorous statistical protocol (binomial tests, FDR) that a purely input-side intervention can measurably reduce a backbone’s inherited accent bias without retraining it. The empirical scope (twelve accents, objective and subjective evaluation, systematic ablation) is a genuine strength, though the human listening panels are small (as few as two female and two male listeners per accent for two of the four tested accents).
Field Significance
moderate — CLARITY demonstrates that accent bias inherited from an imbalanced training corpus can be partially mitigated at inference time, without backbone retraining, by supplying better-targeted prompts and text. This gives the field a concrete, backbone-agnostic mitigation recipe and a reusable statistical protocol (binomial bias test plus Fairness Discrepancy Rate) for quantifying accent bias in instruction-guided TTS, though the underlying components (LLM-as-judge selection, retrieval-augmented prompting) are adapted from established techniques rather than newly invented.
Claims
- supports: Retrieval-based selection of speech prompts, guided jointly by an accent-confidence score and text-content similarity, substantially improves accent fidelity in zero-shot instruction-guided TTS compared to selecting prompts without such signals.
Evidence: In the ablation, accent accuracy rises from 45.6% (no accent or text-similarity signal in prompt selection) to 61.74% once accent-confidence-guided selection is added, a roughly 16-point gain before any text adaptation is applied. (§III-B.3, Table I)
- supports: Jointly adapting the input text toward the target dialect on top of accent-consistent prompt retrieval yields a further, though comparatively smaller, gain in accent fidelity beyond acoustic conditioning alone.
Evidence: Adding GPT-adapted text on top of accent-score-guided RAAP raises accent accuracy from 61.74% to 63.36% (about 1.6 points), a smaller increment than the gain contributed by retrieval-based prompt selection itself; NISQA quality is not harmed by the addition (4.25 to 4.31). (§III-B.3, Table I)
- complicates: Instruction-guided zero-shot TTS backbones trained on accent-imbalanced data exhibit strong, statistically significant bias toward majority accents that persists even when explicit contrary accent instructions are given, and an input-side mitigation framework can reduce but not eliminate this bias for some accents.
Evidence: A binomial test against a 1/12 random-chance baseline shows the CosyVoice2 backbone has extreme, consistent bias toward US/CA accents across all ten tested non-US/CA accents (p as low as 3.6e-67); CLARITY’s mitigation lowers the bias substantially but remains statistically significant for Chinese, Spanish, Korean, and Portuguese accents. (§III-D, Table II/VII)
- complicates: Automatic accent classifiers used both to drive retrieval-based conditioning and to evaluate accent fidelity are less reliable for accents that are phonetically or prosodically close to other accent groups, which limits both the conditioning signal quality and the trustworthiness of the resulting evaluation.
Evidence: The fine-tuned ECAPA-TDNN accent classifier reaches 89-99% accuracy for Indian, Singaporean, and British accents but drops to 50-59% for Japanese, Chinese, and Portuguese accents, with confusion concentrated within phonetically related regional clusters (East Asian, North American, European). (§Appendix B.1, Fig. 10)
Limitations and Open Questions
Human listening panels for two of the four subjectively evaluated accents (Indian and British English) consisted of only two female and two male native listeners each, a small sample that limits the statistical reliability of the subjective accent-authenticity comparisons for those languages.
Beyond the listening-panel size, several accents (Spanish, Japanese, Portuguese) remain below 50% accent accuracy under CLARITY’s best configuration, indicating the framework does not close the gap for all minority accents equally. The bias-reduction effect also varies by accent: CLARITY still shows statistically significant residual bias toward US/CA accents for Chinese, Spanish, Korean, and Portuguese target accents. The framework’s LLM-as-judge text-adaptation stage is itself judge-model dependent: text adapted by LLaMA scored lower when evaluated by a GPT-5 judge, which the authors attribute to possible judge-model mismatch rather than genuinely worse localization, suggesting the automatic scoring pipeline used to select the “best” adapted text may not generalize cleanly across judge/adapter model pairings. Finally, CLARITY was evaluated only on English-language accent variation; the authors note that extending the approach to genuinely multilingual adaptation is left to future work.
Wiki Connections
- Zero-Shot TTS — CLARITY is explicitly designed as a bias-mitigation layer for zero-shot, instruction-guided TTS backbones, operating without any backbone retraining.
- Instruction-Conditioned TTS — the framework’s linguistic-adaptation stage parses free-form natural-language style instructions into structured accent/gender/age metadata that then drives both text localization and prompt retrieval.
- Evaluation Metrics — the paper applies a statistical bias-quantification protocol (binomial testing against random-chance accent prediction, plus Fairness Discrepancy Rate) to instruction-guided TTS, extending fairness evaluation methodology from prior speaker-verification work into synthesis.
- Subjective Evaluation — native-speaker listening tests across four accents provide human validation of naturalness, accent accuracy, and demographic consistency, corroborating the objective accent-classifier results.
- CosyVoice — used as one of two baseline instruction-guided zero-shot TTS backbones (referred to in the paper as CosyVoice2) against which CLARITY’s accent accuracy, NISQA quality, and fairness are directly compared.
- Parler-TTS — used as the second baseline instruction-guided zero-shot TTS system in both objective and subjective evaluations.