arXiv · 2026 · Preprint
Chen et al. (National Taiwan University, Inventec Corporation) · → Paper · Demo: ? · Code: ?
Shows that gender bias in instruction-conditioned TTS (ITTS) emerges from non-additive interactions among compositional social cues (status, career, persona), not just isolated attribute associations, and traces these interactions back to the semantic priors of the pre-trained text encoder used for conditioning.
Problem
Prior audits of gender bias in ITTS systems (e.g., testing whether the word “nurse” biases synthesis toward a female voice) rely on univariate testing: probing one social attribute at a time and treating the resulting associations as the model’s bias profile. Real-world instructions are compositional, mixing status, occupational, and personality descriptors in a single prompt (a “reckless nurse” rather than just “nurse”). The paper argues that univariate testing misses cases where one cue overrides or amplifies another, producing a misleading picture of a model’s bias and, by extension, poorly targeted mitigation strategies.
Method
The paper formalizes an ITTS model as a function synthesizing a waveform from a style instruction and a content transcript, with the content transcript fixed to gender-neutral text so any acoustic gender shift is attributable to the instruction alone. It organizes instructions along three theoretically grounded social axes: Social Status (Weberian stratification, operationalized via Social Dominance Orientation descriptors), Career (occupational stereotypes), and Persona (Big Five personality traits).
The analysis proceeds in two stages. Stage 1 establishes univariate baselines: for each of 69 descriptors across the three axes, the empirical female probability P(x) is measured via a wav2vec 2.0 gender classifier (validated against human judgment on a 10% audited subset, 95% agreement), averaged over 100 synthesized utterances per descriptor (10 gender-neutral transcripts x 10 templates). Stage 2 constructs bi-dimensional and tri-dimensional composite instructions from the most polarizing Stage 1 descriptors and measures deviation from an additive baseline in log-odds space: an interaction term I quantifies how far a composite instruction’s bias departs from the sum of its component univariate biases, with significance assessed via a permutation test. To separate model-behavior bias from encoder bias, the paper additionally computes relative cosine similarity between each descriptor’s contextual embedding (from the text encoder used by each model: BERT, mT5, or Flan-T5) and gender anchor word sets, yielding standardized effect sizes (Cohen’s d).
Four open-source ITTS systems spanning three architectural families are evaluated: VoxInstruct (LLaMA-based autoregressive + non-autoregressive backbone with an mT5 text encoder, ~7B parameters), PromptTTS++ (diffusion + mixture-density-network backbone with a BERT encoder, ~150M parameters), and Parler-TTS Mini and Large (AudioLM-style backbones with a shared Flan-T5-large encoder, 880M and 2.3B parameters respectively).
Key Results
Univariate baselines already diverge sharply across backbones: VoxInstruct is female-skewed overall but shifts moderately and additively with career/persona stereotypes; PromptTTS++ is strongly bimodal, collapsing to P=0.00 for stereotypically male occupations (e.g., electrician) and P=1.00 for stereotypically female ones (e.g., midwife); the Parler-TTS family shows saturated high female probabilities (P >= 0.73) even for male-leaning occupations such as plumber.
At the compositional level, the paper identifies three distinct interaction regimes across the four models. VoxInstruct shows “additive smoothness”: interaction terms stay small and statistically indistinguishable from zero (|I| <= 1.81, p > 0.05). PromptTTS++ shows “asymmetric veto power”: under cue conflict, male-leaning cues systematically override female-leaning ones, with interaction terms as large as I = -5.78 to I = -9.43 for specific tri-dimensional combinations (p < 0.001). The Parler-TTS family shows “prior saturation”: under congruent (same-polarity) cues, interaction terms become strongly negative (I = -6.76 for Parler-Large, I = -7.68 for Parler-Mini) because the model’s already-saturated female prior cannot be pushed further, producing sub-additive rather than actively competitive behavior.
Comparing encoder-level bias scores (Table 6) against acoustic bias patterns (Table 2), the paper reports that text-encoder bias tracks acoustic polarization; for example, BERT’s effect size at the Career axis mirrors PromptTTS++‘s strict bimodal collapse. Training-data annotations alone are argued to be insufficient: many strongly biased descriptors in the study are absent from the PromptTTS++ training corpus’s gender-descriptor pairings, yet still produce polarized outputs.
Novelty Assessment
The contribution is methodological rather than architectural: no new TTS model is proposed. The novelty lies in (1) reframing gender-bias auditing as a compositional, multi-axis problem rather than a univariate one, and (2) an interaction-term formalism (additive log-odds baseline plus permutation-tested deviation) that lets the paper classify each evaluated system into one of three named bias regimes. The three-regime taxonomy (additive smoothness, asymmetric veto power, prior saturation) is a genuinely new descriptive framework for this specific problem, distinguishing it from prior single-attribute bias probes. The attribution of bias to text-encoder priors versus training-data distribution is an empirical correlational finding, not a causal intervention (no ablation retrains a model with a debiased encoder), so this claim should be read as suggestive rather than demonstrated.
Field Significance
moderate — the paper extends an existing univariate gender-bias evaluation paradigm for ITTS into a compositional one, contributing a reusable interaction-term methodology and a three-regime taxonomy that other bias audits of instruction-conditioned speech systems could adopt. Its scope is narrow (a single social dimension, binary gender, three social axes, four evaluated systems), and its causal claims about the origin of bias (text encoder versus training data) rest on correlational evidence rather than intervention.
Claims
- supports: Compositional, multi-attribute testing surfaces bias interaction effects in instruction-conditioned speech synthesis that univariate single-attribute testing does not capture.
Evidence: Interaction terms computed from bi- and tri-dimensional composite instructions reach large, statistically significant magnitudes (e.g., I = -5.78, I = -7.81, I = -9.43 across models) that are invisible to axis-level univariate probabilities alone. (§4.2, Table 5)
- supports: Different generative backbones and conditioning text encoders in instruction-conditioned TTS resolve conflicting compositional social cues through qualitatively distinct, non-uniform mechanisms rather than a single shared bias pattern.
Evidence: The paper categorizes the four evaluated systems into three regimes: additive smoothness (VoxInstruct, |I| <= 1.81, p > 0.05), asymmetric veto power (PromptTTS++, male-leaning cues override at p < 0.001), and prior saturation (Parler-TTS Mini/Large, I as low as -7.68 under congruent female-leaning cues). (§4.2)
- refines: Gender bias in instruction-conditioned speech synthesis traces in part to semantic priors embedded in the pre-trained text encoder used for conditioning, not solely to demographic imbalance in the acoustic training corpus.
Evidence: Standardized bias effect sizes computed from BERT, mT5, and Flan-T5 embeddings correlate with each system’s acoustic gender polarization pattern (e.g., BERT’s Career-axis effect size mirrors PromptTTS++‘s bimodal collapse), and many strongly biased descriptors are absent from the PromptTTS++ training corpus’s own gender-descriptor annotations. (§4.3, Table 6)
- complicates: Naive prompt-based debiasing strategies are insufficient once social cues interact compositionally, limiting the reliability of simple instruction-rewriting as a mitigation approach.
Evidence: The paper reports that conventional/generic prompting fails to resolve the deeper compositional bias layers it identifies, while targeted contextual attribute insertion is described as showing greater feasibility as a mitigation direction. (§1, §5)
Limitations and Open Questions
The attribution of bias to text-encoder semantic priors versus training-data distribution is correlational, not causal: no experiment retrains or swaps a model's text encoder to isolate the encoder's causal contribution, so the relative weight of the two proposed sources remains unverified.
The study operationalizes gender as binary throughout, which the authors acknowledge limits generalization to non-binary or continuous gender representation. The evaluation covers only three social axes (status, career, persona) and four open-source ITTS systems, so the three-regime taxonomy’s generality to other instruction-conditioned architectures or to other bias dimensions (e.g., race, age) is untested. The mitigation claim in the abstract and introduction (generic diversity prompting is insufficient, contextual attribute insertion is more promising) is not accompanied by a dedicated experimental section or table in the paper as parsed, leaving that specific finding relatively underspecified.
Wiki Connections
- Instruction-Conditioned TTS — audits three representative instruction-conditioned TTS systems (VoxInstruct, PromptTTS++, Parler-TTS) for how natural-language style prompts encode unintended social bias.
- Evaluation Metrics — introduces an interaction-term formalism (additive log-odds baseline plus permutation-tested deviation) as a new bias-auditing metric for compositional prompts.
- Subjective Evaluation — validates its automatic wav2vec 2.0 gender classifier against human judgment on a manually audited subset before relying on it for all bias measurements.
- Gender Bias in Instruction-Guided Speech Synthesis Models — extends this precedent’s univariate bias-probing methodology into a compositional, multi-axis framework and interaction-term formalism.
- VALL-E — cited as an example of the neural codec language modeling paradigm underlying modern instruction-conditioned TTS systems.
- F5-TTS — cited as an example of recent high-fidelity TTS architectures contrasted against instruction-based conditioning approaches.