arXiv · 2025 · Preprint
Lee et al. · → Paper · Demo: ? · Code: ?
P2VA introduces a two-strategy framework that converts natural language persona descriptions into explicit voice attribute prompts for instruction-conditioned TTS, closing the usability gap between persona-based AI systems and voice style specification, while exposing systematic demographic biases in LLM-generated voice attributes.
Problem
Prompt-based TTS systems accept fine-grained style instructions in natural language, yet the instructions users can naturally provide are persona descriptions (“a calm and trustworthy healthcare assistant”) rather than explicit voice attribute specifications (“slow pace, low pitch, warm tone”). This mismatch forces a choice: require users to possess specialized knowledge of speech style terms, or accept that TTS systems will misinterpret vague personality-based descriptions. No prior work directly bridges persona descriptions and voice synthesis, leaving the usability gap largely unaddressed.
Method
P2VA (Persona-to-Voice-Attribute) is a pipeline layer that sits between a user’s persona description and an instruction-conditioned TTS model. The framework uses GPT-4o-mini in a prompting configuration (no fine-tuning) and offers two complementary conversion strategies.
P2VA-C (closed-ended form) instructs the LLM to extract structured values for a predefined attribute set: gender, accent (e.g., American, British, Indian), tone (e.g., Analytical, Warm, Engaging), speed, and pitch. These structured attributes are serialized into a JSON representation and then rewritten as a fluent text prompt suitable for the downstream TTS system. P2VA-O (open-ended form) removes the attribute schema constraint, allowing the LLM to produce a free-form style description that can capture voice qualities not expressible through the fixed attribute vocabulary.
Parler-TTS serves as the TTS backend throughout, receiving the P2VA-generated text prompts as conditioning. Evaluation uses persona descriptions sampled from the Persona-1M corpus paired with transcripts from LJSpeech across 1,000 persona-transcript pairs.
Key Results
P2VA-C outperforms the unguided baseline on all reported metrics. WER drops from 22% to 17% (5 percentage points), human MOS improves from 3.09 to 3.42 (+0.33 points), and UTMOS rises from 2.85 to 2.94. P2VA-O is intermediate across all metrics (WER 18%, human MOS 3.23, UTMOS 2.88), reflecting its richer but less constrained output. In a user study, over 90% of participants reported that P2VA-generated speech aligned better with their intended voice style than what they produced without it.
The bias analysis yields sharp findings. When LLMs process persona descriptions that do not include explicit gender cues (81% of the test set), the resulting voice attributes assign male gender 64% of the time and female gender 33%, compared to a source distribution of 10% explicitly male and 9% explicitly female. Accent outputs concentrate 90% in North American or British variants. Gender-conditioned attribute breakdowns further reveal stereotyped co-occurrences: male personas are assigned “Cognitive and Analytical” tone most frequently (41%), while female personas are disproportionately assigned “Warm and Supportive” tone (44%) and high pitch (62%).
Note
Human MOS evaluation uses a small participant pool, and the study does not report inter-rater agreement or statistical significance for MOS gains. UTMOS improvement is acknowledged by the authors to be of uncertain statistical significance.
Novelty Assessment
The framework is a prompt-engineering pipeline over two off-the-shelf components: GPT-4o-mini and Parler-TTS. No model training, no new architecture, and no new evaluation protocol are introduced. The primary contribution is the problem framing (persona-to-voice as a distinct task) and the demonstration that a two-strategy LLM conversion layer meaningfully improves TTS usability for non-expert users. The bias analysis is the most substantively novel finding: it quantifies how LLMs propagate and amplify demographic stereotypes when generating voice attribute prompts, a concern that extends to any system using LLMs to mediate between natural language descriptions and speech style.
The 0.33 MOS gain is meaningful given the simplicity of the intervention, though the evaluation covers only English speech on a single-speaker dataset. The claim of being the first persona-to-voice framework is plausible based on the related work surveyed.
Field Significance
Moderate — P2VA connects two active research directions (persona-based LLMs and instruction-conditioned TTS) and introduces a problem formulation that has not previously been addressed explicitly. Its more lasting contribution may be the bias findings: quantifying how LLMs encode gender and accent stereotypes during voice attribute generation raises a fairness concern relevant to any pipeline that uses language models to mediate voice style, not just persona-driven ones.
Claims
-
supports: An LLM-based intermediate conversion layer can reduce the expertise burden on users of instruction-conditioned TTS without requiring model retraining.
Evidence: P2VA-C converts persona descriptions to structured prompts via GPT-4o-mini zero-shot prompting, improving human MOS from 3.09 to 3.42 and reducing WER from 22% to 17% relative to an unguided baseline on 1,000 LJSpeech-transcript persona pairs. (§4.3, Table 2)
-
complicates: Natural language voice attribute prompts generated by LLMs should not be assumed to be demographically neutral.
Evidence: When personas lack explicit gender cues, LLM-generated voice attributes assign male gender 64% of the time (vs. 10% in source descriptions), and 90% of accent assignments default to North American or British variants. (§5.1, Table 3)
-
refines: The trade-off between structured and free-form style conditioning in instruction-conditioned TTS extends to intermediate conversion layers, not only the TTS interface itself.
Evidence: P2VA-C (structured, predefined attributes) achieves lower WER and higher human MOS than P2VA-O (free-form), while P2VA-O preserves expressivity for attributes outside the predefined vocabulary. (§4.3, Table 2)
-
complicates: Fairness in voice generation systems requires addressing bias in LLM-mediated style inference, not only in training data or TTS model design.
Evidence: Stereotyped gender-tone and gender-pitch co-occurrences (e.g., male assigned “Cognitive and Analytical” tone 41%, female assigned “Warm and Supportive” 44% and high pitch 62%) emerge from the LLM conversion layer itself, independent of the downstream TTS model’s behavior. (§5.1, §5.2, Tables 4-6)
Limitations and Open Questions
Warning
Evaluation is confined to English speech using a single-speaker dataset (LJSpeech), and the bias analysis covers only the five predefined attributes in P2VA-C. The scope of demographic bias in open-ended P2VA-O outputs is not characterized.
The user study does not report participant demographics, task instructions, or statistical significance, limiting the generalizability of the 90% alignment claim. The framework’s dependence on Parler-TTS means results may not transfer to TTS systems with different conditioning interfaces. Bias mitigation methods are identified as future work; this paper provides quantification but no remediation. The attribute preset (P2VA-C) was derived from Parler-TTS training data labels, coupling the framework’s controllability vocabulary to one model’s training choices.
Wiki Connections
- Instruction-Conditioned TTS — P2VA generates natural language style prompts from persona descriptions, enabling non-expert use of instruction-conditioned TTS systems such as Parler-TTS.
- Prosody Control — P2VA-C explicitly maps persona traits to prosodic attributes (speed, pitch, tone), making it a persona-aware prosody conditioning mechanism.
- Evaluation Metrics — the paper reports WER, human MOS, LLM-judged MOS (Gemini), and UTMOS as complementary quality dimensions, illustrating multi-perspective evaluation practice.
- Subjective Evaluation — human MOS ratings and a user study are used to assess both speech naturalness and style alignment with user intent.