arXiv · 2025 · Preprint
Lin et al. · → Paper · Demo: ? · Code: ?
Introduces the E-VOC corpus of 60,000+ human perceptual ratings to quantify the instruction-perception gap across four expressive control dimensions in instruction-guided TTS, finding that only GPT-4o-mini-TTS reliably translates degree modifiers and graded emotion adjectives into perceptually ordered output, while word-level emphasis and speaker age remain challenges for all five systems tested.
Problem
Instruction-guided TTS (ITTS) promises to make speech synthesis more accessible by accepting natural-language style prompts, but whether synthesized speech aligns with what a listener actually perceives from those instructions has been largely unmeasured. Prior ITTS evaluation relied on coarse proxies: accuracy on discrete category labels, embedding-based audio-text similarity, or overall relevance MOS. None of these capture whether degree modifiers such as “slightly” or “extremely” produce perceptually ordered output, nor whether graded emotional adjectives (from “content” to “ecstatic”) translate to smoothly increasing perceived intensity. Fine-grained controls like word-level prosodic emphasis and speaker age group had only been assessed with automated classifiers, which inherit training-data biases rather than reflecting actual listener perception.
Method
This paper introduces an evaluation framework and accompanying corpus rather than a new generative model. The framework defines four control dimensions: (1) adverbs of degree, testing whether degree modifiers (slightly, very, extremely) produce measurable and perceptually ordered changes in loudness, pitch, and speaking rate; (2) emotion-intensity adjectives, using graded synonyms drawn from the NRC Emotion Intensity Lexicon (e.g., Satisfied, Content, Happy, Overjoyed, Ecstatic for the happy category) to test whether systems produce correspondingly scaled emotion intensity; (3) speaker age, asking systems to synthesize voices perceived as Child, Teenager, Adult, or Elderly; and (4) word-level emphasis, testing whether systems can place prosodic prominence on a specified target word.
Five representative ITTS systems were evaluated: Parler-TTS-large-v1 and Parler-TTS-mini-v1 (open-source research), PromptTTS++ (open-source research), UniAudio (general-purpose audio foundation model), and GPT-4o-mini-TTS (commercial API). Eight conversational transcripts covering everyday contexts were generated with Gemini 2.5 Pro and paired with task-specific style prompts to produce the audio stimuli. Human perceptual ratings were collected from native English-speaking raters recruited on Prolific, with quality control through embedded check utterances drawn from CREMA-D, EMNS, and Nexdata.ai. The result is the Expressive VOice Control (E-VOC) corpus: 8,640 utterances with more than 61,000 human perceptual ratings from over 165 raters across all four control dimensions (Table 2). Objective acoustic measures (LUFS loudness, CREPE-estimated F0, words per second) supplement the subjective data for the adverbs-of-degree task.
Key Results
GPT-4o-mini-TTS was the most capable system across all dimensions. On the adverbs-of-degree task, it was the only model producing clear, consistent progressions in both objective acoustics and perceived emotion intensity when prompted with slightly, very, and extremely. For emotion-intensity adjectives, only GPT-4o produced smooth perceptual gradations across graded synonyms for all four emotions; open-source systems produced weak or irregular distinctions between adjacent intensity levels, with UniAudio occasionally showing reversed trends.
Speaker age control was difficult for all systems. Best overall accuracy reached only 0.294 (Parler-TTS-large), with all systems showing a strong adult bias: regardless of the instruction, the most common listener response was “adult” (Table 4). Child voices were particularly poorly realized, with F1-scores near zero across all five systems; even GPT-4o achieved only 0.074 F1 on the child category (Table 3).
Word-level emphasis was the most challenging task. GPT-4o achieved the highest accuracy at 0.265, compared to a random baseline of approximately 0.143 given an average of six candidate words per sentence (Table 3). The remaining systems performed only slightly above or at chance level, indicating that consistent word-level prosodic control remains largely unsolved.
Novelty Assessment
The contribution is methodological: a new perceptual evaluation corpus and analysis framework for ITTS controllability. The E-VOC corpus fills a genuine gap by providing large-scale human judgments on two novel dimensions (adverbs of degree, graded emotion adjectives) and extending perceptual evaluation to dimensions previously measured only with automated classifiers (age, emphasis). The annotation pipeline is carefully designed, with quality-controlled recruitment and inter-rater reliability statistics reported.
The paper introduces no new model, training recipe, or architectural insight. The five ITTS systems evaluated are pre-existing; the paper’s contribution is entirely the evaluation infrastructure and the findings it produces. The adult-voice bias and failure of emphasis control are quantified clearly, even if the underlying causes are not investigated. The comparison across five systems is informative, though the eight-transcript stimulus set limits linguistic diversity.
Field Significance
Moderate - this paper provides a rare large-scale human perceptual study of fine-grained ITTS controllability, surfacing specific and quantified failure modes (adult-voice bias, near-random emphasis accuracy) that coarser evaluation methods miss. The E-VOC corpus can serve as a benchmark for future ITTS systems and as training data for automated perceptual judges, though findings are constrained to five systems and eight conversational transcript types in English.
Claims
-
complicates: Fine-grained prosodic control in instruction-guided TTS does not improve monotonically with overall system capability.
Evidence: GPT-4o-mini-TTS led all five systems on adverbs-of-degree and emotion-intensity tasks yet achieved only 0.265 on word-level emphasis accuracy (barely above the 0.143 random baseline), and all systems showed near-zero F1 on child-voice generation regardless of system tier. (§4.3, §4.4, Table 3)
-
supports: Human perceptual evaluation reveals ITTS alignment gaps that automated classifier-based metrics are likely to underestimate.
Evidence: The E-VOC corpus collected 60,000+ ratings from 165+ raters on Prolific; large-scale human annotation found all five systems defaulted to adult-sounding voices regardless of explicit child or elderly instructions, a systematic bias that per-class classifiers tied to their training distribution would likely miss. (§2.2, §3.3, §4.3)
-
complicates: Current ITTS systems exhibit a systematic adult-voice bias that natural-language age instructions alone cannot overcome.
Evidence: GPT-4o confusion matrix (Table 4) shows utterances labeled Child, Teenager, and Elderly were predominantly perceived as Adult by human listeners; the best child-voice F1 across all five systems was 0.113 (Parler-TTS-large), with GPT-4o reaching only 0.074. (§4.3, Table 3, Table 4)
-
supports: Commercial ITTS systems can achieve perceptually ordered degree-scaled speech control that open-source alternatives fail to replicate at comparable levels.
Evidence: GPT-4o-mini-TTS was the only system producing consistent objective acoustic progressions (LUFS, F0, words per second) and corresponding perceptual gradations under degree modifiers; Parler-TTS variants, PromptTTS++, and UniAudio showed minimal, irregular, or reversed variation. (§4.1)
Limitations and Open Questions
Warning
The study generates audio from only eight conversational transcripts paired with fixed style prompts, which limits the diversity of linguistic contexts covered. Generalisation of findings about speaker age and word-level emphasis to a wider range of sentence types, speaking styles, or TTS voices not included in the study is untested.
Annotation covers only native English speakers from the United States, so judgments of perceived age and prosodic emphasis may reflect dialect-specific or cultural norms. Inter-rater agreement (Cohen’s kappa) was low for the adverbs-of-degree task (0.17) and emotion-intensity task (0.226), indicating that even human raters find degree-scaled emotion perception genuinely ambiguous, which complicates interpretation of model scores on those dimensions. The study measures the instruction-perception gap but does not diagnose whether failures arise from model architecture, training data distribution, or prompt formulation, leaving the design question open for future work.
Wiki Connections
- Instruction-Conditioned TTS - this paper provides the first large-scale perceptual evaluation of fine-grained instruction following across degree modifiers and graded emotion adjectives, quantifying where current ITTS systems succeed and where they fail systematically.
- Evaluation Metrics - the E-VOC corpus introduces human-labeled benchmarks for ITTS controllability across four dimensions, complementing existing MOS-based and embedding-similarity evaluation approaches.
- Subjective Evaluation - the study collects 60,000+ perceptual ratings from 165+ native-English raters on Prolific using a quality-controlled annotation protocol with embedded gold-standard check items.
- Emotion Synthesis - graded emotion intensity (from “satisfied” to “ecstatic”) and adverbially scaled emotional prosody are the two novel evaluation dimensions introduced, with only GPT-4o-mini-TTS achieving reliable perceptual gradations across all four emotion categories.