EMNLP · 2025 · Conference
Anuj Diwan et al. (University of Texas at Austin / New York University) · → Paper · Demo: ✓ · Code: ✓
ParaSpeechCaps introduces the first large-scale (2,709 hours total) open-source dataset annotating speech with 59 rich paralinguistic style tags covering both intrinsic speaker-level and situational utterance-level dimensions, along with an automatic pipeline that scales human-annotated rich style data by 9× (intrinsic) and 3× (situational) using perceptual speaker similarity and audio-LLM acoustic matching.
Problem
Style-prompted TTS requires training data that pairs utterances with natural-language style descriptions rich enough to reflect how humans actually describe voice. Existing large-scale datasets (Parler-TTS training data, PromptTTS2) automate annotation via signal processing, but this limits coverage to basic, extractable tags — pitch level, speed, gender — which do not capture the abstract and subjective qualities (guttural, shrill, pained, sarcastic) that make style prompts expressive. Small-scale human-annotated datasets do cover richer styles, but top out at under 100 hours. The result is a data gap: no large-scale open-source resource covers both rich intrinsic (speaker identity) and rich situational (utterance affect) tags simultaneously, and no principled method existed for scaling such annotations automatically without expensive human labelling at the target scale.
Method
The taxonomy organises 59 style tags along two axes: intrinsic (33 tags: speaker-persistent qualities like pitch texture, clarity, rhythm, accent) vs. situational (26 tags: utterance-level qualities like emotion and expressiveness), and rich (subjective, requires annotation) vs. basic (extractable automatically). Eleven style factors with an average of five tags each cover the full range.
PSC-Base (282 hours, human-annotated) is built by hiring Amazon Mechanical Turk workers to annotate 594 VoxCeleb celebrities and speakers from Expresso and EARS with rich intrinsic tags. Each annotator hears a 20–40 second audio collage and writes at least three style tags; tags are retained if two or more annotators agree. Situational tags come from remapping existing emotion labels in Expresso and EARS. All tag sets are converted to natural-language style prompts using Mistral-7B-Instruct-v0.2.
PSC-Scaled (2,427 hours, automatic) applies two separate scaling pipelines to the 45k-hour Emilia dataset. For intrinsic tags, perceptual speaker embeddings (VoxSim, trained on human perceptual similarity ratings) are computed for each speaker; Emilia speakers reaching cosine similarity ≥0.8 to an annotated VoxCeleb speaker inherit that speaker’s intrinsic tags, multiplying intrinsic data 9×. For situational tags, a three-stage cascade handles the absence of comprehensive emotion classifiers: (1) Expressivity Filtering keeps utterances with extreme dominance-valence-arousal values using an off-the-shelf DVA classifier; (2) Semantic Matching embeds transcripts with SFR-Embedding-Mistral and ranks them against emotion-description queries; (3) Acoustic Matching prompts Gemini 1.5 Flash as an audio LLM to rate whether the speech acoustically matches the target emotion on a 5-point Likert scale, retaining only top-scored examples. This 3× multiplies situational data.
The TTS model is Parler-TTS (880M params), an autoregressive language model that predicts DAC audio tokens with Flan-T5-Large as a frozen text/style encoder and cross-attention to the style prompt. Fine-tuning on ParaSpeechCaps is the core training step. At inference, classifier-free guidance (scale 1.5, zero-tensor unconditioned prompt) improves style adherence — notably, this is effective even without CFG dropout during training.
Key Results
Results are from Table 3, a 246-example holdout evaluation with Amazon MTurk annotators (three annotations per example, 95% CIs reported):
| Model | Consistency MOS | Intrinsic TR | Situational TR | Naturalness MOS |
|---|---|---|---|---|
| Parler-TTS (baseline) | 3.05 | 33.0% | 21.2% | 2.85 |
| +LibriTTS-P, Exp, EARS (best baseline) | 3.55 | 40.7% | 69.7% | 3.10 |
| PSC-Base model | 3.75 | 63.6% | 68.1% | 3.27 |
| PSC-Scaled model | 3.83 | 69.5% | 75.4% | 3.58 |
The Scaled model’s gains over the best baseline are +7.9% Consistency MOS and +15.5% Naturalness MOS. Intrinsic Tag Recall gains (+28.8 percentage points over the best baseline) are especially large, reflecting that PSC-Base’s VoxCeleb intrinsic annotations are of higher quality than LibriTTS-P.
Human evaluation of the automatic scaling process (Table 2) confirms that PSC-Scaled achieves tag recall comparable to PSC-Base: 50.3% vs. 48.7% (intrinsic) and 71.3% vs. 68.1% (situational). Using a standard WavLM ECAPA-TDNN embedder instead of VoxSim drops intrinsic recall to 45.3%, validating that perceptual similarity is a better proxy than speaker verification for style propagation. Removing any of the three situational scaling steps degrades recall, confirming all stages are necessary.
Intelligibility trade-off: models trained on rich style data score lower on both IMOS and WER (8.63% WER for Scaled model) than clean audiobook baselines (4.47%), but this is because they faithfully reproduce naturally less-intelligible styles (non-American accents, slurred speech, extreme emotions) rather than a quality defect.
Novelty Assessment
The dataset and its scaling pipeline are the genuine contribution. The perceptual speaker similarity approach for intrinsic tag propagation is well-motivated and technically sound — standard speaker verification embeddings encode identity rather than perceived vocal style, so VoxSim is a principled choice. The three-stage situational pipeline demonstrates a practical method for using audio LLMs as quality filters for subjective style annotations, which is a reusable design pattern. The TTS modeling component is incremental: Parler-TTS is used as-is, and applying CFG at inference with a zero-tensor unconditioned prompt is a minor trick, not a contribution. The paper’s value is primarily in releasing a resource and demonstrating an annotation methodology, not in advancing model architecture.
Field Significance
Moderate — ParaSpeechCaps fills a concrete gap in the instruction-conditioned TTS landscape: the absence of large-scale open-source data with rich, abstract style tags spanning both speaker identity and utterance affect. The automatic scaling methodology — using perceptual speaker embeddings for intrinsic tag propagation and a cascaded DVA-semantic-acoustic pipeline for situational tags — provides a reusable template for extending style-captioned datasets without proportional annotation cost. The evaluation methodology (Consistency MOS, Tag Recall by tag type) also advances the field’s toolkit for measuring stylistic adherence, where automatic metrics do not yet exist.
Claims
- Large-scale instruction-conditioned TTS datasets with rich abstract tags produce substantially higher style consistency and naturalness than datasets limited to basic signal-extractable tags. (§5.3, Table 3)
- Perceptual speaker similarity, rather than speaker verification similarity, is the appropriate proxy for propagating subjective intrinsic style tags across speakers. (§4.3, Table 2)
- Combining expressivity filtering, semantic matching, and acoustic LLM scoring is necessary for reliable automatic annotation of situational style tags at scale; omitting any stage degrades quality. (§4.3, Table 2)
- Style-expressive TTS models trained on rich paralinguistic data incur intelligibility trade-offs that reflect genuine style fidelity rather than generation quality degradation. (§5.3, §5.5)
- The lack of automatic metrics for style consistency in instruction-conditioned TTS remains a bottleneck: current evaluation relies entirely on expensive crowd-sourced human judgement. (§5.1, Limitations)
Limitations and Open Questions
Warning
All style consistency evaluation is human-annotated on a single holdout set of 246 examples drawn exclusively from the PSC-Base distribution; there is no out-of-domain or in-the-wild evaluation of the trained model.
Coverage is English-only, limiting applicability to multilingual style-prompted TTS. The dataset carries demographic biases baked into the annotation process — shrill is overrepresented for female speakers, guttural for male — which a trained model may amplify when generating novel combinations. The Emilia source corpus used for PSC-Scaled is itself not bias-audited for style diversity. The TTS model evaluation uses the same holdout distribution as the training data; generalisation to novel compositional style prompts remains partially tested (compositional evaluation shows improvement but uses no ground truth audio). The absence of robust automatic metrics for style consistency means large-scale ablations and model comparisons remain costly.
Wiki Connections
ParaSpeechCaps directly informs instruction-conditioned-tts, providing the first large-scale open-source dataset pairing speech with rich abstract style descriptions. The intrinsic tag taxonomy (pitch texture, rhythm, clarity) connects to prosody-control. The situational tags (emotion, affect, expressiveness) connect to emotion-synthesis. The Consistency MOS and Tag Recall evaluation framework is a contribution to subjective-evaluation practice for stylistic TTS. The use of VoxSim (a perceptual speaker similarity model trained on human perceptual ratings) for intrinsic tag propagation highlights the importance of perception-based speaker representations for style data collection.