Speech Generation Research Wiki

CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech

3 min read

arXiv · 2025 · Preprint

Wang et al. · → Paper · Demo: ? · Code: ✓

CapSpeech introduces a large-scale benchmark of over 10 million machine-annotated and 360,000 human-annotated audio-caption pairs across five style-captioned TTS downstream tasks, together with evaluation protocols measuring style consistency, naturalness, and intelligibility.

Citation Stub

This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.

Context in Speech Generation

CapSpeech provides a unified resource for style-captioned TTS (CapTTS), the paradigm in which a natural language description specifies the desired speaking style rather than an audio prompt or categorical label. The benchmark covers five tasks: general CapTTS, TTS with sound events (CapTTS-SE), accent-captioned TTS, emotion-captioned TTS, and TTS for a single expressive virtual agent (AgentTTS). Alongside the dataset, two new human-recorded corpora (CapSpeech-SEDB and CapSpeech-AgentDB) are introduced to fill gaps where no open-source data existed. The benchmark also specifies an evaluation suite combining objective style-classification accuracy (Style-ACC), automatic MOS (UTMOSv2), WER, and three subjective MOS dimensions (SMOS, NMOS, IMOS), providing a reusable evaluation framework for instruction-conditioned TTS systems seeking to assess style fidelity across multiple attribute categories.

Wiki Connections

  • instruction-conditioned-tts — CapSpeech formalises five downstream tasks for caption-guided TTS, providing training data and evaluation protocols for systems that condition on natural language style descriptions.
  • emotion-synthesis — the EmoCapTTS and AgentTTS tasks cover fine-grained emotion categories and non-speech vocalizations, extending emotion-conditioned TTS beyond discrete categorical labels.
  • evaluation-metrics — the benchmark defines Style-ACC as a composite style consistency metric and pairs it with UTMOSv2, WER, and three subjective MOS dimensions (SMOS, NMOS, IMOS) for assessing CapTTS systems.
  • subjective-evaluation — the paper recruits 15 native annotators via Prolific with a qualification gate and reports 95% confidence intervals across three MOS dimensions, documenting a reproducible human evaluation protocol.
  • 2406.05370 (VALL-E 2) — cited as a representative large-scale zero-shot TTS system that motivates the need for style-captioned benchmarking beyond audio-prompt conditioning.
  • 2502.04128 (LLaSA) — cited as a recent TTS system built on LLaMA-based architectures, contextualising the landscape of systems that CapSpeech aims to evaluate.
  • 2412.10117 (CosyVoice 2) — cited alongside other recent TTS systems to motivate the gap in standardised style-captioned benchmarks.
  • 2409.00750 (MaskGCT) — cited as a recent masked generative codec TTS model that the benchmark contextualises.
  • 2406.02430 (Seed-TTS) — cited as a high-quality versatile TTS family demonstrating the state of the field that CapSpeech benchmarks against.
  • 2410.06885 (F5-TTS) — adopted as the non-autoregressive backbone for the CapSpeech-NAR model evaluated on the benchmark.
  • 2402.01912 (Parler-TTS / natural language guidance) — adopted as the autoregressive backbone for CapSpeech-AR and cited as the prior large-scale style-annotation approach that CapSpeech extends.
  • 2503.04713 (ParaSpeechCaps / scaling style-prompted datasets) — used as the primary comparison dataset in pretraining experiments; CapSpeech outperforms it on style consistency and naturalness.
  • 2504.12867 (EmoVoice) — cited as a related system for LLM-based emotional TTS with freestyle text prompting.
  • 2508.05385 (Non-verbal speech generation pipeline) — cited as related work on scaling non-verbal and expressive speech generation.
  • 2505.14648 (Vox-Profile) — cited as a related benchmark for characterising diverse speaker and speech traits.
  • 2407.10759 (Qwen2-Audio) — cited as an audio-language model used to evaluate speech understanding in the benchmark context.
  • 2504.18425 (Kimi-Audio) — cited alongside Qwen2-Audio as a speech understanding model evaluated for style captioning.
  • 1810.04805 (BERT) — the BERT base model is fine-tuned as a duration predictor for the NAR model in the CapSpeech pipeline.
  • 2312.15821 (Audiobox) — cited as a prior system supporting natural language prompts and sound events, providing the closest comparison point for CapSpeech’s scope.

Backlinks

Graph View