arXiv · 2025 · Preprint
Bai et al. · → Paper · Demo: ✓ · Code: ✓
An automated pipeline for constructing large-scale paralinguistic speech datasets, yielding a 118.75-hour Chinese corpus (SynParaSpeech) with 6 paralinguistic categories and precise timestamps, validated through fine-tuning of CosyVoice2 and F5-TTS and through in-context prompt tuning of multimodal LLMs for event detection.
Problem
Natural conversational speech is rich with paralinguistic vocalizations (laughter, sighs, gasps, throat clearing, pauses, and tsks), yet most TTS research focuses exclusively on semantic content. The few existing paralinguistic speech datasets either lack speech context and timestamps (audio-only event collections such as AudioSet and ESC-50), have low sampling rates, cover only a handful of categories, or were constructed via ASR-based annotation methods that suffer from category imbalance and high annotation cost. Methods such as NVS and NVSpeech that expand paralinguistic coverage using ASR re-labelling introduce bias from the imbalance inherent in found media (animations, films) and cannot easily scale to new languages or categories.
Method
SynParaSpeech constructs training data through a three-stage automated synthesis pipeline applied to natural conversational speech. In Stage I, three ASR models (Whisper Large V3, SenseVoice, and Paraformer) produce sentence-level transcriptions by majority voting, VAD segments the audio into clips, and a split-point validation procedure using character error rate (CER below 0.1) ensures accurate timestamped segments. DeepSeek V3 then inserts paralinguistic tags ([laugh], [sigh], [gasp], [throat clearing], [pause], [tsk]) at appropriate text boundaries without modifying the transcript.
In Stage II, isolated paralinguistic audio events are randomly selected from a matching category corpus and voice-converted to match the timbre of the target speaker. The voice conversion pipeline uses Whisper Large V3 for semantic encoding, CAM++ for speaker embedding extraction, and the zero-shot VC model SeedVC to produce timbre-adjusted paralinguistic audio. The converted paralinguistic segment is then inserted at the timestamped position in the clean speech clip.
Stage III applies manual verification by speech professionals across four dimensions: naturalness of speech-to-paralinguistic transitions, timbre consistency, overall audio quality, and accuracy of temporal alignment.
The resulting SynParaSpeech corpus contains 118.75 hours across 79,986 clips in Mandarin Chinese at 24 kHz, distributed across 6 categories (sigh 23.76%, throat clearing 21.43%, laugh 17.55%, pause 15.41%, tsk 12.48%, gasp 9.36%) with balanced representation across categories.
For downstream validation, the authors fine-tune CosyVoice2 (autoregressive multi-stage codec TTS) and F5-TTS (flow-matching non-autoregressive TTS) using supervised fine-tuning (SFT), and extend CosyVoice2 with direct preference optimization (DPO). In the DPO setup, each SynParaSpeech utterance provides a natural positive-negative pair: the original clean speech (without paralinguistic events) serves as the rejected sample and the synthesized version with paralinguistic events serves as the chosen sample. New paralinguistic tokens are initialized from CLAP’s RoBERTa-based text encoder via dimension interpolation.
Key Results
All results are reported on 100 inference texts per paralinguistic category drawn from a paralinguistic rewrite of Seed-TTS-Eval, evaluated by 21 volunteers on four 5-point scales: PMOS (paralinguistic quality), NMOS (naturalness), SMOS (speaker similarity), and QMOS (overall quality), plus objective metrics CER, SECS, and UTMOSv2.
Fine-tuning on SynParaSpeech substantially improves paralinguistic quality over both the baseline (untrained) models and models trained on the NVS dataset. CosyVoice2’s PMOS increases from 1.88 (baseline) to 2.35 (NVS SFT) to 3.31 (SynParaSpeech SFT). F5-TTS shows a similar pattern, rising from a PMOS of 1.16 to 3.10 with SynParaSpeech SFT. DPO further improves CosyVoice2: DPO-Staged reaches PMOS 3.40 and DPO-Joint reaches PMOS 3.46, the best result across all configurations, with accompanying gains in NMOS (4.17), SMOS (4.03), and QMOS (4.12).
A consistent finding is that objective metrics degrade after paralinguistic fine-tuning: CER rises (since laughter may be transcribed as “ha ha”) and UTMOSv2 falls, as these metrics were designed for standard speech. Subjective NMOS and QMOS remain high, indicating that the objective-subjective gap is particularly acute for paralinguistic data.
For paralinguistic event detection, SynParaSpeech prompt tuning improves both accuracy and macro F1 over no-context baselines for Kimi Audio and Qwen 2.5 Omni. Performance peaks at 5-shot context and degrades at 7-shot, indicating an optimal prompt length.
Novelty Assessment
The primary contribution is the automated dataset construction pipeline rather than a new model architecture. The individual components (multi-model ASR majority voting, voice conversion for timbre alignment, LLM-based tag insertion) are all established techniques; the novelty lies in assembling them into a reproducible, scalable pipeline for paralinguistic data synthesis. This is a meaningful engineering contribution, as the field has lacked open-source large-scale paralinguistic datasets with precise timestamps.
The DPO application to paralinguistic speech is incremental but practically valuable: the paired structure of SynParaSpeech (original vs. synthesized utterances) maps naturally to preference pairs, and the gains over SFT are consistent and meaningful. The evaluation framework (PMOS alongside standard MOS dimensions) is straightforward.
Limitations include Chinese-only coverage, a restricted set of 6 paralinguistic categories, and the use of synthetic (rather than naturally occurring) paralinguistic insertions. The gap between objective metrics (CER, UTMOSv2) and subjective ratings also underscores that standard automatic evaluation is insufficient for paralinguistic speech.
Field Significance
Moderate — SynParaSpeech fills a concrete gap by providing the first openly available large-scale paralinguistic speech corpus with precise timestamps derived from natural conversational speech. The automated construction approach can in principle be extended to other languages and paralinguistic categories. The DPO experiments demonstrate that the dataset’s natural pairing structure enables preference learning, a direction that may generalize to other expressive speech synthesis tasks. Its scope is currently limited to Mandarin Chinese and a narrow set of 6 event types.
Claims
-
supports: Automated mixing of isolated paralinguistic audio events with clean conversational speech, combined with voice conversion for timbre alignment, can produce effective fine-tuning data for paralinguistic TTS.
Evidence: CosyVoice2 SFT on SynParaSpeech achieves PMOS 3.31 versus 2.35 for NVS SFT and 1.88 baseline; F5-TTS SFT on SynParaSpeech similarly achieves PMOS 3.10 versus 1.49 for NVS SFT. (§4.2.1, Table 3)
-
supports: DPO with naturally paired utterances (with vs. without paralinguistic events) can improve paralinguistic speech quality beyond SFT alone.
Evidence: CosyVoice2 DPO-Joint achieves PMOS 3.46 and QMOS 4.12 versus SFT-only PMOS 3.31 and QMOS 4.01; joint SFT+DPO outperforms staged DPO across all subjective dimensions. (§4.2.1, Table 3)
-
complicates: Standard automatic speech quality metrics are unreliable for evaluating paralinguistic TTS systems.
Evidence: All models fine-tuned on paralinguistic data show degraded CER, SECS, and UTMOSv2 relative to baselines, despite achieving higher subjective NMOS and QMOS; laughter vocalizations raise CER because ASR transcribes them as “ha ha”. (§4.2.1, Table 3)
-
supports: Few-shot in-context prompting with paralinguistic examples improves paralinguistic event detection in multimodal LLMs.
Evidence: SynParaSpeech prompt tuning raises accuracy and macro F1 for both Kimi Audio and Qwen 2.5 Omni over no-context baselines, with optimal performance at 5-shot context for both models. (§4.2.2, Table 4)
-
complicates: Increasing prompt context length for in-context learning in paralinguistic event detection yields diminishing returns and eventually degrades performance.
Evidence: Kimi Audio accuracy peaks at 5-shot (0.382) and declines at 7-shot (0.371 for Kimi Audio); Qwen 2.5 Omni also peaks at 5-shot (0.473) and declines at 7-shot (0.423), reflecting input overload beyond an optimal context window. (§4.2.2, Table 4)
Limitations and Open Questions
Warning
The corpus is Chinese-only and covers only 6 paralinguistic categories, limiting direct applicability to other languages or richer paralinguistic inventories. The paralinguistic events are inserted synthetically rather than captured in natural speech, which may introduce distribution mismatch relative to genuinely spontaneous vocalizations.
Standard objective metrics (CER, UTMOSv2, SECS) degrade after paralinguistic fine-tuning and do not reflect the true quality improvement observed in subjective evaluation. This leaves the field without reliable automatic metrics for paralinguistic speech, requiring human listener studies for valid comparisons. The paper does not evaluate zero-shot generalization to unseen paralinguistic categories or cross-lingual transfer.
Wiki Connections
- Flow Matching — F5-TTS, a flow-matching TTS model, is fine-tuned on SynParaSpeech and shows large PMOS gains, validating the dataset’s utility for non-autoregressive paralinguistic synthesis.
- Autoregressive Codec TTS — CosyVoice2, an autoregressive LM-based TTS system, serves as the primary fine-tuning backbone and benefits most from DPO applied to SynParaSpeech pairs.
- RLHF Speech — the paper applies DPO using SynParaSpeech’s natural pairing structure (with vs. without paralinguistic events) as positive-negative preference pairs, extending preference alignment to paralinguistic speech quality.
- Emotion Synthesis — paralinguistic vocalizations (laughter, sighs, gasps) overlap substantially with expressive and emotional speech, and SynParaSpeech provides training data for modeling these naturalistic expressive sounds.
- Subjective Evaluation — a 21-listener double-blind study with four MOS dimensions (PMOS, NMOS, SMOS, QMOS) reveals a systematic objective-subjective gap in evaluating paralinguistic TTS.
- NVS (Scalable Pipeline for Non-Verbal Speech) — SynParaSpeech directly compares against NVS as a baseline dataset for fine-tuning, consistently outperforming it on paralinguistic quality metrics.
- The paper follows the protocol of MMSU (a spoken-language-understanding benchmark, out of corpus scope) for selecting Kimi Audio and Qwen 2.5 Omni as evaluation models for paralinguistic event detection.