arXiv · 2025 · Preprint
Runchuan Ye et al. (Tsinghua University) · → Paper · Demo: ✓ · Code: ?
A fully automated annotation pipeline — combining a wav2vec-bert-based frame-level NV detection model and a Temporal-Semantic Alignment (TSA) module — produces NonVerbalSpeech-38K (NVS), a 38K-sample, bilingual (EN/ZH) dataset of naturally-occurring non-verbal vocalizations that improves controllable NV synthesis and NV-aware transcription when used to fine-tune existing TTS and ASR models.
Problem
Existing speech systems largely ignore non-verbal vocalizations (NVs) such as laughter, sighs, coughs, and breathing sounds — phenomena that carry substantial emotional and communicative content in human speech. Building datasets to support NV-aware systems is expensive: manual annotation is slow and limits speaker diversity, rule-based augmentation (e.g., concatenating NV clips onto clean speech) produces unnatural timbre mismatches, and semi-automated approaches that first manually seed a dataset still require significant human effort. Prior datasets such as NonVerbalTTS and SMIIP-NV are either English-only, limited in scale, or confined to a small set of NV categories. The lack of large-scale, naturally-occurring, well-annotated bilingual NV data constrains both the synthesis of controllable NV speech and the development of NV-aware speech understanding models.
Method
The pipeline operates in three stages. In the first stage, in-the-wild audio from movies, cartoons, variety shows, and radio dramas is preprocessed through the Emilia-Pipeline (standardization, source separation, speaker diarization), with quality filtering applied via WER and DNSMOS thresholds to obtain clean single-speaker segments with word-level timestamps.
In the second stage, a unified frame-level NV detection model localises NV events within these segments. The model builds on pretrained wav2vec-bert 2.0: a learnable weighted sum fuses representations across its 25 layers, which are then passed through a transformer encoder. A transformer decoder with N_q = 10 learnable queries (one per NV category) cross-attends to the encoder output, and frame-level predictions are obtained by computing sigmoid similarities between decoder and encoder outputs. The model is trained with binary cross-entropy loss on augmented samples constructed by concatenating or overlaying NV clips from VocalSound, VGGSound, and other sound event datasets onto speech from GigaSpeech. A sliding window strategy (10 s window, 5 s hop) is applied to full segments during inference, with post-hoc filters on minimum duration, energy, and CLAP score.
In the third stage, detected NV segments are matched to adjacent speech regions and their labels are inserted into ASR transcriptions. A straightforward timestamp-based ordering (TBO) method places NV tags by comparing ASR word timestamps against NV boundaries. Because ASR timestamps contain offsets that cause positional drift, the paper introduces the Temporal-Semantic Alignment (TSA) method: speech is cut at NV boundaries, ASR is run independently on each piece, and the resulting segments are merged using edit-distance alignment, combining NV timing precision with full-segment semantic coherence.
For downstream evaluation, F5-TTS (a flow-matching transformer system) is fine-tuned on NVS by extending its vocabulary with NV tag tokens, initialising the new embeddings from CLAP text encoder representations. BigVGAN-v2 is used as vocoder. For NV understanding, Qwen2-Audio and Whisper-Large-V3 are fine-tuned with LoRA and full-decoder fine-tuning respectively, also with expanded NV token vocabularies.
Key Results
On NV speech generation (Table 2), F5-TTS fine-tuned on NVS (TSA) achieves CLAP Scores of 0.187 (EN) and 0.179 (ZH), compared to 0.110/0.100 for the strongest comparable baseline (F5-TTS + NVTTS), representing relative gains of approximately 70% and 79%. IMOS scores of 3.48 (EN) and 3.71 (ZH) are the highest among fine-tuned conditions and outperform CosyVoice2 and Dia on most NV categories. The TSA variant consistently outperforms TBO — CLAP scores improve by roughly 0.07 and IMOS by roughly 0.2–0.3 in both languages — confirming that positional accuracy of NV tag annotations directly drives controllability.
On NV speech understanding (Tables 4–5), NVS (TSA) fine-tuning reduces Tag Position Distance (TPD) by 0.92 for Qwen2-Audio and by 2.64 for Whisper-Large-V3 compared to CapSpeech-trained baselines in English, and achieves competitive NV-aware CER on Chinese despite being smaller than competing datasets. Again, TSA substantially outperforms TBO on TPD and NTD metrics across both languages.
The frame-level detection model achieves approximately 91% F1 across three test splits, with cross-lingual generalisation — training on English data only, the model performs comparably on Chinese test sets.
Note
NVS (TSA) shows somewhat lower SSIM and higher WER than F5-TTS baselines on English, likely due to dataset size constraints; the authors note expansion as future work. Comparison systems differ in their supported NV categories, so aggregate CLAP score rankings exclude partially-supported baselines.
Novelty Assessment
The primary contribution is the NVS dataset and the automated annotation methodology. The pipeline recombines existing components — Emilia preprocessing, Whisper transcription, CLAP-based quality scoring — with two genuinely new elements: the frame-level NV detection model (a specific application of learnable-query transformer decoding to NV event localisation) and the TSA alignment module (a practical fix to the timestamp-drift problem). Neither element is architecturally novel in a broad ML sense; the detector is closely related to T-UAED, and edit-distance-guided merging is a standard NLP technique. The contribution is strongest as an engineering integration that makes fully automated, scalable NV dataset construction practical for the first time. The release of a 38K-sample bilingual dataset covering 10 NV categories, with no human annotation cost, is the work’s most durable contribution to the field.
Field Significance
Moderate — The NVS dataset fills a clear gap: the field has lacked a large-scale, naturally-occurring, bilingual resource for non-verbal vocalizations. By demonstrating that automated annotation can match or exceed manually annotated datasets in downstream task performance, this work lowers the barrier to training expressive NV-capable TTS and ASR systems. Its impact is primarily as a data contribution; the methodological improvements are incremental but practically significant.
Claims
- Automated annotation pipelines for non-verbal vocalizations can match or exceed manually-annotated datasets in downstream NV generation and understanding tasks, while scaling at substantially lower cost. (§5.1, Table 2; §5.2, Tables 4–5)
- Positional accuracy of non-verbal tag annotations is a more important determinant of NV controllability in TTS than dataset size alone. (§5.1, Table 2)
- Frame-level NV detection models trained exclusively on one language can generalise to structurally different languages without retraining, suggesting that acoustic features of non-verbal sounds are largely language-agnostic. (§2.2; §3)
- Rule-based NV data augmentation produces measurably worse NV controllability in fine-tuned TTS systems compared to models trained on naturally-occurring vocalizations. (§5.1, Table 2)
Limitations and Open Questions
Warning
The dataset is heavily skewed toward Chinese (over two-thirds of samples), not because the pipeline is language-dependent but because the crawled source material is predominantly Chinese. This imbalance means English NV performance, while competitive, may underperform on broader English benchmarks, and the paper does not provide results on out-of-domain English test sets.
The detection model is trained and evaluated on simulated test data (augmented speech + NV clips), not on naturally-occurring NV events in the wild. Whether the ~91% F1 reflects real-world performance is untested. The evaluation covers only six NV categories despite the dataset spanning ten. NVS is also relatively small (38K samples, ~131 hours) compared to comparable speech datasets used for general TTS; the observed SSIM and WER gaps relative to the base F5-TTS suggest this is a binding constraint.