arXiv · 2025 · Preprint
Zhou et al. · → Paper · Demo: ? · Code: ✓
Two open-source dual-track conversational speech datasets (10 hours Chinese, 5 hours English) capturing full-duplex spontaneous dialogue with overlaps, backchannels, and interruptions, released to support conversational TTS and spoken dialogue system research.
Problem
TTS training corpora and conversational dialogue datasets overwhelmingly follow turn-based recording conventions, where speakers alternate cleanly and overlapping speech is rare or absent. Classic telephone corpora like Switchboard constrained participants to sequential exchanges, and TTS-focused datasets like DailyTalk target contextual synthesis without annotating interruption dynamics or spontaneous overlap events. This leaves a data gap for systems that aim to model full-duplex conversational behaviour, where more than 40% of speaker turns in natural interaction involve simultaneous speech, and turn transitions occur within hundreds of milliseconds.
Method
Two datasets were collected through a crowdsourcing platform (Magic Data), recruiting pairs of native Mandarin Chinese and native English speakers who were already familiar with each other. Familiarity was prioritised to elicit natural backchannels, interruptions, and affective variation without scripting. Participants in separate rooms recorded simultaneously using mobile phones, producing speaker-isolated dual-mono WAV files (16-bit PCM, 16 kHz) with one track per speaker. This setup preserves overlap events as overlapping time intervals on the two tracks rather than merging them. Conversations proceeded without topic restriction or role assignment, covering diverse daily domains. A human-assisted quality control pipeline filtered recordings with severe noise. Manual transcription by trained annotators captured speaker identity, gender, precise timestamps, and paralinguistic elements (backchannels, laughter, non-verbal vocalizations), with VAD segmentation aligned to semantic completeness. The resulting datasets total 35 conversations (27 Chinese, 8 English) spoken by 14 unique speakers, with 2,306 Chinese utterance segments (10 hours) and 4,149 English utterance segments (5 hours).
To validate utility, the authors fine-tuned a pretrained CosyVoice-300M model separately on each language subset. CosyVoice-300M uses an autoregressive language model for text-to-speech token prediction (cross-entropy loss) and an optimal-transport conditional flow matching module for token-to-mel spectrogram generation. Both modules were trained independently with a learning rate of 1e-5 over 200 epochs.
Key Results
Objective evaluation used four acoustic distance metrics comparing synthesized speech to reference speech: spectrum l2 distance (spectral similarity), F0 Wasserstein distance (prosodic match), ZCR Wasserstein distance (voicing patterns), and energy Wasserstein distance (loudness dynamics). The Chinese fine-tuned model improved F0 distance by 7.08% and spectrum l2 by 1.59%. The English model improved F0 distance by 3.67% and ZCR distance by 7.44%, but English spectrum l2 distance worsened by 0.85%.
Subjective evaluation recruited 10 native listeners per language, who judged 80 A/B pairs (original vs. fine-tuned). For Chinese, 45.0% preferred the fine-tuned model, 33.9% preferred the original, and 21.1% rated them similar. For English, 46.4% preferred fine-tuned, 40.7% preferred original, and 12.9% rated them similar. Complementary MOS scores show small but consistent gains: naturalness improved from 4.37 to 4.40 (Chinese) and 4.47 to 4.51 (English); intelligibility improved from 4.46 to 4.49 (Chinese) and 4.56 to 4.58 (English).
Novelty Assessment
The contribution is a data resource, not an architectural or training innovation. The recording methodology (dual-track isolated speakers via mobile phone) and annotation scheme (VAD-aligned timestamps with paralinguistic tags) are practical but not technically novel. The value lies in the open-source release of naturally overlapping bilingual conversational speech, which is genuinely scarce in TTS and spoken dialogue research. The validation experiment fine-tunes an existing model (CosyVoice-300M) and uses standard metrics; no new evaluation protocol is introduced. The MOS and preference gains are modest in absolute terms, and with only 15 hours of data, the dataset is small by current standards. The significance is primarily as a resource contribution rather than a methodological advance.
Field Significance
Moderate — The datasets address a real and underserved data gap: open-source, speaker-isolated, full-duplex conversational speech with annotated overlaps and backchannels is rare, and its absence has constrained conversational TTS research. The bilingual release (Chinese and English) broadens applicability. The datasets are small (15 hours total), and the fine-tuning gains are incremental, but the resource can serve as a reproducible starting point for models targeting spontaneous dialogue synthesis.
Claims
-
supports: Incorporating spontaneous full-duplex conversational data into TTS training improves the prosodic naturalness of synthesized speech, as measured by both acoustic distance metrics and human preference.
Evidence: Fine-tuning CosyVoice-300M on 15 hours of dual-track conversational speech reduced F0 Wasserstein distance by 7.08% (Chinese) and 3.67% (English), and the fine-tuned model was preferred by a majority in A/B preference tests in both languages (45.0% vs. 33.9% Chinese; 46.4% vs. 40.7% English). (§4.1, §4.2, Table 2)
-
supports: Existing TTS and dialogue corpora lack the annotated overlap and interruption dynamics needed for full-duplex conversational modelling.
Evidence: Turn-based datasets like Switchboard constrain to sequential exchanges with limited overlap, and DailyTalk rarely includes spontaneous overlapping speech or explicit interruption annotations, leaving a data gap for full-duplex conversational TTS. (§1)
-
complicates: Fine-tuning TTS models on small conversational corpora yields perceptible naturalness gains but leaves spectral fidelity inconsistent across languages and acoustic feature dimensions.
Evidence: MOS naturalness improved by only 0.03-0.04 points in both Chinese and English, and English spectrum l2 distance worsened by 0.85% despite improvements in F0 (3.67%) and ZCR (7.44%), suggesting that small-scale fine-tuning does not uniformly improve all acoustic dimensions. (§4.1, §4.2, Table 2, Table 3)
Limitations and Open Questions
The datasets are small by current standards: 27 Chinese conversations (10 hours) with only 6 unique speakers, and 8 English conversations (5 hours) with 8 unique speakers. The limited speaker diversity and total duration constrain how much can be learned about speaker-independent conversational behaviours. The validation experiment evaluates only fine-tuning an existing TTS model; it does not test whether the datasets improve end-to-end full-duplex spoken dialogue systems, which was the stated motivation. The English subset’s spectral similarity worsened after fine-tuning, suggesting that domain mismatch or corpus size may limit generalization in some acoustic dimensions. The persistent DOI link for dataset access was not finalized at the time of submission.
Wiki Connections
- Spoken Language Model — The datasets are designed to support full-duplex spoken dialogue systems that require natural conversational dynamics, including overlaps and backchannels that standard TTS corpora do not provide.
- Subjective Evaluation — The paper conducts A/B preference tests with 10 native listeners per language alongside MOS ratings for naturalness and intelligibility, serving as the primary validation of dataset utility.
- Evaluation Metrics — Four acoustic distance metrics are employed for objective evaluation: spectrum l2, F0 Wasserstein, ZCR Wasserstein, and energy Wasserstein distances, which complement the subjective assessment.