arXiv · 2025 · Preprint
Wenming Tu et al. (Shanghai Jiao Tong University / BIGAI) · → Paper · Demo: ? · Code: ✓
Introduces UltraVoice, an 830-hour synthetic speech-dialogue dataset with instructions across six fine-grained style dimensions (emotion, speed, volume, accent, language, composite), and shows that fine-tuning spoken dialogue models on it improves style controllability without eroding general conversational ability.
Problem
End-to-end spoken dialogue models (Mini-Omni, Moshi, LLaMA-Omni, SLAM-Omni, VocalNet) have advanced low-latency speech interaction but almost universally produce speech with neutral, homogeneous prosody. This stems from the training data: dialogue corpora built by simply applying TTS to text-based conversations (InstructS2S, VoiceAssistant400k) carry no authentic paralinguistic variation, so models learn what to say without learning how to say it. Existing controllable-TTS datasets (SpeechCraft, EmoVoice-DB) do carry stylistic variation but are single-turn, non-interactive corpora; forcing them into a dialogue format degenerates the interactive task into a read-aloud TTS task. No existing resource combines full dialogue context with multi-dimensional, instruction-level style control.
Method
UltraVoice is built with a four-stage synthetic pipeline. First, roughly 200,000 question-answer pairs are curated from UltraChat (filtered to remove URLs, citations, and long quoted text). Second, for six predefined stylistic dimensions (emotion, speed, volume, accent, language, and composite combinations of speed/volume/emotion), GPT-4o generates diverse natural-language style instructions and correspondingly stylized textual responses, with practical TTS-friendly rewrites (spelling out numbers, avoiding code-like text). Third, instruction audio is synthesized with speaker timbres randomly sampled from the SeedTTS test-set-en corpus (real-world background noise included) to simulate realistic user input, while response audio is synthesized with a single fixed timbre for consistency; GPT-4o-audio-preview handles most stylized responses, Edge TTS plus CosyVoice-300M-based voice conversion handles accent-specific responses (Edge TTS itself lacks custom timbre support), and a further 40,000 generic QA pairs from VoiceAssistant400k are resynthesized with CosyVoice-300M for balance. Fourth, all synthesized instruction/response audio is filtered with Whisper-large-v3 ASR: only samples with character error rate below 20% and duration under 30 seconds are retained.
The resulting corpus totals 100,770 dialogue samples (832.92 hours): 84,832 style-instructed samples across the six dimensions plus 15,938 unstyled general-QA samples. Emotion coverage spans seven categories (neutral, happy, sad, angry, surprised, fearful, disgusted); accent coverage spans six English varieties (Australian, Canadian, British, Indian, Singaporean, South African); language-switching covers Chinese, Japanese, and Korean.
The dataset is used to supervised-fine-tune (SFT) four existing end-to-end spoken dialogue models spanning two model families and two LLM backbones: SLAM-Omni-0.5B (Whisper-small-v3 encoder, Qwen2 backbone, CosyVoice1 decoder) and VocalNet-1B/7B/8B (Whisper-large-v3 encoder, LLaMA3.2/Qwen2.5/LLaMA3.1 backbones, CosyVoice2 decoder with multi-token prediction). No architectural modification is introduced; the contribution is entirely in the training data and evaluation protocol. Separately, five of UltraVoice’s stylistic dimensions (speed, volume, emotion, accent, composite) are repurposed as instruction-speech pairs to SFT a pre-trained EmoVoice-0.5B checkpoint (taken before its original EmoVoice-DB training), producing a controllable TTS model (UltraVoice-0.5B-SFT) to test whether the dataset generalizes beyond dialogue training.
Key Results
On a held-out 2,300-sample internal test set (100 examples per fine-grained sub-dimension, no train overlap), SFT improves instruction-following rate (IFR) by 14.61-40.09 percentage points across the four base models, with the largest relative gains on smaller, weaker-baseline models (SLAM-Omni-0.5B IFR: 28.30% → 68.39%). MOS (measured via an automatic audio-language-model judge, see caveat below) improves 29.12-42.33% in relative terms across models, e.g. VocalNet-7B overall MOS rises from 2.73 to 3.59 and VocalNet-8B from 2.85 to 3.68. On URO-Bench, which evaluates general dialogue understanding, reasoning, and conversation ability, all four SFT models improve over their base counterparts (+10.84% average on Basic, +7.87% on Pro), and VocalNet-7B-SFT surpasses strong existing baselines including Qwen2.5-Omni-7B and GLM-4-Voice-9B on several sub-metrics. On the controllable-TTS validation experiment, UltraVoice-0.5B-SFT reduces in-domain WER from 19.82 (EmoVoice-0.5B) to 3.97 while maintaining competitive emotion similarity, emotion recall, and UTMOS against PromptTTS, CosyVoice, CosyVoice2, and EmoVoice-0.5B on the out-of-domain EmoVoice-DB benchmark.
A notable exception cuts across both experiments: language-switching control is inconsistent. LLaMA-backbone models (VocalNet-1B, VocalNet-8B) show a slight MOS decline and near-zero IFR gain on the Language dimension, while Qwen-backbone models (SLAM-Omni-0.5B, VocalNet-7B) show clear gains, which the authors attribute to differing multilingual pretraining exposure in the LLM backbones rather than to the dataset itself.
Novelty Assessment
The contribution is a dataset and an accompanying evaluation protocol, not a new model architecture; all fine-tuned systems reuse existing spoken-dialogue architectures (SLAM-Omni, VocalNet) and an existing TTS pipeline (EmoVoice) unmodified. The genuine novelty is combining full dialogue context, turn-taking structure, and six-dimensional fine-grained style instructions in a single large-scale (830-hour) synthetic corpus, which prior spoken-dialogue datasets lacked entirely and prior controllable-TTS datasets could not provide in an interactive format. The empirical demonstration that the same data usefully transfers to both dialogue fine-tuning and standalone controllable TTS fine-tuning is a secondary, useful validation of the data’s quality rather than a new technical contribution.
Field Significance
High
This paper fills a clearly-scoped and previously unaddressed gap: no existing spoken-dialogue dataset paired full conversational context with instruction-level, multi-dimensional style control, and no controllable-TTS dataset offered a genuinely interactive dialogue structure. UltraVoice’s scale (830 hours, over 100,000 dialogue samples), its demonstrated transfer across four dialogue-model configurations and two LLM backbone families, and its extension into controllable TTS training make it a broadly reusable resource for future expressive spoken-dialogue and TTS work, with data, code, and checkpoints released publicly.
Claims
- supports: Training data lacking authentic paralinguistic variation constrains spoken dialogue models to homogeneous, expressively flat delivery independent of the underlying architecture.
Evidence: Fine-tuning four dialogue models (SLAM-Omni-0.5B, VocalNet-1B/7B/8B) on a style-annotated dialogue dataset raises instruction-following rate by 14.61-40.09 percentage points relative to the same base models trained only on unstyled dialogue data. (§4.2, Table 10)
- supports: Supervised fine-tuning on a style-annotated dialogue dataset can improve fine-grained expressive controllability without degrading general dialogue competence.
Evidence: All four SFT models show simultaneous gains on URO-Bench (Understanding, Reasoning, Oral Conversation; +10.84% average on Basic, +7.87% on Pro) and on stylistic MOS (+29.12% to +42.33% relative). (§4.3, Table 5)
- complicates: Fine-grained language-switching control transfers unevenly via supervised fine-tuning, depending on the multilingual pretraining exposure of the underlying LLM backbone.
Evidence: LLaMA-backbone models (VocalNet-1B, VocalNet-8B) show a slight MOS decline and near-zero IFR gain on the Language control dimension, while Qwen-backbone models (SLAM-Omni-0.5B, VocalNet-7B) show clear gains on the same dimension and dataset. (§4.2, “Cross-Metric Consistency and Limitations”)
- supports: Fine-grained, style-annotated speech data constructed for dialogue training also generalizes to standalone controllable TTS fine-tuning.
Evidence: Repurposing five of UltraVoice’s stylistic dimensions as instruction-speech pairs to fine-tune a pre-trained EmoVoice-0.5B checkpoint reduces in-domain WER from 19.82 to 3.97 while improving MOS and IFR across accent, speed, volume, emotion, and composite dimensions relative to the pre-trained baseline. (§4.4, Tables 6-7)
- complicates: Reported “MOS” scores from automatic audio-language-model judges substitute for, rather than replicate, human perceptual evaluation, and should be interpreted as automated-judge scores rather than listening-test outcomes.
Evidence: Both MOS and instruction-following rate are generated by prompting Gemini-2.5-Flash as an automatic evaluator on the audio output, justified only by a cited correlation between such judges and human ratings in prior work, with no human listening test reported in this paper. (§4.1, “Audio-Language Model (ALM) based Metric”)
Limitations and Open Questions
All reported "MOS" and instruction-following rate (IFR) scores in this paper are generated by an audio-language-model judge (Gemini-2.5-Flash), not by human raters. This is an automated proxy for subjective quality, not a genuine listening test, and should not be read as evidence of human-perceived naturalness or style fidelity on its own.
Further limitations acknowledged or evident in the paper:
- Language-switching control is unreliable and backbone-dependent, likely because the underlying LLM’s multilingual pretraining exposure, not the dataset, is the bottleneck; the authors note the dataset’s own multilingual diversity and volume are also limited (three languages, comparatively small sample counts).
- The dataset is single-turn: each dialogue sample is one instruction-response exchange, with no multi-turn evolution of style within a conversation. The one clear regression observed (SLAM-Omni-0.5B Pro Reasoning on URO-Bench, 24.72 → 20.07) is attributed to this single-turn focus limiting complex multi-turn reasoning transfer.
- The entire pipeline is fully synthetic: text is generated by GPT-4o and speech by TTS/VC systems (GPT-4o-audio-preview, Edge TTS, CosyVoice-300M), so the paralinguistic variation the dataset provides is itself model-generated rather than drawn from real human speakers, which could bound how much authentic human expressive nuance it can transmit to downstream models.
- Quality filtering relies solely on ASR-based CER and duration thresholds (Whisper-large-v3), which screens for transcription accuracy and length but does not directly verify that the intended style (e.g., correct emotion or accent) was actually rendered in the synthesized audio.
Wiki Connections
- Spoken Language Model — fine-tunes existing speech-LLM dialogue architectures (SLAM-Omni, VocalNet) that consume external user speech audio and generate spoken responses, evaluating them under this paradigm with a new style-controlled training corpus.
- Instruction-Conditioned TTS — repurposes UltraVoice’s style instructions as natural-language conditioning to fine-tune a controllable TTS model (UltraVoice-0.5B-SFT), extending instruction-conditioned synthesis into a dialogue-derived dataset.
- Emotion Synthesis — dedicates one of six control dimensions to emotion, with genuine emotion-specific metrics (emotion2vec similarity and recall) reported for the derived TTS model.
- Multilingual TTS — includes an explicit language-switching control dimension (Chinese, Japanese, Korean) with its own fine-tuning data and per-language MOS/IFR results.
- Prosody Control — provides explicit natural-language instruction control over speaking rate and volume as two of the six stylistic dimensions.
- SLAM-Omni — one of two base spoken dialogue model families fine-tuned on UltraVoice to evaluate style controllability gains.
- VocalNet — the other base spoken dialogue model family (three sizes, two LLM backbones) fine-tuned on UltraVoice.
- EmoVoice — its controllable-TTS pipeline and pre-trained checkpoint are reused to validate that UltraVoice data generalizes beyond dialogue training.
- CosyVoice — CosyVoice-300M is used both as a voice-conversion tool for accent-specific response synthesis and to resynthesize the VoiceAssistant400k-derived general-QA samples.
- CosyVoice 2 — serves as the speech decoder (with multi-token prediction) inside the VocalNet base models fine-tuned in this work.
- Seed-TTS — its SeedTTS test-set-en corpus supplies the diverse, real-world-noise speaker timbres sampled for instruction audio.
- GLM-4-Voice — used as a strong existing baseline on URO-Bench against which the fine-tuned VocalNet-7B model is compared.
- Mini-Omni — its VoiceAssistant400k corpus supplies the additional generic QA pairs resynthesized to balance UltraVoice’s unstyled samples.