arXiv · 2024 · Preprint
Wenxi Chen et al. (SJTU / Microsoft) · → Paper · Demo: ? · Code: ?
SLAM-Omni is a spoken conversational agent that combines speaker-decoupled semantic tokens with grouped token prediction and text-only history compression to achieve zero-shot timbre control and competitive spoken dialogue quality using only 60 GPU-hours of single-stage training.
Problem
Prior spoken dialogue models (SDMs) suffer from two compounding problems. First, they require multi-stage training pipelines (modality alignment pre-training on ASR or TTS, followed by dialogue fine-tuning), which is computationally expensive and can erode the original LLM’s instruction-following ability. Second, existing systems generate responses in a fixed system voice, with no mechanism to give users control over the speaker timbre of the system’s output. The frequency mismatch between text tokens (~3 Hz) and semantic audio tokens (50 Hz) further inflates sequence lengths, increasing training costs and inference latency.
Method
SLAM-Omni is built on Qwen2-0.5B and uses Whisper-small as the speech encoder. User speech is encoded at 50 Hz, downsampled by concatenating every 5 consecutive frames, and projected into the LLM’s embedding space. On the output side, the LLM predicts text tokens and CosyVoice supervised semantic audio tokens in parallel, sharing a single expanded vocabulary.
To address the token frequency mismatch, the paper introduces Semantic Group Modeling: rather than predicting one semantic token per LLM step, the model predicts G tokens simultaneously using a lightweight linear projection layer over grouped audio logits. With G = 3, the audio sequence length is reduced by 3x, enabling the model to converge with less than half the GPU hours of the G = 1 baseline while achieving substantially better speech-text alignment (4.54% ASR-WER vs. 18.23% at G = 1).
Timbre control is decoupled from the LLM entirely. Because the semantic tokens encode only linguistic content (not speaker identity), a conditional flow-matching vocoder conditioned on a speaker audio prompt converts semantic tokens to mel spectrograms, which HiFi-GAN then synthesizes to waveform. Users can provide any reference audio clip to select the system’s output voice at inference time.
For multi-turn dialogue, Historical Text Prompting replaces the typical approach of maintaining interleaved audio-text token histories. Instead, only text transcriptions of past dialogue turns are stored in the LLM context, with KV-cache reuse across rounds. This drastically reduces context length and better leverages the LLM’s text-based in-context learning capabilities.
The entire system is trained in a single stage on dialogue data (primarily VoiceAssistant-400K, ~664 hours of instruction audio), with full fine-tuning of all components except the Whisper encoder, which is kept frozen. Training completes in approximately 15 hours on 4 NVIDIA A100 GPUs.
Key Results
SLAM-Omni is evaluated against Mini-Omni, Mini-Omni2, LLaMA-Omni, Freeze-Omni, and GLM-4-Voice using a custom benchmark spanning 8 test sets across three dimensions: understanding, reasoning, and oral conversation. All systems are evaluated on speech-to-speech capability directly (not text-only output).
At 0.5B scale, SLAM-Omni substantially outperforms Mini-Omni and Mini-Omni2 on ChatGPT-based content scoring (39.32 vs. 22.58 and 26.56 overall), achieves the highest UTMOS across all SDMs including the larger 7-9B models (4.45 vs. 4.37 for Freeze-Omni), and the lowest ASR-WER (4.54% vs. 6.05% for Mini-Omni and 10.42% for LLaMA-Omni). The UTMOS and WER advantages hold in multi-turn evaluation as well.
Ablations confirm that G = 3 is a strong operating point: G = 4 gives marginal WER improvement (4.31%) at lower content quality (37.19 ChatGPT score), and G = 5 degrades on all dimensions. The pre-training ablation is notable: adding an ASR or TTS pre-training stage before the dialogue fine-tune hurts ChatGPT content scores by 5 and 12 points respectively while providing no meaningful gain in WER, suggesting the pre-training disrupts instruction-following.
In multilingual evaluation (Chinese), SLAM-Omni achieves the best CER (4.4%) and UTMOS (3.67) among comparable SDMs while trailing larger models on semantic quality.
Novelty Assessment
The three core contributions are each modest but together form a coherent system design. Grouped token prediction for parallel audio-text generation is an extension of the technique used in VALL-E 2 (Chen et al., 2024a) and is straightforward to implement. Historical text prompting is a sensible compression heuristic, essentially trading non-verbal dialogue continuity for context efficiency. Zero-shot timbre control for SDMs is the most genuinely novel element: transferring the speaker-decoupled semantic token paradigm (from CosyVoice) into a dialogue setting had not been demonstrated before.
The single-stage training claim requires nuance. The claim is that multi-stage pipelines are unnecessary and harmful — supported by the ablation results (Table 6). This is a useful negative result for the field, though the baseline against which it is tested (VoiceAssistant-400K only, 0.5B backbone) is modest. It is unclear whether the result generalises to larger models or more diverse training corpora.
The evaluation framework proposed (8 test sets across understanding, reasoning, and oral conversation) is a genuine contribution to SDM benchmarking, addressing a gap identified by the authors.
Field Significance
Moderate: SLAM-Omni addresses practical pain points in spoken conversational agent training (training efficiency, timbre monotony, context compression) and demonstrates that single-stage fine-tuning on dialogue data alone can match or exceed multi-stage pipelines at small scale. The zero-shot timbre control mechanism (decoupling speaker identity into the vocoder rather than the LLM) provides a reusable design pattern for spoken conversational agents seeking controllable output voices without additional LLM parameters. The finding that ASR/TTS pre-training degrades rather than improves instruction-following is a concrete data point that challenges common practice in the field.
Claims
- Decoupling speaker identity from semantic content in spoken dialogue systems enables zero-shot timbre control without modifying the language model or adding speaker-conditioning layers. (§3.4)
- Grouping audio tokens to reduce the frequency mismatch between text and speech representations substantially improves speech-text alignment and training efficiency in parallel audio-text dialogue models. (§3.3, Table 5)
- Single-stage fine-tuning on dialogue data can match or outperform multi-stage pipelines that include ASR or TTS pre-training, because modality-specific pre-training degrades instruction-following and general knowledge retention. (§5.3.2, Table 6)
- Replacing audio-token history with text-only history in multi-turn spoken dialogue models improves the system’s ability to handle longer conversation contexts while leveraging pre-trained LLM in-context learning. (§3.5)
- Current spoken dialogue models consistently underperform text-only LLMs of similar scale on semantic content quality, even after dialogue fine-tuning. (§5.1, Table 3)
Limitations and Open Questions
Warning
Historical text prompting discards all non-verbal information from prior dialogue turns (prosody, emotion, paralinguistic cues). In scenarios where voice-level context matters for dialogue coherence, this strategy may reduce response quality in ways not captured by the text-based evaluation metrics used.
The system is evaluated exclusively at 0.5B scale. The single-stage training advantage may not hold for larger LLMs, where the data volume required for joint audio-text modeling grows substantially.
Training data relies entirely on TTS-synthesized dialogue utterances rather than real recorded speech. Whether the system’s timbre control and speech quality generalise to natural in-the-wild audio inputs at inference time is not tested.
The evaluation benchmark, while more comprehensive than prior SDM evaluations, is still largely built on synthetic TTS-generated test inputs. Evaluation on real end-to-end voice interactions would strengthen the claims.
Wiki Connections
- spoken-language-model — spoken conversational agent with zero-shot timbre control and single-stage training
- speech-to-speech — full speech-in speech-out pipeline; Whisper encoder input, CosyVoice + HiFi-GAN output
- autoregressive-codec-tts — AR generation of CosyVoice supervised semantic tokens from text context
- neural-codec — CosyVoice supervised semantic tokens used as discrete audio representation
- zero-shot-tts — timbre control at inference from a reference audio clip without speaker-specific training
- evaluation-metrics — proposes an 8-dataset SDM benchmark spanning understanding, reasoning, and oral conversation
- 2407.05407 (CosyVoice) — source of supervised semantic tokens and the timbre-decoupled vocoder architecture
- 2410.00037 (Moshi) — concurrent end-to-end dialogue model; contrasted on duplex design
- 2409.06666 (LLaMA-Omni) — SDM baseline at comparable scale
- 2411.00774 (Freeze-Omni) — concurrent SDM baseline with highest UTMOS among 7-9B models
- 2408.16725 (Mini-Omni) — primary same-scale (0.5B) comparison
- 2410.11190 (Mini-Omni2) — same-scale multimodal SDM comparison
- 2412.02612 (GLM-4-Voice) — larger SDM baseline
- 2406.05370 (VALL-E 2) — source of grouped token prediction technique
- 2301.02111 (VALL-E) — foundational codec LM establishing discrete token generation
- 2411.13577 (WavChat) — concurrent survey contextualising SDM landscape