arXiv · 2025 · Preprint
Huang et al. (StepFun) · → Paper · Demo: ? · Code: ✓
Step-Audio is a 130B-parameter unified speech-text model from StepFun that integrates understanding and generation in a single system, combining a novel dual-codebook speech tokenizer with a hybrid AQTA+TTS architecture and RLHF-trained chat model, released as an open-source spoken conversational agent with instruction-following control over dialect, emotion, singing, and tool-calling.
Problem
End-to-end spoken conversational agents faced three compounding problems in early 2025. Cascaded ASR-LLM-TTS pipelines introduced latency and error propagation across modality boundaries. Open-source alternatives to proprietary systems like GPT-4o lacked the data infrastructure to support controllable speech generation across dialects, emotions, and singing styles. And the standard vocabulary of speech tokenizers forced a trade-off: semantic tokens preserved coherence but degraded acoustic quality, while linguistic tokens preserved quality but impaired semantic understanding. No open-source system had demonstrated production-ready integration of all these capabilities in one model.
Method
Step-Audio’s architecture centers on three components: a dual-codebook speech tokenizer, a 130B-parameter LLM backbone, and a 3B-parameter hybrid speech decoder. The tokenizer interleaves linguistic tokens (from a Paraformer encoder at 16.7 Hz, codebook size 1024) and semantic tokens (from a CosyVoice tokenizer at 25 Hz, codebook size 4096) in a 2:3 temporal ratio. This interleaving is the core tokenizer innovation: semantic tokens maintain semantic coherence in generated continuations, while linguistic tokens preserve the acoustic quality needed for intelligible reconstruction. The dual-codebook approach reduced CER on ASR tasks from 25.5 to 18.4 in ablation experiments on a 3B-scale model.
The LLM backbone (Step-1, a 130B text model) is adapted via three-stage continual pretraining on 3.3T tokens of multi-modal data spanning audio continuation, TTS synthesis, ASR, and audio-text alternating sequences. Rather than training a unified speech-in/speech-out model, the system adopts an AQTA (audio-in, text-out) design for the LLM, then routes text output to a separate TTS speech decoder. This choice reflects both the scarcity of high-quality end-to-end dialogue audio and the desire for controllable output timbre via a parametric TTS stage.
The speech decoder is a 3B-parameter autoregressive LM followed by a flow-matching acoustic model and a mel-to-waveform neural vocoder. It receives text or audio tokens and generates dual-codebook representations, then reconstructs waveforms through the flow-matching stage. The same decoder architecture underlies the distilled standalone Step-Audio-TTS-3B model, which is trained on synthetic data produced by the 130B model rather than manually curated recordings.
Instruction-driven control is implemented through tagged SFT format: descriptive tags govern language, dialect, and vocal style (Japanese, Korean, Cantonese, Sichuan dialect, RAP, singing), while comparative tags provide hierarchical control over emotion and speaking rate at five intensity levels. The AQTA system is post-trained with PPO-based RLHF using a two-stage reward model, where a known failure mode (“deaf hacking”) was identified and mitigated by constructing explicit counter-examples in the reward training data.
Real-time inference uses speculative response generation triggered by voice activity detection pauses. The system preemptively generates responses during user pauses; roughly 40% of speculative responses are committed, reducing latency by approximately 500 ms per response. Conversation history is stored as text transcriptions rather than audio tokens (text-to-audio token ratio approximately 1:14), enabling longer context windows without proportional memory cost.
Key Results
On the SEED-TTS benchmark for TTS quality, Step-Audio-TTS-3B achieves 1.31% CER (Chinese) and 2.31% WER (English), outperforming CosyVoice, MaskGCT, and FireRedTTS on intelligibility while remaining competitive on speaker similarity. The 130B version improves further to 1.17% CER and 2.00% WER, demonstrating a clear scaling effect. Speaker similarity scores (0.733 for Chinese, 0.660 for English) are somewhat lower than CosyVoice 2 (0.806/0.736), reflecting a trade-off between intelligibility scaling and acoustic identity preservation.
On the spoken conversational benchmarks, Step-Audio-Chat scores 81% on LLaMA Question, 75.1% on Web Questions, and 58% on TriviaQA, outperforming GLM-4-Voice, Moshi, Freeze-Omni, LUCY, MinMo, and Qwen2-Audio. On the proprietary StepEval-Audio-360 benchmark it achieves factuality of 66.4% and chat score of 4.11 (vs. GLM-4-Voice’s 3.49 and Qwen2-Audio’s 2.27).
On instruction following for voice control (dialect, emotion, singing/RAP), Step-Audio-Chat scores 4.4 MOS on voice control IF versus GLM-4-Voice’s 3.6. Audio quality for singing and RAP reaches 4.0 MOS versus GLM-4-Voice’s 2.4 on the same task.
ASR capability of the pretrained model (Step-Audio Pretrain) averages 4.64% CER/WER across six test sets, comparable to Qwen2-Audio (4.32%) and substantially better than Whisper Large-v3 (7.28%) among discrete-token models, while the chat model degrades slightly to 5.89%.
Novelty Assessment
Step-Audio’s primary contribution is engineering integration at scale: it combines a dual-codebook tokenizer, a large LLM backbone, a flow-matching TTS decoder, RLHF post-training, and speculative streaming inference into a single open-source release. The dual-codebook interleaving approach is genuinely novel at this scale and addresses a real limitation of prior systems (semantic-only or linguistic-only tokenization). The “deaf hacking” failure mode discovered during RLHF training and its proposed mitigation is a useful empirical finding for the field.
The AQTA+TTS architecture (as opposed to fully end-to-end speech generation) is a practical but conservative design choice shared with several contemporaries (LLaMA-Omni, MinMo, LUCY). The claim that this is “the first production-ready open-source solution” is a marketing framing rather than a technical claim; Moshi and GLM-4-Voice also target production-grade quality. What Step-Audio uniquely offers is the breadth of controllable generation (dialects, emotions, singing, RAP, tool-calling) in a single open-source release backed by a large synthetic data generation pipeline.
Field Significance
Tip
High, with caveats. Step-Audio demonstrates that a single large model can credibly handle ASR, TTS, spoken dialogue, singing, RAP, and tool-calling through unified multi-modal pretraining. Its open-source release at 130B scale, with a distilled 3B TTS model and the synthetic data engine that produced it, provides a strong practical reference point for the field. The “deaf hacking” RLHF failure mode and its mitigation also contribute a reusable empirical insight for teams training reward models on speech interaction data.
Claims
- A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality. (§4.4, §6.2.1)
- Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive. (§6.2.2, Table 3)
- RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training. (§5.2.6)
- Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems. (§3.4)
- Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles. (§5.1.1)
Limitations and Open Questions
Warning
The StepEval-Audio-360 benchmark is proprietary and created by the same team; human evaluation results on it cannot be independently reproduced. Open-source benchmark comparisons mix locally re-run models with results taken from original publications, complicating direct numerical comparison.
Speaker similarity scores for the distilled Step-Audio-TTS-3B are noticeably lower than CosyVoice 2 on both Chinese and English SEED-TTS tests, suggesting that the dual-codebook approach trades some acoustic identity preservation for intelligibility gains. The AQTA+TTS design still relies on a cascade: errors in ASR transcription of history or in text generation propagate to the TTS stage. The paper’s future work section acknowledges that purely end-to-end audio-in/audio-out (AQAA) remains unsolved. Evaluation for singing, RAP, and dialect control is limited to instruction following scores without reference audio; absolute quality in these dimensions is difficult to assess from the reported numbers alone.
Wiki Connections
Core spoken conversational agent references: 2410.00037 (Moshi), 2409.06666 (LLaMA-Omni), 2411.00774 (Freeze-Omni), 2408.16725 (Mini-Omni), 2412.02612 (GLM-4-Voice).
TTS and codec foundations: 2407.05407 (CosyVoice), 2412.10117 (CosyVoice 2), 2406.02430 (Seed-TTS), 2407.10759 (Qwen2-Audio).
Broader model context: 2407.21783 (LLaMA 3), 2410.21276 (GPT-4o).
Concept pages: spoken-language-model, autoregressive-codec-tts, neural-codec, instruction-conditioned-tts, streaming-tts, rlhf-speech.
2507.16632 — Step-Audio 2 Technical Report