arXiv · 2025 · Preprint
Wu et al. (StepFun) · → Paper · Demo: ? · Code: ✓
Step-Audio 2 is an end-to-end large audio language model that unifies speech understanding and generation through interleaved discrete text and audio token modeling, augmented with retrieval-augmented generation and tool-calling capabilities including a novel audio search mechanism for timbre switching.
Problem
Large audio language models face two persistent gaps: they either handle audio understanding without expressive speech generation, or they generate speech without adequately capturing and responding to paralinguistic signals such as emotion, intonation, and speaking style. Existing systems also suffer from hallucination when responding to factual queries, and offer limited control over output voice characteristics beyond a fixed speaker embedding. Cascaded ASR-LLM-TTS pipelines address some of these gaps but introduce latency and modular mismatch. End-to-end approaches like the original Step-Audio addressed unification but did not fully integrate audio token generation into the language modeling core.
Method
Step-Audio 2 is a multi-modal LLM that processes raw audio as input and generates interleaved sequences of discrete text and audio tokens. The architecture comprises four components: a frozen audio encoder pretrained on ASR and audio classification tasks (25 Hz output, downsampled to 12.5 Hz by a lightweight adaptor), an LLM decoder, and an audio detokenizer.
The audio encoder connects to the LLM via a trainable adaptor that bridges the audio feature space to the text embedding space. The LLM’s vocabulary is extended with 6.6K audio tokens from the CosyVoice 2 tokenizer, and text and audio tokens are interleaved at a fixed ratio in the output sequence. The audio detokenizer uses a flow-matching module to generate mel spectrograms from output audio tokens, followed by HiFi-GAN for waveform synthesis. The flow-matching component incorporates a CNN-based encoder layer after each self-attention module, improving mel spectrogram reconstruction and pronunciation accuracy.
A key capability addition is tool calling, including a novel audio search tool that maintains a library of hundreds of thousands of speech samples with descriptions. During inference, the retrieved speech is appended to the input context, enabling timbre switching and speaking style mimicry based on voice retrieval. Other tools cover web search, weather, and date/time retrieval.
Training proceeds in four stages: adaptor alignment on 100B ASR tokens with frozen encoder and LLM; vocabulary expansion and joint training on 256B mixed tokens; main pre-training on 800B tokens across ASR, TTS, translation, and speech-to-speech conversation data; and a cooldown phase on 200B high-quality tokens. Supervised fine-tuning follows on 4B tokens, succeeded by multi-stage reinforcement learning using proximal policy optimization (PPO) and group relative policy optimization (GRPO) to improve reasoning efficiency and audio perceptual ability. The total training corpus spans 1.356T tokens from text and 8 million hours of speech and audio.
Key Results
On English ASR, Step-Audio 2 achieves an average WER of 3.14% across LibriSpeech, FLEURS, and Common Voice, outperforming both Kimi-Audio (4.18%) and Qwen-Omni (5.35%) and competitive with GPT-4o Transcribe (4.5%) (Table 1). On Chinese ASR, the average CER of 3.08% leads all baselines including Doubao LLM ASR (3.81%) and Kimi-Audio (3.75%). For low-resource dialects and accented Mandarin, Step-Audio 2 achieves 8.85% average CER versus 25.52% for Kimi-Audio and 40.49% for GPT-4o Transcribe on in-house test sets.
On the MMAU audio understanding benchmark, Step-Audio 2 scores 78.0 average, leading all compared systems including Omni-R1 (77.0), Qwen2.5-Omni (71.5), and Gemini 2.5 Pro (71.6) (Table 3). On the authors’ StepEval-Audio-Paralinguistic benchmark evaluating 11 paralinguistic dimensions, Step-Audio 2 achieves 83.09% average accuracy compared to 49.64% for Kimi-Audio and 44.18% for Qwen-Omni (Table 2).
For speech-to-text translation on CoVoST 2, Step-Audio 2 achieves 39.26 BLEU (average of English-Chinese and Chinese-English), ahead of Qwen2.5-Omni (35.40) and GPT-4o Audio (29.61). On speech-to-speech translation via CVSS, it achieves 30.87 BLEU average versus 23.68 for GPT-4o Audio (Table 4).
On URO-Bench speech-to-speech conversation, Step-Audio 2 leads the Chinese track with 83.32 (basic) and 68.25 (pro), significantly above GPT-4o Audio (78.59 / 67.10). In English, it scores 83.90 (basic) and 66.07 (pro), just below GPT-4o Audio (84.54 / 67.51) (Table 6). An open-source mini variant initialized from Qwen2.5-7B achieves competitive results within a few points of the full model across all benchmarks.
StepEval-Audio-Paralinguistic and StepEval-Audio-Toolcall are benchmarks introduced by the authors. Results on these proprietary benchmarks should be interpreted with caution pending independent evaluation.
Novelty Assessment
The contribution is primarily engineering integration at scale. The core architectural choices (frozen pretrained audio encoder, adaptor, LLM decoder with extended vocabulary, flow-matching detokenizer) are established components assembled from prior work including CosyVoice 2, HiFi-GAN, and earlier Step-Audio systems. The multi-stage training pipeline (alignment, joint pre-training, SFT, RL) follows patterns established by Qwen2-Audio, Moshi, and similar systems.
The genuinely novel elements are: (1) the audio search tool, which enables voice-query-driven timbre switching and style retrieval at inference time using a library of hundreds of thousands of samples; (2) the two new benchmarks (StepEval-Audio-Paralinguistic covering 11 paralinguistic dimensions, StepEval-Audio-Toolcall for tool invocation in speech), which provide structured evaluation criteria where none previously existed. The integration of CoT reasoning and RL into a speech-generative LALM is novel in its combination but builds on GRPO/PPO methods from text LLM work.
The scale of training data (8M hours, 1.356T tokens) and the comprehensiveness of evaluation across ASR, audio understanding, translation, tool calling, and conversation is a distinguishing feature of this report relative to most academic work, though the underlying components are not architecturally new.
Field Significance
Tip
High (engineering-integration + evaluation-contribution) — Step-Audio 2 demonstrates that the LALM paradigm can simultaneously achieve strong speech understanding, expressive speech generation, and tool-augmented factual grounding within a single end-to-end system, advancing the practical viability of spoken conversational agents. The two new benchmarks (StepEval-Audio-Paralinguistic, StepEval-Audio-Toolcall) provide evaluation infrastructure for capabilities that have been difficult to measure systematically, particularly fine-grained paralinguistic comprehension and voice-triggered tool invocation.
Claims
- Interleaving discrete audio tokens with text tokens in a shared language model vocabulary enables end-to-end spoken dialogue systems that respond coherently to paralinguistic cues without requiring separate speech synthesis pipelines. (§3.1, §4.6)
- Reinforcement learning applied to audio language models can improve reasoning efficiency in complex acoustic scenarios while preserving generation quality. (§3.4)
- Retrieval-augmented generation and external tool calling can substantially reduce hallucination and expand capability (timbre switching, web-grounded responses) in large audio language models without architectural redesign. (§3.1, §4.5)
- Comprehensive multi-task pre-training across ASR, TTS, translation, and conversation significantly improves spoken dialogue performance in low-resource languages and accented speech. (§3.2, §4.1)
- Existing audio language model benchmarks fail to capture fine-grained paralinguistic comprehension and voice-triggered tool invocation, leaving important capability dimensions systematically unmeasured. (§4.2, §4.5)
Limitations and Open Questions
Warning
Two of the main evaluation benchmarks (StepEval-Audio-Paralinguistic, StepEval-Audio-Toolcall) are introduced by the authors themselves and have not been validated independently. Results on these benchmarks may overstate absolute capability levels even if relative comparisons are informative.
Model size is not reported, which makes parameter-count comparisons with other open-source systems (Kimi-Audio, Qwen2.5-Omni) difficult to interpret fairly. The open-source mini variant uses Qwen2.5-7B as its backbone, but the full model’s parameter count remains undisclosed.
The audio search tool relies on a proprietary library of hundreds of thousands of speech samples, limiting reproducibility of this capability. Latency characteristics are not reported; real-time performance is claimed via VAD and deployment infrastructure inherited from Step-Audio but not benchmarked here.
Evaluation is primarily Chinese-English bilingual. Performance on other language families, especially those with minimal pre-training coverage, is largely uncharacterized. The tool-calling evaluation uses synthesized speech conversations generated from text scripts rather than natural speech, which may not reflect real-world tool invocation patterns.
Wiki Connections
This paper builds directly on 2502.11946 (Step-Audio) and its predecessor for unified audio understanding and generation.
Core system components draw on 2412.10117 (CosyVoice 2) for the audio tokenizer and on 2010.05646 (HiFi-GAN) for the vocoder. The flow-matching detokenizer follows patterns from 2407.05407 (CosyVoice).
Contemporary spoken conversational agent systems evaluated as baselines include 2410.00037 (Moshi), 2409.06666 (LLaMA-Omni), 2411.00774 (Freeze-Omni), 2408.16725 (Mini-Omni), 2503.20215 (Qwen2.5-Omni), and 2504.18425 (Kimi-Audio).
The URO-Bench evaluation benchmark comes from 2502.17810. Related LALM precedents include 2306.12925 (AudioPaLM), 2402.05755 (SpiritLM), 2501.06282 (MinMo), and 2305.11000 (SpeechGPT). Neural codec references include 2210.13438 (EnCodec), 2308.16692 (SpeechTokenizer), and 2409.05377 (BigCodec).
Concept pages: spoken-language-model, speech-to-speech, neural-codec, autoregressive-codec-tts, evaluation-metrics, self-supervised-speech
2301.02111 — Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers 2305.07243 — Better speech synthesis through scaling 2406.02430 — Seed-TTS: A Family of High-Quality Versatile Speech Generation Models 2409.00750 — MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer 2503.01710 — Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens