arXiv · 2026 · Preprint
Dongchao Yang et al. · → Paper · Demo: ✓ · Code: ✓
Proposes a unified audio-text foundation model built on a factorized audio tokenizer that separates text-aligned reasoning tokens from acoustic reconstruction tokens, paired with a functionally-specialized autoregressive backbone, and trains it on 100B text and 60B audio tokens for speech, sound, and music understanding and generation.
Problem
Prior audio language models face a representation dilemma: continuous self-supervised features (HuBERT, Whisper encoders) preserve rich perceptual information but are difficult to model autoregressively, while discrete acoustic codecs (EnCodec, DAC-style tokenizers) support scalable generation but encode mostly low-level acoustic detail with little text-aligned, high-level abstraction useful for understanding. On the architecture side, most unified audio-text language models process text and audio tokens uniformly across all transformer layers, which the authors argue is suboptimal: lossy audio tokens propagated through every layer limit perceptual abstraction, and aligning text and audio throughout the whole network risks rapidly forgetting the pretrained LLM’s textual knowledge. UniAudio 2.0 targets both problems together, aiming to build an audio foundation model that generalizes to few-shot and zero-shot tasks the way large text-only LLMs do.
Method
The system has two components. First, ReasoningCodec factorizes an audio waveform into two token streams: reasoning tokens r, produced by a reasoning branch that fuses frozen Whisper and music-SSL encoder features through a query-based transformer (4 layers, learnable queries, interleaving factor K=5) compressed to a 5 Hz frame rate and quantized with 8-codebook RVQ; and reconstruction tokens s, produced by a reconstruction branch that extracts phone-level (WavLM), music-structure, and environmental-sound features from frozen multi-expert encoders and quantizes them with group-wise VQ (one layer for phone semantics, one for music structure, 6 residual layers for environmental/acoustic detail). The reconstruction branch is conditioned on the quantized reasoning tokens via FiLM modulation to reduce redundancy between the two streams, and waveforms are recovered only from the reconstruction tokens through a flow-based scalar latent diffusion decoder trained with a flow-matching objective plus a semantic feature-matching loss. The reasoning branch itself is trained in two stages: supervised fine-tuning on audio understanding tasks (ASR, captioning), followed by GRPO-style reinforcement learning using an LLaMA 3.1-8B-Instruct verifier as a reward model to encourage grounded, detailed analytical descriptions.
Second, UniAudio 2.0 packs text, reasoning, and reconstruction tokens into a 9-stream per-timestep representation (8 audio codebook streams plus one text stream) fed to a single autoregressive transformer initialized from LLaMA 3.2 3B. Rather than treating all layers uniformly, the backbone is partitioned into three functional blocks: lower layers act as audio-understanding experts (randomly initialized, audio-only computation), middle layers act as cross-modal experts (initialized from the pretrained LLM to preserve textual knowledge), and upper layers act as audio-generation experts (randomly initialized, audio-only) that condition a lightweight local decoder for frame-level parallel prediction of the K token streams, following the local-decoder strategy used in prior codec-LM systems. A masking mechanism restricts the understanding and generation experts to update only audio positions, leaving text positions untouched. Text tokens are predicted with a standard token-level cross-entropy loss; audio frames are predicted with a frame-level loss over the local decoder’s K parallel streams. Training proceeds in four stages: audio-understanding warm-up (with an auxiliary semantic-distillation objective reconstructing WavLM/music-SSL features), audio-generation warm-up, joint audio-text pretraining (context length 1024), and a mid-training stage that extends context to 2048 tokens and introduces “auditory sentences” — long, multi-segment sequences linking audio/text/caption spans that are used to construct diverse multi-task training signal without hand-designed task pipelines. The final model is trained on 100B text tokens and 60B audio tokens.
Key Results
ReasoningCodec’s reconstruction quality is reported against DAC-Codec, EnCodec, MimiCodec, Higgs-Audio, X-Codec, and ALMTokenizer under matched token-rate settings, with the strongest PESQ (2.36/2.93 across the two reported PESQ variants), STOI (0.85), UT-MOS (3.91) and SPK-SIM (0.97) among discrete tokenizers on speech, and the best subjective MUSHRA scores at matched token rate (speech 90.5, sound 80.8, music 86.6) against MimiCodec, X-Codec, Higgs-Audio, EnCodec, and DAC (Table 1, Table 4). On downstream LLM-based audio understanding (ASR/emotion/audio/music classification), the combined reasoning+reconstruction tokenizer scores close to the fully continuous Whisper baseline and clearly ahead of purely acoustic tokenizers such as DAC (Table 3). For the full UniAudio 2.0 model, the paper reports ASR and TTS results competitive with or ahead of MiMo-Audio-7B-Instruct and Qwen2.5-Omni-7B while using only 3B parameters (e.g., LibriSpeech test-clean ASR WER 2.7 vs. MiMo-Audio’s 3.5; TTS WER on the LS-clean condition 3.46 vs. MiMo-Audio’s 4.74) (Table 5). On few-shot tasks (speech denoising, voice conversion, emotion and sound classification) UniAudio 2.0 is generally strongest in the 1-shot setting against MiMo-Audio and UniAudio 1.5, though 2-shot gains are mixed and MiMo-Audio wins some 2-shot cells (Table 6). On zero-shot tasks — MMLU, speech-to-speech/text instruction following (InstructS2S-Eval), dysarthric speech recognition, and two newly designed tasks (audio-prompt/caption-guided TTS and speech-plus-sound generation) — the model retains most of its base LLM’s MMLU accuracy (44.10 vs. 47.63 for LLaMA 3.2 3B) and reports a large WER improvement over Qwen2.5-Omni on dysarthric speech recognition (19.4 vs. 80.6). On the InstructS2S-Eval GPT-score comparison it is mixed: UniAudio 2.0 beats SpeechGPT on speech-to-text instruction following (3.66 vs. 2.98) but trails LLaMA-Omni on both the speech-to-speech (2.16 vs. 3.47) and speech-to-text (3.66 vs. 3.99) scores (Table 7).
Novelty Assessment
The core contribution is architectural: factorizing a discrete audio codec into a text-aligned reasoning stream and an acoustic reconstruction stream, coupled via FiLM, is a genuinely new codec design distinct from prior semantic/acoustic split tokenizers, and the functional layer-specialization of the autoregressive backbone (understanding/cross-modal/generation experts operating on disjoint layer ranges with audio-only masked updates) is a structural departure from the uniform-layer designs used by most prior unified audio-text LMs. The multi-stage training recipe and “auditory sentence” task-construction strategy are comparatively more of an engineering/data contribution layered on top of the architectural core. Baseline comparisons for the full system rely heavily on numbers reported in cited papers’ own tables (MiMo-Audio, Qwen2.5-Omni) rather than uniformly re-run comparisons, and the paper is explicit that its compared-model set is not exhaustive; some results are mixed rather than uniformly favorable — in the 2-shot voice conversion setting, for instance, MiMo-Audio reports better WER and SPK-SIM than UniAudio 2.0, a result the paper reports alongside its stronger 1-shot numbers rather than omitting.
Field Significance
Tip
High — this paper is the most substantial architectural update to the UniAudio line, replacing the original’s largely acoustic-codec-based representation with an explicit reasoning/reconstruction token factorization and replacing a uniform transformer backbone with layer-specialized experts. It demonstrates that a 3B-parameter unified model can match or exceed contemporary 7B unified audio-language models on several in-domain speech tasks, and it introduces concrete new zero-shot evaluation tasks (dysarthric speech recognition, audio-prompt-and-caption-guided TTS, speech-plus-sound generation) that other unified audio-LM work can adopt as benchmarks.
Claims
- supports: Explicitly factorizing a discrete audio codec into a text-aligned high-level stream and a separate low-level acoustic-reconstruction stream can close much of the gap between discrete tokenizers and continuous SSL representations on LLM-based audio understanding tasks.
Evidence: On ASR/emotion/audio/music classification, the combined reasoning+reconstruction tokenizer scores close to the fully continuous Whisper baseline (e.g., ASR 9.0 vs. Whisper’s 8.5) and well ahead of purely acoustic tokenizers such as DAC (ASR error 93.2) (§5.3, Table 3).
- supports: Partitioning a unified autoregressive backbone into layer ranges specialized for audio understanding, cross-modal alignment, and audio generation, rather than processing text and audio uniformly across all layers, improves downstream task performance without added inference-time infrastructure complexity.
Evidence: Ablating the specialized experts and training only the cross-modal block under Stage 3 causes a significant performance drop across MMLU, ASR, TTS, audio/music generation, and song generation benchmarks (§5.7, Table 8).
- complicates: Reasoning-token-style semantic abstraction that improves understanding and generation-token prediction accuracy does not straightforwardly transfer to solving arbitrary unseen audio tasks; task coverage in zero-shot settings remains bounded by what the multi-task training data implicitly supports.
Evidence: The model cannot perform speech diarization in zero-shot settings, which the authors attribute to the absence of diarization- or duration-related supervision during training, despite strong zero-shot results on other newly designed tasks (§Appendix C, Limitation).
- complicates: Flow-based diffusion decoders used to recover high-fidelity waveforms from discrete semantic/acoustic tokens introduce a multi-step decoding cost that trades off against inference latency.
Evidence: The paper states the multi-step decoding procedure in the flow-based decoder increases inference latency for generation, and flags exploring few-step (e.g., two-step) decoding as necessary future work (§Appendix C, Limitation).
Limitations and Open Questions
Warning
Due to GPU resource constraints, the paper only trains and ablates 1B and 3B backbone variants; no scaling-law analysis is reported, so it is unclear how the architecture’s advantages hold or change at 7B+ scale where several of its comparison baselines (MiMo-Audio-7B, Qwen2.5-Omni-7B) already operate.
Beyond scaling, the authors report several other open issues in their own limitations section: zero-shot task coverage is bounded by training-data composition (the model cannot perform speech diarization, a task absent from its supervision); sound and music performance lags speech performance due to relatively less sound/music training data; the work focuses on pretraining design choices (tokenizer, unified architecture) and does not explore post-training strategies such as multi-task SFT or RL beyond the codec’s own GRPO stage; and the set of compared baseline systems is acknowledged as non-exhaustive, since many related unified models are not publicly available for direct comparison.
Wiki Connections
- Neural Audio Codec — introduces ReasoningCodec, a discrete codec that explicitly factorizes reasoning and reconstruction token streams rather than relying on a single semantic-or-acoustic hierarchy.
- Spoken Language Model — builds a unified autoregressive LLM that consumes external speech (via reasoning tokens) for understanding tasks and is evaluated on zero-shot spoken dialogue (InstructS2S-Eval).
- Speech-to-Speech — evaluates zero-shot speech-to-speech and speech-to-text instruction following against dedicated dialogue systems as a held-out generalization test.
- Instruction-Conditioned TTS — supports instructed TTS (style-following via text instructions) and a new audio-prompt-and-caption-guided TTS task that combines timbre transfer with natural-language style control.
- Zero-Shot TTS — the audio-prompt-guided TTS task uses a short reference clip to control timbre at inference without task-specific fine-tuning.
- Subjective Evaluation — reports MUSHRA listening tests comparing ReasoningCodec against six prior codecs across speech, sound, and music reconstruction.
- Voice Conversion — evaluates few-shot voice conversion on VCTK using WER, speaker similarity (SIM), and DNS-MOS, following the MiMo-Audio protocol.
- Singing Voice Synthesis — supports lyric-to-song generation, evaluated against the SongGen benchmark with WER and AudioBox aesthetics scores (CE/CU/PQ).
- UniAudio — direct predecessor; UniAudio 2.0 replaces its largely acoustic-codec-based tokenization and uniform transformer backbone with the reasoning/reconstruction factorization and layer-specialized architecture described here.
- ALMTokenizer — the query-based compression and quantization scheme used in ReasoningCodec’s reasoning branch, and the query-based tokenizer itself, serve as a direct methodological basis and reconstruction-quality baseline.
- Moshi — MimiCodec is used as a reconstruction-quality baseline, and the local audio decoder strategy for frame-level parallel token prediction follows the approach introduced in Moshi.
- MiMo-Audio — the primary contemporary unified audio-language model baseline compared across seen-task, few-shot, and zero-shot evaluations.
- Qwen2.5-Omni Technical Report — a second primary baseline for ASR, TTS, captioning, and zero-shot dysarthric speech recognition comparisons.
- Kimi-Audio Technical Report — cited as a contemporary representative of the unified discrete-sequence audio-language-model paradigm this paper also pursues.
- LLaMA-Omni — used as a zero-shot speech-to-speech/speech-to-text baseline on the InstructS2S-Eval benchmark.
- SpeechGPT — used as a second zero-shot speech dialogue baseline alongside LLaMA-Omni.