arXiv · 2025 · Preprint
Tongyi Fun Team (Alibaba Group) · → Paper · Demo: ✓ · Code: ✓
A large audio language model that processes speech at an efficient 5Hz frame rate in its shared LLM backbone while generating high-quality 25Hz speech tokens through a dedicated refinement head, using a two-stage cocktail fine-tuning strategy and multi-task DPO post-training to retain text-LLM knowledge while achieving top-tier results among similarly-scaled models on spoken question answering, audio understanding, function calling, instruction-following, and voice empathy benchmarks.
Problem
Joint speech-text models that let an LLM directly comprehend and generate speech face three interlocking problems: a temporal resolution mismatch between speech tokens (typically 25Hz) and text tokens (roughly 3Hz) dilutes semantic information and prevents the LLM from fully exploiting its reasoning capabilities; continual pre-training or post-training a text-LLM backbone into a multimodal model tends to cause catastrophic forgetting of the original text knowledge; and the high frame rates most joint models operate at (12.5Hz-25Hz) impose substantial computational cost, limiting practical deployment. Most recent large audio language models address audio capability development through large-scale audio-text pre-training, which is itself expensive and adds another axis of cost.
Method
Fun-Audio-Chat scales up two architectural innovations from the authors’ prior work DrVoice to millions of hours of training data and larger model sizes (dense 8B and MoE 30B-A3B). A Whisper-Large-v3 speech encoder and adapter convert user audio into continuous representations for comprehension, while a frozen CosyVoice 3 speech tokenizer converts assistant-side audio into discrete semantic tokens. Dual-Resolution Speech Representations (DRSR) groups five consecutive 25Hz speech tokens into a single embedding via a learned linear projection, letting the shared LLM backbone process both input and output at an efficient 5Hz frame rate; a separate Speech Refined Head (SRH) then takes the shared LLM’s final hidden state, splits it back into five group-sized segments, and autoregressively predicts the five original 25Hz speech tokens the group represented, recovering fine-grained acoustic detail that the grouping operation would otherwise discard. A frozen CosyVoice 3 detokenizer (a flow-matching model producing mel-spectrograms, followed by a HiFi-GAN vocoder) converts the recovered speech tokens back to a waveform, using speaker-specific embeddings for timbre. Following Moshi, the model is a parallel joint speech-text architecture: at each decoding step, speech-token and text-token embeddings are summed into one composite input, with a special silence token padding whichever stream is shorter, so speech and text generation proceed as one unified autoregressive process rather than two separate streams. Post-training proceeds in three stages: Pre-alignment trains only the speech encoder, adapter, and SRH on large-scale speech-text pairs while the shared LLM stays frozen; Core-Cocktail Training performs full supervised fine-tuning in two phases (an aggressive high-learning-rate phase, then a weighted parameter merge of the fine-tuned model back toward the original pretrained LLM to recover lost text knowledge, followed by a low-learning-rate refinement phase), using speech synthesized from billions of text tokens via CosyVoice 3 and filtered by synthesis WER; and Multi-Task DPO Training applies Direct Preference Optimization across four simultaneous preference dimensions (robustness to noisy/diverse real speech, instruction-following for emotion/style/prosody control, audio-understanding accuracy, and empathetic response quality) in a single unified training stage. A full-duplex variant, Fun-Audio-Chat-Duplex, extends the architecture with a parallel speech-text input stream that lets the model ingest user speech while it is still generating its own response, trained on full-duplex conversation data synthesized by augmenting half-duplex dialogue data with simulated overlapping-turn behavior.
Key Results
Fun-Audio-Chat-8B achieves the best overall performance among roughly-8B-scale open-source models on OpenAudioBench (76.61%) and VoiceBench (83.21%), outperforming GLM-4-Voice, MiniCPM-o 2.6, Baichuan-Omni-1.5, Kimi-Audio, Step-Audio2-Mini, and MiMo-Audio on both benchmarks, while the larger 30B-A3B variant achieves results competitive with closed-source GPT-Audio and Gemini-2.5-Pro. On audio understanding, Fun-Audio-Chat-30B-A3B achieves the best results among all evaluated models on MMAU (77.9%), MMAU-Pro (59.9%), and MMSU (70.1%), ahead of Kimi-Audio, Audio-Flamingo-3, Step-Audio2-Mini, and MiMo-Audio; ASR performance on LibriSpeech/Common Voice is competitive but not best-in-class (e.g. WER 1.71 on LibriSpeech-clean for the 8B model, versus 1.28 for Kimi-Audio). On speech function calling, Fun-Audio-Chat-30B-A3B achieves the highest overall score (79.63%), and the 8B model outperforms top-tier closed-source GPT-Audio and Gemini-2.5-Pro specifically on parallel function calling (BFCL-Parallel, 87.63%). On the VStyle instruction-following/empathy benchmark, Fun-Audio-Chat substantially outperforms open-source Baichuan-Audio and Kimi-Audio while remaining competitive with commercial GPT-4o and Doubao, though individual sub-dimensions (e.g. role-play, some empathy categories) show more mixed results against the strongest commercial baselines. The full-duplex variant, Fun-Audio-Chat-Duplex-30B-A3B, achieves the highest knowledge-understanding scores (S2M-T 54.89%, S2M-S 49.28%) and a perfect 100% turn-taking success rate, both ahead of Moshi and FreezeOmni. Speech generation quality is reported as UTMOS 4.37 and ASR-WER 4.32% on the Llama Q. speech-to-speech test set, and the dual-resolution design is reported to cut training GPU hours by roughly 50% relative to models operating at higher uniform frame rates.
Novelty Assessment
The two architectural pillars, Dual-Resolution Speech Representations and Core-Cocktail Training, are both explicitly inherited from the authors’ own earlier DrVoice work rather than introduced here; this paper’s contribution on that front is demonstrating the two techniques scale effectively to millions of hours of data and to substantially larger models (8B dense, 30B-A3B MoE) without losing their efficiency or forgetting-mitigation properties. The genuinely new elements are Multi-Task DPO Training, a single DPO stage that jointly optimizes four distinct preference dimensions (robustness, instruction-following, audio understanding, empathy) rather than relying on supervised fine-tuning alone, and the full-duplex extension (Fun-Audio-Chat-Duplex) with its parallel speech-text input stream. The breadth of the evaluation, spanning spoken QA, audio understanding, speech function calling, instruction-following, voice empathy, and full-duplex interaction against a wide range of open- and closed-source baselines, is itself a substantial empirical contribution distinct from the underlying architecture.
Field Significance
high — Fun-Audio-Chat demonstrates that a dual-resolution speech representation and cocktail-merging fine-tuning strategy, first validated at smaller scale in DrVoice, retain their efficiency and anti-forgetting properties when scaled to millions of hours of data and tens of billions of parameters, while extending post-training with a unified multi-dimension DPO stage that measurably improves instruction-following and empathy without a separate large-scale audio-text pre-training phase.
As an unreviewed industry technical report from a single lab, its state-of-the-art claims rest on the authors’ own benchmark selections and comparisons rather than independent replication, and several results (e.g. VStyle role-play, some emotion sub-categories) are more mixed against strong commercial baselines than the paper’s overall framing suggests. Its main value to the field is a validated recipe, efficient dual-resolution processing plus staged cocktail fine-tuning plus multi-task preference optimization, that other groups building large audio language models without large-scale audio-text pre-training budgets can draw on directly, along with the open-sourced 8B checkpoint and code.
Claims
- supports: A joint speech-text language model can process speech at a substantially reduced input/output frame rate for its core reasoning backbone (via token grouping) while still recovering full-resolution, high-quality speech tokens through a separate lightweight generation head, without sacrificing either computational efficiency or speech quality.
Evidence: Processing speech at 5Hz in the shared LLM backbone (versus 6.25-25Hz in compared systems) reduces training GPU hours by approximately 50%, while the Speech Refined Head still recovers 25Hz output tokens that achieve UTMOS 4.37 and ASR-WER 4.32% on the Llama Q. speech-to-speech test set. (§2.2, §3.2, §3.7)
- supports: Combining a two-stage fine-tuning schedule with an intermediate weighted merge back toward the original pretrained text-LLM parameters can mitigate catastrophic forgetting during multimodal post-training at scale, extending beyond the smaller-scale setting where the technique was first validated.
Evidence: Core-Cocktail Training (Stage 1 high-learning-rate fine-tuning, an α=0.5 weighted merge with the original pretrained LLM, then Stage 2 low-learning-rate refinement), scaled to 8B dense and 30B-A3B MoE models trained on millions of hours of data, enables Fun-Audio-Chat to achieve top text-benchmark-adjacent results (e.g. IFEval 78.52 on VoiceBench for the 8B model) alongside strong audio capabilities. (§2.4, Table 2)
- supports: A single, multi-dimensional Direct Preference Optimization stage that jointly targets robustness, instruction-following, audio understanding, and empathy can improve a spoken dialogue model’s real-world conversational capabilities beyond what supervised fine-tuning alone achieves.
Evidence: Following Core-Cocktail supervised fine-tuning with Multi-Task DPO Training, Fun-Audio-Chat substantially outperforms open-source Baichuan-Audio and Kimi-Audio on the VStyle instruction-following/empathy benchmark (overall scores 3.35/3.46 for Fun-Audio-Chat-8B vs. 2.50/2.25 and 2.54/3.11 respectively, English/Mandarin) while remaining competitive with commercial GPT-Audio and GPT-4o. (§2.4, §3.5, Table 5)
- complicates: State-of-the-art performance on aggregate benchmark scores can mask meaningfully mixed results on individual sub-dimensions, particularly against strong commercial baselines, so overall-score comparisons alone can overstate a model’s relative capability on any specific skill.
Evidence: Despite leading open-source models on VStyle overall scores, Fun-Audio-Chat-8B trails commercial GPT-4o and Doubao on several individual sub-dimensions such as English role-play scenario/character scores (2.50/3.06 vs. GPT-4o’s 3.89/3.83) and several individual emotion-empathy categories (e.g. English anxiety empathy 2.90 vs. GPT-4o’s 5.00). (§3.5, Table 5)
Limitations and Open Questions
The paper’s own limitations section identifies three open issues: the model occasionally loses context in complex multi-turn conversations requiring long-context comprehension; speech instruction-following shows instability in expressiveness, with generated speech sometimes not fully capturing the intended emotional nuance, style, or prosody; and voice empathy performance is inconsistent across different emotional scenarios despite competitive benchmark averages. The dual-resolution architecture and Core-Cocktail Training strategy are both inherited from the authors’ own prior work (DrVoice) rather than validated here as new architectural ideas, so this paper’s novelty claim rests primarily on scaling and on the added Multi-Task DPO stage and full-duplex variant. As an internal industry technical report, baseline comparisons and evaluation protocols are entirely the authors’ own selection, with no independent third-party replication of the reported results.
Wiki Connections
- Spoken Language Models — presents a parallel joint speech-text LLM that consumes external user speech and generates synchronized speech-and-text output within a single autoregressive process, following the Moshi architectural lineage.
- Streaming TTS — generates speech tokens autoregressively at a reduced 5Hz backbone frame rate with a dedicated refinement head recovering full 25Hz output resolution, explicitly targeting reduced latency and compute versus higher-frame-rate joint speech-text models.
- RLHF for Speech — introduces Multi-Task DPO Training, a single Direct Preference Optimization stage jointly targeting robustness, instruction-following, audio understanding, and empathy preferences.
- Multilingual TTS — trained and evaluated across English and Mandarin, including per-language breakdowns on the VStyle instruction-following and voice-empathy benchmarks.
- CosyVoice 3 — its pretrained speech tokenizer and detokenizer are used frozen throughout training as Fun-Audio-Chat’s speech representation and waveform-synthesis components, and it is also used to synthesize WER-filtered training speech.
- Moshi — its parallel speech-text architectural approach (explicit text streams providing semantic guidance for speech generation) is directly adopted, and Moshi serves as a baseline for the full-duplex evaluation.
- VoiceBench — used as a primary Speech-to-Text spoken question answering evaluation benchmark across instruction-following, knowledge, safety, and robustness dimensions.
- VStyle — used as the primary benchmark for evaluating speech instruction-following and voice empathy capabilities across acoustic attributes, instruction-following, role-play, and empathy dimensions.
- Kimi-Audio — a primary open-source baseline compared against across spoken question answering, audio understanding, and speech instruction-following/empathy benchmarks.
- Step-Audio 2 — a primary open-source baseline compared against across spoken question answering, audio understanding, and speech function calling benchmarks.
- GLM-4-Voice — an early end-to-end spoken chatbot baseline compared against on spoken question answering benchmarks.
- MMAU-Pro — used as an audio understanding evaluation benchmark stressing long-form, spatial, and overlapping-sound conditions.