ACL · 2025 · Conference
William Held et al. · → Paper · Demo: ✓ · Code: ✓
DiVA trains an end-to-end speech large language model using only ASR data and cross-modal knowledge distillation from a frozen text LLM, entirely bypassing the need for annotated speech instruction data.
Problem
Existing Speech LLMs trained with supervised fine-tuning (SFT) on speech instruction data suffer from two compounding problems. First, they forget the general capabilities of their underlying text LLMs because the available annotated speech instruction corpora are narrow in task coverage, small in scale, and drawn from a limited speaker pool. Freezing the LLM weights is insufficient to prevent forgetting, as the new audio encoder induces distributional shift. Second, the speaker demographics in these datasets are skewed, which is likely to amplify biases in speech processing across regional accents and dialects. The root cause in both cases is the same: there is no large, diverse, and broadly annotated speech instruction corpus, and attempts to construct one using SFT workarounds (task transformation, synthetic generation) produce imbalanced coverage that weakens non-target task performance.
Method
DiVA (Distilled Voice Assistant) reframes speech LLM training as a cross-modal distillation problem. The system consists of three components: a frozen Whisper-Large-v3 encoder (1.5B params) that extracts mel-spectrogram audio features; a Q-Former connector module that compresses those features into a fixed number of text-length tokens for the downstream LLM; and the Llama 3 8B Instruct model, whose weights remain entirely frozen throughout training.
The key architectural innovation is in initializing the Q-Former from the Whisper decoder rather than from scratch. Whisper’s decoder was already trained to map audio embeddings to discrete text tokens for ASR, so its cross-attention key/value projection matrices already encode a meaningful audio-to-text mapping. DiVA repurposes this by replacing the decoder’s autoregressive input tokens with static learnable query vectors Q, producing |Q| = 448 audio tokens that represent a compressed speech embedding for the LLM.
Training uses only CommonVoice 17 English (3.5k hours of crowdsourced, globally diverse, permissively licensed speech) paired with its text transcripts. No annotated responses are required. Instead, DiVA optimizes two loss functions simultaneously.
The first is an input token alignment loss (L_con): an L2 distance between the final N audio token embeddings and the corresponding N text token embeddings from the frozen LLM’s embedding layer. This forces the audio tokens to lie near the LLM’s input distribution, enabling the model to follow text instructions when audio is the input. Empirically, this loss alone produces incoherent generations.
The second is an output embedding distillation loss: rather than computing the full KL divergence over the vocabulary (prohibitively expensive at V·d operations), DiVA exploits the fact that, when the output embedding matrix is frozen (shared between teacher and student), minimizing the L2 distance between the LLM’s final hidden state on audio input (h_s) versus text input (h_t) is equivalent to minimizing a subset of the KL divergence minimization problem. This reduces computation from O(V·d) to O(d) and produces smoother gradients than direct KL optimization. In practice, only the first predicted next-token position is used for this loss, following evidence that single-token probabilities encode substantial information about the full output distribution.
The combined two-loss training runs for 4,300 steps (batch size 512, ~2 epochs) on a TPU v4-256 pod in approximately 12 hours — over 100× less compute than the Qwen 2 Audio training budget.
Key Results
DiVA is compared against BLSP (hard distillation), UltraVox (output-only soft distillation), SALMONN, Qwen Audio, and Qwen 2 Audio — all of which use direct supervision on the evaluated tasks.
Speech translation (CoVoST 2, SacreBLEU): Results are mixed across languages. DiVA achieves the best scores on Tamil (3.22) and Turkish (11.74) but underperforms Qwen models for Chinese (12.22 vs 25.11) and Japanese (6.17 vs 22.48), both cases attributable to Llama 3’s bias toward Latinized romanization (Pinyin, Romaji) rather than native script. Both UltraVox (also Llama 3 based) shows the same artifact. Hard distillation baselines (BLSP, UltraVox) achieve very low BLEU across all languages, showing that output distillation alone is insufficient for translation.
Spoken question answering (HeySquad, SDQA across 10 accent regions): DiVA outperforms all baselines by at least 10 percentage points on every accent and both datasets (e.g., 54.6% vs 44.2% for Qwen 2 Audio on SDQA-USA, 55.2% vs 46.2% on HeySquad). Qualitative analysis shows Qwen Audio exhibits severe capability forgetting — 30% of responses transcribe the question rather than answering it. SALMONN reduces this to 8% via inference-time LoRA ablation, Qwen 2 Audio to 4% via DPO, while DiVA achieves zero instruction-ignoring outputs.
Speech classification (emotion, sarcasm, humor): DiVA significantly outperforms all baselines on emotion recognition (IEMOCAP: 50.6 F1 vs 42.9 for the next-best BLSP; MELD: 41.3 F1 vs 40.1). No model achieves better than chance on sarcasm (MUSTARD); Qwen Audio Chat is the only model better than chance on humor (URFunnyV2). These results suggest current speech LLMs cannot reliably capture complex social signals in speech tone.
User preference study (vs. Qwen 2 Audio, 522 judgments, 53 Prolific participants): DiVA achieves a 72% win rate at the preference level; 77% of users (41/53) preferred DiVA for the majority of their inputs. This holds despite Qwen 2 Audio outperforming DiVA on several automatic benchmarks.
Ablations confirm that: (1) output distillation loss alone fails at translation and emotion (1.4% correct language output); (2) input alignment alone produces mostly incoherent generations; (3) Q-Former initialized from scratch drops QA by 13.7%; (4) replacing Q-Former with a simple MLP drops QA by 38.1% further.
Novelty Assessment
The central contribution is methodological: demonstrating that cross-modal context distillation from a text LLM to the speech modality — using only ASR-paired data and without any annotated response labels — can match or exceed SFT-trained speech LLMs on most tasks. This is genuinely novel in its scope and practical simplicity.
The architectural contribution (initializing Q-Former from Whisper decoder weights) is also novel and well-motivated, though incremental relative to Q-Former itself. The O(d) KL approximation via hidden-state L2 is elegant and rigorously motivated, though it relies on the specific assumption that the output embedding matrix is frozen.
The user preference result (72% win rate vs. 100× more compute Qwen 2 Audio) is the strongest empirical claim, though this is a relative preference study with Prolific workers rather than a domain-expert panel, and the test prompts are not released for replication.
The work is primarily a training-recipe contribution: the architecture components (Whisper encoder, Q-Former, Llama 3) are all off-the-shelf. The value is showing these components can be combined effectively with under 3.5k hours of unannotated, permissively licensed data.
Field Significance
Tip
High — DiVA demonstrates that cross-modal context distillation from a frozen text LLM can replace supervised fine-tuning for speech instruction following, achieving stronger user preference than SFT-trained systems trained on over 100× more compute. This challenges the assumption that large annotated speech instruction corpora are a prerequisite for capable spoken conversational agents, and provides a practical path for teams without access to proprietary instruction data or large GPU budgets.
Claims
- Cross-modal context distillation using only ASR transcription pairs can transfer instruction-following capabilities from a text LLM to a speech encoder without any annotated speech responses. (§3.2, §5.1.2)
- SFT-trained speech LLMs exhibit systematic capability forgetting — reverting to transcription behavior rather than answering instructions — that distillation-based training avoids. (§5.1.2, Table 3)
- Joint input-token alignment and output-distribution distillation are both necessary: output distillation alone fails at translation and emotion tasks, while input alignment alone produces incoherent generations. (§6, Table 5)
- Initializing the Q-Former cross-attention from pretrained Whisper decoder weights substantially improves spoken question answering compared to random initialisation, demonstrating that pretrained audio-to-text alignment is a transferable inductive bias. (§6, Table 5)
- User preference ratings can diverge substantially from automatic benchmark performance, with a model performing worse on several task benchmarks still achieving a 72% win rate in side-by-side evaluation. (§5.2)
Limitations and Open Questions
The distillation objective inherently bounds DiVA to signals detectable in text: any paralinguistic information (prosody, fine-grained emotional tone, speaker intent) that has no mutual information with the transcript cannot be learned. DiVA outperforms baselines on emotion despite this constraint, likely because many emotional utterances carry semantic cues, but sarcasm and humor detection both fail, suggesting paralinguistic understanding remains an open problem.
Evaluation covers only single-turn interactions; multi-turn conversation handling is acknowledged as untested. Translation quality for logographic languages (Chinese, Japanese) is limited by Llama 3’s romanization bias — a base LLM limitation, not a distillation limitation. Whether DiVA could be adapted with a more multilingual base LLM is not explored.
The approach requires that the teacher LLM responds meaningfully to transcripts without audio context. For tasks where audio context would substantially change the LLM’s response (e.g., emotion-specific queries, follow-up questions about speaker affect), the distillation target may be incorrect. This creates a ceiling on what the distillation approach can achieve relative to a fully supervised system with audio-grounded responses.
Wiki Connections
- spoken-language-model — DiVA proposes a new training paradigm (cross-modal context distillation without instruction data) for speech LMs, directly addressing the capability-forgetting problem that SFT-based systems exhibit
- speech-to-speech — the end-to-end speech-in, text-out architecture and Q-Former bridge design connect to the broader pipeline of spoken conversational agents
- subjective-evaluation — the user preference study (side-by-side Prolific ratings, 72% win rate metric) illustrates a non-MOS alternative for evaluating spoken conversational agent quality, and shows automatic benchmarks can diverge from human preferences