arXiv · 2025 · Preprint
Lu et al. (NTU / NVIDIA) · → Paper · Demo: ? · Code: ?
DeSTA2 shows that a spoken language model with strong instruction-following and paralinguistic understanding can be trained using only automatically generated speech caption data, eliminating the need for manually annotated speech instruction-tuning pairs.
Problem
Existing end-to-end spoken language models (SLMs) typically rely on large volumes of speech instruction-tuning data, annotated either by human labelers or generated through LLM-prompted QA pipelines. Both approaches carry significant costs: manual annotation introduces task-specific biases and format constraints, while automated QA generation often produces textual distributions that diverge from the base LLM’s pretraining corpus. This distributional mismatch can cause catastrophic forgetting of the LLM’s original capabilities, including complex instruction following and chain-of-thought reasoning. The open question is whether the instruction-following abilities already present in a frozen LLM can be transferred to the speech domain without a dedicated instruction-tuning stage.
Method
DeSTA2 introduces a data construction pipeline that sidesteps speech instruction-tuning entirely. Rather than generating task-specific question-answer pairs, the method assembles a “seed transcript” for each utterance: a structured text record encoding time-stamped transcription alongside paralinguistic attributes extracted by specialized models (gender classifier, emotion recognizer using emotion2vec, Whisper ASR, SNR predictor via Brouhaha, and room acoustics estimation). This seed transcript, together with the single prompt “What can you hear from the audio?”, is fed to the underlying Llama3-8B-Instruct to produce a free-form speech caption. Because the target text is generated by the same LLM used in the SLM, the distribution of training responses aligns closely with Llama3’s pretraining regime, minimizing textual mismatch and preserving the model’s original formatting and reasoning capabilities.
The end-to-end model integrates a frozen Whisper-small encoder (244M parameters) with a frozen Llama3-8B-Instruct. A modality adapter consisting of a 2-layer transformer decoder block uses 64 trainable query vectors in a Q-Former configuration to extract speech features from intermediate Whisper encoder layers; these are then aggregated by a learned weighted sum and projected linearly into Llama3’s input space. At inference, the concatenation of continuous speech features and Whisper text transcription substitutes the seed transcript, enabling the model to respond to any natural language instruction over speech. Only the 22.3M adapter parameters are trained, leaving both Whisper and Llama3 frozen throughout. Training uses 155 hours of annotated speech across six datasets covering accent, emotion, dialogue, prosody, speaker identity, and acoustic conditions, plus noise-augmented variants.
Key Results
On Dynamic-SUPERB, DeSTA2 achieves an overall accuracy of 56.78%, the highest among all end-to-end systems evaluated (Table I), clearly outperforming Qwen2-Audio (51.69%), BLSP (46%), and SALMONN (36.44%). The gain is especially large on the paralinguistic dimension (43.14% vs. Qwen2-Audio’s 29.21%), reflecting DeSTA2’s rich paralinguistic metadata. On AIR-Bench-Chat, DeSTA2 scores 7.16 against Qwen2-Audio’s 7.18, a near-tie despite using no speech instruction-tuning data. Ablations (Table III) confirm that the key design choice is the LLM-generated free-form caption target: using structured seed transcripts directly as targets, or generating QA pairs with constrained prompts, both yield substantially lower accuracy (33.13% and 50.33-50.56% respectively). Notably, adding more QA pairs per sample (Open QA (3) vs. Open QA (1)) improves degradation tasks but hurts content and semantic dimensions, illustrating the language-ability vs. speech-understanding trade-off that DeSTA2 avoids. When further fine-tuned on Dynamic-SUPERB’s training set with LoRA, DeSTA2 reaches 74.45% overall accuracy, exceeding the prior DeSTA system (67.63%) and an adapter-only fine-tune baseline (70.86%) (Table IV).
Novelty Assessment
The contribution is primarily one of data engineering: the finding that using the SLM’s own frozen LLM to generate unconstrained speech captions reduces distributional mismatch and preserves general instruction-following is the key insight. The model architecture (Whisper + Q-Former adapter + frozen LLM) follows a well-established pattern originating from BLIP-2 and used in SALMONN, Qwen-Audio, and DeSTA. What is new is the deliberate elimination of any task-specific prompting or postprocessing in the data construction step, and the demonstration that a single generic speech captioning prompt suffices for competitive multi-task performance. The evaluation coverage is broad and comparisons across end-to-end systems appear fair given shared benchmark conditions. However, the approach is not structurally innovative, and its advantage over Qwen2-Audio on AIR-Bench-Chat is marginal (0.02 points).
Field Significance
Moderate — DeSTA2 provides useful evidence that the instruction-following ability of a frozen LLM can be leveraged to bootstrap spoken language model training without expensive annotation pipelines, challenging the assumption that large-scale speech instruction-tuning data is necessary. The empirical gains on Dynamic-SUPERB are substantial, but the architecture and training setup are incremental relative to the DeSTA line of work and the broader Whisper-LLM integration literature.
Claims
-
supports: Speech language models can acquire broad instruction-following capabilities over paralinguistic attributes without task-specific speech instruction-tuning data.
Evidence: DeSTA2 achieves 56.78% overall accuracy on Dynamic-SUPERB across 48 tasks, surpassing all prior end-to-end systems that used extensive instruction-tuning, using only LLM-generated speech captions with a single generic prompt. (§V.A, Table I)
-
complicates: Automated speech instruction-tuning pipelines that use constrained prompts or QA formatting may degrade the base LLM’s general language capabilities.
Evidence: Ablations show that training with open-ended QA pairs (Open QA variants) improves degradation task performance but causes measurable drops in content and semantic dimensions relative to DeSTA2’s unconstrained generation approach. (§V.B, Table III)
-
supports: Distributional alignment between training targets and the base LLM’s pretraining regime reduces catastrophic forgetting in speech-extended language models.
Evidence: DeSTA2 preserves chain-of-thought reasoning and output format following (e.g., computing words-per-second, responding in all-caps on demand), capabilities absent from Qwen2-Audio despite its larger training budget, because training targets are generated by the same LLM. (§V.D, Table V)
-
refines: Rich paralinguistic metadata coverage, rather than task diversity, is the primary driver of broad spoken language model performance on multi-dimensional speech benchmarks.
Evidence: DeSTA2 uses a single captioning task with 12 paralinguistic attributes (gender, emotion, pitch, SNR, accent, etc.) and outperforms systems with diverse task-specific instruction sets on the paralinguistic dimension (43.14% vs. 29.21% for Qwen2-Audio). (§IV.A, §V.A, Table I)
Limitations and Open Questions
Warning
Speaker verification performance remains below specialized baselines: DeSTA2 scores 42.5% on the speaker dimension of Dynamic-SUPERB, compared to 47.9% for Qwen2-Audio, because the approach does not address multi-speaker comparison tasks. The training setup does not cover cross-utterance speaker verification.
The model relies on Whisper for both feature extraction and text transcription, making it unclear how the system performs when ASR quality degrades (non-native accents, noisy conditions). The training corpus is relatively small (155 hours) and English-centric; generalization to other languages or low-resource acoustic conditions has not been tested. Code and model weights were not released at submission time. The evaluation benchmarks (Dynamic-SUPERB, AIR-Bench-Chat) use GPT-4 and GPT-4o as judges, introducing potential instability in score reproducibility across evaluation runs.
Wiki Connections
- Spoken Language Model — DeSTA2 demonstrates that a frozen LLM’s instruction-following can be extended to the speech domain via lightweight adapter training without task-specific speech annotations.
- Evaluation Metrics — the paper evaluates on Dynamic-SUPERB (48-task classification benchmark) and AIR-Bench-Chat, contributing evidence on how LLM-judged metrics behave for multi-task speech understanding.
- Subjective Evaluation — AIR-Bench-Chat uses GPT-4 agreement scoring as a proxy for subjective quality, and DeSTA2’s results raise questions about the reliability of LLM judges compared to human listeners.