ACL · 2025 · Conference
Chen et al. (Columbia University and Google) · → Paper · Demo: ? · Code: ?
DAMSEL combines three auxiliary tasks derived from existing speech recordings to improve multimodal LLM fine-tuning for spoken question answering, achieving state-of-the-art results on Spoken-SQuAD with open-weight models using only 10% of training data, alongside ASK-QA, the first multi-turn spoken dialogue dataset with ambiguous user requests.
Problem
Adapting multimodal LLMs (MLLMs) to spoken contexts requires models to robustly comprehend audio signals that encode both semantic content and paralinguistic features such as speaking rate and pitch. Standard fine-tuning with cross-entropy loss on spoken QA data fails to explicitly teach the cross-modal skills that underlie strong performance, particularly when training data is scarce. Prior work addressed this with large-scale multi-modal pre-training on additional datasets, an approach that is computationally expensive and impractical for task-specific customization of closed-source models or smaller open-weight systems. Existing spoken dialogue datasets also lacked ambiguous, multi-turn interactions with diverse speakers, limiting evaluation of real-world spoken conversational agent behaviour.
Method
DAMSEL (Data-centric Multitask Speech Learning) reframes MLLM speech fine-tuning as a three-task problem without collecting any additional audio recordings. From a fixed set of speech data, three auxiliary training objectives are derived: (1) Listening Comprehension, which asks the model to transcribe the spoken context, directly targeting the ASR-level errors that cascade into downstream QA failures; (2) Cross-Modal Commonsense Reasoning, which frames dialogue response selection as a multiple-choice task to teach the model to align spoken and textual representations; and (3) Response Generation, the standard end-to-end QA objective. These tasks are implemented purely as modifications to the training data mixture, making DAMSEL applicable to both closed-source commercial APIs (Gemini Pro and Gemini Flash via Vertex AI) and open-weight adapters.
For the open-weight setting, the paper introduces Speech-Qwen: a 17.8M-parameter linear projection layer bridging a frozen WavLM-Large audio encoder to a frozen Qwen 2.5 7B-Instruct decoder. The projection layer is first pre-trained on large-scale ASR data before downstream task fine-tuning, following conventions established for visual MLLMs. A Speech-Phi variant using Phi 3.5 Mini (3B parameters) is also evaluated.
ASK-QA is constructed from Abg-CoQA, a text-based conversational QA dataset, by synthesising speech using Google Cloud TTS with 64 unique voice types, randomly sampled speaking rates and pitches, and injected disfluencies. The result is 221.8 hours of multi-turn spoken dialogue across 7,830 conversations, with an average audio input length of 1 minute 41 seconds. The dataset supports both single-turn and a novel dynamic multi-turn evaluation where a prompted action classifier and LLM-based user simulator interactively extend the spoken conversation in response to the model’s clarifying questions.
Key Results
On Spoken-SQuAD, Speech-Qwen with DAMSEL surpasses the prior state-of-the-art open-source system (DDNet: 64.1 EM / 77.1 F1) using only 10% of available training data (66.38 EM / 78.80 F1). With full data, Speech-Qwen+DAMSEL reaches 72.13 EM / 82.36 F1 (Table A7). Consistent multi-task gains appear across all data regimes and both open-weight and closed-source base models.
On ASK-QA, DAMSEL yields up to 13.3% relative improvement over single-task fine-tuning with Speech-Qwen across multi-turn trajectory-level similarity (Table A6). Even Gemini Pro, which already has large-scale multi-modal pre-training, benefits by 5.7% relative in the 1% data regime, indicating that multi-task auxiliary supervision teaches something not covered by generic pre-training.
On SD-QA (five languages, 24 regional dialects), DAMSEL improves Gemini Pro exact match by 16.13% relative in the limited data setting (46.33 to 55.24, Table 2) and Speech-Qwen by up to 52.8% relative with 1% of training data, suggesting that the gains are especially pronounced when base models have limited multilingual exposure.
Ablations in Table 3 show that each auxiliary task contributes independently, with Listening Comprehension (story transcription) producing the largest single-task degradation when removed, reinforcing the finding that ASR-level comprehension is the primary bottleneck in spoken QA.
Novelty Assessment
The DAMSEL approach is an engineering integration: multi-task learning with auxiliary tasks is a well-established training paradigm (Caruana 1997), and the Speech-Qwen architecture (frozen SSL encoder + linear projection + frozen LLM) follows the SLAM-LLM pattern from Ma et al. (2024). The contribution is the specific design of auxiliary task types for the spoken QA setting and the demonstration that these tasks transfer gains even to frontier closed-source models. The data-centric framing (no additional data collection) is practically useful but methodologically incremental.
ASK-QA is the more substantial contribution: it is the first spoken dialogue corpus featuring ambiguous mixed-initiative conversations with diverse speaker profiles, multi-turn dynamic evaluation, and long audio contexts. Existing spoken QA datasets (Spoken-SQuAD, SD-QA) are either single-speaker, single-turn, or lack free-form ambiguous requests. The construction pipeline (TTS with acoustic diversity plus LLM-based paraphrasing and filtering) provides a template for scalable spoken dialogue dataset creation.
Field Significance
Moderate — The paper addresses a genuine gap in data-efficient MLLM adaptation for spoken conversational agents, demonstrating that targeted auxiliary task design can substitute for large-scale data collection. ASK-QA provides the field with a harder benchmark for multi-turn spoken dialogue evaluation than previously available. The DAMSEL approach is straightforward to apply but its design choices are specific enough to be non-obvious, and the evaluation across two commercial and two open-weight model families provides unusually broad empirical coverage.
Claims
-
supports: Explicit listening comprehension objectives improve cross-modal understanding in multimodal LLMs fine-tuned for spoken QA.
Evidence: Removing the story listening comprehension sub-task from DAMSEL causes the largest single-task performance degradation (Table 3, ablations on ASK-QA with Speech-Qwen), confirming that transcription supervision is the most critical auxiliary signal. (§3.4, Table 3)
-
supports: Multi-task auxiliary task design can substitute for large-scale data collection in adapting MLLMs to spoken question answering.
Evidence: DAMSEL with Speech-Qwen surpasses the prior state-of-the-art on Spoken-SQuAD using only 10% of training data (66.38 EM vs. DDNet 64.1 EM trained on 100% of data). (§3.2, Figure 4, Table A7)
-
supports: Data-centric multi-task fine-tuning yields consistent gains even for frontier MLLMs with extensive prior multi-modal pre-training.
Evidence: Gemini Pro fine-tuned with DAMSEL on 1% of ASK-QA improves 5.7% relative over single-task tuning; improvement persists with full data (1.6% relative), despite Gemini having seen large-scale multi-modal training data and ASK-QA being newly synthesised. (§3.1)
-
complicates: TTS-synthesised speech datasets have quality ceilings that limit model generalisation to naturalistic paralinguistic variation.
Evidence: The paper notes that WER-filtered ASK-QA transcriptions may not perfectly match synthesised speech, and that current TTS lacks perfect controllability for speaking rate and pitch. The dynamic evaluation pipeline also depends on TTS to regenerate spoken turns, compounding generation quality issues across multi-turn rollouts. (§Limitations, §B.1)
-
refines: Spoken QA performance gains from multi-task learning are largest in the low-data regime and diminish as training data scales.
Evidence: Gemini Pro’s DAMSEL advantage over single-task tuning narrows from 16.13% relative EM improvement (1% data) to 1.7% (full data) on SD-QA, and from 5.7% to 1.6% relative on ASK-QA. (§3.1, §3.3, Table 2)
Limitations and Open Questions
Warning
Performance relies on ground-truth transcriptions as listening comprehension targets. In practice, transcriptions from Spoken-SQuAD and SD-QA are derived from ASR systems with known errors. The paper acknowledges it is unclear whether slight transcription noise improves robustness or degrades performance.
A key open question is whether the DAMSEL data generation process scales to MLLM post-training. The paper demonstrates gains with fine-tuning but notes that verifying post-training effects is computationally infeasible in this work. The dynamic multi-turn evaluation for ASK-QA also depends on an LLM-based action classifier and user simulator, introducing additional sources of evaluation variance that are not thoroughly characterised.
The model currently targets auditory semantic understanding. The paper notes that more nuanced SCA tasks (such as monitoring user frustration in task guidance) require sensitivity to different paralinguistic dimensions not addressed by the current auxiliary tasks.
Wiki Connections
- Self-Supervised Speech — Speech-Qwen uses WavLM-Large as its frozen audio encoder, making self-supervised speech pre-training a core architectural dependency of the entire open-weight system.
- Evaluation Metrics — The paper introduces a dynamic multi-turn evaluation pipeline for ASK-QA in which an LLM action classifier and user simulator interactively extend spoken dialogue to assess models on ambiguous request resolution.
- Multilingual TTS — The SD-QA evaluation covers five languages and 24 regional dialects; DAMSEL’s gains in the multilingual low-data regime suggest that auxiliary task design can partially compensate for limited cross-lingual speech training coverage.