ACL · 2025 · Conference
Pandey et al. (Amazon AGI) · → Paper · Demo: ✗ · Code: ?
SIFT-50M is a 50M-example multilingual instruction dataset for fine-tuning speech-text LLMs, spanning five languages and covering both speech understanding and controllable generation; it ships with EvalSIFT, a benchmark for evaluating instruction-following capability, and SIFT-LLM, a 7B-parameter demonstration model.
Problem
Speech-text LLMs are typically pre-trained on task-specific corpora designed for automatic speech recognition, which provide audio-label pairs rather than natural language instructions. The resulting data scarcity and homogeneity limits generalization to broader speech understanding tasks and to controllable generation scenarios that require following natural language prompts. Existing instruction datasets for speech (such as OpenASQA at 2.7M examples) are small relative to text-domain instruction sets and cover a narrow slice of acoustic and paralinguistic dimensions.
Method
SIFT-50M is constructed from three publicly available multilingual speech corpora: Multilingual LibriSpeech, Common Voice, and VCTK, collectively providing around 14K hours of speech across English, German, French, Spanish, and Italian. The construction pipeline augments source metadata (transcription, speaker gender, accent, age) with automatically extracted acoustic features including pitch (via PYYAPT), intensity (Parselmouth), speaking rate, noise level, reverberation, and distortion level. Continuous statistics are bucketed into categorical labels to enable natural language description. An LLM (Mixtral 8x7B for most categories, Amazon Nova Pro for open-ended and comparison tasks) then generates up to ten question-answer pairs per utterance from the enriched metadata.
Instructions are organized into three high-level categories. Closed-ended instructions cover four sub-types: acoustic-level attributes (pitch, speaking rate, intensity), content-level attributes (transcription, gender, age, accent), word-level alignment comparisons, and cross-utterance comparisons of two audio clips. Open-ended instructions ask the model to synthesize and reason from metadata. Controllable generation instructions specify target acoustic characteristics and ask the model to produce matching speech. The resulting dataset totals 55.6M training examples and 470K development examples.
The demonstration model, SIFT-LLM, uses Whisper-medium as the frozen audio encoder, a linear projection layer mapping 1024-dimensional speech embeddings into the 3584-dimensional input space of Qwen2.5-7B-instruct. For speech output, the model generates discrete tokens from either HuBERT k-means clusters (2000 tokens, primarily semantic) or X-codec2 (65536 tokens, fusing semantic and acoustic codes); HuBERT codes are converted to waveforms via a BigVGAN vocoder. LoRA fine-tuning (rank 16) yields 8.7M trainable parameters for the understanding configuration, rising to 590M for the controllable generation variant (LoRA rank 128 with the LLM embedding layer unfrozen).
Training proceeds in two stages: continued pre-training on a multi-task mixture (ASR, emotion recognition, speaker-attributed ASR, intent classification, speech translation, TTS) to align the audio embedding space with the LLM, followed by instruction fine-tuning on SIFT-50M.
Key Results
On instruction-following benchmarks, SIFT-LLM outperforms publicly available 7B-scale speech-text LLMs across all evaluations. On Dynamic-Superb (closed-ended), SIFT-LLM scores 57.4% accuracy versus 48.0% for Qwen2-Audio-7B-Instruct and 34.7% for SALMONN-7B. On EvalSIFT closed-ended, SIFT-LLM reaches 46.1% versus 25.1% and 21.9% for those baselines. The direct comparison against O-ASQA-LLM, which uses the same pre-trained backbone but fine-tunes on the smaller 2.7M-example OpenASQA dataset, is the clearest evidence for the dataset’s contribution: SIFT-LLM exceeds O-ASQA-LLM on every reported benchmark.
On foundational tasks, no single model dominates. SALMONN-7B achieves lower WER on English LibriSpeech (2.5% clean vs. 3.5% for SIFT-LLM), while SIFT-LLM substantially outperforms on multilingual ASR (FLEURS-5: 11.4% vs. 23.7% for SALMONN) and emotion recognition (53.6% vs. 38.5% accuracy on MSP-test1). Qwen2-Audio-Instruct leads on speech translation.
For controllable generation, the X-codec2 variant achieves markedly better controllability over pitch variation (QWK 0.69 vs. 0.15 for HuBERT codes) and speaking rate (QWK 0.73 vs. 0.46), with 95.8% gender accuracy versus approximately 50% for the HuBERT-code variant.
Ablations confirm that comparison data and open-ended data are both important: removing comparison instructions drops EvalSIFT closed-ended accuracy from 45.4% to 34.3%, and removing open-ended data drops the open-ended LLM score from 8.0 to 6.1.
Novelty Assessment
The dataset itself is the primary contribution. At 50M examples spanning five languages with instruction categories that extend beyond transcription to word-level alignment and cross-utterance comparison, SIFT-50M is substantially larger and more diverse than prior speech instruction datasets. The dataset construction methodology (acoustic feature extraction, categorical bucketing, LLM-generated QA pairs with iterative prompt refinement) is systematic but conceptually straightforward, drawing on prior work such as OpenASQA.
SIFT-LLM is an engineering integration of established components: Whisper encoder, a transformer LLM backbone, LoRA fine-tuning, and an existing codec. The architecture introduces no structural novelties. The EvalSIFT benchmark is a useful evaluation contribution, particularly for multilingual instruction-following evaluation where Dynamic-Superb and AIR-Bench provide only English data.
The finding that semantic-only HuBERT codes are insufficient for reliable speaker-dependent controllable generation is a practically important negative result, validated by systematic ablation.
Field Significance
Moderate - SIFT-50M provides the speech-text LLM community with a substantially larger and more linguistically diverse instruction corpus than previously available, directly addressing the data scarcity that constrains spoken conversational agents. The accompanying EvalSIFT benchmark fills a gap in multilingual instruction-following evaluation. The work is primarily a data and benchmarking contribution; the modeling choices are conventional.
Claims
-
supports: Large-scale instruction datasets that span diverse acoustic and paralinguistic dimensions improve speech-text LLM generalization on instruction-following benchmarks beyond what smaller task-specific corpora achieve.
Evidence: SIFT-LLM outperforms O-ASQA-LLM (same backbone, fine-tuned on 2.7M-example OpenASQA) across all benchmarks, including 46.1% vs. 22.9% on EvalSIFT closed-ended and 57.4% vs. 45.9% on Dynamic-Superb. (§5.2, Table 3)
-
complicates: Instruction fine-tuning on diverse speech tasks reduces specialized performance on foundational tasks relative to the pre-trained model checkpoint.
Evidence: SIFT-LLM WER on LibriSpeech test-clean rises from 2.5% (50K pre-training checkpoint) to 3.5% after instruction fine-tuning, consistent with behavior observed in Qwen2-Audio and its instruction fine-tuned variant. (§5.3, Table 4)
-
supports: Codec representations that jointly encode acoustic and semantic content enable more accurate controllable speech generation than semantic-only representations.
Evidence: X-codec2 (fused semantic and acoustic codes) achieves QWK of 0.69 for pitch variation versus 0.15 for HuBERT codes (semantic-only) on SIFT-LLM GEN controllable generation evaluations. (§5.4, Table 6)
-
complicates: Semantic-only discrete speech representations are insufficient for reliable speaker-dependent controllable generation, as they primarily encode content rather than prosodic and acoustic identity.
Evidence: The HuBERT-code vocoder setup achieves approximately 50% gender accuracy against instruction-specified gender, compared to 95.8% for X-codec2; pitch variation QWK from original audio degrades substantially in HuBERT re-synthesis (0.11 QWK, Table 18 Appendix F.3). (§5.4, §F.3)
Limitations and Open Questions
Warning
SIFT-LLM does not achieve state-of-the-art results on foundational speech tasks; the instruction fine-tuning trades off specialized ASR and translation capability, limiting its use as a drop-in replacement for task-specialized models.
Controllable generation evaluation relies on automatically extracted acoustic features compared against instruction-specified targets, which indirectly assesses controllability but cannot measure perceptual quality or naturalness. The QWK and MAE metrics measure categorical agreement rather than human-perceived fidelity.
The LLM-as-a-judge evaluation methodology for open-ended and classification tasks introduces judge model dependency; results may shift with different judge LLMs, and the interaction between judge capability and task difficulty is not systematically studied.
Hallucination when queried about content unrelated to the input audio is noted as an open issue; attempts to mitigate it by adding off-topic instructions with higher weight degraded speech understanding performance, suggesting a robustness-capability tension.
Wiki Connections
- Spoken Language Model - SIFT-50M is a training resource specifically designed for speech-text LLMs that emit discrete speech tokens, directly contributing to the spoken language model paradigm.
- Instruction-Conditioned TTS - the controllable generation subset of SIFT-50M provides 5.6M natural language instructions for generating speech with specified acoustic characteristics (pitch, speaking rate, gender, accent, intensity).
- Multilingual TTS - SIFT-50M covers five languages (English, German, French, Spanish, Italian) with both understanding and generation instructions, extending multilingual speech-text LLM capability.
- Evaluation Metrics - EvalSIFT introduces a structured benchmark for evaluating instruction-following in speech-text LLMs, including multilingual closed-ended, open-ended, and controllable generation sub-sets totaling 30K examples.
- Self-Supervised Speech - SIFT-LLM uses HuBERT k-means discrete tokens as one of its two codec options for speech generation, directly incorporating self-supervised speech representations as a core component.