ACL · 2025 · Conference
Hei et al. · → Paper · Demo: ? · Code: ?
A query rewriting framework with multi-LLM knowledge fusion that transforms out-of-distribution text instructions into TTS-friendly form, enabling automated construction of high-quality speech instruction datasets for training large speech language models.
Problem
Large speech language models struggle to follow speech instructions because suitable training data is scarce and expensive to collect at scale. Existing approaches either repurpose ASR datasets by having models transcribe speech (which biases training toward language recognition rather than instruction following) or use LLMs to continue the linguistic content of speech to generate responses (which amplifies the gap between machine-generated outputs and real human answers). Using TTS synthesis to generate speech instruction data from existing text QA corpora is a natural and cost-effective alternative, but TTS models have limited vocabulary and cannot properly render out-of-distribution text such as compound words, abbreviations, and mathematical expressions. In practice, this results in roughly 72% of synthesized speech passing quality thresholds, leaving a substantial proportion of the data unusable.
Method
The framework operates in three stages. First, multi-agent annotation and verification addresses the unreliability of single-model ASR quality assessment by using three architecturally distinct ASR models (Whisper-large-v3, Canary-1B, and Parakeet-TDT-1.1B) together with three text embedding models to compute a Semantic Similarity metric (SIM). For each synthesized utterance, the best recognition result across ASR models is selected, and its similarity to the original text is scored by averaging cosine similarity across all three embedding models. This SIM score replaces strict WER-based filtering, which rejects semantically equivalent speech due to incidental ASR errors.
Second, query rewriting applies three LLMs in parallel (Llama-3-8B-Instruct, Phi-3-small-8k-instruct, and Qwen2-7B-Instruct) to independently rewrite each original text instruction into a form more suitable for TTS synthesis. All four candidates (original plus three rewrites) are passed to the TTS model, and the output with the highest SIM score is selected. In the multi-speaker setting, GPT-4 generates 192 natural language speaker style descriptions used with Parler-TTS-Large to produce stylistically diverse speech.
Third, knowledge fusion handles the hardest-to-rewrite samples: a Llama-3-8B model is fine-tuned with LoRA on pairs of (original text, successfully rewritten text) derived from the previous stage. This model learns to generalise rewriting strategies from examples where individual LLMs succeeded, and is then applied to samples where all three LLMs had failed.
The resulting speech instruction data is used to fine-tune Qwen2-Audio-7B-Instruct across seven QA datasets.
Key Results
The proposed method raises average data usability (Pass, the fraction of samples with SIM score above 0.9) from 72% to 93% and improves average embedding-space similarity from 93.06% to 97.98% in the multi-speaker setting (Table 2). The SIM metric achieves consistency with WER above 98% when strict linguistic equivalence holds and above 92.5% in all tested conditions across three TTS models (Table 1).
When used to fine-tune Qwen2-Audio-7B-Instruct, high-quality rewritten data with human-annotated golden answers achieves average ROUGE-L of 63.04 across DROP, Quoref, ROPES, and NarrativeQA, compared to 39.77 for the untuned baseline and 52.54 for LLM-continued alignment targets (Table 3). The gap between LLM-continued and golden alignment targets persists across all data quality thresholds and configurations, indicating that alignment target quality matters independently of speech quality.
The multi-agent ASR ensemble reduces average WER from 9.6–10.2% (individual models) to 8.36% (Table 4). Human evaluation confirms that rewriting improves linguistic consistency from 0% to 69–79% on the hardest subset of samples where original TTS failed entirely (Table 10).
Novelty Assessment
The contribution is engineering integration: the pipeline assembles existing LLMs, TTS systems, ASR models, and embedding models into a data construction workflow. No individual component is new. The SIM metric extends standard semantic textual similarity methods by averaging across multiple embedding models, which is a modest but practical refinement. The core design insights — that parallel LLM rewriting benefits from architectural diversity, that multi-model ASR averaging reduces annotation errors, and that LoRA fine-tuning on successfully rewritten samples recovers failing cases — are each individually intuitive. The practical value lies in the demonstration that this combination raises data usability by 21 percentage points.
The most generalizable result is the finding that LLM-continued alignment targets cannot substitute for human-annotated responses, even when synthesized speech is high-quality. This provides a concrete empirical data point against a common shortcut in LSLM training data construction.
Field Significance
Moderate — This paper addresses a real data bottleneck in spoken language model training by providing a practical, low-cost pipeline for constructing speech instruction datasets from text QA corpora. Its primary value is empirical: the quantification of the alignment target gap (LLM-continued vs. human-annotated) is a useful result for practitioners designing LSLM training pipelines. The evaluation scope is narrow (English-only, short QA questions, four comprehension benchmarks), which limits how broadly the findings generalise.
Claims
-
supports: Semantic similarity in embedding space can serve as a reliable proxy for linguistic equivalence when filtering TTS-synthesized speech instruction data, avoiding the over-rejection caused by WER-based criteria.
Evidence: The SIM metric achieves over 98% consistency with WER when strict linguistic equivalence holds, and above 92.5% across all conditions and three TTS models on seven QA datasets. (§5.2, §5.3, Table 1)
-
supports: Training large speech language models on LLM-continued alignment targets produces systematically weaker instruction-following capability than training on human-annotated responses, regardless of the speech synthesis quality.
Evidence: Models trained with LLM-continued answers achieve 52.54 average ROUGE-L vs. 60.5 with human-annotated golden answers at identical data quality thresholds; the gap persists across all ablation settings. (§5.3, Table 3)
-
supports: Parallel query rewriting with multiple structurally diverse LLMs improves TTS data usability compared to any single-LLM rewriting approach, due to complementary performance across text types.
Evidence: Using Llama-3, Phi-3, and Qwen2 together raises average Pass rate by 2–3 percentage points over the best single-LLM baseline across MeloTTS and ParlerTTS-Large in the single-speaker setting. (§5.4, Table 2)
-
complicates: WER-based quality filtering of TTS-synthesized speech instruction data causes inappropriate data rejection due to incidental ASR errors, distorting the resulting dataset distribution.
Evidence: Strict WER filtering rejects samples where ASR recognition errors occur despite semantic equivalence; SIM-based filtering recovers these samples while maintaining downstream task performance, as confirmed by case analysis in Table 13. (§3, §5.3, Table 13)
Limitations and Open Questions
The framework is validated only on short questions (under 100 words); the authors note that constructing spoken instructions from longer texts through rewriting remains an open problem. LSLM evaluation covers only four reading comprehension tasks measured with ROUGE-L, which may not reflect instruction-following quality in conversational or multi-turn settings. All experiments are English-only. The full multi-speaker pipeline requires approximately 558 GPU-hours on a single NVIDIA A40, which is non-trivial for reproduction at scale (Table 8).
Wiki Connections
- Spoken Language Model — this paper directly targets a training data bottleneck for spoken language models, proposing a pipeline to construct high-quality speech instruction datasets without human audio collection.
- Evaluation Metrics — the paper introduces the SIM metric, an averaged embedding-space cosine similarity score, as a more robust alternative to WER for assessing the linguistic fidelity of TTS-synthesized training data.
- Subjective Evaluation — human annotation is used to validate rewriting quality on 100 samples per dataset, confirming that automatic SIM improvements translate to linguistically consistent speech (Table 10).
- Instruction-Conditioned TTS — the multi-speaker pipeline uses Parler-TTS-Large conditioned on GPT-4-generated natural language speaker style descriptions to produce stylistically diverse training utterances.