arXiv · 2025 · Preprint
Wang et al. (Nankai University / Microsoft) · → Paper · Demo: ? · Code: ✓
Proposes SpeechLLM-as-Judges, a paradigm for LLM-based interpretable speech quality evaluation, introducing the SpeechEval multilingual benchmark (32,207 clips, 128,754 annotations) and SQ-LLM, a model trained with chain-of-thought reasoning and reward optimization to assess, compare, suggest improvements for, and detect deepfakes in speech.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Existing automatic speech quality metrics such as MOS predictors and signal-based measures produce scalar scores without explanations, making it difficult to attribute quality failures to specific perceptual dimensions. This paper introduces a framework that treats speech quality evaluation as a structured natural language generation task, enabling dimension-wise reasoning over eight quality sub-dimensions spanning intelligibility, distortion, speech rate, dynamic range, and subjective experience. The SpeechEval dataset provides multilingual training and evaluation coverage across Chinese, English, Japanese, and French, including four task types (quality assessment, pairwise comparison, improvement suggestion, and deepfake detection) with human-annotated natural language explanations rather than scalar labels. SQ-LLM, built on Qwen2.5-Omni-7B with LoRA fine-tuning, achieves the highest Pearson correlation with human overall quality ratings on the assessment task, outperforming specialist MOS predictors such as UTMOS and Audiobox Aesthetics as well as general-purpose audio LLMs evaluated without task-specific tuning.
Wiki Connections
- evaluation-metrics — introduces SpeechEval, a multilingual benchmark with multi-task annotations that extends the scope of automatic speech quality evaluation beyond scalar MOS prediction.
- subjective-evaluation — frames speech quality evaluation as structured natural language generation grounded in human-annotated dimension-level judgments, bridging subjective assessment methodology with automatic evaluation.
- 2502.05139 (Meta Audiobox Aesthetics) — used as a baseline expert MOS predictor and as a speech encoder variant in the comparison study; SQ-LLM outperforms it on overall quality correlation.
- 2407.10759 (Qwen2-Audio) — used as one of the directly evaluated multimodal LLM baselines and as a fine-tuned variant; SQ-LLM substantially improves on it after task-specific training.
- 2305.11000 (SpeechGPT) — cited in the broader context of LLM-based speech understanding as an early example of spoken language modelling capable of multi-turn interaction.