arXiv · 2025 · Preprint
Manakul et al. · → Paper · Demo: ? · Code: ?
A systematic study of Large Audio Model (LAM) as a judge for speech evaluation, finding that audio concatenation combined with in-context learning enables a multi-aspect ensemble to achieve up to 0.91 Spearman correlation with human preferences on a speech-in speech-out system ranking benchmark.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
AudioJudge investigates whether a single large audio model, prompted without any task-specific fine-tuning, can replace the collection of specialized models (UTMOS, MOSANET+, speaker verification systems) that TTS and spoken conversational agent papers currently assemble for evaluation. The paper identifies two evaluation scenarios: example-level audio characteristic detection (pronunciation accuracy, speaking rate, speaker identity, speech quality) and system-level human preference simulation for automated benchmarking. A key finding is that audio concatenation, where test and in-context example audio clips are merged into a single continuous stream before being passed to the LAM, produces substantial gains over naive multi-file prompting, with the best four-shot configuration achieving 66.5% pronunciation accuracy and 0.91 Spearman correlation on the SpeakBench multi-aspect benchmark. The paper also introduces SpeakBench, a 82-instruction paralinguistic-focused benchmark covering pronunciation, speaking style, prosody, and non-linguistic sounds across 13 speech-in speech-out systems, and quantifies verbosity and positional biases in LAM-based evaluation that require mitigation when using this paradigm.
Wiki Connections
- evaluation-metrics — this paper provides evidence that LAM-based pairwise judgment can serve as an alternative to specialized MOS predictors such as UTMOS, offering cross-domain consistency at the cost of increased API latency and verbosity/positional bias.
- subjective-evaluation — AudioJudge proposes system-level human preference simulation as a scalable substitute for costly listening tests, reaching 0.91 Spearman correlation against human judgments on SpeakBench.
- spoken-language-model — the paper evaluates 13 speech-in speech-out systems including end-to-end spoken language models (GPT-4o-Audio, Gemini-2.0-Flash, Moshi, LLaMA-Omni) to validate AudioJudge as a benchmarking tool for this class of system.
- 2410.17196 (VoiceBench) — cited as a benchmark for general voice assistant capabilities; AudioJudge targets the complementary problem of human preference simulation and paralinguistic assessment not captured by objective benchmarks.
- 2410.00037 (Moshi) — included as one of the 13 speech-in speech-out systems evaluated on SpeakBench, where it scores lowest in both automated and human win rates.
- 2409.06666 (LLaMA-Omni) — evaluated as a speech-in speech-out baseline on SpeakBench, where it ranks near the bottom on both automated and human preference scores.
- 2407.10759 (Qwen2-Audio) — evaluated in the cross-modality consistency analysis, where it shows significant performance degradation when moving from text to audio inputs relative to proprietary models.
- 2005.07143 (ECAPA-TDNN) — referenced as a specialized speaker verification model that AudioJudge can potentially replace through prompting-based speaker identification.
- 2410.19168 (MMAU) — cited as an existing audio understanding benchmark that assesses objective capabilities rather than subjective human preferences, motivating AudioJudge’s system-level evaluation approach.
- 2402.07729 (AIR-Bench) — cited as a comprehensive audio reasoning benchmark focused on generative comprehension rather than the paralinguistic preference simulation that SpeakBench targets.
- 2504.12867 (EmoVoice) — cited as a corpus paper using AudioJudge-style evaluation for emotional TTS, one of the prior works that prompted individual task applications without systematic study.