arXiv · 2025 · Preprint
Ge et al. (Northeastern University, China / NiuTrans Research) · → Paper · Demo: ? · Code: ?
SageLM is an end-to-end audio-native LLM judge that jointly evaluates the semantic and acoustic dimensions of speech-to-speech dialogue responses, trained via rationale-augmented supervised fine-tuning on a new 324,774-instance preference dataset.
Problem
Evaluating end-to-end speech-to-speech (S2S) spoken dialogue systems is unresolved. Human evaluation is the gold standard but is too slow and costly for the rapid iteration cycles needed to develop S2S models. The dominant automated alternative, a cascaded pipeline that transcribes model output with ASR and then judges the transcript with a text LLM, is blind to acoustic information (tone, emotion, prosody) and propagates ASR transcription errors into the judgment. Neither approach is explainable at scale, and no large preference dataset exists that pairs speech responses with judgments spanning both content quality and acoustic appropriateness.
Method
SageLM operates end-to-end on the “LLM-as-a-judge” paradigm: given a textual query/instruction and a pair of candidate speech responses, it directly consumes the audio (rather than an ASR transcript) and outputs a structured comparison across five aspects (truthfulness, honesty, helpfulness, instruction following, and speech instruction following) together with a textual rationale for each judgment. Textual, rather than spoken, queries are used as the fixed input side of the comparison so that evaluation is not confounded by the specific TTS system used to synthesize each S2S model’s training data.
To train the judge, the authors construct SpeechFeedback, a 324,774-instance preference dataset with two components. The semantic component synthesizes UltraFeedback’s text preference data into speech using seven TTS systems (CosyVoice, CosyVoice2, SparkTTS, ChatTTS, F5-TTS, IndexTTS, and gpt-4o-mini-tts), after filtering out math/code/multilingual content and discarding utterances with high Whisper ASR word-error rate or sub-0.2s duration; absolute quality scores are converted to pairwise win/lose/tie labels, and rationales are rewritten into comparative form via Qwen2.5-32B-Instruct. The acoustic component covers three speaking-style categories (emotion, gender, cartoon voice) across three task formats (explicit TTS, explicit dialogue, implicit dialogue), with ground-truth control labels used to construct win/lose/tie pairs and GPT-4o-generated rationales.
The authors first run a preliminary comparison of instruction tuning (full-parameter SFT) against reinforcement learning (GRPO with a rule-based accuracy-plus-format reward) for the judgment task itself, motivated by the “easy verification, hard generation” property that has favored RL in other domains. Based on this analysis (described in Key Results below), SageLM’s production training uses a two-stage, rationale-augmented SFT curriculum rather than RL: stage one trains on the abundant semantic preference data (four aspects), and stage two adds the smaller acoustic preference data, introducing the fifth (speech instruction following) aspect while preserving stage-one semantic performance. The backbone is instantiated from either Qwen2-Audio-Instruct-7B or Qwen2.5-Omni-3B/7B; the best-performing configuration (Qwen2.5-Omni-7B-SFT) is designated SageLM.
Key Results
On a human-annotated semantic test set (728 instances), SageLM (Qwen2.5-Omni-7B-SFT) reaches 82.79% average agreement with human judgments across the four semantic aspects, exceeding the strongest cascaded baseline (Whisper + GPT-4o, 75.37% average agreement) by at least 7.42 points and the best speech-to-text LLM baseline (Qwen2.5-Omni-7B prompted directly) by at least 26.20 points (§Experimental Results, Table 1). On the acoustic test set (410 instances) covering emotion, gender, voice, implicit emotion, and mixed control, SageLM reaches agreement rates of 91–100% across aspects, far above the uninstructed S2T LLM baselines that score 43–57% (Table 2); cascaded ASR-based baselines are not applicable here since transcription discards acoustic information entirely.
A preliminary ablation directly compares GRPO (RL) against SFT for the judgment task: GRPO outperforms label-only SFT on strict scoring accuracy, but on pairwise comparison (the format SageLM actually uses), GRPO and SFT are comparable, and SFT augmented with GPT-4-generated rationales outperforms both (§RL or SFT? Empirical Analysis, Figure 3). A manual audit of 100 GRPO outputs found the generated rationale contradicted the model’s own final judgment label in 39% of cases, a failure mode not present with rationale-supervised SFT.
On out-of-distribution data (AlpacaEval prompts with S2S responses from Kimi-Audio and Qwen2.5-Omni, evaluated per VoiceBench methodology), SageLM surpasses the Whisper + GPT-4o cascade by 18.67 points in agreement and 27.21 points in accuracy (Table 3), with a case study attributing part of the cascade’s failure to Whisper mistranscribing Kimi-Audio’s speaking rate and prosody, which biased GPT-4o’s downstream judgment. Agreement with human/GPT-4o annotations declines slightly as the combined length of the compared response pair increases, and remains stable up to roughly 60 seconds of combined audio (the model’s truncation window) before degrading further (§Analysis, Figures 5 and 7).
Novelty Assessment
The architectural contribution is modest: SageLM fine-tunes existing audio-capable LLM backbones (Qwen2-Audio, Qwen2.5-Omni) rather than proposing a new architecture, placing it in the same lineage as prior audio-native judge/reward models such as WavReward. The paper’s genuine contributions are (1) SpeechFeedback, a substantially larger and more diverse speech preference dataset than prior work, spanning both semantic and acoustic dimensions with structured five-aspect labels; and (2) a direct empirical comparison of RL and SFT for this specific judgment task, yielding a negative result for rule-based RL (reward-label inconsistency, no benefit on the pairwise-comparison format actually used) that runs against the general expectation that RL suits tasks with an easy-verification/hard-generation gap. The multi-aspect, explainable, jointly-semantic-and-acoustic framing is an incremental but meaningful expansion over single-score reward models in the same space.
Field Significance
moderate — SageLM extends the small but growing line of audio-native LLM judges for spoken dialogue evaluation with a larger, more structured preference dataset and an explicit accounting of why rationale-supervised SFT was chosen over RL for this task. It provides a concrete, testable data point on RL-versus-SFT trade-offs for LLM-as-judge training in the speech modality, and demonstrates generalization from synthetic training data to real S2S model outputs on an out-of-distribution benchmark.
Claims
- supports: End-to-end audio-native LLM judges that consume speech directly outperform cascaded ASR-then-text-LLM pipelines for evaluating spoken dialogue responses, because cascaded pipelines discard acoustic information and propagate transcription errors into the final judgment.
Evidence: SageLM (Qwen2.5-Omni-7B-SFT) reaches 82.79% average agreement with human judgments on the semantic test set, exceeding the strongest cascaded baseline (Whisper + GPT-4o) by 7.42 points; a case study shows Whisper mistranscribed Kimi-Audio’s speaking rate and prosody, biasing GPT-4o’s downstream comparison against it. (§Experimental Results, §Analysis, Table 1, Table 3)
- refines: The general expectation that reinforcement learning outperforms supervised fine-tuning on tasks with an easy-verification/hard-generation gap does not hold for training an LLM judge on pairwise speech-response comparison, where rationale-augmented SFT matches or exceeds rule-based RL.
Evidence: GRPO (RL) beats label-only SFT on strict scoring accuracy but is only comparable to SFT on pairwise comparison accuracy/agreement, and SFT augmented with GPT-4-generated rationales outperforms both GRPO and label-only SFT on the pairwise format actually used by the judge. (§RL or SFT? Empirical Analysis, Figure 3)
- complicates: Rule-based RL reward signals for LLM judgment tasks can decouple the model’s generated rationale from its own predicted label, undermining explainability even when label accuracy is comparable to alternatives.
Evidence: A manual analysis of 100 GRPO-trained judgment outputs found the generated rationale contradicted the model’s final win/lose/tie label in 39% of cases. (§RL or SFT? Empirical Analysis)
- complicates: Audio-native LLM judges’ agreement with human preference degrades as the total duration of the compared speech responses grows, so a fixed audio context window imposes a practical ceiling on reliably judgeable response length.
Evidence: On both the 728-example in-domain test set and a supplementary 100-example set spanning response-length intervals, agreement with human/GPT-4o annotations declines slightly as combined response-pair duration increases; with a 60-second truncation window, agreement is stable up to about 60s combined length and declines beyond it. (§Analysis, Figure 5, Figure 7)
- supports: Synthesizing speech preference data by rendering an existing large-scale text preference corpus through multiple heterogeneous TTS systems, then filtering by ASR-measured intelligibility, is a viable way to bootstrap training data for speech evaluation models without collecting new human speech-preference annotations at scale.
Evidence: SpeechFeedback derives 324,774 preference instances from UltraFeedback by synthesizing responses through seven TTS systems (CosyVoice, CosyVoice2, SparkTTS, ChatTTS, F5-TTS, IndexTTS, gpt-4o-mini-tts) and discarding utterances with high Whisper ASR word-error rate or duration below 0.2s. (§Data Construction and Pre-processing)
Limitations and Open Questions
The evaluation protocol deliberately uses textual, not spoken, queries as the fixed input side of every comparison, so SageLM has not been validated on judging fully spoken-query-to-spoken-response interactions, which is the setting most S2S systems are ultimately deployed in.
Agreement degrades for longer response pairs, and the model’s fixed input window caps reliable evaluation at roughly 60 seconds of combined audio, limiting applicability to longer-form spoken interactions. The authors also note SageLM has not been extended to multi-turn dialogue history, multilingual or code-switched speech (constrained by current TTS capability for code-switching), or full-duplex conversation, all left as future work.
Wiki Connections
- Spoken Language Model — SageLM adapts existing speech-text LLM backbones (Qwen2-Audio, Qwen2.5-Omni) to consume external speech audio directly for a judgment task rather than for generation.
- Evaluation Metrics — introduces SpeechFeedback and a five-aspect Accuracy/Agreement evaluation protocol as an alternative to cascaded ASR-based and human-only evaluation of spoken dialogue systems.
- RLHF Speech — provides a direct empirical comparison of rule-based RL (GRPO) against rationale-augmented SFT for training a speech-domain judge, finding RL prone to reward-hacking-like rationale/label inconsistency.
- WavReward — closely related prior audio-native reward/judge model on a similar Qwen2.5-Omni backbone; SageLM extends the approach to five aspects with explicit rationale supervision and a larger preference dataset.
- Kimi-Audio — one of the real S2S LLMs whose outputs SageLM judges in the out-of-distribution generalization experiment on AlpacaEval.
- VoiceBench — supplies the evaluation methodology used to construct SageLM’s out-of-distribution test comparing Kimi-Audio and Qwen2.5-Omni responses.
- Qwen2-Audio — used both as a judge backbone for SageLM’s fine-tuned variants and, in its instruction-tuned form, as a direct-inference baseline that SageLM substantially outperforms.
- CosyVoice — one of seven TTS systems used to synthesize the semantic portion of the SpeechFeedback preference dataset.
- F5-TTS — one of seven TTS systems used to synthesize the semantic portion of the SpeechFeedback preference dataset.
- FLEXI — companion full-duplex benchmarking work by an overlapping author team, cited as context for the full-duplex evaluation challenges SageLM leaves to future work.