arXiv · 2025 · Preprint
Yifei Cao et al. (Fudan University) · → Paper · Demo: ✗ · Code: ✗
Reformulates six heterogeneous MOS datasets into a unified preference-comparison benchmark and systematically compares scalar, semi-scalar, and generative reward-modeling paradigms for automatic speech quality assessment.
Problem
MOS remains the default way to assess synthetic speech quality, but it is expensive to collect and, because rating scales and listener pools differ across datasets, MOS values are not directly comparable across studies. Automatic MOS predictors trained to regress absolute scores inherit these dataset-specific biases, and prior efforts (bias-aware losses, score-alignment frameworks such as AlignNet) only partially correct for it. Meanwhile, reward modeling, an approach that has matured substantially in text and vision alignment, has seen little systematic exploration for speech: there is no established benchmark for training or comparing reward models that judge speech quality, and it is unclear which reward-modeling paradigm (scalar, language-augmented, or fully generative) is best suited to the task.
Method
The paper builds MOS-RMBench by converting five MOS datasets (BVCC, NISQA, SingMOS, SOMOS, TMHINT-QI) spanning natural speech, TTS, VC, singing voice synthesis/conversion, and speech enhancement, plus the VMC’23 challenge set held out for out-of-domain (OOD) evaluation, into a single preference-comparison format. Rather than using raw MOS values, sample pairs are grouped by shared content, shared speaker/system, or neither, and the sample with the higher MOS is labeled “chosen” against the lower-scoring “rejected” sample; datasets are rebalanced so no single source dominates. This yields 55,333 training, 9,905 development, and 6,240 in-domain test pairs, plus 3,000 OOD test pairs, across six languages.
On top of this benchmark, the paper trains and evaluates three reward-modeling paradigms, all built on the Qwen2-Audio audio-language model backbone. Classic scalar reward models output one quality score per sample, trained with either a Bradley-Terry (BT) pairwise-ranking loss or an MSE regression loss against MOS. Semi-scalar (“Cloud”-style) models first learn to generate a natural-language critique across four quality dimensions (noise, distortion, naturalness, continuity), authored by Gemini-2.5-Pro on the training data, before passing that representation through a reward head. Generative reward models (GRMs) take both members of a pair as joint input, are supervised fine-tuned on Gemini-2.5-Pro-annotated paired critiques and scores, and are then optimized with GRPO or DAPO reinforcement learning using a rule-based “Accuracy reward” that only checks whether the chosen sample was scored higher than the rejected one. To address the observed weakness on similar-quality pairs, the paper adds an MOS-difference reward term (a cosine-shaped function of the normalized ground-truth MOS gap) to the RL objective, producing an MOS-aware GRM that scales the reward magnitude by how difficult each pair is. Baselines include UTMOS as a standalone MOS predictor and LLM-as-a-judge evaluation with Gemini-2.5-Pro and Qwen2.5-Omni-7B, which output a direct pairwise preference without a scalar score.
Key Results
Across MOS-RMBench-Eval, Classic scalar reward models with BT loss reach the highest overall accuracy (80.04%), ahead of semi-scalar Cloud models with BT loss (78.82%) and GRMs (74.63% with SFT alone, rising to 76.94-77.60% with GRPO/DAPO). BT loss consistently beats MSE loss by 4+ points overall, with the advantage widest on the OOD VMC’23 set. UTMOS performs strongly on BVCC (86.70%) but drops sharply on other datasets (60.83-73.17%), and the LLM-as-a-judge baselines (Gemini-2.5-Pro, Qwen2.5-Omni-7B) stay below 70% overall, both well behind every trained reward-model paradigm. A percentile-based error analysis shows all models are markedly weaker on pairs with small MOS differences, with the best model’s error rate exceeding 40% in the lowest-difference bin and dropping steadily as the gap widens. Adding the MOS-difference reward to GRM training improves overall accuracy by roughly 0.5-1 point (to 78.00-78.08%) and yields larger, targeted gains on the hardest pairs (MOS difference ≤ 0.5), for example 57.44% to 60.74% on TMHINT-QI under GRPO and 62.79% to 65.12% on BVCC under DAPO.
Novelty Assessment
The genuinely new contribution is MOS-RMBench itself: no prior benchmark reformulates multiple heterogeneous MOS datasets into a single preference-comparison protocol for training and evaluating speech quality reward models, and doing so sidesteps the cross-dataset scale-inconsistency problem that has hampered absolute-MOS regression. The systematic comparison of scalar, semi-scalar, and generative reward paradigms is an empirical contribution rather than an architectural one; all three paradigms reuse the Qwen2-Audio backbone and RL recipes (GRPO, DAPO) imported directly from text-domain LLM alignment work. The MOS-aware reward function is a modest, well-targeted refinement of the RL objective (a cosine-weighted difficulty term added to an existing rule-based reward) rather than a new training paradigm, and its gains, while consistent, are small in absolute terms (roughly 0.5-3 points depending on the difficulty subset).
Field Significance
Moderate — this paper establishes the first systematic, cross-dataset benchmark and evaluation protocol for speech quality reward modeling, filling a gap the authors correctly identify as underexplored, and provides a useful empirical ranking of reward-modeling paradigms with a concrete methodological baseline (Classic scalar + BT loss) for future work to beat. Its scope is narrow (reward modeling for quality assessment rather than generation itself) and the headline accuracy numbers (74-80%) leave clear room for improvement, so its influence is currently confined to a specific evaluation sub-problem rather than a shift in how speech generation systems are built.
Claims
- supports: Automatic reward models for speech quality generalize noticeably worse to synthetic speech than to natural or mixed human speech, indicating a persistent domain gap in learned quality judgments.
Evidence: Across all paradigms, accuracy on synthetic-speech-heavy datasets such as SOMOS (74.30-76.80%) and the VC/SVC-heavy VMC’23 OOD set (69.23-77.73%) trails accuracy on BVCC (82.60-86.70%) and NISQA (77.81-83.93%). (§5.1, Table 2)
- complicates: Preference-based reward models for speech quality remain unreliable at fine-grained discrimination, limiting their use for comparing systems of similar quality.
Evidence: A percentile-based error analysis shows error rates above 40% for the top-performing scalar model on the smallest-MOS-difference bin, declining sharply only as the MOS gap widens. (§5.2, Figure 3)
- supports: Explicitly weighting a reward model’s training signal by the magnitude of the ground-truth quality gap improves its ability to discriminate perceptually similar speech samples.
Evidence: Adding a cosine-shaped MOS-difference reward term to GRM training raises accuracy on hard pairs (MOS difference ≤ 0.5) from 57.44% to 60.74% on TMHINT-QI (GRPO) and 62.79% to 65.12% on BVCC (DAPO), while improving overall accuracy by 0.48-1.06 points across all six evaluation sets. (§5.3, Table 3, Figure 4)
- complicates: General-purpose audio-language models used directly as pairwise judges are unreliable proxies for speech quality assessment without task-specific reward training.
Evidence: Gemini-2.5-Pro and Qwen2.5-Omni-7B, evaluated as LLM-as-a-judge, both score below 70% overall accuracy on MOS-RMBench-Eval, trailing every trained scalar, semi-scalar, and generative reward model by 5-19 points. (§5.1, Table 2)
Limitations and Open Questions
MOS-RMBench covers only six languages and focuses on overall perceptual quality, leaving prosody, emotion, and style-level judgments unmodeled. The authors also note that converting absolute MOS scores into pairwise preferences changes the original score distribution, which may affect how models learn to reason about absolute quality levels rather than relative ones. Code and the benchmark dataset were not yet released at the time of writing, and no explicit parameter counts are given for the Qwen2-Audio-based reward models used throughout.
Wiki Connections
- Evaluation Metrics — introduces MOS-RMBench, a benchmark that converts absolute MOS scores from six datasets into a unified preference-comparison protocol to enable consistent cross-dataset evaluation of automatic speech quality metrics.
- Subjective Evaluation — directly addresses inconsistency and poor reproducibility in MOS-based human listening test data by reformulating it as pairwise preferences rather than absolute scores.
- RLHF Speech — trains generative reward models for speech quality via GRPO and DAPO reinforcement learning against a rule-based accuracy reward, with an added MOS-difference reward term for difficulty-aware training.
- Qwen2-Audio — used as the shared audio-language model backbone for all scalar, semi-scalar, and generative reward models evaluated in this paper.
- Qwen2.5-Omni — evaluated as one of two LLM-as-a-judge baselines, which underperform all trained reward-model paradigms.
- Gemini 2.5 — used both as the annotator generating natural-language critiques and paired quality scores for training data, and as an LLM-as-a-judge baseline.
- UTMOS — evaluated as a standalone MOS-prediction baseline, showing strong in-domain performance but poor cross-dataset generalization compared to the paper’s reward models.
- DeepSeekMath — source of the GRPO reinforcement learning algorithm adapted here to train generative reward models on paired speech quality judgments.