ASRU · 2025 · Conference
Wang et al. (National Taiwan Normal University) · → Paper · Demo: ? · Code: ✓
QAMRO augments standard regression-based MOS prediction with a pairwise ranking loss that adapts its margin to the perceptual score gap between samples and up-weights pairs involving high-quality audio, improving system-level correlation with human judgments across TTS, TTM, and TTA.
Problem
Automatic MOS prediction has primarily been framed as a regression problem, with training objectives such as MAE or MSE minimising absolute prediction errors. These losses ignore the relative ordering of samples, which is often more perceptually consistent than absolute scores across annotators and datasets. Standard ranking losses that have been applied in adjacent fields use fixed margins and treat all sample pairs uniformly, discarding information about the perceptual distance between samples and the particular importance of high-quality content in discrimination tasks.
Method
QAMRO introduces a pairwise ranking loss with two enhancements over the standard margin ranking loss. First, the margin is made data-dependent: it scales with the absolute difference between ground-truth MOS scores for a pair, so pairs that are far apart in perceived quality incur a larger penalty when their relative ordering is violated. Second, a quality-aware weighting term assigns higher weight to pairs that include at least one high-quality sample, reflecting the intuition that ranking errors near the top of the quality scale are more costly in practice.
The framework is evaluated on two backbone models. For text-to-music (TTM) assessment, CLAP is used: audio clips are encoded by the CLAP audio encoder and prompts by the CLAP text encoder, with separate MLP heads predicting musical impression (audio only) and text alignment (audio + text). The training objective combines the QAMRO loss with Huber regression. For the broader multi-task setting covering TTS, TTA, and TTM, the pre-trained Audiobox-Aesthetics model serves as backbone, with four MLP heads predicting production quality, production complexity, content enjoyment, and content usefulness. All models are trained exclusively on the AudioMOS Challenge 2025 closed-set data (MusicEval for TTM; AES-Natural for the general task). Hyperparameters α = 0.2 (margin scaling) and β = 7.0 (quality preference weight) are selected by validation performance.
Key Results
On MusicEval, QAMRO achieves SRCC scores of 0.972 (musical impression) and 0.916 (textual alignment), compared to 0.845 and 0.779 for the MusicEval baseline that also uses Huber regression, making this a fair comparison on the same backbone. On AES-Natural, QAMRO reaches SRCC of 0.883, 0.942, 0.869, and 0.852 across the four evaluation dimensions, versus 0.864, 0.938, 0.845, and 0.809 for the Audiobox-Aesthetics baseline. Gains are most pronounced on content usefulness (0.052 SRCC improvement), which is also the hardest dimension for baselines.
Ablation studies confirm both components contribute independently: removing quality-aware weighting causes a noticeable drop, and further removing adaptive margin results in an additional degradation. Results hold across both backbone architectures, suggesting the approach is model-agnostic.
Novelty Assessment
The QAMRO loss function is a sensible and well-motivated extension of the standard margin ranking loss. Both constituent ideas — score-gap-proportional margins and quality-upweighting of pairs — have analogues in the learning-to-rank literature outside audio. The novelty lies in applying them to the perceptual audio quality estimation problem and demonstrating that they transfer well there. The paper does not propose a new backbone architecture or a new evaluation protocol; the AudioMOS Challenge 2025 dataset and evaluation setup are external. The contribution is therefore a targeted training objective improvement for an existing class of MOS prediction systems, with clean ablations that isolate the components and honest comparison against baselines using the same regression objective.
Field Significance
Moderate — QAMRO offers a concrete, reproducible improvement over regression-only MOS predictors by injecting ranking structure into training, a methodological gap that had been underexplored in automatic speech and audio quality evaluation despite being well-studied in retrieval and detection. The gains are consistent across backbone architectures and audio types, adding credibility beyond a single model evaluation. The closed-set evaluation limits claims about generalisation to out-of-distribution systems, but the direction is likely to influence future automatic evaluator training in the field.
Claims
- Incorporating pairwise ranking objectives into MOS prediction training improves system-level ranking consistency with human judgments beyond what standard regression losses achieve. (§II.A, Table I)
- Adaptive margins that scale with perceptual score gaps outperform fixed-margin ranking losses for audio quality assessment. (§II.A, Table I — “w/o Adaptive Margin” ablation)
- Quality-aware upweighting of high-scoring sample pairs is a consistently beneficial component in automatic perceptual quality evaluation. (§IV.A, §IV.B — ablation results in both tables)
- Automatic MOS predictors built on pre-trained audio-text representations can be meaningfully improved by training objective changes without modifying the backbone architecture. (§II.B, Table II)
Limitations and Open Questions
Warning
All training and evaluation is conducted on the AudioMOS Challenge 2025 closed-set data only. The MusicEval and AES-Natural sets cover limited system diversity and are collected under specific annotation conditions; how well QAMRO-trained predictors generalise to unseen TTS or VC systems outside this distribution is untested.
The paper evaluates only system-level SRCC, which averages predictions over all clips per system before comparing to human-annotated system scores. Utterance-level accuracy and failure modes for individual clips are not examined. The pairwise ranking strategy also scales quadratically with batch size; the paper uses a fixed batch size of 256 and does not discuss scalability to larger training sets. The authors identify listwise ranking as a natural extension that could better capture global ranking structure.
Wiki Connections
Concept pages: evaluation-metrics · subjective-evaluation · rlhf-speech
Related papers: 2410.06885 (F5-TTS, cited as a TTS system; the evaluation framework targets systems of this class)