arXiv · 2026 · Preprint
Wei Wang et al. (Shanghai Jiao Tong University / Carnegie Mellon University / TU Braunschweig / Meta / Waseda University / VUI Labs) · → Paper · Demo: ? · Code: ?
Introduces UrgentMOS, a unified speech quality assessment model that jointly learns to predict a broad set of objective and perceptual quality metrics while explicitly tolerating missing labels for any subset of metrics, and additionally predicts comparative MOS (CMOS) preferences via a naturalness-conditioned pairwise module trained on preference pairs derived automatically from absolute MOS data.
Problem
Automatic speech quality assessment has become essential for evaluating generative speech systems (synthesis, enhancement, coding) as human listening tests remain costly and hard to scale. But existing learning-based predictors rely mainly on scarce human-annotated MOS data, which is often poorly aligned across datasets due to differing annotation protocols and listener populations, limiting robustness when trained on heterogeneous data. Separately, as synthesized speech quality approaches natural-sounding levels, absolute rating scales saturate and perceptual differences between systems become too subtle for listeners (or single-score predictors) to reliably discriminate, motivating a shift toward comparison-based evaluation that directly assesses relative quality. No prior framework unifies robust multi-source absolute-score training with native comparative/preference prediction in one model.
Method
UrgentMOS has two coupled modules operating on a shared feature representation. A multi-branch, architecture-agnostic feature extractor processes each input waveform through several heterogeneous pretrained encoders in parallel (WavLM, Kimi-Audio, Qwen3-Omni-Captioner, Audio Flamingo), each producing layer-wise representations optionally aggregated by a weighted sum; all representations are then temporally aligned via length interpolation to a common sequence length and fused into a unified representation shared by both downstream modules.
The Absolute Metric Prediction Module (AMPM) organizes 15 tracked speech-quality metrics into five semantically coherent groups (Noise & Distortion, Naturalness, Intelligibility, Speaker Characteristics, Spectral Accuracy), each with a shared metric encoder producing a group-specific latent representation and metric-specific prediction heads producing individual scalar predictions. Range-Constraining Activations are applied after each head (a scaled sigmoid for bounded metrics, softplus-based transforms for one-sided bounds) to keep predictions within each metric’s valid numerical range while preserving smooth gradients near the boundaries, unlike hard clipping. Because different training datasets provide different, often incomplete, subsets of these 15 metrics (intrusive metrics need reference signals; MOS labels are scarce at scale), UrgentMOS masks unavailable labels with a per-sample binary validity mask and computes each metric’s regression loss only over valid samples, normalized by the count of valid samples, so no training example needs to be discarded for missing annotations.
The Naturalness-Conditioned Preference Module (NCPM) operates specifically on the naturalness-related metric group’s representations (which include human MOS and learned naturalness predictors) for a pair of speech samples, using cross-attention between the paired representations to produce comparative predictions such as a preference label or comparative MOS (CMOS). Because dedicated comparative (CCR-style) datasets are scarce relative to abundant absolute-score (ACR) datasets, UrgentMOS derives additional preference supervision automatically from ACR data: given two samples’ MOS scores, a preference label (A≻B, B≻A, or tie) is assigned based on whether their score difference exceeds a tunable threshold δ, under three matching strategies of increasing strictness (arbitrary pairing, same-dataset pairing, or same-reference-content-different-system pairing). To encourage order-invariant preference predictions without constraining the network architecture, each preference-labeled pair is duplicated in reversed order with a correspondingly reversed label, as an explicit symmetric data-augmentation step rather than an architectural constraint.
Key Results
Trained across a diverse collection of TTS, VC, speech-enhancement, and simulated-distortion datasets (spanning both absolute-score and preference-labeled data), UrgentMOS consistently outperforms conventional single-target MOS predictors (DNSMOS, UTMOS, SCOREQ, Distill-MOS, NISQA-MOS) and dataset-specific preference models (SpeechEval, SpeechJudge trained only on their own benchmarks) on both correlation-based (LCC/SRCC) and preference-accuracy metrics across SOMOS, TMHINT-QI, URGENT24-SQA, BVCC, NISQA-FOR, SpeechEval, and SpeechJudge test sets. Dataset-specific models (SpeechEval, SpeechJudge) show strong in-domain performance but noticeably weaker generalization to other domains, while UrgentMOS variants maintain consistently strong performance across domains. Most existing MOS predictors perform worse than random guessing on the tie-allowing accuracy metric for the SpeechJudge test set (synthesized speech from recent TTS systems), because they compress predicted scores for high-quality modern samples into a narrow band, producing ambiguous tie predictions that are penalized on a test set without tie annotations. Naturalness-focused multi-metric supervision (grouped metrics, C5/M5) consistently outperforms MOS-only supervision (C1/M1), but extending supervision to all 15 tracked metrics (M15) does not always help and can degrade performance, particularly with a single shared feature encoder, likely due to weak or inconsistent correlations among some metrics. The choice of preference-pair construction strategy (reference-level, corpus-level, or unconstrained matching) has limited impact on downstream accuracy, suggesting the fixed tie threshold is sufficient to separate perceptually distinct samples even under cross-corpus annotation-scale inconsistencies. Using four feature extractors instead of one (F4 vs. F1) provides consistent, if modest, gains across nearly all evaluation datasets.
Novelty Assessment
The architectural contribution is a specific, well-motivated combination rather than a single new mechanism: a metric-grouped multi-task prediction module (AMPM, adapted from but architecturally and formulation-wise distinct from prior multi-metric work Uni-VERSA) with an explicit missing-label-tolerant masked loss, coupled to a cross-attention preference module (NCPM) that conditions specifically on naturalness-related representations rather than the full metric space. The automatic derivation of comparative preference supervision from existing absolute MOS datasets, via a tunable tie threshold and multiple matching strategies, is a practical and reusable technique for addressing the scarcity of dedicated CCR-style datasets. The released preference-annotated dataset derived from existing ACR data is a concrete artifact for future comparative speech-quality benchmarking work. Individual components (range-constraining activations, symmetric pair augmentation) are incremental engineering refinements rather than conceptually new ideas, but the overall system is validated with a broad and carefully designed ablation matrix (feature-extractor count, metric grouping, supervision scope, pairing strategy).
Field Significance
high — this paper demonstrates both that heterogeneous multi-metric supervision with explicit tolerance for missing labels improves robustness over MOS-only training, and that preference/comparative supervision can be derived automatically and at scale from existing absolute MOS datasets rather than requiring dedicated comparative annotation efforts, directly addressing the increasingly important problem (documented in this paper's own results) that single-score MOS predictors lose discriminative power on today's near-natural-quality synthesized speech.
Claims
- supports: Jointly training a speech-quality predictor on multiple heterogeneous objective and perceptual metrics, while explicitly tolerating missing labels for arbitrary metric subsets per sample, improves robustness and cross-domain generalization compared to training solely on scarce human MOS annotations.
Evidence: UrgentMOS variants trained with naturalness-related multi-metric supervision consistently outperform MOS-only-supervised counterparts and conventional single-metric predictors (DNSMOS, UTMOS, SCOREQ, Distill-MOS, NISQA-MOS) across correlation and preference-accuracy benchmarks spanning TTS, VC, speech-enhancement, and simulated-distortion domains. (§5.1-5.2, Tables 4-5)
- complicates: Existing single-target MOS predictors lose discriminative sensitivity on modern, high-quality synthesized speech, where perceptual differences between systems become too subtle for the predictors’ typical output range to separate.
Evidence: Most existing MOS predictors perform worse than random guessing under a tie-allowing preference-accuracy metric on the SpeechJudge test set (recent TTS systems), because they compress scores into a narrow band, producing ambiguous tie predictions penalized against a test set with no tie annotations. (§5.1)
- complicates: Expanding supervision to a large, heterogeneous set of quality metrics does not monotonically improve a unified speech-quality predictor, and can degrade it when correlations among the added metrics are weak or inconsistent.
Evidence: Extending supervision from naturalness-focused metrics to all 15 tracked metrics does not consistently improve, and in the single-shared-encoder configuration measurably degrades, performance on several evaluation datasets relative to the naturalness-only variant. (§5.1-5.2)
- supports: Preference labels for comparative speech-quality assessment can be derived automatically from existing absolute MOS annotations via a tunable tie threshold, producing training supervision that is largely insensitive to whether sample pairs are matched within a corpus, across corpora, or by shared reference content.
Evidence: UrgentMOS variants trained with reference-level, corpus-level, or unconstrained pairing strategies achieve comparable preference-prediction accuracy across evaluation datasets, indicating a fixed δ=0.5 threshold suffices to separate perceptually distinct samples even under cross-corpus annotation-scale inconsistencies. (§5.1)
- supports: Incorporating multiple complementary pretrained audio feature extractors, rather than relying on a single encoder, provides consistent gains for speech-quality prediction, at the cost of increased inference overhead.
Evidence: The four-feature-extractor variant outperforms the single-extractor counterpart on both preference accuracy and correlation metrics across nearly all evaluated datasets, though the paper’s own limitations note this increases inference cost, potentially restricting use in latency-sensitive settings. (§5.1-5.2, Tables 4-5; Limitations)
Limitations and Open Questions
The authors’ own limitations note that UrgentMOS does not provide natural-language explanations for its quality judgments, unlike recent language-based evaluation approaches (PAM, SpeechLLM-as-Judges), though its structured multi-metric predictions could support downstream reasoning modules. Incorporating multiple feature extractors improves robustness but increases inference cost, potentially limiting use in large-scale or latency-sensitive data-filtering scenarios. The authors also flag that identifying which metric subsets to include, or developing training strategies that better resolve conflicts between weakly- or inconsistently-correlated metrics, remains an open problem, since naively including all 15 tracked metrics does not reliably outperform a curated naturalness-focused subset.
Wiki Connections
- Evaluation Metrics — introduces a unified speech-quality assessment model spanning multi-metric absolute prediction and comparative/preference prediction, trained across TTS, VC, and speech-enhancement quality datasets with a missing-label-tolerant objective.
- Seed-TTS — cited as an example of the class of modern high-quality speech generation systems that motivate the need for more discriminative, preference-based quality assessment.
- Audio Flamingo 3 — one of the pretrained audio encoders used as a heterogeneous feature extractor in UrgentMOS’s multi-branch feature-extraction stage.
- Kimi-Audio — one of the pretrained audio encoders used as a heterogeneous feature extractor in UrgentMOS’s multi-branch feature-extraction stage.
- F5-TTS — cited as an example of modern flow-matching-based TTS systems producing the near-natural-quality speech that motivates comparison-based evaluation.
- SpeechLLM-as-Judges — cited as a complementary language-based evaluation approach (natural-language explanations) that UrgentMOS’s structured multi-metric predictions could support but does not itself provide.