arXiv · 2022 · Preprint
Saeki et al. (University of Tokyo) · → Paper · Demo: ? · Code: ✓
UTMOS is an ensemble MOS prediction system combining SSL fine-tuning with weak learners that ranked first on multiple metrics in VoiceMOS Challenge 2022 and became the community’s primary automatic speech quality predictor.
Problem
Subjective listening tests are the gold standard for TTS evaluation but are expensive and slow. Prior neural MOS predictors lacked generalization across different listening test conditions and underperformed on out-of-domain data. The VoiceMOS Challenge 2022 framed this as a formal prediction task with both in-domain (main track) and out-of-domain (OOD) conditions, motivating development of robust, generalizable automatic predictors.
Method
UTMOS combines two tiers of learners in a stacking ensemble. The strong learners fine-tune a pretrained SSL model (wav2vec 2.0 base) with several enhancements over the SSL-MOS baseline. Rather than averaging frame-level SSL features into a single utterance embedding, UTMOS feeds them through bidirectional LSTM layers and computes frame-level scores, averaging only at inference time. This frame-level regression target is combined with a contrastive loss that penalizes pairwise ranking errors between utterances, which is specifically suited to improving Spearman correlation metrics.
To handle variation across listeners and listening test domains, the model concatenates listener and domain embeddings with SSL features before prediction. During training, a “mean listener” embedding is also included as a target; at inference this is used since individual listener identity is unknown. A phoneme encoder (BLSTM over ASR-derived phoneme sequences) provides an auxiliary linguistic input, with the reference phoneme sequence estimated by clustering ASR outputs using DBSCAN and Levenshtein distance when ground-truth text is not available. Data augmentation via speaking-rate perturbation and pitch-shifting addresses the limited size of the OOD training set (136 labeled utterances).
The weak learners use mean-pooled SSL embeddings from eight pretrained models (four wav2vec 2.0 variants, two HuBERT, two WavLM) fed into six families of classical regressors (ridge, SVR, random forests, LightGBM, kernel SVR, Gaussian process regression), yielding 48 weak learners per domain. The final ensemble uses multi-stage stacking: stage 0 predictions from strong and weak learners are combined by a meta-learner, which is itself combined at stage 3. For the OOD track, separate weak learner sets are trained per domain and integrated at stage 2.
Key Results
On the VoiceMOS 2022 main track, UTMOS ranked first in utterance-level MSE (0.165), utterance-level SRCC (0.897), and system-level MSE (0.090), and third in system-level SRCC (0.936) among 21 teams. On the OOD track, it ranked first in utterance-level MSE (0.162), system-level MSE (0.030), and system-level SRCC (0.988), and second in utterance-level SRCC (0.893) among 15 teams.
Ablation results (Tables 2 and 3 in the paper) show that listener dependency is the most impactful component: removing listener IDs causes system-level SRCC to drop from 0.972 to 0.931 on OOD and degrades utterance-level MSE substantially on both tracks. Contrastive loss helps more on the OOD track than on the main track, particularly for ranking metrics. External data collection (having Chinese annotators rate the 540 unlabeled OOD utterances) provides a clear boost for OOD performance. The phoneme encoder shows mixed results: slightly degrading main-track metrics while improving OOD.
Stacking ensemble results (Tables 4 and 5) confirm that combining more strong and weak learners consistently lowers MSE on the main track, with the 17-strong + 48-weak configuration achieving utterance-level MSE of 0.165 vs. 0.216 for a single strong learner alone.
Novelty Assessment
The individual components — SSL fine-tuning for MOS prediction, listener-dependent modeling, contrastive loss for ranking, and ensemble stacking — each draw on prior work. The paper’s contribution is their careful combination and evaluation under a controlled challenge setting, plus the introduction of phoneme encoding as an auxiliary input to leverage linguistic intelligibility cues. The frame-level loss formulation (extending the scalar MOS target across frames) is a straightforward but effective modification to the SSL-MOS baseline. The external data collection for the OOD track is a practical contribution relevant to low-resource evaluation settings. Overall this is an engineering-integration contribution that demonstrated strong empirical results within the challenge framework.
Field Significance
Tip
High — UTMOS emerged from VoiceMOS 2022 as the community’s reference automatic MOS predictor. Its public code release and strong correlation with human judgments make it the most frequently used automatic speech quality metric in TTS evaluation pipelines, cited extensively across the corpus. For researchers who need a proxy for subjective quality without running listening tests, UTMOS provides a reliable and reproducible baseline that generalises beyond the challenge’s original data distribution.
Claims
- Listener-dependent modeling substantially improves MOS prediction accuracy, particularly for out-of-domain data with limited labeled samples. (§3.1.3, Table 2b, Table 3b)
- Contrastive pairwise ranking losses improve Spearman correlation in MOS prediction more than standard regression losses alone. (§3.1.2, §4.3, Table 2)
- SSL-based features from multiple heterogeneous pretrained models (wav2vec 2.0, HuBERT, WavLM) combined via ensemble stacking yield more robust MOS predictions than any single model. (§3.3, §4.4, Tables 4–5)
- Phoneme-level linguistic inputs improve MOS prediction robustly in low-resource conditions but may not benefit in-domain prediction when sufficient labeled data is available. (§3.1.4, §4.3, Table 2)
Limitations and Open Questions
Warning
The UTMOS model is trained and evaluated exclusively on synthetic speech data from Blizzard and Voice Conversion Challenges; its generalization to modern neural TTS systems (LLM-based, flow-matching) with different quality failure modes is not characterized.
The OOD evaluation uses Chinese synthetic speech while the strong learner SSL backbone (wav2vec 2.0) was pretrained primarily on English. Generalization to other languages depends on the SSL model’s cross-lingual transfer capability, which is not analyzed here. The external data collection for OOD (32 Chinese listeners, ~2 ratings per utterance) is low-coverage and introduces annotation noise. The paper proposes larger-scale general-purpose data collection as future work but does not address it here. Additionally, data augmentation (pitch-shifting, time-stretching) is assumed not to change perceptual quality, but this assumption is not verified through listening tests.
Wiki Connections
Concept pages this paper most directly informs: evaluation-metrics, subjective-evaluation, self-supervised-speech.
Related in-corpus paper cited: 1412.6980 (Adam optimizer used in training).
2306.00814 — Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis