arXiv · 2025 · Preprint
Wen-Chin Huang (Nagoya University) · → Paper · Demo: ? · Code: ✓
A perspective review of four years of the VoiceMOS and AudioMOS Challenge series alongside a survey of open-source speech quality assessment toolkits, arguing that shared benchmarks and accessible implementations are the primary drivers of progress in automatic speech quality assessment.
Problem
Automatic speech quality assessment (SQA) has become increasingly important as generative AI systems proliferate, yet the field has historically suffered from fragmented evaluation protocols, limited labeled data for training predictors, and poor out-of-domain generalisation. Human listening tests remain costly and slow to integrate into development pipelines. The lack of standardised datasets and shared implementations created barriers to both participation and reproducibility, slowing the field relative to adjacent areas like automatic speech recognition. The key question the paper addresses is: what community infrastructure — challenges, datasets, open-source toolkits — has accelerated progress most, and what remains unsolved?
Method
This is a perspective paper, not a system paper; it presents no new model or architecture. The paper reviews the VoiceMOS Challenge (VMC) series from 2022 through 2024 and its 2025 rebranding as the AudioMOS Challenge (AMC), summarising task design, evaluation settings, and key findings across nine tracks. Coverage spans TTS and VC naturalness prediction (English, French, Chinese), singing voice quality, noisy and enhanced speech, text-to-music (TTM), and multi-axis audio aesthetics evaluation.
Challenge evaluation used system-level Spearman rank correlation coefficient (SRCC) and mean squared error (MSE) between system predictions and human MOS ratings as primary metrics. The paper also surveys open-source SQA toolkits — MOSNet, DNSMOS, NISQA, SSL-MOS, UTMOS, TorchAudio-Squim, SHEET, and VERSA — comparing their capabilities across inference support, training support, multi-model integration, and multi-dataset support. VERSA is noted as the most comprehensive, supporting 65 metrics across 729 configuration variants as of publication.
Key Results
In VMC 2022, the best participant system achieved a system-level SRCC of 0.939 on the main English TTS/VC track, versus 0.921 for the top baseline. Fine-tuning self-supervised learning (SSL) models for MOS prediction emerged as the dominant and most effective approach, achieving high correlation even with very small amounts of labeled OOD data (down to 136 training samples). In VMC 2023, the central finding was that no team achieved uniformly strong performance across all three tracks from a single model trained on shared data, confirming that general-purpose MOS prediction across diverse domains remains an open problem. VMC 2024’s “zoomed-in” track — evaluating high-quality systems near human naturalness — proved particularly difficult, indicating that current SQA methods struggle to discriminate among the best modern TTS systems. AMC 2025 attracted 24 teams, the highest across all editions, and the top TTM quality track submission exceeded 0.95 SRCC on both evaluation axes, though organisers noted the in-domain setting was likely too easy.
Note
Challenge results describe the best-performing participant submissions, not a single proposed system. Numbers are not directly comparable across years due to different datasets, tracks, and evaluation protocols.
Novelty Assessment
This paper makes no architectural or modelling contribution. Its value is retrospective and organisational: it is the first consolidated account of the VoiceMOS/AudioMOS Challenge series covering all four editions, and provides the most comprehensive survey of open-source SQA toolkits available at the time of publication. The key insight — that standardised challenges and open-source implementations accelerate field progress more reliably than any single modelling advance — is argued through citation trend data rather than controlled experiments. The paper frames community infrastructure as a first-class research contribution, which is a useful perspective but not an empirical finding.
Field Significance
Moderate — This perspective paper documents the evolution of automatic speech quality evaluation infrastructure across four years of shared benchmarks, providing the field with a reference point for understanding what evaluation dimensions remain hard (cross-domain MOS prediction, high-quality system discrimination) and what is now largely solved (in-domain MOS prediction for conventional TTS). Its contribution is primarily curatorial and organisational rather than technical. The observation that no single SQA model generalises well across all audio domains and listening test conditions has practical implications for any paper relying on a single automatic MOS predictor as its primary evaluation signal.
Claims
- Fine-tuning self-supervised speech representations is sufficient to achieve high MOS prediction correlation in-domain, even with small labeled datasets, but this does not transfer reliably to out-of-domain evaluation settings. (§II.A)
- Automatic MOS predictors consistently fail to discriminate among high-quality speech generation systems whose naturalness approaches human levels, indicating a ceiling effect in current SQA methodology. (§II.C)
- Community challenges with well-documented baselines and open datasets accelerate research progress in evaluation methodology more reliably than individual paper contributions. (§I, §II.E)
- Expanding automatic quality assessment beyond speech to music and general audio requires rethinking evaluation dimensions beyond MOS, as production quality, content enjoyment, and textual alignment reflect distinct perceptual dimensions that do not collapse to a single score. (§II.D, §IV)
Limitations and Open Questions
Warning
The paper’s central argument — that challenges and open-source toolkits drive progress — is supported by citation count trends rather than a controlled study. Alternative explanations (larger models, more data, general SSL pre-training improvements) are not ruled out.
The paper does not provide an empirical comparison of SQA toolkit accuracy or calibration quality. The survey of toolkits (Table II) covers feature availability but not predictive performance on held-out data. Several tracks across the challenge series were identified as too easy (in-domain, strong baseline), raising questions about how to design more discriminative future tracks. The extension to music and general audio quality opens fundamentally different perceptual dimensions that MOS-style scales may not adequately capture. The authors acknowledge that expressive TTS, multilingual speech, and prompt-based TTS remain underserved by current SQA benchmarks.
Wiki Connections
-
evaluation-metrics — This paper surveys the landscape of automatic MOS predictors and discusses when they succeed and fail, directly informing the state of evaluation methodology for speech generation.
-
subjective-evaluation — The VoiceMOS/AudioMOS Challenge series establishes controlled protocols for collecting and benchmarking subjective speech quality ratings, extending the concept of listening test standardisation.
-
voice-conversion — VC quality assessment is a recurring track across VMC editions, and the paper notes particular interest in singing voice conversion (SVCC 2023), making it relevant to VC evaluation methodology.