arXiv · 2025 · Preprint
Hu et al. (Zhejiang University) · → Paper · Demo: ? · Code: ?
A MoE-based multi-task MOS prediction framework, augmented with synthetic speech data from commercial TTS systems, achieves top system-level accuracy in a competitive evaluation while revealing that utterance-level prediction improvements remain elusive despite architectural and data-augmentation enhancements.
Problem
Automatic MOS prediction exhibits large performance gaps between evaluation granularities. System-level prediction, which averages scores across utterances to rank synthesis systems, is relatively tractable; utterance-level prediction, which requires modelling fine-grained quality variation within a single sample, remains substantially harder. Existing self-supervised speech models fine-tuned for MOS prediction achieve reasonable correlation at the system level but degrade at utterance level, and cross-rater generalization (training and test annotators are disjoint sets) further limits reliability.
Method
The proposed framework extends a wav2vec2 backbone with three additions: a Mixture of Experts (MoE) classification head, a multi-task auxiliary objective, and a data augmentation strategy. The MoE head consists of N independent expert networks, each a small multi-layer perceptron, gated by a lightweight softmax gating network that dynamically weights experts based on the input feature vector. This allows the model to specialize different experts for different audio types or quality ranges.
The auxiliary task requires the model to classify which synthesis system generated a given utterance, on the premise that different TTS systems leave detectable acoustic fingerprints that help the model learn discriminative quality-related features. A joint adaptive loss combines the regression loss for MOS prediction (smooth L1) with cross-entropy for model classification, with weights dynamically shifted from auxiliary-heavy in early training to MOS-heavy in later stages.
Data augmentation doubles the original 400-sample challenge dataset to 800 by generating additional samples with four commercial TTS systems: CosyVoice-300M, CosyVoice2-0.5B, FireRedTTS, and f5tts at 24kHz. Training follows a three-stage curriculum: auxiliary pre-training on model classification, joint pre-training on the combined objective, and final fine-tuning focused on MOS prediction with AdamW and cosine annealing.
Key Results
In the system-level evaluation, the proposed system achieves MSE 0.056 (rank 1 among seven participating teams), LCC 0.978 (rank 2), SRCC 0.913 (rank 5), and KTAU 0.758 (rank 4). The MSE improvement over the strongest baseline (MSE 0.071) is substantial in absolute terms, suggesting the MoE architecture effectively reduces prediction bias.
At utterance level, the system places mid-tier: MSE 0.277 (rank 3), LCC 0.811 (rank 6), SRCC 0.716 (rank 3), KTAU 0.529 (rank 4). The paper attributes this to a train/test annotator mismatch where raters 0-9 annotate training data and raters 10-19 annotate the test set, meaning the model cannot learn to emulate the scoring tendencies of test raters. Utterance-level rankings are more variable across metrics than system-level rankings, indicating that no single approach dominates at fine granularity.
Novelty Assessment
The application of MoE gating to a self-supervised MOS prediction backbone is a straightforward engineering extension; MoE classification heads and multi-task learning are established techniques. The auxiliary model classification task is an interesting inductive bias but draws on the broader multi-task learning literature rather than introducing a new mechanism. The data augmentation strategy, generating samples from commercial TTS models, is practically useful but not architecturally novel.
The paper’s more durable contribution is analytical: it directly confronts the system-level versus utterance-level performance gap, attributes it to both the fundamental difficulty of capturing micro-quality variation and a concrete rater generalization failure, and sketches directions such as rater-ID embeddings and fine-grained acoustic feature extraction. These observations are valuable for the evaluation metrics sub-field, even if the proposed model does not resolve them.
Field Significance
Low — This challenge system paper provides empirical evidence that MoE-based multi-task learning can push system-level MOS prediction accuracy while leaving utterance-level performance largely unimproved. The finding that annotator-set mismatch is a dominant failure mode for utterance-level predictors adds a concrete, reproducible data point to a known difficulty, but the small dataset, unnamed challenge context, and absence of comparison to standard benchmarks (such as the VoiceMOS challenge test sets) limit generalisability.
Claims
-
supports: Multi-task learning with an auxiliary synthesis-model classification objective can improve absolute MOS prediction accuracy at the system level by encouraging the model to learn discriminative acoustic quality features.
Evidence: The MoE system with auxiliary classification achieves system-level MSE 0.056 (rank 1 of seven teams), a 21% reduction in MSE over the strongest single-objective baseline (MSE 0.071). (§III.C, Table 2)
-
complicates: Improvements in system-level MOS prediction accuracy do not reliably transfer to utterance-level prediction performance.
Evidence: Despite ranking first on system-level MSE, the proposed model places 3rd-6th across utterance-level metrics; the authors identify disjoint train/test annotator sets as a primary cause, not merely architectural limitations. (§IV, Table 1)
-
refines: The performance gap between system-level and utterance-level MOS prediction reflects fundamentally different task requirements, not a data-scaling problem alone.
Evidence: Doubling training data from 400 to 800 samples and introducing MoE architecture yields only marginal utterance-level gains, while system-level gains are large; the paper attributes residual difficulty to micro-feature variability and rater-specific bias. (§IV)
-
complicates: High correlation between predicted and actual MOS at the system level does not imply high correlation at the utterance level for the same model.
Evidence: The proposed model achieves LCC 0.978 on system-level evaluation but only LCC 0.811 on utterance-level evaluation, with similarly large gaps in SRCC and KTAU, confirming that aggregation masks prediction errors at finer granularity. (§IV, Tables 1-2)
Limitations and Open Questions
Warning
The training and test annotator pools are completely disjoint (raters 0-9 for training, raters 10-19 for testing). This means utterance-level results measure how well the model generalises to unseen rater subjectivity, an inherently harder problem than standard train/test splits assume. Utterance-level claims should be interpreted with this constraint in mind.
The dataset is small (800 samples across 12 systems, derived from 30 text entries), making it difficult to assess whether findings generalise to larger and more diverse evaluation sets. System and sample diversity are limited: all audio is generated from the same text prompts, which may suppress naturalness variation that appears in open-domain evaluation.
The challenge setup is unnamed in the paper, making it impossible to compare directly against published baseline systems outside the competition tables. The paper proposes rater adaptation and fine-grained acoustic features as future directions but does not experiment with them.
Wiki Connections
- Evaluation Metrics — proposes a MoE-based MOS predictor that improves system-level accuracy, contributing to the sub-field of automated speech quality assessment.
- Subjective Evaluation — directly addresses the challenge of replacing expensive human MOS annotation with automated prediction, and analyses why utterance-level prediction lags system-level.
- Self-Supervised Speech — uses wav2vec2 as the feature extraction backbone for MOS prediction, making SSL pre-training a core dependency of the proposed system.
- CosyVoice (CosyVoice) — uses CosyVoice-300M as one of four commercial TTS systems for training data augmentation.