arXiv · 2025 · Preprint
Wisnu et al. (Academia Sinica) · → Paper · Demo: ? · Code: ?
A multi-axis perceptual quality predictor for generative audio that uses triplet loss with buffer-based sampling over BEATs self-supervised embeddings to generalise from natural training data to unseen synthetic evaluation data across TTS, text-to-audio, and text-to-music domains.
Problem
Automatic speech quality predictors trained on natural audio struggle to generalise to the synthetic output of modern TTS and audio generation systems. The AudioMOS Challenge 2025 Track 2 formalises this problem as a four-axis prediction task — Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness — evaluated exclusively on synthetic audio despite participants having access only to natural-audio training data. Standard regression on SSL embeddings is not explicitly shaped to respect the relative ordering of perceptual scores, which limits generalisation when the input distribution shifts.
Method
The system, called AESA-Net (Audio Aesthetics Assessment Network), builds on BEATs — a transformer-based audio representation model pretrained on AudioSet — as its feature backbone. Hidden states from all BEATs transformer layers are combined via a learnable weighted sum with softmax-normalised scalars, allowing the model to emphasise layers most informative for perceptual quality. These weighted features are projected by a linear adapter layer and then encoded by a two-layer bidirectional LSTM that captures temporal structure across variable-length audio. A shared fully connected layer with dropout produces a common representation, which is then branched into four task-specific heads, one per aesthetic axis. Each head contains a multi-head self-attention layer, a frame-level linear-sigmoid scoring layer, and adaptive average pooling to produce a clip-level scalar prediction.
Training minimises a combined loss of MSE regression and a triplet loss applied to intermediate embeddings. The triplet loss uses a FIFO memory buffer that stores recent embedding-score pairs; at each step an anchor’s positive example is drawn from the buffer where the score difference is less than a threshold epsilon, and a negative from where it exceeds epsilon. This enforces perceptual ordering in the latent space without requiring contrastive batch construction. The final loss is L_total = L_MSE + 0.2 * L_triplet. The model is trained with the Adam optimiser at 1e-4 learning rate with early stopping on validation MSE; batch size is 1 due to variable-length audio. No synthetic data is used during training.
Key Results
On the official AudioMOS Challenge 2025 evaluation set (3,000 synthetic samples), the proposed system outperforms the challenge baseline on rank-based metrics across all four perceptual axes. On Production Quality, SRCC improves from 0.866 to 0.896 and KTAU from 0.683 to 0.737. On Content Usefulness, SRCC improves from 0.810 to 0.894 and KTAU from 0.629 to 0.724. Production Complexity achieves SRCC 0.928, essentially matching the baseline’s strong 0.934 while reducing MSE (0.198 vs. 0.226). The one regression is MSE for Content Enjoyment (3.991 vs. baseline 1.142), though rank-based metrics still improve (SRCC 0.904 vs. 0.841), suggesting the model ranks perceptual quality correctly but with larger absolute deviations (§IV.C, Table II).
Note
The comparison baseline is the official challenge system; other submitted systems are not compared. The evaluation set composition and the challenge’s exact scoring protocol are provided by the organisers and not fully detailed in this paper.
Novelty Assessment
The contribution is engineering integration: BEATs embeddings, bidirectional LSTMs, multi-task regression, and triplet loss are all established techniques. Their combination for multi-axis audio aesthetic assessment is the paper’s novelty, along with the buffer-based triplet sampling strategy adapted from face recognition (FaceNet) and instance discrimination literature. The core hypothesis — that metric learning over perceptual scores improves domain generalisation — is well-motivated and supported by the results, but the contribution is incremental rather than architectural. The architecture closely follows HAAQI-Net by the same research group.
Field Significance
Moderate — This paper provides practical evidence that triplet-loss-based metric learning over self-supervised audio representations improves domain generalisation for automatic perceptual quality prediction, a persistent challenge in automated speech evaluation. Its main value to the field is as a challenge system description that demonstrates this technique in the specific AudioMOS 2025 setting; the evaluation is narrow (one challenge track) and the comparisons are limited to the official baseline.
Claims
- Self-supervised audio representations trained on natural audio can generalise to predicting perceptual quality dimensions of synthetic speech and audio without synthetic training examples. (§IV.C, Table I)
- Triplet loss applied to intermediate embeddings with score-based sampling improves rank correlation with human perceptual ratings over MSE-only training objectives. (§III, §IV.C, Table II)
- Multi-axis perceptual quality prediction benefits from task-specific attention heads over a shared backbone, reflecting the partial independence of production and content quality dimensions. (§II.C)
- Domain shift between natural and synthetic audio is more problematic for absolute score prediction than for rank-based ordering of perceptual quality. (§IV.C, Table II)
Limitations and Open Questions
Warning
The model is trained exclusively on 2,700 natural audio samples, yet evaluated entirely on synthetic audio. The challenge protocol validates that this generalises in rank-order terms, but the large CE MSE gap (3.991 vs. 1.142 baseline) indicates that absolute score calibration is unreliable for content enjoyment specifically.
Evaluation is limited to the AudioMOS Challenge 2025 protocol; no comparison against other published automatic quality predictors (UTMOS, DNSMOS) is included, making it difficult to assess how this approach stands relative to the broader field. The model relies on BEATs pretrained on AudioSet rather than speech-specific SSL models, and the authors note that integrating WavLM for speech segments is deferred to future work. Batch size of 1 due to variable-length audio constrains training efficiency. Code and pretrained weights are not released as of the preprint date.
Wiki Connections
This paper contributes to evaluation-metrics and subjective-evaluation by extending automatic quality prediction beyond naturalness MOS to multi-axis aesthetic dimensions. The use of self-supervised-speech features (BEATs) as the representation backbone connects to broader trends in SSL-based evaluation. The AudioMOS Challenge 2025 builds on the VoiceMOS challenge series, which established the benchmark infrastructure for non-intrusive quality prediction in this field.