arXiv · 2026 · Preprint
Xi Wang et al. (Tsinghua University / Xiaomi) · → Paper · Demo: ? · Code: ✓
Fine-tunes a 7B audio language model to score Mandarin TTS output on 12 explicitly defined perceptual dimensions, writing a criterion-grounded rationale before each score, and uses the resulting diagnostic profiles to separate six modern TTS systems that look nearly identical on basic quality.
Problem
A single MOS value says little about where a near-human TTS system still fails. Global scalar predictors and preference models average over the utterance, which dilutes short, localized artifacts. Recent multi-dimensional and explanation-producing judges target high-level perceptual attributes rather than acoustic detail or language-specific phonetics, and because their scoring criteria are not explicit, the rationales they produce tend to be formulaic. Existing quality datasets are mostly English and skewed toward good samples, so decision boundaries for rare, subtle defects are poorly defined. For Mandarin specifically, errors such as tone sandhi mistakes, polyphone misreadings and n/l confusion are not captured by ASR-based intelligibility metrics.
Method
Schema. TTS-PRISM defines a two-layer, 12-dimension taxonomy with explicit, acoustically anchored criteria for each score level. The Basic Capability layer (scored 1-5) covers Audio Clarity, Pronunciation Accuracy, Prosody Accuracy (split into Intonation, Pauses and Speech Rate) and intra-utterance Consistency (split into speaker, style and emotion consistency). The Advanced Expressiveness layer (scored 0-2 as a bonus, with 0 meaning neutral rather than a penalty) covers Stress, Lengthening, Paralinguistics and Emotion Expression. Each level is tied to tolerance thresholds, for example which artifact types a score of 4 still permits.
Data. Source texts come from literary, conversational and web corpora. Positive anchors include high-fidelity human speech and output from leading TTS systems, with NVSpeech and FireRedTTS-2 output setting the ceiling for paralinguistic and emotional expression, and in-house professional recordings serving as gold anchors for Stress and Lengthening. Negatives come from injected prosody and rhythm perturbations, pronunciation and audio-quality degradations, consistency breaches, and the perturbation subset of the Intelligibility Preference Speech Dataset. Gemini-2.5-Pro labels each sample through 12 separate dimension-wise prompts. Human-instructed rationale refinement corrects its hallucinations on Stress and Lengthening, and an 11k-sample expert-annotated “Pronunciation Gold Subset” adds Mandarin tone sandhi and polyphone knowledge. The result is 200k aligned samples. Most training labels are therefore produced by an LLM with targeted human correction, rather than collected from listeners.
Model. The scorer is MiMo-Audio (7B), fine-tuned with full-parameter SFT (AdamW, batch size 1, fixed learning rate 1e-6). The target sequence interleaves rationale and score for all 12 dimensions, [R1, S1, …, R12, S12], so a single forward pass produces the full diagnosis. Each rationale is conditioned on the explicit scoring criterion and comes before its score, which the authors present as a logical regularizer. This differs from the open-ended chain-of-thought (CoT) used by generalist audio LLMs.
Key Results
On the 1,600-sample Mandarin Gold Test Set, which is disjoint from training, includes 20% out-of-distribution samples (unseen TTS systems and real recordings) and carries consensus expert labels, TTS-PRISM (7B) has the highest LCC of the four judges on 8 of 12 dimensions (Table 1). Its clearest margins are on Audio Clarity (LCC 0.815 vs. 0.756 for Gemini-2.5-Pro), the consistency dimensions, Stress and Lengthening. Gemini-2.5-Pro still leads on Pronunciation Accuracy (0.613 vs. 0.511), Intonation, Pauses and Paralinguistics. Step-Audio-R1 (33B) and Qwen3-Omni (30B) have lower LCC than both models on every dimension. Averaged over all 12 dimensions, TTS-PRISM reaches LCC 0.717, SRCC 0.721 and normalized MSE 0.044 (Table 4). The comparison is not symmetric. Baselines run 12 separate dimension-wise inferences while TTS-PRISM uses one pass. Gemini-2.5-Pro is also a baseline, the main source of TTS-PRISM’s training labels, and the judge behind the RSC rationale metric.
Rationale Support Consistency (RSC), which uses Gemini-2.5-Pro to check whether a rationale supports the ground-truth score, is 0.98 for TTS-PRISM versus 0.91 for Step-Audio-R1 and 0.88 for Qwen3-Omni. The authors read the baselines’ combination of high RSC and low alignment as reasoning that is coherent but not grounded in the audio (§4.1). On the OOD subset, average LCC drops modestly from 0.729 to 0.690 (Basic Capability) and from 0.716 to 0.675 (Advanced Expressiveness) (Table 3).
Ablations (Table 4) show that training without negatives collapses alignment (LCC 0.150), which is below even the un-tuned backbone (0.320). Removing rationale generation lowers LCC from 0.717 to 0.662.
Profiling 500 utterances from each of F5-TTS, CosyVoice 3, MaskGCT, Qwen3-TTS, FireRedTTS-2 and IndexTTS2 (Table 2) shows a ceiling effect in the Basic layer: all systems score above 4.9 on speaker consistency and between 4.79 and 4.86 on Pronunciation Accuracy. The Advanced layer separates them much more. Paralinguistics ranges from 0.114 (F5-TTS) to 0.735 (CosyVoice 3), and Lengthening from 0.067 (MaskGCT) to 1.033 (IndexTTS2). Human evaluators turn these profiles into qualitative flags such as “Stable but Flat” for F5-TTS and “Highly Expressive” for IndexTTS2. For the Advanced layer, each system was run under its most expressive available controls and the best average was kept, so these numbers reflect a per-system ceiling under different configurations rather than a controlled comparison.
Novelty Assessment
The main contribution is the evaluation design: a fine-grained, Mandarin-specific schema with explicit per-level criteria, plus a data pipeline that deliberately fills in the low-quality tail with targeted perturbations. The modelling is standard. An existing audio LLM receives full SFT on an interleaved rationale-then-score target, much like the explanation-producing speech judges that the paper cites as prior work. The ablations do add useful evidence: without hard negatives, a fine-tuned judge does worse than no fine-tuning at all, and criterion-anchored rationales give a measurable gain over direct score regression. The label pipeline relies heavily on Gemini-2.5-Pro, so the model is partly a distilled, single-pass version of a commercial judge. Its advantage over that judge is mixed across dimensions rather than uniform.
Field Significance
Moderate. TTS-PRISM provides an open-checkpoint, multi-dimensional diagnostic judge for Mandarin TTS. It shows that Advanced Expressiveness dimensions separate current zero-shot systems that are effectively saturated on basic quality. It also provides evidence that targeted negative data and criterion-grounded rationales are the components that make an audio-LLM judge track expert labels.
Claims
- supports: Fine-grained, multi-dimensional diagnosis separates modern TTS systems that are indistinguishable on basic quality measures, because expressive attributes vary far more than stability attributes.
Evidence: Across six systems, speaker and emotion consistency scores sit near the 5-point ceiling and pronunciation scores span only 4.797 to 4.860, while Paralinguistics ranges from 0.114 to 0.735 and Lengthening from 0.067 to 1.033 on the 0-2 scale. (§4.3, Table 2)
- supports: Learned speech-quality judges need deliberately constructed low-quality and hard-negative examples; training on mostly good samples produces conservative predictions that track human judgments poorly.
Evidence: Training TTS-PRISM without negative samples drops average LCC to 0.150, below the un-tuned MiMo-Audio backbone (0.320) and far below the full model (0.717), under a compute-matched schedule. (§3.1, §4.2, Table 4)
- supports: Requiring an audio-LLM judge to produce a rationale grounded in explicit scoring criteria before each score improves agreement with human labels over direct score prediction.
Evidence: Removing rationale generation lowers average LCC from 0.717 to 0.662 and SRCC from 0.721 to 0.654. (§4.2, Table 4)
- complicates: Generalist audio LLMs can give internally coherent quality rationales that are not grounded in the actual acoustics, so rationale plausibility does not show that an automatic speech judge is accurate.
Evidence: Step-Audio-R1 and Qwen3-Omni reach rationale support consistency of 0.91 and 0.88 yet have much lower per-dimension LCC than TTS-PRISM or Gemini-2.5-Pro (for example, Qwen3-Omni at 0.169 LCC on Pronunciation Accuracy). (§4.1, Table 1)
- complicates: Audio-LLM judges built on ASR-oriented pre-training have difficulty detecting fine-grained pronunciation defects, because recognition objectives reward tolerating exactly those deviations.
Evidence: Pronunciation Accuracy is TTS-PRISM’s weakest dimension (LCC 0.511 vs. 0.613 for Gemini-2.5-Pro) despite an 11k expert-annotated pronunciation subset, which the authors attribute to a pre-training bias that instruction tuning could not override. (§4.1, §5, Table 1)
Limitations and Open Questions
Warning
Gemini-2.5-Pro generates most of the 200k training labels, acts as one of the baselines, and is the judge behind the RSC metric. The Gold Test Set carries expert-validated labels, but how much of TTS-PRISM’s alignment reflects human perception versus Gemini’s own annotation behaviour cannot be separated from the reported experiments.
- The schema, data and evaluation are Mandarin-only. Transfer to other languages, especially the Mandarin-specific pronunciation criteria, is untested.
- The paper does not report inter-annotator agreement or the number of experts for the Gold Test Set, so the human ceiling for each dimension is unknown.
- Baselines receive 12 separate prompts while TTS-PRISM is evaluated in a single pass. This is meant to favour the baselines, but it means the setups differ.
- The profiling Advanced-layer scores use each system’s best configuration (audio prompts, style tags), which varies across systems. Profiles are averages from TTS-PRISM itself rather than listener scores, and the diagnostic flags are assigned by humans reading those scores, not by listening.
- The training dataset is described in detail, but the paper only commits to releasing the criteria, code and checkpoints.
- The authors propose reinforcement learning to further calibrate diagnostic precision as future work.
Wiki Connections
- Evaluation Metrics — Proposes a 12-dimension learned diagnostic judge for Mandarin TTS that produces a rationale and score for each dimension and is validated against expert labels with LCC, SRCC and normalized MSE. Applied to six TTS systems, it profiles them and finds that they converge on basic stability but differ strongly on stress, lengthening and paralinguistic expressiveness.
- Subjective Evaluation — Breaks the single MOS scalar into explicitly anchored perceptual dimensions, uses consensus expert annotation of speech as the ground truth for its Gold Test Set, and uses expert annotations for its pronunciation training subset.
- MiMo-Audio — Serves as the 7B audio language model backbone that TTS-PRISM fine-tunes with full parameters.
- Gemini 2.5 — Gemini-2.5-Pro labels the training data dimension by dimension, is the strongest baseline judge, and scores rationale support consistency.
- Step-Audio-R1 — Reasoning-enhanced audio LLM baseline that gives coherent rationales but weaker alignment with expert labels.
- Qwen3-Omni — Generalist omni-modal baseline judge with the weakest alignment, particularly on pronunciation.
- SpeechLLM-as-Judges — Prior multi-dimensional, explanation-producing speech judge; TTS-PRISM argues that such schemas omit fine-grained acoustic and language-specific criteria.
- NVSpeech — Supplies positive anchor samples that define the ceiling for paralinguistic expression in the training data.
- Intelligibility Preference Alignment — Its Intelligibility Preference Speech Dataset perturbation subset is reused to sharpen sensitivity to Mandarin homophone and sub-phoneme errors.
- IndexTTS2 — Profiled system that receives the highest Emotion Expression and Lengthening scores and the “Highly Expressive” flag.