arXiv · 2025 · Preprint
Ismail Rasim Ulgen et al. (Johns Hopkins University; National University of Singapore) · → Paper · Demo: ? · Code: ?
Proposes TTScore, a pair of text-conditioned sequence-to-sequence token predictors whose conditional log-likelihood serves as a reference-free, aspect-specific objective metric for intelligibility and pitch-related prosody in synthesized speech.
Problem
Objective evaluation of synthesized speech has relied on two narrow families of metrics: text-domain intelligibility measures (WER, CER) that ignore acoustic and temporal artifacts and become less discriminative as ASR and TTS systems both improve, and reference-dependent prosody measures (F0-RMSE, F0 correlation) that require aligned ground-truth utterances, are sensitive to alignment error, and cannot account for the fact that a single text input may have multiple valid prosodic renditions. A prior likelihood-based alternative, SpeechLMScore, models unconditional token generation with a decoder-only language model and provides only a general-purpose quality signal, without targeting intelligibility or prosody specifically. This leaves a gap between broad, interpretable subjective MOS ratings and narrow, weakly-correlated objective metrics, with no reference-free, aspect-specific, and speech-domain metric for either dimension.
Method
TTScore reframes both intelligibility and prosody evaluation as conditional generation tasks. Two BART-based sequence-to-sequence transformers (6 encoder layers, 6 decoder layers, 512 hidden dimension) are trained to autoregressively predict discrete speech tokens from an input phoneme sequence, using large-scale paired text-speech data (LibriSpeech-960). TTScore-int’s generator predicts HuBERT-derived content tokens obtained via k-means quantization (the paper sweeps HuBERT layers 3/9/12 and cluster sizes k=50/500); TTScore-pro’s generator predicts prosody tokens derived from FACodec, the residual-vector-quantized, disentangled prosody codebook introduced in NaturalSpeech 3. Frame-level FACodec prosody features are pooled to the phoneme level using Montreal Forced Aligner boundaries before quantization, simplifying the prediction target and shortening sequence length.
At inference time, for a synthesized utterance, the generators are used to compute the average log-likelihood (via teacher forcing) of the utterance’s own extracted content or prosody token sequence, conditioned on its input text. Because the tokenizer already discards most non-phonetic acoustic detail, the generator’s job is to model a text-conditioned distribution over a compressed, aspect-specific token space rather than reconstruct full speech, which the authors argue is a simpler and more interpretable modeling problem than either full TTS or unconditional token language modeling. High conditional likelihood on the content-token stream is taken as evidence of intelligibility; high likelihood on the prosody-token stream, of appropriate pitch-related prosodic structure. Both metrics are reference-free with respect to ground-truth audio, since they only require the input text and the synthesized speech being evaluated.
Key Results
For intelligibility, TTScore-int correlates far more strongly with ASR-based WER/CER than either SpeechLMScore or an ablated unconditional token-LM baseline (uLM) trained under identical settings: on VoiceMOS22 utterance-level correlation with WER reaches up to 0.78 for TTScore-int versus 0.44 for the best uLM configuration and 0.32 for SpeechLMScore (Table 1). TTScore-int also correlates more strongly with human MOS ratings than WER or CER across both SOMOS and VoiceMOS22 system-level and utterance-level splits (Table 2), and remains competitive with UTMOS (a MOS-supervised predictor) despite UTMOS’s direct access to MOS training labels; UTMOS’s correlation degrades sharply out of its training distribution (SOMOS) while TTScore-int’s does not.
For prosody, TTScore-pro assigns systematically higher scores to real speech than synthesized speech, and higher scores to originally-pitched resynthesis than to pitch-perturbed (inverted or flipped F0) resynthesis, validating the metric’s sensitivity to prosodic degradation (Figure 2). TTScore-pro also shows consistently positive correlation with MOS on both SOMOS and VoiceMOS22 (up to LCC 0.45 system-level on VoiceMOS22, Table 4) where the reference-dependent F0-RMSE and F0-correlation baselines show near-zero or unstable correlation (Table 3), and, unlike the baselines, correlates in the correct direction with TTS Arena ELO scores, whereas F0-RMSE shows an unexpected positive (wrong-sign) correlation with ELO (Table 5).
Novelty Assessment
The contribution is evaluation methodology rather than a new speech-generation architecture: the underlying sequence-to-sequence predictor is a standard, unmodified BART encoder-decoder, and the discrete tokenizers (HuBERT k-means, FACodec) are both taken directly from prior work. The genuine novelty is the reframing of evaluation as text-conditioned generation over aspect-specific token streams, in contrast to unconditional language modeling (SpeechLMScore) or reference-dependent signal comparison (F0-RMSE). The phoneme-level pooling step for prosody tokens is a modest but sensible engineering simplification rather than a new representation. The empirical case is reasonably thorough (three independent benchmarks, an ablation over tokenizer layer/cluster size, and three complementary prosody sanity checks), though the correlations for prosody remain moderate in absolute terms (LCC around 0.2-0.5) even where they beat the baselines.
Field Significance
Moderate — the paper demonstrates that conditioning discrete-token likelihood on input text produces a more discriminative and better human-aligned signal than either unconditional token-LM scoring or reference-based F0 comparison, for two long-standing weak spots in objective TTS evaluation. Its contribution is confined to evaluation methodology: it does not introduce a new generative architecture, and the reported prosody correlations, while consistently better than baselines, remain modest in absolute magnitude.
Claims
- supports: Conditioning a discrete-token likelihood model on the input text produces a substantially more discriminative and human-aligned intelligibility signal than unconditional token language modeling.
Evidence: TTScore-int reaches utterance-level correlation with WER up to 0.78 on VoiceMOS22 (system-level up to 0.96), compared to at most 0.44/0.55 for an identically-trained unconditional token-LM (uLM) baseline and 0.32/0.42 for SpeechLMScore. (§V.A, Table 1)
- supports: MOS predictors trained directly on MOS-labeled data generalize poorly across benchmarks relative to reference-free metrics with a narrower, aspect-specific training objective.
Evidence: UTMOS, trained on VoiceMOS22 data, achieves high correlation with MOS on VoiceMOS22 but notably lower correlation on the out-of-distribution SOMOS dataset, while TTScore-int retains stronger correlation with MOS on SOMOS. (§V.B, Table 2)
- supports: Reference-free, text-conditioned prosody scoring can track prosodic appropriateness where reference-dependent F0-based metrics fail, including on benchmarks lacking aligned reference audio.
Evidence: TTScore-pro shows positive correlation with MOS on both SOMOS and VoiceMOS22 and correct-sign correlation with TTS Arena ELO scores, while F0-RMSE shows near-zero or negative correlation with MOS and an unexpected positive (wrong-sign) correlation with ELO. (§VI.C, Tables 3-5)
- complicates: Objective prosody metrics that isolate a single acoustic dimension (pitch) correlate only moderately with overall perceived naturalness, since naturalness depends on additional prosodic factors beyond F0.
Evidence: TTScore-pro’s correlations with MOS remain in the 0.2-0.5 range even where it clearly outperforms F0-RMSE and F0-correlation baselines, and the paper attributes the residual gap to prosody encompassing rhythm, energy, and phrasing beyond the F0-based prosody tokens used here. (§VI.C, Table 3)
Limitations and Open Questions
Warning
The method targets only F0-related prosody; rhythm, energy, and phrasing, which the authors acknowledge are also core prosodic dimensions, are not modeled or evaluated.
The text-to-token generators are trained exclusively on a large-scale, clean, English-only corpus (LibriSpeech-960), so the metric’s reliability in noisy, spontaneous, or non-English settings is untested. Because the token distributions are learned from data, the authors note the approach may inherit whatever biases exist in that training corpus. The paper also does not report whether code will be released, and the demo/code availability fields could not be confirmed from the paper text.
Wiki Connections
- Evaluation Metrics — introduces TTScore-int and TTScore-pro, text-conditioned conditional-likelihood metrics that reframe intelligibility and prosody evaluation as discrete-token generation tasks.
- Self-Supervised Speech — builds TTScore-int on HuBERT-derived content tokens obtained via k-means quantization, sweeping layer depth and cluster count to identify effective configurations.
- Neural Audio Codec — derives its prosody tokens from FACodec’s disentangled residual-vector-quantized prosody codebook, pooled to the phoneme level for tractable sequence prediction.
- NaturalSpeech 3 — reuses the FACodec prosody tokenizer introduced in this paper as the discrete representation underlying TTScore-pro.
- UTMOS — compares against UTMOS as a MOS-supervised baseline, showing that UTMOS’s correlation with human ratings degrades more on out-of-distribution data (SOMOS) than the proposed reference-free metric.