arXiv · 2025 · Preprint
Hitomi Jin Ling Tee et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ? · Code: ?
Introduces SP-MCQA, a spoken-passage multiple-choice question-answering evaluation that measures whether synthesized speech conveys key information correctly, exposing cases where low WER coexists with poor real-world intelligibility.
Problem
Word Error Rate and word-level intelligibility MOS remain the dominant proxies for whether synthesized speech is understandable, but both operate at the word level and do not check whether a listener actually receives the information that matters in a passage: names, dates, quantities, and other proper-noun or digit content. Existing TTS test sets (LibriSpeech-, LibriTTS-, and GigaSpeech-derived sets, plus Seed-TTS’s harder patterns) are either too simple or, even when they contain tongue twisters and repetitions, still under-represent irregular text such as digits and proper nouns that frequently carry the essential content of informative speech. As word-level accuracy on standard benchmarks approaches ceiling for state-of-the-art systems, the paper argues that WER can no longer distinguish models on the dimension that matters most to real listeners: whether they walked away understanding the message.
Method
SP-MCQA (Spoken-Passage Multiple-Choice Question Answering) is a subjective evaluation framework, not a trained model. It pairs a purpose-built benchmark dataset, SP-MCQA-Eval, with a human-rater evaluation pipeline.
SP-MCQA-Eval is built from National Public Radio news audio and its manually annotated transcripts, chosen for conversational speech that is rich in contextual, information-dense content. The construction pipeline follows Emilia’s pre-processing conventions: background music is removed with Ultimate Vocal Remover, and WhisperX handles ASR and timestamp extraction. A regular-expression filter retains only “information paragraphs” containing at least one number with three or more digits and at least two uppercase letters outside sentence-initial position, which biases the corpus toward passages carrying proper nouns and digits. From the filtered pool, 550 paragraphs (65 to 260 words, roughly 30 seconds to 2 minutes of speech) are segmented into speaker-attributed, sentence-level audio using Pydub, pyannote speaker diarization, and NLTK sentence splitting, yielding 5,805 utterances across 483 speakers and 8.76 hours of audio.
For each paragraph, GPT-4o-mini automatically generates two to ten multiple-choice questions targeting the key information, with four options per question: the correct answer, an “Other” catch-all, and two distractors representing distinct error types (phonetic, semantic, syntax, or grammar confusions). All questions are manually inspected and hallucinated or malformed items are removed.
In the evaluation pipeline, 40 human annotators, screened for native English proficiency or an IELTS listening score of 8.0 or above, listen to synthesized speech and answer the associated text-based multiple-choice questions, with per-question accuracy scored 1 or 0. Each task is assigned to two annotators initially, with a third or fourth added on disagreement (never beyond four). Golden test questions are seeded into 10% of tasks to filter out annotators who fall below 100% accuracy on general-knowledge checks, and their tasks are reassigned. SP-MCQA ACC is the average accuracy across qualified annotators for a given system. Three standard objective metrics (WER via Whisper-large-v3, speaker similarity via WavLM-TDNN embeddings, and DNSMOS P.835) are reported alongside SP-MCQA ACC to characterize intelligibility, speaker coherence, and audio quality independently.
Key Results
Four zero-shot TTS systems (MaskGCT, F5-TTS, CosyVoice 2, and FishSpeech V1.4) are evaluated on SP-MCQA-Eval alongside ground-truth recordings. Ground-truth itself scores 92.045% SP-MCQA ACC and 8.067% WER, with the paper attributing the non-trivial ground-truth error rate to imprecise timestamp extraction during preprocessing rather than genuine intelligibility loss. Among the synthesized systems, CosyVoice 2 achieves the highest SP-MCQA ACC (90.399%) despite a middling WER (9.044%), while FishSpeech V1.4 has the lowest WER (5.739%) but the worst SP-MCQA ACC (81.194%) of the four systems (§4.4, Table 3). A per-error-type breakdown shows phonetic errors dominate across all systems, followed by structural (syntax and grammar) and semantic errors, and that non-autoregressive systems (F5-TTS, MaskGCT) show a higher rate of semantic/structural “hallucination”-type errors than the autoregressive systems tested (CosyVoice 2, FishSpeech) (§4.4, Table 4). Qualitative annotator comments (Table 5) attribute FishSpeech’s low SP-MCQA ACC to mid-sentence word drops and normalization failures on abbreviations (e.g., “Ala.” rendered as silence), CosyVoice 2’s near-top score to correctly expanding abbreviations like “Ala.” to “Alabama” despite occasional trailing noise artifacts, and F5-TTS’s errors partly to an overly fast speaking rate that impedes recognition of key content.
Novelty Assessment
The contribution is a new subjective evaluation methodology and an accompanying benchmark dataset, not a new TTS architecture. The core idea, replacing or complementing word-by-word transcript metrics with comprehension-style multiple-choice questions over synthesized speech, is adapted from listening-comprehension test formats in language assessment and from spoken multiple-choice QA (SMCQA) research, but the paper’s specific hybrid format (spoken passage, text-based questions and options) and its targeted focus on digit- and proper-noun-dense “key information” for TTS evaluation are new instantiations of that idea. The dataset curation pipeline (regex-based filtering for information density, LLM-generated questions with manually curated distractor error types, multi-annotator agreement protocol with golden-question quality control) is a carefully engineered evaluation asset rather than an architectural advance. The most substantive empirical finding, that WER and key-information accuracy can diverge and even rank systems in opposite orders, is an evaluation-methodology result rather than a modeling one.
Field Significance
moderate — This paper provides a concrete, reproducible demonstration that word-level accuracy and comprehension-relevant accuracy are dissociable in current TTS systems, giving the field a benchmark and protocol for measuring the gap directly rather than assuming WER saturation implies solved intelligibility. Its contribution is bounded by scale (a single-language, news-domain benchmark) and by the human-annotation cost inherent to its own protocol, which the paper itself flags as a barrier to routine use.
Claims
- supports: Word-level accuracy metrics such as WER can diverge from listeners’ ability to correctly retrieve key information from synthesized speech, so low WER does not guarantee high real-world comprehension.
Evidence: FishSpeech V1.4 achieves the lowest WER (5.739%) among the four evaluated systems but the lowest SP-MCQA ACC (81.194%), while CosyVoice 2 has higher WER (9.044%) but the highest SP-MCQA ACC (90.399%) (§4.4, Table 3).
- complicates: Benchmarking TTS systems on standard read-speech test sets can understate intelligibility failures that occur specifically on irregular text such as proper nouns, digits, and abbreviations.
Evidence: Annotator comments attribute system errors to mispronounced or dropped proper nouns and numbers (e.g., “Ala.” rendered as “Alala” by MaskGCT or as silence by FishSpeech), a category of failure the benchmark specifically targets via its regex-based filtering for uppercase and multi-digit content (§3.1, Table 5).
- refines: Text normalization and phonetic accuracy on rare or irregular text remain unresolved failure modes for autoregressive and non-autoregressive TTS architectures alike, rather than being specific to one architecture family.
Evidence: Phonetic errors are the most prevalent error type across all four evaluated systems regardless of architecture, though non-autoregressive systems (F5-TTS, MaskGCT) show comparatively more semantic and structural errors than autoregressive systems (CosyVoice 2, FishSpeech) (§4.4, Table 4).
- complicates: Multiple-choice comprehension-based subjective evaluation of synthesized speech requires substantial manual annotation effort, limiting its use as a routine, frequently-run metric compared to automated scores.
Evidence: The evaluation pipeline requires 40 screened human annotators with disagreement-driven re-annotation (up to four annotators per task) and golden-question quality filtering, which the authors identify as a scalability limitation motivating future work on Audio LLM-based automation (§5).
Limitations and Open Questions
The benchmark and evaluation protocol are English-only and drawn from a single domain (NPR news speech), so the reported gap between WER and key-information accuracy has not been shown to generalize to other languages, speaking styles, or lower-resource domains. The paper explicitly notes that the substantial manual effort required for human evaluation is a limitation of the approach itself, and proposes future work on using Audio LLMs to automate assessment and on extending the framework to other languages. The ground-truth recordings also show a non-trivial error rate (7.955% on SP-MCQA ACC, 8.067% WER) attributed to preprocessing artifacts (imprecise timestamp extraction), which sets a noisy upper bound against which the four TTS systems are compared rather than a clean ceiling.
Wiki Connections
- Evaluation Metrics — proposes SP-MCQA ACC as a comprehension-based complement to WER, directly addressing the inability of word-level metrics to capture key-information accuracy.
- Subjective Evaluation — relies entirely on screened human annotators answering multiple-choice questions about synthesized speech content, with explicit inter-annotator disagreement resolution and quality control via golden questions.
- Zero-Shot TTS — benchmarks four current zero-shot TTS systems (MaskGCT, F5-TTS, CosyVoice 2, FishSpeech V1.4) against the same speaker-conditioned prompts, exposing intelligibility gaps not visible from WER alone.
- Seed-TTS — cited as prior work that introduced harder test patterns (word repetitions, tongue twisters) but still under-represents digit- and proper-noun-heavy content, motivating SP-MCQA-Eval’s targeted filtering.
- F5-TTS — evaluated as one of four benchmarked systems; SP-MCQA analysis attributes its errors partly to an overly fast speaking rate that impedes recognition of key information.
- CosyVoice 2 — evaluated as one of four benchmarked systems and the top performer on SP-MCQA ACC despite a middling WER, illustrating the paper’s central WER-versus-comprehension divergence finding.
- MaskGCT — evaluated as one of four benchmarked systems, with annotator comments noting proper-noun mispronunciation (e.g., “Ala.” rendered as “Alala”) as a source of SP-MCQA errors.