arXiv · 2026 · Preprint
Shijun Luo (NetEase Cloud Music) · → Paper · Demo: ✗ · Code: ✓
Introduces an open, automatically scored benchmark that checks whether commercial Chinese TTS products read compact news written forms (sports scores, hyphenated model names, ranges, units, generation labels) with the intended spoken meaning from raw, unnormalized text, and finds strict target accuracy ranging from 0.244 to 0.879 across seven products.
Problem
Chinese news text is full of compact written forms whose surface marks are ambiguous: a hyphen can mark a score (96-91 should be read 九十六比九十一, not as the range 九十六到九十一), a range, subtraction, or part of a model designator (苏-27 is a model name, not “Su minus 27”); “80后” is conventionally read digit by digit as a generation label; “620N·m” must be expanded as a unit name. A TTS system can reproduce every character and still change the meaning. Global quality measures such as MOS or MOS predictors are too coarse to track these specific reading decisions, and a generic ASR round-trip comparison cannot decide whether a particular target was read in the intended news form. Prior text-normalization and Chinese G2P/polyphone benchmarks test isolated frontend components, not the end-to-end behaviour of deployed TTS APIs given raw text.
Method
Raw Input Product Track. Every system receives the same original Chinese news-style text with one fixed voice chosen in advance. Provider-internal normalization and frontends are allowed; external rule frontends, LLM rewriting, SSML pronunciation hints, and manual text edits are not. The author is explicit that this measures product behaviour, not model weights in isolation (§2.1).
Data. The v0.1 sentences are synthetic news-style text produced by a released, seeded generator (11 template families, 86 lexicon entries) covering sports scores, military and vehicle models, abbreviations, mixed Chinese-Latin-digit brands, unit symbols, percentages, ranges, year ranges, and generation labels. The templates are described as diagnostic motifs from production risk, not as a sample of real news frequency. The release has a 200-record dev split (248 auto-evaluable targets) and an 800-record public split (992 auto-evaluable targets); each target carries offsets, a category, and lists of positive (correct) and negative (known wrong) readings. Same-character polyphones (20 optional targets in total) are included for audit but excluded from the automatic score because ordinary ASR text often hides tone (§2.2, Table 1, Table 2).
Scoring. Each generated utterance is transcribed by three heterogeneous ASR routes: MiMo API ASR (commercial API), SenseVoiceSmall, and Paraformer-zh (open-source local models). After Unicode, case, punctuation, and whitespace normalization, each route checks negative patterns before positive ones and returns correct, wrong, or unknown; a target’s final label is the majority if at least two routes agree, else unknown (§3.2). Three metrics are reported together: strict accuracy (correct over all auto-evaluable targets, so unknowns count against the system), coverage ((correct + wrong) over all auto-evaluable targets), and resolved accuracy (correct over correct + wrong). The ASR transcripts are fixed and released (16,800 successful rows for seven systems), so the leaderboard can be re-scored without re-calling the commercial APIs (§3.1). Generated audio is not redistributed because of provider terms (§4).
Systems evaluated. Seven commercial products, each with one fixed voice and default normalization: Volcano/Doubao TTS (seed-tts-2.0-standard), Azure Speech TTS, Google Cloud TTS (Chirp 3 HD), MiniMax TTS (speech-2.8-hd), Aliyun CosyVoice (cosyvoice-v3-plus), MiMo TTS (mimo-v2.5-tts), and AWS Polly (polly-neural) (Table 8). No open-weight research TTS model is evaluated.
Key Results
Leaderboard (Table 5, public split, 992 targets). Strict accuracy: Volcano 0.879 [Wilson 95% CI .857 to .898], Azure 0.756, Google 0.604, MiniMax 0.548, Aliyun CosyVoice 0.472, MiMo 0.275, AWS Polly 0.244. Volcano has the highest strict accuracy and the highest coverage (0.913) of the seven. Azure has zero majority-voted wrong targets (resolved accuracy 1.000) but 242 unknowns, so its strict accuracy is 0.756; Google and MiniMax reach higher coverage than Azure (0.861, 0.850 vs. 0.756) but with many wrong targets (255 and 299). Aliyun CosyVoice combines high resolved accuracy (0.885) with the lowest coverage (0.533, 463 unknowns). The author uses these cases to argue that strict accuracy, coverage, and resolved accuracy must be read together. Wilson intervals separate most adjacent ranks, but two adjacent pairs overlap: Google vs. MiniMax ([.573,.634] vs. [.517,.579]) and MiMo vs. AWS ([.248,.304] vs. [.218,.272]).
Category difficulty (Table 6, pooled over systems). Percentages are fully solved (strict 1.000), vehicle models (0.929) and abbreviations (0.865) largely so. Sports scores are the hardest category by strict accuracy (0.233) and have the highest wrong rate (0.494); unit symbols have the highest unknown rate (0.472), which the author attributes to ASR not exposing symbol-level readings.
Per-system failure modes (Table 7). Volcano reads sports scores almost perfectly (0.996) while mixed brands remain weak (0.391); Google, MiniMax, MiMo, and AWS score 0.000 on sports scores, with the dominant failure being reading the score hyphen as a range. Across routes, range negatives outnumber subtraction negatives 1,496 to 266 (§4).
ASR protocol diagnostics (Tables 3 and 4). Pooled over 6,944 target instances, the three individual routes give similar strict accuracy (0.513 to 0.543) but different coverage: MiMo API ASR is conservative (coverage 0.569, resolved 0.902) while the two local routes resolve more (0.748, 0.728). Paraformer-zh and SenseVoiceSmall agree on 0.959 of tri-labels, versus 0.595 and 0.604 between MiMo API ASR and the local routes; 41.7% of instances have at least one route disagreement. Dropping MiMo API ASR (P+S) changes 88 of 6,944 final labels (1.27%, all from resolved to unknown), and every two-route subset preserves the full seven-system rank order. The lower M+S and M+P scores (e.g. Volcano 0.575 and 0.570 vs. 0.879 with three routes) reflect the stricter two-route agreement requirement rather than different system behaviour.
No human listening test is reported, and the automatic labels are not compared against human judgments of the same targets.
Novelty Assessment
The contribution is an evaluation resource: a target-level schema with explicit positive and negative readings, a raw-input product track, and a three-ASR voting scorer with released transcripts. Target-level pronunciation checking and ASR-based intelligibility scoring are both established ideas; the new parts are the focus on compact Chinese news written forms evaluated end to end through deployed APIs, the inclusion of negative patterns so the scorer can distinguish “wrong reading” from “not resolvable”, and the transparent reporting of unknowns through coverage and resolved accuracy. The ASR-subset ablation is a useful stability check, but it tests agreement among ASR systems, not agreement with human listeners, so scorer validity rests on the hand-written pattern lists. The data are template-generated rather than drawn from real news, and the v0.1 public split is small per category (32 to 224 targets).
Field Significance
Low. The benchmark gives a concrete, reproducible way to measure raw-text reading errors in Chinese TTS products and shows large spread across seven commercial systems, and its reporting of unknowns alongside accuracy is a useful methodological caution for ASR-scored pronunciation tests. It is a single-author, single-language, template-generated automatic benchmark with no human validation of the scorer, so it primarily adds a narrow diagnostic data point rather than new understanding of how TTS systems handle text normalization.
Claims
- complicates: End-to-end TTS products that accept raw text still mis-read compact written forms (scores, hyphenated model designators, units, generation labels), so text normalization remains a practical failure source even for current commercial systems.
Evidence: On 992 auto-evaluable targets, strict accuracy ranges from 0.879 (Volcano) to 0.244 (AWS Polly), with four of seven systems below 0.60; four systems score 0.000 on sports scores, mostly by reading the score hyphen as a range. (§4, Table 5, Table 7)
- refines: ASR-based pronunciation scoring should report coverage and resolved accuracy alongside strict accuracy, because a system can look error-free on the resolvable subset while many targets remain undecidable.
Evidence: Azure has zero majority-voted wrong targets (resolved accuracy 1.000) but 242 unknown targets, giving strict accuracy 0.756; Aliyun CosyVoice has resolved accuracy 0.885 but coverage only 0.533. (§3.2, §4, Table 5)
- complicates: ASR transcripts are an imperfect proxy for pronunciation correctness: tonal distinctions and symbol-level readings are often invisible in ASR text, leaving some target categories largely unresolvable.
Evidence: Same-character polyphones are excluded from the main score because ASR text hides tone; unit symbols have an unknown rate of 0.472 and military models 0.413, and the author states the benchmark should not replace human listening tests. (§2.2, §4, Table 6, §Limitations)
- supports: System rankings from multi-ASR voting can be stable to the choice of ASR subset even when individual recognizers differ substantially in coverage and agreement.
Evidence: Pairwise tri-label agreement is 0.959 between the two local routes but about 0.6 between the API route and each local route, yet all three two-route subsets preserve the complete seven-system rank order, and the P+S subset changes only 1.27% of final labels. (§3.3, Table 3, Table 4)
Limitations and Open Questions
Warning
The scorer is never validated against human listeners: there is no listening test and no measurement of how often the ASR-voted correct/wrong labels match human judgments of the same audio. All reported accuracies therefore depend on the hand-written positive/negative pattern lists and on ASR behaviour.
- The data are synthetic template sentences; the author notes they are diagnostic motifs, not a sample of real news frequency, so the leaderboard does not estimate real-world error rates.
- Only commercial APIs with one fixed voice and default settings are evaluated, at fixed dates (TTS generated 2026-06-22); provider frontends and models change, which the author handles by recommending dated leaderboard entries. Results conflate frontend normalization and acoustic model behaviour by design.
- One of the three ASR routes (MiMo API ASR) carries the same MiMo product name as one evaluated TTS system (MiMo TTS); the paper does not discuss whether a shared vendor could bias that route.
- Generated audio is not released, so third parties can re-score transcripts but cannot re-transcribe or listen to the original outputs.
- The public split is open and could be overfit; a hidden split is proposed only as future work. Tonal polyphone errors are outside the automatic score.
Wiki Connections
- Evaluation Metrics — defines strict accuracy, coverage, and resolved accuracy for target-level pronunciation scoring with three-ASR majority voting, and checks rank stability under ASR-subset ablations.
- Seed-TTS — the top-ranked product (Volcano/Doubao, strict 0.879) is served as “seed-tts-2.0-standard”, a later product version of the ByteDance Seed-TTS family; the paper does not cite the Seed-TTS report.
- CosyVoice 3 — Aliyun’s cosyvoice-v3-plus is scored at strict 0.472 with high resolved accuracy but the lowest coverage among the seven products.
- MiniMax-Speech — MiniMax speech-2.8-hd, a later product version from the same vendor, is scored at strict 0.548 and fails all sports-score targets.
- FunAudioLLM — SenseVoiceSmall, described in this report, is one of the two open-source local ASR routes in the scoring ensemble.
- Paraformer — Paraformer-zh is the other local ASR route; it agrees with SenseVoiceSmall on 0.959 of tri-labels.