arXiv · 2026 · Preprint
Shijun Luo et al. (NetEase Cloud Music) · → Paper · Demo: ✗ · Code: ✓
Uses audio-first human listening audits to show that ASR-roundtrip scoring of Chinese news TTS can report a span as correct when listeners hear a wrong reading, because the ASR model uses sentence context or written-form conventions to recover the intended text.
Problem
ASR-roundtrip evaluation (synthesize, transcribe, compare to the reference) is a cheap, scalable proxy for TTS intelligibility, and transcript agreement is often read as evidence that the audio is correct. Chinese news text contains short written forms whose correct reading depends on context or domain convention: 13-11 as a snooker score (十三比十一) rather than a range (十三至十一), 伊尔-76 as an aircraft model rather than a negative number, or unit and membership strings such as 350Wh/kg and 88VIP. A TTS system can speak such a span fluently but wrongly, and an ASR model with strong language and normalization priors can still write the surface-correct form. The paper asks whether such false negatives exist, what produces them, and whether they depend on the ASR system. It explicitly does not propose a new TTS frontend.
Method
Risk taxonomy. The authors define Context-Dependent Reading Decisions (CDRD): written spans whose correct reading depends on context beyond the local character sequence. Cases carry one of three labels: CDRD-entity (scores, military and aircraft models, financial quarters, generation labels, entity-dependent hyphenated forms), CDRD-polyphone (Chinese polyphones, closer to G2P), and CDRD-adjacent (units, percentages, mixed-script strings, memberships, abbreviations, foreign names). Each risk span is annotated with surface form, expected reading, known wrong readings where available, context evidence, and whether transcript-based scoring is allowed (§2.1).
Data. Starting from 108,124 company-produced Chinese news scripts used in a production TTS workflow, taxonomy-driven rule mining and a scripted text-only pipeline select a 500-case real-news candidate pool; the selection uses only text and risk-span metadata, before any TTS or ASR output exists. A 5K synthetic hard-case pool is also built. The frozen 200-case benchmark holds 155 real-news and 45 synthetic cases (85 CDRD-entity, 35 CDRD-polyphone, 80 CDRD-adjacent/non-CDRD) across seven news domains (§2.2).
Systems and conditions. The primary pipeline pairs the MiMo-V2.5-TTS API with the audio-capable MiMo mimo-v2.5 API prompted for verbatim transcription under a “strict” protocol that discourages numeral normalization (mimo-v2-omni as fallback and ablation route). Controls are Whisper-small as a non-MiMo ASR, CosyVoice-300M-SFT as a second TTS (Raw input only), and Paraformer-zh v2.0.4 and Qwen3-ASR-1.7B run on the 220 TTS-specific Raw files from the two targeted audits and on 46 aligned MiMo clips, with no source text or hotwords. Two text conditions are compared: Raw (original title and summary, no rewriting of risk spans) and Structured (risk spans replaced by their expected spoken forms from the same rules that define the benchmark targets). The authors frame Structured as an oracle-style upper bound, not a deployable frontend (§3.1).
Human audits. One primary annotator listened to Raw and Structured audio for all 200 benchmark cases and counted correct risk spans, judging from audio before consulting ASR transcripts; a 30-case stratified subset was independently reviewed by two further annotators for inter-annotator agreement. The score is case-macro risk-span audio accuracy (mean over cases of correct spans divided by risk spans) (§3.2). Separately, a 110-row targeted audit over high-risk candidates (mined from risk-span rules and suspected masking signals such as Raw/Structured disagreement) labels each Raw TTS file as confirmed masked (audio wrong, at least one ASR route writes the expected or surface-correct form), exposed TTS error, no Raw TTS error, or uncertain. All 110 labels were checked by a second reviewer, and a 30-row label-blind relabel subset tests label stability (§3.3). This is a span-level correctness audit by listeners, not a MOS or preference test.
Context isolation. For the 46 confirmed masked MiMo cases, target-span clips are cut either as a rough 6-second window or by Whisper-small timestamp alignment and re-transcribed with the main strict ASR, to test whether removing sentence context re-exposes the wrong reading (§4.3).
Key Results
Masked false negatives exist and are reported with a full denominator. On the 110-case targeted pool, MiMo Raw audio yields 46 confirmed masked, 9 exposed TTS errors, 55 no Raw TTS error and 0 uncertain; CosyVoice Raw yields 51 confirmed masked, 27 exposed, 30 no error and 2 uncertain (Table 1). The authors stress that these are targeted yields, not production prevalence. By label, MiMo outcomes (masked/exposed/no error) are 33/6/42 for CDRD-entity, 3/1/7 for CDRD-polyphone and 10/2/6 for CDRD-adjacent (§4.1). Masking is protocol-sensitive: a separator-normalized match finds surface-correct recovery in 34/46 masked cases under default ASR and 27/46 under the MiMo-V2-Omni strict route (§4.1). The 46 MiMo cases are sports scores (20), kW/kWh unit strings (10), military models (9), hyphen-range or torque forms (3), and single financial-quarter, compute-unit, membership and voltage cases (§4.2). Representative cases include F-35 heard as F 杠三五 but transcribed F-35, and 88VIP heard as 八十八伏 IP but transcribed 88VIP (Table 2).
Isolation re-exposes many errors. With the full sentence, all 46 cases are masked; aligned clips expose 18, stay masked in 12, return no output in 13 and something else in 3; rough 6-second clips expose 16, mask 11, return no output in 17 and something else in 2 (Table 3). The full-sentence route is case-specific while clips all use main strict ASR, so this comparison is cross-route (§4.3).
Masking depends on the ASR system. On the 97 confirmed-masked files, Qwen3-ASR-1.7B surface-recovers (masks) 19/46 MiMo and 21/51 CosyVoice files, 40/97 in total, while Paraformer-zh does so in 0/46 and 2/51, 2/97 in total (Table 4). Paraformer preserves a wrong or non-canonical form in 88/97 cases and Qwen in 56/97; Paraformer’s ITN toggle changes 0/220 full-file transcripts. Of Qwen’s 19 full-context MiMo recoveries, aligned-span transcription re-exposes 12, reproducing the isolation effect with the same decoder. Among the 51 CosyVoice masked cases, MiMo strict ASR masks 37, Whisper-small 36 and both 22 (§4.5).
ASR scores are miscalibrated relative to listeners. Human case-macro accuracy is 0.8889 for Raw and 0.9503 for Structured (+0.0614, 95% CI [+0.0352, +0.0891]); the authors report their largest per-label gain on CDRD-entity cases (0.7887 to 0.9146, +0.1259). Automatic scores range from 0.495/0.528 (Raw/Structured) under default ASR to 0.728/0.805 under MiMo-V2-Omni strict ASR, with the Raw-to-Structured direction stable across protocols. Under MiMo strict ASR, Raw scores 0.6121 and Structured 0.7826; on the aligned 196-case subset, human accuracy minus strict-ASR score is +0.2745 for Raw and +0.1667 for Structured, which the authors describe as a calibration gap rather than a masking rate (§4.4).
Annotation reliability. On the 30-case IAA subset, pairwise exact agreement on correct counts is 0.900 to 0.933 for Raw and 0.833 to 0.900 for Structured, with Pearson correlations from 0.848 to 0.948 (§4.4). Blind relabeling of 30 audit rows reaches 23/30 exact agreement (κ = 0.634) and 27/30 on confirmed-masked versus other outcomes (κ = 0.800) (§3.3).
Novelty Assessment
The contribution is a documented evaluation failure, not a new system. Concerns that WER from ASR-roundtrip does not fully capture TTS quality predate the paper (its references [1, 2]); what is new is a specific mechanism, context- and convention-driven transcript recovery of wrong readings, demonstrated with human-confirmed cases, a complete outcome denominator, a context-isolation diagnostic, and cross-TTS and cross-ASR controls showing that the effect recurs but its size depends on the evaluator. The CDRD taxonomy and the audio-first span-audit protocol are reusable, and prompts, transcripts, labels, audio and scoring code are released. The evidence is deliberately narrow: one language and domain, two TTS systems, targeted rather than random sampling, and one primary annotator for the 200-case pass.
Field Significance
Moderate — The paper gives concrete, human-verified evidence that transcript agreement can hide fluent wrong readings in Chinese news TTS, and that how much is hidden depends on which ASR model scores the audio, from 2/97 recoveries for Paraformer-zh to 40/97 for Qwen3-ASR-1.7B on the same files. It is a cautionary data point for anyone using ASR-roundtrip WER or CER as the sole correctness measure for text-normalization-heavy content. Its scope (one language, a targeted 110-case pool, two TTS systems) limits it to existence and mechanism, as the authors state; it does not estimate how often the failure occurs.
Claims
- complicates: ASR-roundtrip transcript agreement is not sufficient evidence that synthesized speech realizes context- or convention-dependent written spans correctly, because ASR can recover the intended written form from a wrong spoken reading.
Evidence: In an audio-first listening audit of 110 high-risk Chinese news cases, 46 MiMo-V2.5-TTS and 51 CosyVoice-300M-SFT Raw outputs were heard as wrong while at least one ASR route wrote the expected or surface-correct form, against 9 and 27 errors that ASR exposed. (§4.1, Table 1)
- refines: Transcript-level masking of TTS reading errors is driven largely by the ASR model’s use of sentence context and written-form conventions, not by the acoustic evidence of the target span alone.
Evidence: Re-transcribing Whisper-small-aligned clips of the 46 masked MiMo cases re-exposed 18 errors (16 with rough 6-second clips); with Qwen3-ASR-1.7B as the same decoder, aligned-span transcription re-exposed 12 of its 19 full-context recoveries. (§4.3, §4.5, Table 3)
- complicates: How many synthesis errors an ASR-roundtrip metric hides depends strongly on the ASR system and transcription protocol, so scores from different evaluators are not interchangeable.
Evidence: On the same 97 confirmed-masked files, Qwen3-ASR-1.7B surface-recovered 40 and Paraformer-zh 2; a separator-normalized match found recovery in 34/46 MiMo masked cases under default ASR versus 27/46 under the strict route, and Raw/Structured scores ranged from 0.495/0.528 to 0.728/0.805 across protocols. (§4.1, §4.4, Table 4)
- supports: ASR-roundtrip scoring can still rank text-processing conditions in the same direction as listeners even when its absolute values are miscalibrated against human judgments.
Evidence: The Raw-to-Structured improvement direction held under every ASR protocol tested, while human case-macro accuracy exceeded strict-ASR scores by +0.2745 (Raw) and +0.1667 (Structured) on the aligned 196-case subset. (§4.4)
Limitations and Open Questions
Targeted, not random, sampling
The 110-row audit pool was mined for high-risk spans and suspected masking signals, and the CosyVoice audit reuses the same pool. The counts (46/110, 51/110) establish that the failure exists and how it arises; they are not occurrence rates for production Chinese news TTS, as the authors state.
- The 200-case span audit has one primary annotator; agreement is measured on a 30-case subset with two further annotators, and all reviewers see expected and known wrong readings, so judgments are not blind to the targets (only the 30-row relabel subset is label-blind).
- The Structured condition derives its readings from the same rules that define the benchmark targets, so its human and ASR gains are an oracle upper bound, not evidence for a deployable frontend.
- The primary TTS and ASR are commercial MiMo APIs whose internal normalization the strict prompt reduces but cannot fully suppress; the context-isolation comparison is cross-route for MiMo, and isolated clips frequently return no output (13/46 aligned, 17/46 rough).
- Only two TTS systems and a small set of ASR systems are tested, in one language and domain; risk spans are found by rules, so failure types outside the taxonomy are not covered. The 108,124-script source export is not released.
- The paper does not propose a replacement metric; how to scale human-grade reading-error detection beyond span audits is left open.
Wiki Connections
- Evaluation Metrics — Shows, against human listening judgments, that ASR-roundtrip WER-style scoring can miss context-dependent reading errors and that the miss rate depends on the ASR evaluator.
- Subjective Evaluation — Uses audio-first span-level listening audits of synthesized speech (one primary annotator, a second reviewer and a 30-case three-annotator agreement subset) as the reference that ASR scores are checked against.
- CosyVoice — CosyVoice-300M-SFT serves as the second TTS system, audited with Raw input on the same 110-case pool, where it yields 51 masked and 27 exposed errors.
- Whisper — Whisper-small is the non-MiMo ASR control and supplies the timestamp alignment used to cut target-span clips for the isolation diagnostic.
- Paraformer — Paraformer-zh is one of the two open-source ASR systems re-transcribing the confirmed-masked files, and it surface-recovers only 2/97, preserving most wrong readings.