arXiv · 2026 · Preprint

Yuxuan Wu et al. (Southeast University) · → Paper · Demo: ✓ · Code: ✗

Re-Sonance chains unmodified Whisper ASR, a Qwen LLM correction step and CosyVoice TTS into an overlapping (asynchronous) pipeline that re-speaks Mandarin dysarthric speech as fluent synthetic speech; it raises rated intelligibility for mild and moderate dysarthria but not for severe cases.

Problem

People with dysarthria struggle to take part in real-time professional speaking, such as meetings and presentations. Augmentative and alternative communication (AAC) tools based on typing or eye-tracking replace the user’s speech, which slows communication and can undermine self-identity. Speech-driven alternatives exist in two forms. Signal-level conversion (dysarthric-to-healthy voice conversion, whisper-to-normal conversion) is resource-heavy and may not improve intelligibility in practice. ASR-TTS cascades are simpler, but ASR errors on dysarthric speech pass straight through to the synthesized output. The paper asks whether an LLM correction step between ASR and TTS can repair those errors while keeping latency low enough for live use. The authors report that the paper was accepted at NCMMSC 2025.

Method

Re-Sonance is a three-stage cascade of off-the-shelf components, none of which is fine-tuned:

  • ASR: the unmodified Whisper-Turbo model transcribes the dysarthric speech, although dysarthria-tuned Whisper variants exist.
  • LLM correction: Qwen-Plus, an Alibaba Cloud model accessed as a service, corrects and refines the transcript through prompt engineering. The paper does not publish the prompt, and its size is not reported.
  • TTS: CosyVoice (cited as the original CosyVoice) synthesizes the corrected text. The paper notes that CosyVoice supports voice cloning, but it does not say which voice the system outputs or whether the user’s voice is cloned.

The pipeline is “asynchronous” in the sense of non-blocking and overlapping. Whisper emits partial text segments, each segment goes to the LLM without waiting for the utterance to end, and corrected segments go to CosyVoice incrementally. The paper also describes a mobile and desktop interface and a public demo. The frontmatter architecture tags describe the inherited components (Whisper encoder-decoder, the autoregressive LLM, CosyVoice’s language model plus flow-matching decoder); the paper itself contributes no model.

The evaluation simulates use on the Chinese Dysarthria Speech Database (CDSD), with no live users. Twenty native Mandarin listeners, trained beforehand with annotated examples, rate samples from 10 dysarthric speakers on 5-point intelligibility and naturalness scales (Table 1). They also judge semantic relevance to the speaker’s intended content. Three conditions are rated: the original dysarthric speech, a baseline ASR-TTS cascade without the LLM, and Re-Sonance. Objective tests report WER, match error rate (MER) and word information lost (WIL) on 30 speakers split into mild, moderate and severe groups. Latency is measured per stage on 200 clips of about 10 seconds each.

Key Results

Subjective scores (§4.1), Re-Sonance versus the ASR-TTS baseline without LLM correction:

Severity (speakers)IntelligibilityNaturalnessSemantic association
Mild (3)4.79 vs 3.244.61 vs 4.3098.3% vs 78.8%
Moderate (4)4.55 vs 3.144.70 vs 4.5788.6% vs 68.2%
Severe (3)1.24 vs 1.094.36 vs 4.036.1% vs 3.0%

The LLM step lifts intelligibility by about 1.4 to 1.6 points for mild and moderate dysarthria. Naturalness changes much less, because both cascades output TTS speech. Figure 3 shows both cascades rated well above the original dysarthric recordings on naturalness, so most of the naturalness gain comes from re-synthesis itself, not from the LLM. For severe dysarthria the output stays unintelligible (1.24) yet is rated natural (4.36): the system produces fluent speech with largely wrong content.

Transcription accuracy (§4.2, Table 2). In Table 2, Δ is Re-Sonance minus the Whisper baseline, so negative values are improvements:

GroupWER (ASR → Re-Sonance, Δ)MER ΔWIL Δ
G1 mild21.58 → 13.74 (-7.84)-10.66-18.00
G2 moderate23.70 → 17.88 (-5.82)-10.18-13.54
G3 severe83.77 → 84.40 (+0.63)+3.46-0.51

The text calls these “relative improvements”, but the Δ values equal the absolute differences between the table’s two columns, so they are percentage-point changes. The paper does not say whether the Re-Sonance column scores the LLM-corrected text or a re-transcription of the synthesized speech.

Latency (§4.2, Figure 5): the mean real-time factor, defined as total processing time over speech duration, is 0.8189 (median 0.7800, SD 0.2396). Figure 5 shows TTS as the slowest of the three stages.

The comparison is fair in one respect: the baseline is the same cascade with the LLM removed. It is weak in others. The paper calls the subjective gains “significant” but reports no statistical test. Each severity group in the listening test has only 3 or 4 speakers. The paper does not compare against any dysarthric speech reconstruction or conversion system.

Novelty Assessment

The contribution is engineering integration. Inserting LLM error correction between ASR and TTS, and pipelining the stages, uses established components with no training, no new model and no published prompt. The new part is applying and evaluating this cascade for dysarthric speakers in real-time professional settings, with a listening test split by severity. Because the TTS and ASR are untouched, the speech-generation evidence is narrow. It shows what text-mediated re-synthesis can and cannot preserve, and it does not advance synthesis methods.

Field Significance

Low — The paper adds a small, single-language data point on cascaded speech-to-speech reconstruction for pathological speech. Its most reusable finding is the severe-dysarthria failure mode: re-synthesis keeps output naturalness high even when the recovered content is almost entirely wrong, so naturalness ratings alone can hide failure in text-mediated reconstruction.

Claims

  • supports: In cascaded speech-to-speech reconstruction of impaired speech, correcting the intermediate transcript with an LLM improves the intelligibility of the re-synthesized output over a plain ASR-TTS cascade.

    Evidence: With the same Whisper and CosyVoice components, adding Qwen-Plus correction raises listener intelligibility from 3.24 to 4.79 (mild) and 3.14 to 4.55 (moderate), and semantic association from 78.8% to 98.3% and 68.2% to 88.6%. (§4.1, Figure 3)

  • complicates: Text-mediated re-synthesis separates output naturalness from content fidelity, so naturalness ratings can stay high when the reconstructed message is wrong.

    Evidence: For severe dysarthria, Re-Sonance output is rated 4.36 for naturalness but 1.24 for intelligibility, with 6.1% semantic association; WER rises from 83.77 to 84.40 after LLM correction. (§4.1, §4.2, Table 2)

  • complicates: Gains from cascaded speech reconstruction depend on recognition accuracy at the input, so they shrink as the input speech becomes more impaired.

    Evidence: WER falls by 7.84 and 5.82 points for mild and moderate groups, but for the severe group WER and MER rise (+0.63, +3.46) and only WIL improves slightly (-0.51). (§4.2, Table 2)

  • supports: A pipelined ASR-LLM-TTS cascade that streams partial segments between stages can run faster than real time.

    Evidence: Over 200 clips, the mean real-time factor is 0.8189 (median 0.7800), with TTS the slowest stage. (§3.2, §4.2, Figure 5)

Limitations and Open Questions

Warning

The evidence comes from simulated use on recorded CDSD samples, with 3 or 4 speakers per severity group in the listening test, no statistical test behind the reported “significant” gains, and Mandarin only. No live dysarthric users tried the system, and it is not compared against any dysarthric speech reconstruction or conversion method.

  • The output voice is not described or evaluated. CosyVoice voice cloning is mentioned, but the paper reports no speaker similarity to the user, so identity preservation, one of the paper’s own motivations, is untested.
  • Severe dysarthria is not helped. The authors attribute this to recognition failure and propose ASR fine-tuning and personalization as future work.
  • The LLM prompt is not published. Section 5.2 credits structured prompts with improving alignment to user intent, but no prompt ablation is shown.
  • The point where WER is measured (corrected text or a re-transcription of the synthesized speech) is not specified.
  • An LLM that “infers missing semantic content” can put words in the user’s mouth. The paper reports semantic association but does not measure hallucinated content separately.
  • Cross-language effectiveness and real-world, longitudinal use are left to future work.

Wiki Connections

  • Speech-to-Speech — Re-Sonance is a cascade-style speech-to-speech system that turns dysarthric input speech into re-synthesized speech through an intermediate corrected transcript.
  • Subjective Evaluation — Twenty trained native listeners rate intelligibility, naturalness and semantic relevance of original, baseline-cascade and Re-Sonance speech, split by dysarthria severity.
  • Whisper — The unmodified Whisper-Turbo model is the ASR front end, and its raw transcripts are the baseline in the transcription-accuracy comparison.
  • Qwen — The Qwen technical report is the paper’s citation for Qwen-Plus, the LLM that corrects ASR errors through prompting.
  • CosyVoice — CosyVoice is the TTS stage that synthesizes the corrected text, used without fine-tuning.