ACL 2026 (Findings) · 2026 · Conference

Changhao Pan et al. (Zhejiang University / ByteDance) · → Paper · → Preprint · Demo: ✓ · Code: ?

SwanBench-Speech is a bilingual (Chinese/English) benchmark of 1,101 long-form texts across 17 scenarios. It scores single-speaker and dialogue generation on seven automatic metrics, checks those metrics against expert listeners, and shows that 26 system entries still trail real recordings mainly in expressiveness and prosodic coherence, not in intelligibility or fidelity.

Problem

Most TTS test sets are sentence-level and single-speaker. Long-form test sets such as MinutesSpeech and LibriSpeech-Long cover only one scenario each (Table 1). Sentence-level metrics like WER are close to saturation and say little about what fails over minutes of speech: speaker drift, changes in the acoustic environment, flat paragraph-level prosody, and monotonous delivery. Human listening tests can capture these failures, but they are expensive and not standardized. Existing LLM-as-judge evaluators mostly give coarse pairwise judgments and ignore consistency over time. The paper asks for a scalable automatic protocol that targets long-form failure modes and agrees with human perception.

Method

Test set. SwanBench-Speech has 1,101 text samples (49.3% Chinese, 50.7% English) in 17 scenarios. The scenarios are grouped by the challenge each one stresses (§3.1, Appendix B):

  • Acoustics: customer service, podcast, chat, debate, audiobook, interview.
  • Semantics: lesson, popular science, presentation, seminar, news.
  • Expressiveness: drama, talk show, hosting, speech, live streaming, sportscast.

The samples cover single-speaker long-form speech, two-speaker dialogue, and 101 three- or four-speaker samples. Mean text length is 271.8 characters for Chinese and 174.6 words for English, with most samples between 80 and 500. This range targets minute-level speech (Appendix B.2). The texts come from three sources:

  • web text, cleaned and proofread by annotators;
  • web audio, processed by ZipEnhancer denoising, a DNSMOS ≥ 3.5 filter, 3D-Speaker diarization, and SenseVoice transcription (from FunAudioLLM), then manually corrected;
  • GPT-5 generations from scenario/topic prompts.

All samples then go through SentenceBERT de-duplication, a GPT-5 quality filter, a DeepSeek-V3.2 privacy and ethics pass, and a final manual review (§3.2–3.3, Appendix A). The transcripts of the web audio also serve as “Real Speech” and “Real Dialogue” reference rows in the results.

Seven metrics (§3.4, Appendix C):

  • Timbre Consistency: mean pairwise cosine similarity between WavLM-TDNN speaker embeddings of 3 s windows (2 s stride) within one output. For dialogue, forced alignment (Paraformer for Chinese, WhisperX for English) splits the audio by speaker, and the score is the average of the per-speaker scores. This differs from speaker similarity to a prompt.
  • Reverb Consistency: standard deviation of SRMR across 3 s windows, after a VAD filter. Lower is better. The metric assumes the acoustic environment should stay fixed within one sample.
  • Sound Fidelity: reference-free SQUIM-PESQ.
  • Content Accuracy: CER for Chinese and WER for English, using FunASR-Nano transcripts and text normalization.
  • Prosodic Coherence: a 1–5 score from SpeechJudge (a fine-tuned Qwen2.5-Omni-7B). Its prompt is rewritten around coherence and flow, rhythmic hierarchy, and naturalness, and the score is averaged over 10 runs.
  • Expressive Richness: Gemini-3-Pro scores 10 s chunks on a 1–5 scale for emotional resonance, character portrayal, and storytelling, then the chunk scores are averaged.
  • Expressive Hierarchy: Gemini-3-Pro scores the full utterance on a 1–5 scale for emotional variation, vocal dynamics, and scene appropriateness.

Human alignment (§3.5, Appendix D). Ten expert listeners (audio engineers, live-streaming specialists, signal-processing researchers) rate generated audio to validate the automatic scores:

  • Timbre maintenance, 50 samples: SRCC 0.75.
  • Clarity and fidelity against SQUIM-PESQ, 50 samples: SRCC 0.72, PLCC 0.47.
  • Prosodic-coherence preference on 50 same-text pairs, compared with the difference in metric scores: SRCC 0.82.
  • Expressiveness, 200 clips: used to choose the judge model. Four MOS predictors and seven LALMs are compared, and Gemini-3-Pro is selected (Tables 4–5).

Systems evaluated (§4.1, Appendix E). All open-source systems run zero-shot with their default settings. Each one is run with all 25 reference prompts (from Emilia, AISHELL-3, NCSSD, LibriSpeech, MSP-Podcast, ChildMandarin, and model repositories), and the results of its best-performing voice are reported. Closed-source systems use their designated official voices.

Key Results

Single-speaker long-form (Table 2). Rankings differ by metric. Among open-source systems:

MetricBestNext / tied
Expressive RichnessVibeVoice 3.71IndexTTS-2 3.59
Expressive HierarchyVibeVoice 3.34IndexTTS-2 2.96
Prosodic CoherenceVibeVoice 3.90FishSpeech 3.80
Sound FidelityFishSpeech 4.10GLM-TTS 3.95
Reverb Consistency (lower is better)GLM-TTS 1.62IndexTTS-2 1.72
CER (Chinese)CosyVoice-2 0.032IndexTTS-2 0.033
WER (English)MegaTTS-3 0.108VibeVoice 0.111
Timbre Consistency0.94 three-way tie (CosyVoice-3, GLM-TTS, IndexTTS-2)

Among closed-source systems:

MetricBestNext / tied
Expressive RichnessGemini-2.5-pro-preview-tts 4.14Minimax 3.80
Expressive HierarchyGemini-2.5-pro-preview-tts 3.51Minimax 3.26
Prosodic CoherenceMinimax 3.95Gemini and OpenAI 3.91
Reverb Consistency (lower is better)Minimax 1.38Gemini 1.44
CER (Chinese)Minimax 0.032
WER (English)Inworld 0.113ElevenLabs 0.115
Sound FidelityElevenLabs 4.02
Timbre ConsistencyElevenLabs 0.96

Gap to real speech (Table 2). The reference recordings score above every system on Richness (4.35 vs at most 4.14), Hierarchy (3.94 vs at most 3.51), and Prosody (4.04 vs at most 3.95). The real-minus-closed-average gap is 0.93 on both Richness and Hierarchy, and 0.25 on Prosody. The other metrics do not favour the recordings:

  • Several systems have lower CER than the recordings (0.032 vs 0.070) and lower reverb variation (1.38 vs 1.91).
  • Several systems have higher fidelity (4.10 vs 3.62). The paper attributes this to noisy web-crawled reference audio (Appendix G.1).
  • Timbre Consistency is 0.96 for both the recordings and ElevenLabs, a tie.
  • English WER is lowest for the recordings (0.074 vs at least 0.108).

Dialogue (Table 3). Among open-source systems, SoulX-Podcast is first or tied for first on six of seven metrics:

  • Prosody 4.01, against ZipVoice-Dialog 3.67.
  • Hierarchy 3.71, against VibeVoice 3.37.
  • Fidelity 3.96.
  • CER 0.061 and WER 0.090.
  • Timbre 0.93, tied with FireRedTTS-2.

The two exceptions are Richness, where VibeVoice leads (3.76 vs 3.44), and Reverb, where MoonCast is lowest (3.06). Among closed-source systems, Gemini leads Prosody (4.06), Richness (4.06), Hierarchy (4.02), and WER (0.092). SeedTTS-Podcast leads Reverb (2.85), Fidelity (3.89), and CER (0.063).

All ten dialogue systems have higher reverb variation than Real Dialogue (2.85–4.43 vs 2.73). The averages are 3.45 for open-source and 3.36 for closed-source systems. The paper reads this as difficulty keeping one acoustic scene across speaker turns. Real Dialogue still leads Richness (4.42) and Hierarchy (4.17). On Prosody, however, Gemini (4.06) and SoulX-Podcast (4.01) score above Real Dialogue (3.95).

Expressiveness profile (Tables 2–3). In single-speaker generation, Hierarchy averages below Richness (open 2.67 vs 3.03, closed 3.01 vs 3.42), although some rows reverse this (FishSpeech, ElevenLabs). Dialogue systems average higher Hierarchy than single-speaker systems (open 3.06, closed 3.76).

Scenarios and length (Figures 3, 4 and 10, §5.1, Appendix F.3). The paper reports these trends:

  • Most metrics drop in the expressiveness scenarios, Richness most of all. Sportscast, hosting, and talk show are the hardest (Figure 11).
  • Reverb Consistency, Prosodic Coherence, and Expressive Hierarchy fall the most as input length grows. Timbre consistency and timbre similarity stay relatively stable.
  • Spark-TTS loses content accuracy as length grows (its Table 2 CER/WER is 0.329/0.240).

Judge selection (Tables 4–5). For Expressive Richness, the four MOS predictors correlate near zero or negatively with expert MOS:

  • UTMOS: SRCC −0.043
  • UTMOSv2: SRCC −0.079
  • SQUIM-MOS: SRCC −0.277
  • DNSMOS: SRCC −0.019

Gemini-3-Pro reaches 0.708. The other LALMs range from 0.170 (Qwen3-Omni-Flash) to 0.527 (Gemini-3-flash). For Hierarchy, Gemini-3-Pro reaches 0.623. The next best are Gemini-3-flash (0.392) and Gemini-2.5-pro (0.374).

Speed (Tables 9–10). The non-autoregressive models have lower RTF than the autoregressive ones:

  • Non-autoregressive: F5-TTS 0.198, MegaTTS-3 0.172, ZipVoice 0.338.
  • Autoregressive: 0.747 (CosyVoice-3) to 3.801 (VibeVoice).

Window ablation (Table 11). With windows of 2 s or shorter, real speech scores lower Timbre Consistency than CosyVoice-3 (e.g. 0.901 vs 0.911 at 2 s). At the chosen 3 s / 2 s setting, the order flips to 0.955 vs 0.929.

Novelty Assessment

The new element is the measurement design for long-form speech. Timbre Consistency (consistency within an utterance, not similarity to a prompt) and Reverb Consistency (SRMR variance over time) target drift that sentence-level metrics cannot see. The two expressiveness dimensions separate average expressiveness per chunk from structure across a whole paragraph. The underlying tools are all existing components (WavLM speaker embeddings, SRMR, SQUIM-PESQ, FunASR, SpeechJudge, Gemini) combined into one pipeline. The paper does not propose a new model.

The validation is more substantial than in many LLM-judge benchmarks: the authors pay ten expert listeners to rate generated audio for each metric family and compare 11 candidate scorers for expressiveness. The validation sets are small, though: 50 samples or pairs for timbre, fidelity, and prosody, and 200 clips for expressiveness.

The scenario coverage is broader than earlier long-form test sets: 17 scenarios against at most three in Table 1, plus two- and four-speaker dialogue.

The system comparison is not like-for-like. Open-source systems are reported with the best of 25 prompt voices, while closed-source systems use fixed official voices.

Field Significance

Moderate — SwanBench-Speech provides a bilingual long-form and dialogue test set with automatic metrics checked against human ratings, together with a 26-entry comparison. Across those systems, the remaining gap to real recordings lies in expressive richness, paragraph-level hierarchy, and (for single-speaker speech) prosodic coherence, while content accuracy and fidelity are at or above the reference recordings. Its expressiveness metrics depend on a proprietary judge, which limits how stable the scores will be over time.

Claims

  • supports: In long-form speech generation, intelligibility and signal fidelity no longer separate current systems from human speech; expressiveness and paragraph-level prosody do.

    Evidence: Several systems match or beat the reference recordings on Chinese CER (0.032 vs 0.070) and SQUIM-PESQ fidelity (4.10 vs 3.62). The recordings exceed every system on Expressive Richness (4.35 vs at most 4.14), Expressive Hierarchy (3.94 vs at most 3.51), and Prosodic Coherence (4.04 vs at most 3.95). (§5.1, Table 2)

  • complicates: Standard automatic MOS predictors cannot stand in for human judgments of speech expressiveness.

    Evidence: Against expert MOS on 200 generated clips, UTMOS, UTMOSv2, SQUIM-MOS, and DNSMOS all give negative SRCC for expressive richness (−0.019 to −0.277). A large audio-language-model judge (Gemini-3-Pro) reaches 0.708, but other LALM judges range from 0.170 to 0.527. (§3.5, §D.4, Table 4)

  • supports: Multi-speaker long-form generation has difficulty keeping one consistent acoustic environment across speaker turns.

    Evidence: All ten dialogue systems show higher SRMR variation than real dialogue (2.85–4.43 vs 2.73), while single-speaker systems are close to real speech on the same metric (averages 1.95 and 1.96 vs 1.91). The user-study feedback reports sound-field and volume mismatches between speakers. (§5.1, §G.1, Tables 2–3)

  • supports: Speaker identity holds up better than acoustic environment, prosody, or expressive structure as long-form generation length grows.

    Evidence: In the length study on five systems, Reverb Consistency, Prosodic Coherence, and Expressive Hierarchy degrade the most with more sentences, while timbre consistency and timbre similarity stay relatively stable. (§4.2, §F.3, Figures 4 and 10)

  • refines: For long-form TTS, non-autoregressive generation is faster and keeps content accuracy stable with length, while autoregressive systems vary more in content robustness; the expressiveness advantage of autoregressive systems is not uniform.

    Evidence: Non-autoregressive RTF is 0.172–0.338 against 0.747–3.801 for autoregressive systems. Spark-TTS loses content accuracy with length (CER/WER 0.329/0.240), while the non-autoregressive models stay stable. On Expressive Hierarchy, F5-TTS (2.77) scores above four of the seven open-source autoregressive systems (CosyVoice-2 2.76, GLM-TTS 2.54, CosyVoice-3 2.45, Spark-TTS 2.38). (§5.2, Tables 2, 9, Figure 4)

Limitations and Open Questions

Warning

Expressive Richness and Hierarchy, the two metrics that separate systems from real speech most clearly, are scored by a proprietary model (Gemini-3-Pro). The authors note that API updates threaten reproducibility. Expressiveness scores are also validated on only 200 clips (SRCC 0.71 and 0.62), and open-source systems are reported with the best of 25 prompt voices while closed-source systems use fixed official voices.

  • Reference recordings are weak upper bounds. The real-speech rows are web-crawled. They score below several systems on fidelity, and Real Dialogue’s English WER (0.137) is higher than every dialogue system except MOSS-TTSD (Table 3). Scores above the “real” row on these metrics do not mean the system exceeds human quality.
  • Some prose does not match the tables. §5.2 states that “F5TTS, despite being the top-performing NAR model, lags significantly behind most AR counterparts in expressive hierarchy”. In Table 2, F5-TTS (2.77) scores above four of the seven open-source autoregressive systems. §5.2 also says CosyVoice3 “significantly lags behind other models in reverb consistency, as shown in Table 2”. Table 2 lists CosyVoice-3 at 2.26, with CosyVoice-2 (2.35) and ElevenLabs (3.05) worse.
  • The timbre thresholds conflict. Appendix D.1 lists “Score < 0.85: Indicates significant timbre drift” and also “Score < 0.93: Demonstrates superior timbre maintenance, with performance comparable to ground truth recordings”. As printed, these overlap and contradict each other.
  • Coverage is narrow in places. Only Chinese and English are covered. The paper evaluates “minute-level” synthesis (texts mostly 80–500 words or characters), not hour-long audiobook generation. The multi-speaker (3–4 speaker) subset is scored only on three closed-source systems (Table 13).
  • Reverb Consistency assumes a fixed acoustic environment. It penalizes intentional scene changes, which the authors acknowledge for scenarios such as outdoor live streaming (Appendix C.2).
  • Some metrics are not robust to periodic artifacts. Timbre Consistency is a global average and can miss periodic timbre fluctuations (Appendix D.1).
  • Release is not yet complete. The test set and evaluation code are described as to be released (Appendix A.4); a demo page is linked.

Wiki Connections

  • Evaluation Metrics — proposes long-form metrics (timbre consistency within an utterance, SRMR-variance reverb consistency, LALM-judged expressive richness and hierarchy) and reports their correlation with expert ratings.
  • Subjective Evaluation — uses ten expert listeners rating generated audio (MOS and pairwise prosody preference) to validate each automatic metric and to select the expressiveness judge.
  • SpeechJudge — provides the prosodic-coherence scorer, used with a re-designed long-form prompt.
  • TTSDS2 — compared in Table 1 as a sentence-level benchmark with four scenarios and 60 clips.
  • Long-Form Speech Generation with Spoken Language Models — source of LibriSpeech-Long, a single-scenario long-form test set that SwanBench-Speech positions itself against.
  • VibeVoice — evaluated in both tracks; the highest open-source scores on single-speaker expressiveness and prosody, and on dialogue richness.
  • SoulX-Podcast — evaluated dialogue system; first or tied for first among open-source dialogue models on six of seven metrics.
  • F5-TTS — evaluated non-autoregressive baseline, used in the paper’s argument about the trade-off between autoregressive and non-autoregressive systems.
  • Spark-TTS — evaluated autoregressive system whose content accuracy degrades with length; the paper links this to short-segment training data.
  • MoonCast — evaluated dialogue system with the highest Chinese CER in the dialogue track (0.313).