NAACL · 2025 · Conference
Siddhant Arora et al. (Carnegie Mellon University) · → Paper · Demo: ✓ · Code: ✓
A unified, open-source Gradio toolkit that lets researchers interact with and automatically evaluate both cascaded (ASR + LLM + TTS) and end-to-end spoken dialogue systems through a single standardized web interface.
This is a toolkit and demo paper. Its contribution is infrastructure (a unified evaluation interface for spoken dialogue systems) rather than a novel model architecture or training method.
Problem
Spoken dialogue systems are increasingly built either as cascaded pipelines (voice activity detection, ASR, dialogue response generation, TTS) or as end-to-end (E2E) audio foundation models that process and generate speech directly. Each released system typically ships with its own bespoke web interface, its own input/output conventions, and its own backend infrastructure, which makes it difficult for researchers to directly compare cascaded and E2E approaches or to identify which design choices drive observed differences in quality. Prior comparison platforms address only adjacent problems: ChatBot Arena benchmarks text-only LLM chatbots, TTS Arena benchmarks TTS models in isolation, and TalkArena benchmarks audio foundation models but is limited to audio-in/text-out, single-turn interactions. No existing platform supports multi-turn, full-duplex-style interaction with multiple cascaded and E2E spoken dialogue systems side by side, nor provides built-in, on-the-fly automated evaluation across the ASR, dialogue, and TTS sub-components.
Method
ESPnet-SDS is built on top of the existing ESPnet speech processing toolkit and adopts a modular wrapper design. Each functional stage of a spoken dialogue system (VAD, ASR, text dialogue response generation, TTS) is implemented as an abstract wrapper class (AbsASR, AbsLLM, AbsTTS, AbsVAD) under espnet2/sds, with concrete implementations provided for specific models (e.g., ESPnet ASR, OWSM, OWSM-CTC, Whisper for ASR; HuggingFace causal LMs for dialogue; ESPnet TTS and ChatTTS for synthesis; WebRTC VAD for turn detection). A separate AbsE2E wrapper class allows E2E spoken dialogue models (demonstrated with Mini-Omni) to be integrated directly from their public checkpoints and inference code, bypassing the cascaded sub-modules. All wrappers are unified under a single ESPnet_SDS_Model_Interface class that supports both cascaded and E2E systems and manages real-time, VAD-driven conversation flow, including barge-in (user audio is continuously collected to allow interruptions). A “Template” module (egs2/TEMPLATE/sds1) packages the interface wrapper, evaluation metric functions, and a Gradio-based App.py into a reusable recipe structure that researchers can configure via a bash launch script without needing to modify the underlying wrapper code.
The toolkit reports practical scale: 294 supported ASR models, 36,464 addressable HuggingFace LLMs for dialogue generation, and 80 ESPnet TTS models, plus ChatTTS, yielding 41 selectable system configurations in the authors’ example demo (40 cascaded combinations of 5 ASR models × 2 LLMs × 4 TTS models, plus 1 E2E system). On-the-fly automated evaluation is implemented at three levels: submodule-specific (ASR via WER/CER against judge-model transcripts; TTS via the VERSA toolkit’s intelligibility and audio-quality metrics, including UTMOS, DNSMOS variants, PLCMOS, and SSQA; dialogue-text quality via perplexity, VERT diversity, and BERT-based context similarity), conversation-level (turn-taking statistics, speaking rate, backchannel rate), and human-in-the-loop (a 4-point naturalness/relevance rating mechanism, with optional, consent-gated logging of interaction data to a remote HuggingFace dataset backend). Models are loaded on the fly to limit memory footprint when switching between the many supported system configurations.
Key Results
Using the toolkit’s own evaluation suite on the Switchboard Eval 2000 dataset (11 hours of human-human conversational telephone speech), the authors report ASR module comparisons (Table 2) where OWSM 3.1 achieves the lowest WER/CER (14.4%/11.2%), ahead of OWSM-CTC 3.2 (15.9%/10.3%), OWSM-CTC 3.1 (18.1%/14.4%), Whisper-large (22.5%/22.3%), and a degenerate ESPnet LibriSpeech ASR baseline (342.2%/298% WER/CER, reflecting domain mismatch with conversational telephone speech). For text dialogue modules (Table 3), LLaMA-3.2-1B produces less diverse but more coherent and context-relevant responses than SmolLM v2-1.7B, while the E2E system Mini-Omni produces highly coherent, contextually relevant responses but shows significant self-overlap and template-like repetition across turns. For TTS modules (Table 4), the single-speaker LJSpeech VITS model achieves the best intelligibility and audio quality among the compared TTS systems, while Mini-Omni’s directly synthesized speech is intelligible but has markedly lower audio quality than dedicated TTS models (UTMOS 2.88 vs. 3.97-4.07 for the VITS variants); the authors note this comparison is not fully fair since Mini-Omni conditions on user audio rather than ground-truth text. A small pilot study (4 participants, ~2 hours of human-AI interaction with a WebRTC VAD + OWSM-CTC + LLaMA-3.2-1B + LJSpeech VITS cascaded system) found higher latency-driven gaps and far less overlapping speech than natural human-human conversation (Table 9), and a slower system speaking rate (138.7 vs. 204.5 words/min for humans); 75.8% of system outputs were rated “very natural” and 71.9% “highly relevant” by the small participant pool.
Novelty Assessment
The contribution is an engineering integration: the toolkit assembles existing open-source ASR, LLM, TTS, and E2E speech foundation models behind a common wrapper interface and a single Gradio front end, rather than proposing a new model, architecture, or training method. Its claimed novelty over prior comparison platforms (Table 1) is the combination of supporting speech input and output, multi-turn interaction, multiple systems, and on-the-fly evaluation simultaneously, none of which prior platforms (ChatBot Arena, TTS Arena, TalkArena, Moshi’s own demo, Hugging Face’s Speech-to-Speech demo) combine. The evaluation metrics themselves (WER/CER, UTMOS, DNSMOS, PLCMOS, SSQA, perplexity, VERT, BERT similarity) are all adopted from prior work via the VERSA toolkit and standard NLP/ASR practice rather than newly proposed; the contribution here is packaging them into a unified, automated, on-the-fly comparison pipeline. The Switchboard analysis functions primarily as a demonstration of the toolkit’s evaluation capability rather than a rigorous, large-scale benchmark study: the human pilot involves only 4 participants drawn from the author group and collaborators, and several comparisons (notably Mini-Omni vs. dedicated TTS models) are explicitly flagged by the authors as not directly comparable.
Field Significance
Moderate — the toolkit lowers the barrier to comparing cascaded and end-to-end spoken dialogue systems under a single interface and a shared set of automated metrics, addressing real fragmentation in how such systems are typically demonstrated and evaluated. Its value is infrastructural: it provides a reusable, extensible recipe template (with documented pull-request-based extension for new E2E systems) rather than new modeling techniques, and the reported comparisons of ASR, TTS, and dialogue submodules serve mainly to validate that the toolkit’s evaluation pipeline produces interpretable, actionable signal.
Claims
- supports: Cascaded spoken dialogue pipelines built from dedicated, task-specific ASR and TTS components can produce substantially higher audio output quality than current end-to-end spoken dialogue systems that generate speech directly from audio input.
Evidence: On Switchboard Eval 2000, the directly-synthesized speech from the E2E system Mini-Omni scores UTMOS 2.88, versus 3.97-4.07 for cascaded pipelines using LJSpeech, LibriTTS, or VCTK VITS as the TTS module. (§7, Table 4)
- complicates: End-to-end spoken dialogue systems that respond directly to audio input without explicit dialogue-context modeling can produce coherent individual responses while still exhibiting unnatural conversational behavior such as repeated or templated phrasing across turns.
Evidence: The text dialogue evaluation finds that Mini-Omni “produces highly coherent and contextually relevant responses” but “exhibits significant overlap in its responses, often generating template-like outputs,” which the authors note could make conversations feel less natural in deployment. (§7, Table 3)
- complicates: Automatic ASR module benchmarking on conversational telephone speech corpora can produce highly misleading word error rates when the candidate ASR model’s training domain does not match the conversational, disfluent register of the test data.
Evidence: An ESPnet LibriSpeech-trained ASR model evaluated on Switchboard Eval 2000 yields a WER of 342.2% and CER of 298%, far worse than domain-matched models like OWSM 3.1 (14.4% WER), illustrating a severe train/test domain mismatch effect rather than a genuine architecture comparison. (§7, Table 2)
- supports: The choice of input transcript quality substantially affects downstream TTS intelligibility metrics independent of the TTS model itself, because more grammatically well-formed text is easier for TTS systems to synthesize accurately.
Evidence: Replacing ground-truth (disfluent, spontaneous) Switchboard transcripts with LLM-generated text responses as TTS input improves intelligibility across all tested TTS models, e.g. LJSpeech VITS WER drops from 16.1% to 6.0% under OWSM 3.1 hypotheses. (§A.4, Table 8)
- complicates: Text dialogue response generators integrated into a cascaded spoken dialogue pipeline are not strongly degraded by upstream automatic speech recognition errors relative to using ground-truth transcripts, though smaller language models may be more sensitive to this noise than larger ones.
Evidence: When substituting OWSM 3.1 ASR transcripts for ground-truth transcripts as dialogue context, LLaMA-3.2-1B perplexity changes only slightly (48.2 to 47.5) while SmolLM v2-1.7B perplexity degrades substantially (113.7 to 285.1). (§A.3, Table 7)
Limitations and Open Questions
Warning
The headline comparative analysis (Section 7) is conducted on a single dataset (11 hours of Switchboard Eval 2000 telephone speech) with only one example E2E system (Mini-Omni) and a small set of cascaded module combinations, and the human pilot study involves just 4 participants drawn from the author group and research collaborators; the reported findings should be read as a demonstration of the toolkit’s evaluation capability rather than a generalizable benchmark of cascaded versus E2E spoken dialogue systems.
The paper itself notes that the range of currently supported E2E audio foundation models is limited (only Mini-Omni is demonstrated), that performance has not been assessed in low-resource language, noisy-environment, or multi-speaker settings, and that several comparisons (e.g., Mini-Omni’s TTS-specific metrics against dedicated TTS models) are not strictly fair since the systems take different inputs (audio vs. ground-truth text). Human feedback collection and interaction-data logging are disabled by default for privacy reasons, which limits the scale at which human-in-the-loop evaluation data can currently be gathered through the public demo.
Wiki Connections
- Spoken Language Model — provides a shared evaluation harness for comparing end-to-end speech language models like Mini-Omni against cascaded ASR+LLM+TTS pipelines on identical conversational data.
- Speech-to-Speech — unifies cascaded and end-to-end spoken dialogue interaction under one interface, directly addressing the lack of standardized comparison tooling across speech-to-speech system designs.
- Evaluation Metrics — packages a broad set of automated submodule- and conversation-level metrics (WER/CER, UTMOS, DNSMOS, PLCMOS, SSQA, perplexity, VERT, turn-taking statistics) into a single on-the-fly evaluation pipeline.
- Subjective Evaluation — implements a human-in-the-loop feedback mechanism collecting 4-point naturalness and relevance ratings, paired with a small pilot study against Switchboard human-human conversation as a reference.
- Zero-Shot TTS — integrates multi-speaker zero-shot-capable TTS models (e.g., VCTK VITS) as interchangeable submodules within its cascaded pipeline configurations.
- Mini-Omni — wrapped as the toolkit’s example end-to-end spoken dialogue system and directly compared against cascaded pipelines on text-dialogue and TTS-specific metrics.
- Moshi — cited as a representative full-duplex end-to-end spoken dialogue model motivating the need for standardized cross-system comparison tooling.
- SpeechGPT — cited as an early end-to-end speech-text model limited to single-channel speech modeling, contextualizing the toolkit’s framing of the cascaded-to-E2E evolution.
- ESPnet-SpeechLM — a companion toolkit from the same research group that builds the underlying speech language models this demo’s E2E wrapper interface is designed to host and evaluate.