arXiv · 2025 · Preprint
Liu, Duan, Wang et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓
An open-source, event-driven modular cascaded speech-to-speech dialogue framework that argues cascaded pipelines, when systematically re-engineered, can match or exceed end-to-end omni-models on latency, paralinguistic awareness, and tool-augmented capability, challenging the field’s prevailing shift toward monolithic end-to-end architectures.
Problem
Spoken dialogue systems split into two paradigms: cascaded pipelines (ASR → LLM/NLU → TTS) and end-to-end (E2E) “omni-models” that unify perception, reasoning, and generation in a single network. The field’s momentum has shifted toward E2E systems on the premise that they offer lower latency, better capture of paralinguistic cues, and more empathetic responses. In practice, E2E systems face prohibitive training costs, a tendency toward intelligence degradation relative to text-only LLM backbones, and unreliability that limits real-world deployment, while conventional cascaded systems are typically implemented as a rigid three-stage pipeline that discards paralinguistic information and struggles to integrate modern LLM capabilities like retrieval-augmented generation and tool use. No prior system had demonstrated that a cascaded architecture, carefully re-engineered, could close the gap with E2E systems on latency and paralinguistic awareness while retaining modularity.
Method
X-Talk decomposes the speech-to-speech dialogue process into four core functional modules: a Speech Interaction Frontend (client-side streaming speech enhancement via FastEnhancer and voice activity detection via Silero VAD, chosen to minimize round-trip latency for interruptions), Speech Understanding (multi-backend ASR including Zipformer, SenseVoice, and Paraformer with a pseudo-streaming mechanism for offline ASR models, plus speaker identification via a Pyannote/WeSpeaker pipeline, and acoustic-scene/paralinguistic captioning via Omni-Captioner), an LLM-Driven Conversational Agent (an LLM augmented via LangChain with web search, local RAG retrieval, on-the-fly timbre switching, emotion switching, and an explicit event-driven “thinking” module decoupled from normal response latency), and Speech Generation (a coordinated “chorus” of TTS backends, including IndexTTS 1.5/2, CosyVoice, GPT-SoVITS, ElevenLabs, and Edge TTS, served via vLLM). These modules are orchestrated through an Event-driven Architecture: all component inputs and outputs are encoded as structured events propagated over a centralized, layered event bus (Frontend, Event Center, Managers, Agents, and Models layers) communicating asynchronously via publish-subscribe, which decouples components and enables independent retraining, swapping, or scaling of any module without touching the rest of the pipeline. To reduce perceived and actual latency, the system streams every stage (incremental ASR hypotheses, token-level LLM output, clause-segmented TTS playback), injects buffered filler utterances to mask tool-call latency, and implements a VAD-driven barge-in preemption loop with rule-based false-interruption filtering (minimum audio duration, empty/filler-only ASR transcripts).
Key Results
Across combinations of ASR (Paraformer, SenseVoice streaming/offline), LLM (Qwen3-8B/30B/80B), and TTS (IndexTTS 1.5/2, CosyVoice) backends, X-Talk’s default configuration (SenseVoice streaming, Qwen3-30B, IndexTTS 1.5) achieves end-to-end response latencies of 272-610 ms across 5-60 second Chinese and English inputs, i.e. consistently sub-second. Latency is largely insensitive to input audio length when streaming ASR is used, since it is dominated by LLM first-sentence generation and TTS first-sentence synthesis rather than full-utterance transcription; switching to offline (non-streaming) ASR makes latency grow sharply with input duration (635 ms at 5s to 3798 ms at 60s), and switching to a larger LLM (80B via API) or a slower TTS backend (IndexTTS 2, CosyVoice API) increases latency, quantifying each component’s contribution to the end-to-end budget. No comparison against actual end-to-end omni-model systems’ measured latency is reported in this version; the paper’s claims of parity with E2E systems rest on these absolute sub-second numbers and qualitative feature comparisons (paralinguistic awareness via dedicated understanding modules, tool use via RAG and web search) rather than head-to-head latency benchmarking against a specific E2E baseline.
Novelty Assessment
None of X-Talk’s individual components are new: it integrates existing ASR (Zipformer, SenseVoice, Paraformer), TTS (IndexTTS, CosyVoice, GPT-SoVITS), speaker/embedding, and VAD models via off-the-shelf checkpoints. The paper’s contribution is the systems-level design: an event-driven, layered, publish-subscribe architecture that both minimizes latency (via client-side frontend processing, streaming at every stage, and buffered filler utterances) and preserves fine-grained modularity (via interface protocols that decouple layers so any component can be swapped without retraining others). This directly targets and empirically pushes back against a specific claim common in recent E2E dialogue system papers, that only end-to-end modeling can achieve the responsiveness and paralinguistic sensitivity needed for natural spoken interaction.
Field Significance
high — X-Talk provides a fully open-sourced, systematically re-engineered cascaded architecture that directly challenges the field's prevailing assumption that end-to-end omni-models are necessary for low-latency, paralinguistic-aware spoken dialogue, backed by concrete sub-second latency measurements and a public demo.
Beyond its individual latency numbers, the paper’s event-driven, layered orchestration design offers a reusable architectural template for the field: a way to compose specialized understanding, reasoning, and generation components (including tool use and retrieval, which remain difficult to integrate into fully differentiable E2E speech models) without sacrificing the responsiveness that has been E2E’s main selling point.
Claims
- contradicts: Low-latency, paralinguistically-aware spoken dialogue does not require a single, jointly-trained end-to-end model; a carefully re-engineered cascaded pipeline with streaming at every stage can achieve comparable responsiveness while retaining full modularity.
Evidence: X-Talk’s default cascaded configuration (streaming ASR, LLM, TTS) achieves 272-610 ms end-to-end response latency across 5-60 second Chinese and English inputs, using off-the-shelf ASR, LLM, and TTS components connected through an event-driven architecture rather than a single jointly-trained network. (§5.2, Table 2)
- supports: Decoupling a spoken dialogue system’s speech understanding into separate specialized components (recognition, speaker identification, emotion/paralinguistic captioning, acoustic scene analysis) can recover paralinguistic and environmental information that a conventional ASR-only cascaded pipeline discards, without requiring a unified end-to-end model.
Evidence: X-Talk’s Speech Understanding module integrates dedicated ASR, speaker recognition, and acoustic-scene captioning (Omni-Captioner) components whose outputs are injected into the LLM’s context, explicitly designed to recover paralinguistic and environmental cues that “conventional cascaded systems often rely solely on ASR” and thereby lose. (§3.2.2)
- complicates: Streaming-capable automatic speech recognition is important for maintaining low, duration-insensitive end-to-end latency in a cascaded spoken dialogue pipeline; switching to non-streaming ASR causes latency to scale sharply with input length.
Evidence: Replacing streaming SenseVoice with SenseVoice in offline mode increases latency from 284 ms to 635 ms at 5-second inputs and from 449 ms to 3798 ms at 60-second inputs in otherwise identical LLM/TTS configurations. (§5.2, Table 2)
- complicates: Reporting an end-to-end dialogue system’s own absolute latency numbers is not equivalent to demonstrating latency parity with competing end-to-end architectures, absent a direct head-to-head comparison under matched conditions.
Evidence: The paper reports X-Talk’s sub-second latency across multiple internal ASR/LLM/TTS configurations but does not report measured latency for any specific end-to-end omni-model system under the same experimental protocol, so its claim of matching or exceeding E2E latency is not directly benchmarked in this version. (§5.2)
Limitations and Open Questions
The latency evaluation is entirely internal, comparing configurations of X-Talk against each other rather than against measured latency of any specific end-to-end omni-model baseline (Moshi, GLM-4-Voice, Qwen2.5/3-Omni, etc.), so the paper’s central claim of parity or superiority relative to E2E systems on latency is argued qualitatively and by absolute latency figures rather than demonstrated head-to-head. No quantitative evaluation of dialogue quality, paralinguistic accuracy, empathy, or barge-in reliability is reported; these capabilities are described architecturally (module design) rather than measured against ground truth or human judgment. Several described components (target speaker extraction, acoustic echo cancellation, additional generative models) are explicitly noted as under active development rather than complete at the time of writing.
Wiki Connections
- Speech-to-Speech Systems — presents a fully-engineered cascaded speech-to-speech dialogue system, explicitly framed as a challenge to the field’s shift toward end-to-end omni-model architectures for this task.
- Streaming TTS — implements streaming synthesis via clause-level segmentation and sequential playback for TTS backends without native streaming support, alongside streaming ASR and token-level LLM output, to achieve sub-second end-to-end latency.
- Spoken Language Models — contrasts its explicit, event-driven “thinking” module against end-to-end spoken language models that attempt simultaneous thinking-and-talking within a single unified model.
- FireRedChat — cited as a comparable recent cascaded, LLM-enhanced full-duplex voice interaction system representing the same architectural paradigm X-Talk advances.
- IndexTTS — IndexTTS 1.5 serves as the default TTS backend in X-Talk’s baseline configuration and is directly benchmarked in the system’s latency experiments.
- IndexTTS2 — integrated as an alternative TTS backend, benchmarked against IndexTTS 1.5 and CosyVoice in the latency experiments and used for native emotion-vector control in the Emotion Switching feature.
- CosyVoice 2 — integrated as an alternative TTS backend (CosyVoice-v3-flash, accessed via API) and directly compared against IndexTTS variants in the latency benchmark.
- Mini-Omni-Reasoner — cited as a representative end-to-end approach that unifies “thinking” and “talking” within a single model, contrasted against X-Talk’s explicit, decoupled event-driven thinking module.