arXiv · 2026 · Preprint
Qiu et al. (Salesforce AI Research) · → Paper · Demo: ✗ · Code: ✓
Provides a from-scratch technical tutorial and latency benchmark for building a fully self-hosted, streaming enterprise voice agent, showing that a cascaded STT→LLM→TTS pipeline remains the only viable fully self-hosted architecture for sub-second time-to-first-audio because no open native speech-to-speech model yet ships a self-hostable low-latency audio decoder.
Problem
Enterprise interest in voice-based conversational agents has grown alongside LLM reasoning and tool-use capability, but building a real-time, fully self-hosted voice interface remains poorly documented. Open-source speech-to-speech models proliferate, but none ships a tutorial for building a realtime voice agent; production orchestration frameworks such as Pipecat and LiveKit Agents provide working components but do not explain the underlying streaming mechanisms; and existing enterprise voice-agent evaluation frameworks assume turn-based, non-streaming pipelines with function calling. No open codebase walks through the construction of a complete, streaming, function-calling-capable enterprise voice agent end to end, explaining every component.
Method
The paper first surveys the deployment landscape and benchmarks the closest available native speech-to-speech (S2S) candidate before settling on the architecture it develops in depth: a cascaded streaming pipeline. It classifies open speech-to-speech systems into three levels: Level 1, models that reason natively in speech tokens with no mandatory text intermediate (e.g., Moshi, Spirit LM); Level 2, text LLMs augmented with jointly trained speech encoders/decoders that “think” in text but read and write speech (e.g., Qwen2.5-Omni, Qwen3-Omni, GLM-4-Voice, Kimi-Audio, Step-Audio, LLaMA-Omni, Freeze-Omni); and Level 3, cascaded pipelines that chain separate ASR, LLM, and TTS models. It then benchmarks Qwen3-Omni, the only Level 2 model in the survey with function calling, across three deployment configurations: a self-hosted vLLM deployment that serves only the Thinker (text generation from audio input, no audio output); a self-hosted HuggingFace Transformers deployment that runs the full Thinker-Talker pipeline (audio in, audio out) on an unoptimized backend; and the cloud-only DashScope Realtime API, which streams optimized audio-to-audio output over WebSocket with server-side voice activity detection (VAD).
Having found no fully self-hosted native S2S option that meets realtime targets, the tutorial builds and open-sources a Level 3 cascaded pipeline from independently instrumented components: Deepgram Nova-3 for streaming speech-to-text over a persistent WebSocket (20ms PCM chunks, distinguishing partial from final transcripts), a self-hosted LLM served by vLLM behind an OpenAI-compatible streaming chat-completions API with function-calling support, and ElevenLabs’ eleven_turbo_v2_5 for streaming text-to-speech. The critical orchestration primitive is a sentence buffer that accumulates streamed LLM tokens, detects sentence boundaries while filtering abbreviation and decimal-number false positives, and forwards each completed sentence to TTS as soon as it is ready, so TTS synthesis of earlier sentences overlaps with LLM generation of later ones. Turn-taking is handled by a Silero VAD state machine (idle → listening → processing → speaking, with an interruption path back to listening), and the browser client uses AudioWorklet processors for low-latency capture and jitter-buffered playback over a binary WebSocket protocol. An enterprise agent layer adds a recursive tool-use loop, following the OpenAI function-calling protocol, over a five-tool hospital-receptionist scenario (check_availability, schedule_appointment, cancel_appointment, get_patient_info, get_doctor_info). All components and the full 9-chapter tutorial are released as open-source code.
Key Results
Native S2S evaluation: Qwen2.5-Omni-7B’s DiT-based Talker runs at roughly 0.5x realtime (about 2s to generate 1s of audio), giving a roughly 13.2s time-to-first-audio (TTFA) even with per-sentence streaming, and offers no function calling and no incremental audio output. Qwen3-Omni-30B-A3B (a 30B-total/3B-active MoE Thinker-Talker with a multi-codebook Talker and native function calling) is markedly faster, but only in specific deployments: the self-hosted vLLM deployment serving only the Thinker reaches 516ms audio-to-text TTFA at 168 tokens/second but produces no audio output; the self-hosted Transformers deployment running the full Thinker-Talker pipeline produces audio but at roughly 145,694ms TTFA (2.91x real-time factor), reflecting a roughly 36x throughput gap versus vLLM (4.6 vs. 168 tokens/second); the cloud-only DashScope Realtime API reaches 702ms audio-to-audio TTFA with streaming and partial function-calling support, but is not self-hostable.
Cascaded pipeline (the paper’s own build): individually measured components are Deepgram STT (402ms P50, 184ms minimum), LLM time-to-first-token (457ms P50 with a cloud OpenAI-compatible API; 337ms P50 with self-hosted vLLM after warmup, though vLLM cold start reaches 4.3s), and ElevenLabs TTS time-to-first-byte (219-221ms P50, under 20% variance, the most consistent stage). The measured end-to-end pipeline achieves 755ms TTFA (729ms best case) with full function-calling support, below the sequential (non-overlapping) estimate of roughly 950-1050ms, which the paper attributes to streaming overlap between stages. Comparing across all five approaches, only the DashScope cloud API (about 702ms) and the self-hosted cascaded pipeline (about 755ms) reach sub-second TTFA with function calling and streaming audio; of these, only the cascaded pipeline is simultaneously self-hosted, function-calling-capable, and audio-streaming.
Novelty Assessment
The contribution is explicitly not architectural: every component (Deepgram, vLLM, ElevenLabs, Silero VAD) is an existing off-the-shelf tool, and the sentence-buffering and VAD state-machine patterns are acknowledged as adaptations of patterns already used by Pipecat and LiveKit. The paper’s value is in (1) a landscape survey that catalogues and levels 25+ speech-to-speech models and 30+ frameworks that individually document their own components but not how to compose them into a realtime agent; (2) an original, reproducible latency comparison across native S2S deployment configurations that is not otherwise available, since the vLLM-Thinker-only and Transformers-full numbers for Qwen3-Omni required the authors’ own instrumented runs rather than being reported in the model’s technical report; and (3) a fully worked, tested reference implementation released as open-source code, in a more transparent and pedagogical form than existing production frameworks provide. Whether the specific latency numbers generalize beyond the authors’ single-GPU hardware and specific API tiers is not established; the comparison is a snapshot of one hardware/software configuration rather than a systematic sweep.
Field Significance
Moderate — the paper does not advance TTS, VC, or spoken-conversational-agent modeling itself, but it provides a rare direct, apples-to-apples latency comparison between native speech-to-speech deployment options and a cascaded pipeline, at a moment when native speech-to-speech models are approaching but have not yet reached fully self-hosted realtime viability. Its practical value is primarily as a reproducible reference architecture and benchmark rather than as new modeling insight.
Claims
- supports: Sentence-level streaming that overlaps LLM token generation with downstream TTS synthesis substantially reduces perceived response latency in cascaded speech agent pipelines relative to fully sequential, turn-based execution.
Evidence: The measured end-to-end time-to-first-audio of 755ms undercuts the sequential sum-of-stage estimate of roughly 950-1050ms, attributed to overlapping STT, LLM first-sentence generation, and TTS time-to-first-byte across stages. (§8, Table 4)
- complicates: Native speech-to-speech models that jointly generate audio and text from a single backbone can face a sharp trade-off between deployment self-hostability and real-time audio-generation latency, even when the underlying architecture is designed for streaming.
Evidence: Qwen3-Omni’s multi-codebook Talker reaches 702ms audio-to-audio latency only via the cloud-hosted DashScope Realtime API; the same model’s fully self-hosted HuggingFace Transformers deployment produces audio at roughly 145,694ms TTFA (2.91x real-time factor), and its self-hosted vLLM deployment supports text-only output (no Talker, no audio) at 516ms. (§3.2-3.3, Table 3)
- complicates: Serving-backend efficiency, not only model architecture, is a first-order determinant of whether a speech-capable model can meet real-time latency targets.
Evidence: The same Qwen3-Omni model produces 4.6 tokens/second via the unoptimized HuggingFace Transformers backend versus 168 tokens/second via vLLM, a roughly 36x gap that turns an architecturally streaming-capable Talker into a non-realtime one purely because of serving-backend inefficiency. (§3.2, §B, Table 3)
- supports: A cascaded ASR-LLM-TTS pipeline built entirely from independently hosted, off-the-shelf streaming components can reach sub-second time-to-first-audio with full function-calling support, without requiring an end-to-end trained model.
Evidence: The pipeline combining Deepgram Nova-3 streaming STT, a vLLM-served LLM with OpenAI-compatible function calling, and ElevenLabs streaming TTS achieves a measured 755ms time-to-first-audio (729ms best case) on a hospital-receptionist enterprise agent with five callable tools. (§7-8, Table 4)
Limitations and Open Questions
Warning
All latency numbers are a single-hardware, single-vendor-tier snapshot (one NVIDIA H200 GPU and specific Deepgram/ElevenLabs/OpenAI-API configurations) rather than a systematic sweep across hardware, model sizes, or API tiers, and the enterprise agent evaluation covers only one scenario (a five-tool hospital receptionist) with no reported functional-correctness or task-success metric for the tool-calling loop itself. (§7-8)
The tutorial reports no formal human evaluation (no MOS or user study) of the resulting agent’s conversational quality, and its VAD-based interruption/turn-taking state machine is described architecturally but not benchmarked for interruption-handling accuracy. Because the central finding, that a cascaded pipeline remains necessary, depends on the current absence of a self-hostable, optimized native speech-to-speech Talker, the paper’s practical recommendation is explicitly time-sensitive and could be superseded by future serving-stack support (e.g., vLLM adding Talker support) that the authors themselves anticipate.
Wiki Connections
- Speech-to-Speech — builds and benchmarks a cascaded (STT→LLM→TTS) speech-to-speech system as the practical alternative to native end-to-end audio-to-audio models, directly instantiating the cascade sub-paradigm.
- Spoken Language Model — empirically benchmarks the Qwen3-Omni Thinker, a text LLM adapted to consume raw audio input, isolating its audio-to-text latency and throughput from the full Thinker-Talker pipeline.
- Streaming TTS — measures ElevenLabs’ streaming text-to-speech time-to-first-byte and integrates it into a sentence-buffered pipeline that begins synthesis before the LLM finishes generating a full response.
- Evaluation Metrics — introduces a latency-decomposition benchmarking methodology (per-component P50/mean/min/max plus end-to-end time-to-first-audio) for comparing native and cascaded speech agent architectures.
- Qwen3-Omni — benchmarks this model directly across three deployment configurations (vLLM Thinker-only, Transformers full pipeline, DashScope Realtime API), finding it the first open Level 2 speech-to-speech model with function calling but without a self-hostable low-latency Talker.
- Qwen2.5-Omni — benchmarks this model as the baseline native S2S candidate, measuring its DiT-based Talker at roughly 0.5x realtime and finding it lacks function calling and incremental audio output.
- Moshi — cites this Level 1, text-free speech-to-speech model as the architecturally purest alternative, achieving roughly 200ms latency but with more limited reasoning capability than text-centric cascaded agents.
- WavChat — draws on this survey of spoken dialogue models for background on the native speech-to-speech landscape that motivates the tutorial’s own model-vs-framework gap analysis.