arXiv · 2026 · Preprint
Kamahori et al. · → Paper · Demo: ✗ · Code: ✓
Introduces VoxServe, a unified serving system for Speech Language Models that decouples model architecture from system-level optimizations and achieves 10-20x higher throughput than existing per-model serving implementations at comparable latency.
Problem
Speech Language Models (SpeechLMs) combine an LLM backbone with audio-specific modules (codebook-structured token representations, audio detokenizers) into multi-stage inference pipelines with heterogeneous compute, memory, and I/O characteristics. Unlike text-only LLMs, no standardized serving framework exists for this class of models: new SpeechLM releases typically ship with bespoke inference stacks that support only one architecture, forcing developers to reimplement serving-related optimizations (batching, caching, scheduling) whenever they adopt a new model family. This fragmentation is compounded by streaming applications’ unique performance demands: audio playback must begin with minimal delay (low Time-To-First-Audio) and subsequent chunks must arrive fast enough to sustain uninterrupted playback, requirements not captured by standard text-LLM serving metrics like TTFT/TPOT.
Method
VoxServe is a serving system built around a model-execution abstraction that separates model-specific logic from system-level optimization. Each supported SpeechLM implements a common interface with four stages: Preprocess (prompt formatting, tokenization, optional audio-encoder inference), LLM Forward (backbone decoding over a 2D token/codebook tensor with optional continuous features and masks), Sampling (temperature/top-k/top-p decisions plus repetition penalty, producing inputs for the next forward pass), and Postprocess (batched, fixed-shape audio detokenization into streamed waveform chunks, with optional cached state for detokenizers like Mimi’s or CosyVoice’s that require cross-chunk context). An optional depth-forward/sampling method handles SpeechLMs that autoregressively sample multiple codebooks via a depth-wise submodel.
At the system level, the execution process is organized into a Scheduler (orchestrates the request lifecycle and decides which requests run the LLM or detokenizer at each iteration), a Worker (manages GPU resources and executes prefill/decode/detokenize operations), and the Model abstraction itself.
Because the model interface standardizes tensor contracts (input tokens, features, masks) and uses fixed execution shapes per policy, VoxServe places the LLM Forward and Postprocess stages on CUDA-graph-captured fast paths (using FlashInfer for attention) to reduce kernel-launch overhead, while control-flow-heavy stages (preprocess, sampling) stay outside CUDA graphs.
For streaming performance specifically, the Scheduler distinguishes a TTFA-critical startup phase (before the first audio chunk) from a streaming-viability-critical steady-state phase (subsequent chunks), and dynamically reprioritizes requests: since streaming viability is a binary per-chunk property, requests with slack can be deprioritized without degrading their quality of service, freeing resources for requests closer to their latency deadline. Separately, VoxServe adopts an asynchronous pipeline that schedules the LLM backbone forward pass and detokenizer forward pass as distinct GPU tasks with explicit per-request state dependencies, overlapping GPU computation with independent CPU-side work to reduce pipeline bubbles.
VoxServe is implemented in ~20,000 lines of Python/PyTorch and supports seven open-source TTS/STS SpeechLM architectures: Chatterbox TTS, CosyVoice 2.0, CSM 1B, GLM-4-Voice, Orpheus 3B, StepAudio 2, and Zonos-v0.1.
Key Results
Against official per-model serving baselines (each combining an LLM-serving system for the backbone with a custom detokenizer engine, evaluated on the three models with available open-source baselines: CosyVoice 2.0, Orpheus 3B, Step-Audio 2), VoxServe achieves 10-20x higher request rate at comparable p90/p99 TTFA while maintaining high streaming viability, measured on a single H100 GPU with request rates sampled from LibriTTS (TTS models) and the VoiceBench AlpacaEval subset (STS models). For CosyVoice, the baseline reaches 500ms p90 TTFA at ≈0.4 req/s, while VoxServe sustains the same TTFA up to 4.0 req/s with 100% streaming viability. For Orpheus, p90 TTFA stays below 500ms up to 10 req/s, but baseline streaming viability degrades past 8.0 req/s due to Orpheus’s 86 tokens/s token rate; VoxServe delivers more than 10x higher throughput at a given TTFA. Step-Audio 2 achieves the lowest sustained request rate of the three due to its 9B parameter size, but VoxServe still outperforms its baseline.
Ablations isolate the contribution of each design choice: the optimized streaming-aware scheduler yields ≈2.5x TTFA reduction at fixed request rate, and the asynchronous pipeline contributes a further ≈15% TTFA reduction at high request rate. Under data parallelism across up to four H100 GPUs, VoxServe shows near-linear scaling: at a 500ms TTFA constraint, DP=4 sustains ≈4x the single-GPU request rate (16 req/s vs. 4 req/s). In a throughput-oriented configuration that maximizes batch sizes for both LLM backbone and detokenizer, VoxServe reaches 134x realtime factor versus 10x for the official baseline and 53x for VoxServe without its scheduling optimizations. Appendix results extend evaluation to Chatterbox TTS, CSM, GLM-4-Voice, and Zonos-v0.1 (no official serving baselines available for comparison) and to additional input datasets (Hi-Fi Multi-Speaker English TTS, LJSpeech), showing consistent low-TTFA, high-viability serving and robustness to input distribution shifts.
Novelty Assessment
The contribution is systems-level rather than a new generative model: VoxServe introduces no new speech generation architecture, training objective, or inference algorithm for producing audio. Its novelty lies in (1) a model-execution abstraction that is general enough to unify seven architecturally distinct SpeechLMs (differing in codebook count, continuous-vs-discrete audio representations, detokenizer design, and depth-wise submodels) under one serving framework without model-specific serving code, and (2) a scheduling policy and asynchronous pipeline purpose-built for streaming-specific metrics (TTFA, streaming viability) rather than adapted from throughput-oriented text-LLM serving. The individual optimization primitives used (batching, CUDA graphs, KV/activation caching) are established techniques from LLM serving, but their systematic, model-agnostic application across the full SpeechLM inference pipeline, and the streaming-aware scheduling policy built on top of them, is the paper’s own claimed first.
Field Significance
Tip
High — VoxServe addresses a genuine infrastructure gap in deploying SpeechLMs at scale: prior serving stacks were architecture-specific and did not jointly optimize LLM backbone and detokenizer scheduling. It also contributes streaming-specific evaluation metrics (TTFA, Streaming Viability) that give the field a way to reason about SpeechLM serving performance distinct from text-LLM serving metrics.
This paper demonstrates that a single, model-agnostic serving abstraction can achieve substantial (10-20x) throughput gains over bespoke per-model serving stacks while preserving streaming latency guarantees, across models spanning different token rates, codebook structures, and detokenizer designs. It provides a reusable systems foundation and a metric vocabulary (TTFA, Streaming Viability) for evaluating SpeechLM deployment, independent of any specific generation architecture.
Claims
- supports: Decoupling system-level serving optimizations from model-specific architecture enables a single serving framework to efficiently support heterogeneous generative speech architectures without reimplementing optimizations per model.
Evidence: A unified model-execution interface (preprocess/forward/sampling/postprocess stages) lets one scheduler, worker, and CUDA-graph optimization path serve seven architecturally distinct SpeechLMs, differing in codebook count, continuous-vs-discrete representations, and detokenizer design, without model-specific serving code. (§3, §3.1)
- supports: Streaming speech-serving performance requires latency metrics distinct from standard text-LLM serving metrics, since audio playback continuity depends on binary per-chunk deadline satisfaction rather than average per-token latency.
Evidence: Introduces Time-To-First-Audio (TTFA) and a binary Streaming Viability metric (per-chunk deadline satisfaction), explicitly contrasted with TTFT/TPOT from text-LLM serving, and uses them to show baseline systems violate streaming viability at request rates where the proposed system does not. (§2.3, §4, Figure 6)
- supports: Jointly scheduling the LLM backbone and audio detokenizer as a single coordinated pipeline, rather than composing independently optimized LLM-serving and detokenizer engines, substantially improves streaming throughput at fixed latency.
Evidence: Against baselines that each combine an off-the-shelf LLM serving system with a custom detokenizer engine, the proposed system sustains up to 4.0 req/s at the same 500ms p90 TTFA where the CosyVoice baseline is limited to ≈0.4 req/s, and delivers more than 10x higher throughput at matched TTFA for Orpheus. (§4, Figure 6)
- complicates: Achieving high serving throughput for streaming SpeechLMs is architecture-sensitive: detokenizer cost, token rate, and model size independently bound the achievable request rate even under an optimized serving system.
Evidence: Orpheus 3B’s streaming viability drops past 8.0 req/s due to its 86 tokens/s token rate despite low TTFA, and Step-Audio 2’s 9B parameter size yields the lowest sustained request rate among the three evaluated models even with the proposed system’s optimizations. (§4)
Limitations and Open Questions
Direct baseline comparison is limited to three models (CosyVoice 2.0, Orpheus 3B, Step-Audio 2), the only ones with official open-source serving implementations available at time of writing; four additional supported models (Chatterbox TTS, CSM 1B, GLM-4-Voice, Zonos-v0.1) are evaluated only in isolation, without a baseline to quantify relative speedup. Multi-GPU results cover data parallelism up to four GPUs and disaggregated inference across two GPUs for a single model each (CosyVoice, Step-Audio respectively), leaving larger-scale or heterogeneous multi-GPU deployment unaddressed. The paper focuses entirely on serving-system latency and throughput; it does not evaluate whether any of its scheduling or batching optimizations affect generated audio quality relative to each model’s reference (non-served) implementation.
Wiki Connections
- Streaming Text-to-Speech — proposes streaming-aware scheduling and TTFA/Streaming Viability metrics purpose-built for evaluating and optimizing low-latency, continuous audio delivery from SpeechLMs.
- Spoken Language Models — builds a unified serving infrastructure spanning seven speech LM architectures (TTS and speech-to-speech), addressing a deployment gap specific to this model class.
- Neural Audio Codecs — its model-execution abstraction must accommodate diverse detokenizer/codec designs (Mimi, CosyVoice’s flow-matching detokenizer, SNAC) with different codebook counts and caching requirements.
- Evaluation Metrics — introduces Time-To-First-Audio and Streaming Viability as new metrics for evaluating SpeechLM serving performance, distinct from generation-quality metrics.
- CosyVoice 2 — one of three models directly benchmarked against its official serving baseline, and the model used for most ablation and scaling experiments.
- GLM-4-Voice — one of seven SpeechLM architectures supported by the unified serving interface.
- Step-Audio 2 — one of three models directly benchmarked; its 9B parameter size and detokenizer caching requirements are used to illustrate architecture-dependent serving bottlenecks.
- Moshi — cited as part of the SpeechLM landscape motivating the need for architecture-agnostic serving.
- VoiceBench — its AlpacaEval subset is used as the request workload for evaluating speech-to-speech model serving performance.
- LibriTTS — used as the request workload for evaluating TTS model serving performance and robustness to input distribution.
- On the Landscape of Spoken Language Models — cited as background framing for the SpeechLM architectural diversity that motivates the paper’s unified abstraction.