arXiv · 2025 · Preprint
Shufan Li et al. (University of California, Los Angeles) · → Paper · Demo: ✗ · Code: ✓
Introduces input-time speculative decoding for cascade voice-chat pipelines, generating and verifying candidate LLM responses while the user is still speaking so that text-to-speech synthesis of the first sentence can begin with minimal delay after the user finishes.
Problem
Cascade voice-chat systems (ASR → LLM → TTS) invoke the TTS model sentence-by-sentence, so the user-perceived latency is dominated by the time the LLM needs to produce a complete first sentence after the user stops speaking (time-to-first-sentence). Integrated speech-language models avoid this delay by generating audio before finishing the first sentence, but they require expensive end-to-end training on audio-text data, tend to lag behind state-of-the-art text-only LLMs, and are inflexible to swapping in newer base models or task-specific customizations. Standard speculative decoding accelerates token throughput after a complete prompt is received but does not address the latency incurred while waiting for the full first sentence, and it assumes idle GPU capacity is unavailable during the input phase, an assumption that does not hold for single-user, local, consumer-GPU deployments where the GPU sits idle while the user is speaking.
Method
The paper models user input as a stream of partial prompts arriving at a fixed rate (approximating 120 words per minute of conversational speech) rather than as a single complete request. Upon receiving each new partial prompt, PredGen runs a verifier that checks how many tokens of a previously generated candidate response remain consistent with the updated prompt, then invokes a predictive-generation subroutine to produce an updated candidate response and pre-synthesize the audio for its first sentence. This loop repeats until the user’s input is complete, at which point a final verification decides whether the pre-synthesized audio can be output directly (best case, no added latency), whether only part of the accepted tokens must be regenerated (partial acceptance), or whether the full first sentence must be regenerated from scratch (worst case, small overhead versus an unoptimized baseline).
Three verifier designs are studied: greedy verification (exact token match against the LLM’s own next-token argmax, which makes the method lossless relative to the un-accelerated baseline), top-K verification (a relaxed variant that accepts any token within the top-K most likely predictions, trading a small amount of output fidelity for higher acceptance rates), and a self-reflection verifier that prompts the LLM itself to judge semantic consistency between the partial prompt and the candidate first sentence rather than relying on token-level agreement. For the predictive-generation subroutine, the paper compares standard autoregressive decoding (which requires no fine-tuning but can degrade on partial, out-of-distribution prompts, mitigated with a system prompt instructing the model to guess user intent) against Jacobi decoding adapted via the CLLM (Consistent Large Language Models) technique, which requires LoRA fine-tuning on a held-out budget (kept under $1,000 of compute) but converges to the correct continuation in fewer forward passes.
For text-to-speech, the system uses Zonos, a 1.6B-parameter Mamba-Transformer hybrid open-source model, conditioned on text tokens and a reference audio prompt for voice characteristics, generating audio autoregressively. The base language model used in most experiments is Qwen2.5-7B-Instruct; the paper also evaluates generalization to Mistral-7B-Instruct, LLaMA3-8B-Instruct, and OLMo-2-7B-Instruct without any additional fine-tuning for the training-free (AR) variants. ASR is not simulated; partial user input is streamed as text directly to the LLM, except in one appendix experiment that pipes synthetic audio through a real streaming ASR system.
Key Results
On Lmsys and MT-Bench, all PredGen variants achieve at least a 1.6x reduction in audio latency relative to an un-accelerated cascade baseline using the same Qwen2.5-7B-Instruct and Zonos TTS components. The lossless greedy variant achieves 1.9x speedup on Lmsys and 1.6x on MT-Bench with identical GPT-4-judged output quality to the baseline; top-K and self-reflection verifiers push speedups to 2.8x and 2.3x on Lmsys respectively, at a negligible or zero drop in judged quality; the CLLM/Jacobi variant reaches 1.8-2.3x speedup while matching or slightly exceeding baseline quality scores. Speedups are smaller on GSM8K and MMLU-Pro (1.1-2.5x for greedy/top-K), which the authors attribute to these tasks’ prompts (e.g., specific numerical values) being harder to predict from partial input than open-ended conversational prompts. Generalizing across base models on Lmsys, training-free PredGen-Greedy achieves speedups on Mistral-7B-Instruct (2.7x), LLaMA3-8B-Instruct, and OLMo-2-7B-Instruct without any base-model-specific fine-tuning. An appendix experiment with real streaming ASR transcription (rather than simulated text streaming) shows comparable speedups (1.9-2.3x) with a small, baseline-shared drop in output quality attributable to ASR noise rather than PredGen itself.
Novelty Assessment
The core contribution is a new inference-time algorithm and system design, not a new neural architecture: PredGen introduces input-time (pre-completion) speculative generation and verification for cascade voice-chat pipelines, a setting the paper distinguishes from standard speculative decoding (which only accelerates post-prompt throughput) and from prior streaming-input work such as Livemind (which targets server-side, top-tier-GPU deployment rather than single-user, consumer-GPU voice interaction). The three verifier designs and the integration of pre-synthesized TTS audio buffering with a Jacobi/CLLM predictive-generation path are genuinely new combinations, but every individual component (speculative decoding, Jacobi/consistency decoding via CLLM, autoregressive TTS via Zonos, LoRA fine-tuning) is drawn from existing work. The contribution is therefore best characterized as a novel system-level inference procedure built from established building blocks, evaluated only in simulation (with one small-scale real-ASR appendix experiment) rather than in a live end-to-end deployment.
Field Significance
Moderate — this paper identifies and directly targets a latency bottleneck specific to single-user, local cascade voice-chat systems (idle GPU time during user input) and demonstrates a training-free, base-model-agnostic method to exploit it, offering a concrete alternative to the more expensive path of training integrated speech-language models for low latency. Its contribution is architectural in the sense of a new inference pipeline design, but the evaluation is simulation-based across four text-domain benchmarks rather than a live spoken-interaction user study.
Claims
- supports: In cascade voice-chat systems, the first-sentence generation latency of the language model, not the TTS synthesis step, dominates overall user-perceived response latency when the LLM is large relative to the TTS model.
Evidence: On the same hardware, Zonos TTS exhibits similar latency (around 200ms) on both an H100 and an RTX 3090, while the 7B-scale LLM sees roughly 4x higher throughput on the H100, showing the LLM’s first-sentence generation time as the dominant, hardware-sensitive bottleneck. (§1)
- supports: Speculative verification of candidate LLM outputs generated before a user’s input is complete can eliminate response latency entirely for a meaningful fraction of real conversational turns.
Evidence: On MT-Bench, more than 10% of first-round conversation turns and an even higher share of second-round turns achieve NFETFS = 1 (single verification step, no regeneration), allowing pre-synthesized audio to be played with no added delay; up to 14% of turns reach this best case. (§5.1, Figure 3b)
- complicates: Latency reductions from input-time speculative generation depend heavily on how predictable the completion of a partial prompt is, and are substantially smaller for tasks with low content predictability.
Evidence: Average speedups on GSM8K (1.1-2.5x) and MMLU-Pro (1.5-4.6x, high variance) are lower and less consistent than on conversational benchmarks Lmsys (1.8-2.8x) and MT-Bench (1.6-2.6x), attributed to prompts containing hard-to-predict content such as specific numerical values. (§4.2, Table 1)
- complicates: Loosening the token-acceptance criterion in speculative verification to increase acceptance rate and speedup trades off measurable output quality, even when the underlying language model is unchanged.
Evidence: Increasing K in the top-K verifier yields further 10-16% speed-ups on Lmsys and MT-Bench at the cost of a slight reduction in GPT-4-judged sample quality, unlike the greedy verifier, which is lossless by construction. (§B.1, Figure 4)
- refines: Adapting a language model for accelerated (Jacobi-style) decoding to reduce voice-interaction latency can be substantially cheaper than training an integrated end-to-end speech-language model for the same purpose.
Evidence: CLLM fine-tuning of the 7B base model via LoRA with DeepSpeed ZeRO-3 Offload on 4 A6000 GPUs for 6k steps costs an estimated 10,000+ (8 A100s, 40,000 steps) reported for training an integrated system such as Mini-Omni. (§C.1)
Limitations and Open Questions
The evaluation is almost entirely simulation-based: user input is streamed as pre-tokenized text at an assumed fixed rate rather than captured from live spoken interaction, and ASR is not simulated in the main experiments (only in one appendix ablation), so real-world latency and quality under live microphone input and imperfect, self-correcting ASR transcripts remain only partially characterized.
The method’s benefits shrink for tasks with less predictable content (math word problems, knowledge-intensive questions), and the paper explicitly leaves this as future work. The experiments are restricted to a single-user, batch-size-1 deployment scenario; the authors note that extending PredGen to multi-user settings (e.g., a shared household voice assistant) would require additional scheduling and batching strategies not addressed here. The CLLM/Jacobi variant currently only supports greedy-style verification, and its fine-tuning is specific to the calibration dataset used (Lmsys), leaving open how well it transfers to different conversational domains.
Wiki Connections
- Streaming TTS — PredGen targets the sentence-by-sentence streaming invocation pattern used by cascade voice-chat systems, specifically reducing the delay before the first streamed sentence is available for TTS synthesis.
- Speech-to-Speech — the paper positions its cascade (ASR-LLM-TTS) design as a flexible, lower-training-cost alternative to integrated speech-to-speech language models, explicitly comparing training cost and adaptability trade-offs.
- Zero-Shot TTS — the TTS component (Zonos) is conditioned on a reference audio prompt for voice characteristics, a zero-shot voice-conditioning mechanism the paper treats as an off-the-shelf building block rather than a novel contribution.
- LLaMA-Omni — cited as a representative integrated speech-language model requiring end-to-end audio-text training, which PredGen’s cascade-based approach is designed to avoid while still reducing latency.
- Mini-Omni — used as a concrete cost comparison point: its reported training setup (8 A100s, 40,000 steps) is contrasted against PredGen’s under-$500 fine-tuning budget for the CLLM variant to argue for cascade-system cost efficiency.
- SLAM-Omni — cited as another integrated voice-interaction system following the speech-tokenizer-plus-LLM-plus-vocoder pattern that PredGen’s cascade approach avoids training end-to-end.
- SpeechGPT — cited as an early integrated speech-language model using a discrete speech tokenizer and vocoder, representative of the architecture class PredGen’s cascade design is positioned against.