arXiv · 2025 · Preprint
Chen et al. (Xiaohongshu) · → Paper · Demo: ✓ · Code: ?
A complete, pluggable full-duplex voice interaction system that upgrades existing half-duplex pipelines with a streaming personalized VAD and semantic end-of-turn detector, demonstrated through cascaded and semi-cascaded implementations.
Problem
Prior modular full-duplex voice interaction frameworks (LiveKit, Ten, Pipecat) offer pluggable architectures but depend on non-open components or external providers, preventing holistic optimisation of the interaction chain. End-to-end speech-to-speech approaches bypass this problem but introduce their own: they are difficult to control, require complex multi-stage training, and any functional update demands full retraining. What was missing was a complete, deployable framework that is fully open internally, combines pluggable control modules with heterogeneous interaction pipelines, and supplies all components from a single team.
Method
FireRedChat decomposes full-duplex voice interaction into three independently optimisable components: a turn-taking controller, an interaction module, and a dialogue manager.
The turn-taking controller provides two functions. First, a streaming personalized VAD (pVAD) targets the primary speaker using an ECAPA-TDNN speaker embedding concatenated with causal-convolution mel features, followed by a GRU classifier that outputs speaking probability at 10 ms resolution. Trained on 2,000 hours of Mandarin and English speech with simulated interfering speakers and background noise at 0-30 dB SNR, pVAD distinguishes the primary speaker from ambient sounds and competing talkers to suppress false barge-ins. Crucially, pVAD timestamps are used only to gate barge-in decisions; original, non-denoised audio is forwarded downstream so that paralinguistic cues are preserved for later stages. Second, a semantic end-of-turn (EoT) detector fine-tunes a 170M-parameter BERT classifier on approximately 830,000 text instances (partial utterance spans labelled “continue”, complete utterances labelled “stop”), enabling semantic rather than purely acoustic turn boundary detection.
The interaction module supports three pipeline variants, any of which can slot in under the same controller. The cascaded configuration uses FireRedASR, Qwen2.5, and FireRedTTS-1s with streaming decoding. The semi-cascaded configuration replaces ASR and LLM with an in-house AudioLLM (trained from scratch following the Qwen2-Audio architecture on large-scale multi-task audio data and 250k colloquial dialogue instances), then synthesises responses with FireRedTTS-2, which is conditioned on the user’s input audio to capture emotional tone and paralinguistic context during generation. The semi-cascaded pipeline thus aligns both the comprehension stage and the synthesis stage to the same acoustic cues, yielding more contextually coherent responses. A dialogue manager built on Dify handles tool invocation (e.g., web search) and context management across turns.
Key Results
Barge-in performance (Table 2): FireRedChat achieves a 10.2% false barge-in rate, compared with 33.4% for LiveKit and 78.1% for Ten. The cost is a modestly higher T90 of 170 ms versus 140 ms (LiveKit) and 90 ms (Ten), reflecting a design trade-off between response speed and robustness to non-primary-speaker interference. Ten’s very low T90 of 90 ms indicates high sensitivity to any acoustic event, leading to its elevated false barge-in rate.
End-of-turn detection (Table 3): The 170M-parameter EoT model achieves 96.0% average accuracy on Chinese and 94.9% on English, matching Ten’s 7B-parameter model (95.8%/94.4%) and substantially outperforming LiveKit’s 373M-parameter model (70.8%/86.2%). This demonstrates that compact, task-specific classifiers can match large general-purpose models for semantic stop detection.
Latency (Table 4): FireRedChat achieves P50/P95 latencies of 2.341 s / 3.015 s, compared with 3.598 s / 4.649 s for LiveKit and 3.375 s / 3.802 s for Ten. The industrial reference DouBao achieves 2.075 s / 2.407 s, leaving a gap the authors attribute to streaming ASR output, streaming TTS input, and smaller VAD chunk sizes as identified engineering headroom.
Novelty Assessment
The primary contribution is engineering integration: a complete, self-contained full-duplex pipeline assembled from components the team develops in-house, enabling end-to-end control and optimisation that prior open frameworks (which rely on third-party providers) do not offer. The pVAD design adds target-speaker conditioning to an existing streaming VAD framework; the EoT classifier applies standard BERT fine-tuning to a curated corpus. Neither component introduces a new architectural paradigm. The semi-cascaded configuration is conceptually interesting as a middle ground between cascaded and end-to-end approaches, but the underlying AudioLLM and FireRedTTS-2 models are described as in-house pre-existing work. The system-level evaluation metrics (barge-in T90, false barge-in rate, end-to-first-response P50/P95) are a genuine contribution: they measure the interaction experience as a whole rather than individual component quality, and the comparison methodology (standardising the LLM across frameworks) improves fairness.
Field Significance
Moderate — FireRedChat provides a self-contained reference implementation for pluggable full-duplex spoken conversational agents, demonstrating that a modular turn-taking controller can upgrade heterogeneous half-duplex pipelines to full-duplex without retraining the underlying models. It also contributes a system-level evaluation methodology for barge-in and end-of-turn accuracy that can be applied across full-duplex frameworks. The work narrows the gap between open frameworks and industrial systems on end-to-end latency, though it does not introduce new architectural principles.
Claims
-
supports: Speaker-conditioned VAD substantially reduces false barge-in rates in full-duplex spoken conversational agents compared to standard VAD, at the cost of slightly higher target-speaker response latency.
Evidence: pVAD achieves a 10.2% false barge-in rate versus 33.4% (LiveKit SileroVAD) and 78.1% (Ten VAD), with T90 of 170 ms versus 140 ms and 90 ms respectively. (§3.1, Table 2)
-
supports: Compact task-specific classifiers for semantic end-of-turn detection can match parameter-heavy general-purpose language models while imposing negligible latency overhead.
Evidence: A 170M-parameter BERT-based EoT model achieves 96.0%/94.9% average accuracy on Chinese/English, on par with a 7B-parameter model (95.8%/94.4%), while using far fewer parameters and thus adding minimal latency to the pipeline. (§3.2, Table 3)
-
supports: Semi-cascaded architectures that replace the ASR+LLM cascade with a single AudioLLM offer paralinguistic awareness and reduced error propagation while maintaining competitive end-to-end latency.
Evidence: The semi-cascaded variant conditioned on user speech emotion achieves a simpler pipeline with potential latency benefits, mitigates ASR error propagation, and enables paralinguistic-consistent TTS synthesis, without sacrificing the modularity needed for independent component optimisation. (§2.3.2)
-
complicates: End-to-end speech-to-speech models face training and architectural challenges that limit their use as a foundation for full-duplex interaction systems.
Evidence: The authors identify catastrophic forgetting in LLM backbones and token length mismatch between text and speech sequences as key drawbacks of end-to-end approaches, motivating the pluggable modular controller design. (§1)
-
supports: System-level evaluation of full-duplex voice interaction requires metrics beyond component-level quality scores, covering barge-in accuracy, semantic end-of-turn detection, and end-to-end latency jointly.
Evidence: FireRedChat introduces three interaction-level metrics (barge-in T90, false barge-in rate, and end-to-first-response P50/P95) that capture naturalness, control accuracy, and efficiency independently of individual ASR, LLM, or TTS scores. (§3)
Limitations and Open Questions
Warning
All evaluation uses proprietary test data for barge-in and latency experiments. The barge-in test set is 1,000 utterances per language constructed by the authors, and latency is measured on 25 synthetic utterances. External reproducibility of these results is not possible.
Latency remains above industrial-grade systems (DouBao) by 0.3 s (P50) and 0.6 s (P95). The authors identify three concrete engineering improvements, streaming ASR output, streaming TTS input, and reduced VAD chunk size, as a path to closing this gap, but these are not demonstrated. The semi-cascaded pipeline’s paralinguistic advantages are described qualitatively; no objective metric (MOS, emotion accuracy) quantifies the improvement in response coherence or emotional consistency. The AudioLLM and FireRedTTS-2 components are in-house proprietary models not released, limiting direct reproducibility of the semi-cascaded results.
Wiki Connections
- Spoken Language Model — the AudioLLM component in the semi-cascaded pipeline acts as a speech-input LLM that perceives paralinguistic and semantic cues jointly, fitting within the spoken language model paradigm.
- Speech-to-Speech — FireRedChat contextualises its design against end-to-end speech-to-speech systems, arguing that pluggable modular control offers better controllability and upgrade flexibility.
- Streaming TTS — the cascaded pipeline uses FireRedTTS-1s with streaming decoding, and latency reduction from streaming TTS input is identified as a key future improvement.
- Evaluation Metrics — the paper introduces system-level metrics (barge-in T90/false rate, EoT accuracy, P50/P95 latency) designed specifically for holistic full-duplex interaction evaluation.
- Emotion Synthesis — the semi-cascaded configuration conditions FireRedTTS-2 on the user’s input audio to capture emotional tone and ensure acoustically consistent response generation.
- FireRedTTS-2 — used directly as the TTS synthesis component in the semi-cascaded pipeline, conditioned on user speech for emotionally consistent dialogue-style generation.
- CosyVoice — referenced as a comparable large-scale TTS framework in the context of the cascaded pipeline’s TTS component choice.
- CosyVoice 2 — referenced as a scalable streaming TTS system relevant to the latency-reduction goals of the cascaded pipeline.