arXiv · 2025 · Preprint
Siddhant Arora et al. (Carnegie Mellon University, Sony Group Corporation) · → Paper · Demo: ? · Code: ✗
Extends chain-of-thought (CoT) training, previously used only in turn-by-turn end-to-end spoken dialogue systems, into a fully duplex, blockwise-streaming setting by using CTC forced alignments to generate block-level intermediate ASR-transcript and text-response targets.
Problem
Turn-by-turn end-to-end spoken dialogue systems rely on an external voice activity detection (VAD) module to decide when the user has finished speaking, but silence alone is a poor cue for turn completion: within-speaker pauses are often longer than between-speaker gaps, and strict listen-then-speak operation cannot produce the overlapping speech common in natural human-human conversation. Full-duplex systems that continuously predict output (including silence) remove the dependency on VAD, but existing duplex architectures either require dual input/output channels that diverge sharply from a pretrained speech-text LM’s training setup (demanding very large compute and data budgets, as in Moshi’s multi-million-hour post-training) or use single-channel time-multiplexing without any structured intermediate reasoning, which the authors show lags noticeably behind turn-by-turn systems in semantic coherence.
Method
The paper builds on a single-channel time-multiplexing duplex formulation: input audio is divided into fixed-duration blocks, and the model alternates between consuming a block of user speech and producing a block of response speech, retaining the autoregressive, single-channel structure that keeps the model close to a pretrained speech-text LM’s original pretraining setup. On top of this, the authors introduce SCoT (Streaming Chain-of-Thought), which incorporates the turn-by-turn CoT idea (predicting an intermediate ASR transcript, then an intermediate text response, then final speech) into each block. Because a full utterance-level transcript or response is not available within a single 2-second block, the method uses frame-level CTC forced alignments (computed with OWSM CTC v4 1B) to produce block-aligned intermediate targets, an aligned ASR-transcript segment and an aligned text-response segment, for every block, giving each block a three-stage decoding process: predict the aligned ASR transcript, predict the aligned text response, then generate the final speech output. Because the optimal reasoning path is unknown a priori, three variants are trained: SCoT-ASR (transcript-only intermediate target), SCoT-Response (text-response-only intermediate target), and SCoT-Full (both).
The backbone is a pretrained speech-text LM (from ESPnet-SpeechLM) initialized from the SmolLM2 1.7B text LLM, using a delay-interleave architecture for multi-stream language modeling over concatenated codec tokens (ESPnet-Codec) and self-supervised tokens (XEUS). Training uses a 2-second block size, with 60-second dialogue contexts constructed from the combined Fisher and Switchboard training sets; words spanning block boundaries are included as intermediate targets in both adjacent blocks. At inference, ASR transcript decoding uses greedy search with hallucination-removal post-processing, text response generation uses top-k sampling (k=30, temperature 0.8) with the same post-processing, and speech generation uses top-k sampling followed by selecting, from 10 candidate samples, the one with highest intelligibility relative to the predicted text response.
Key Results
Against a standard time-multiplexed Duplex E2E baseline (no intermediate reasoning), SCoT-Response cuts perplexity from 313.8 to 42.3 and raises ROUGE-1 from 14.4 to 27.9 on Switchboard’s Eval2000 set, exceeding the ROUGE/METEOR scores of even a turn-by-turn multi-turn CoT system and a ground-truth-transcript SmolLM v2-1.7B text baseline (Table 1). An LLM judge (Qwen2.5-Omni) scores SCoT-Response at 3.24 versus 2.20 for Duplex E2E, while the turn-by-turn CoT E2E baseline scores highest (4.02) but at a real-time factor of 2.95, far above SCoT-Response’s 0.56 RTF, which is itself lower than a standard turn-by-turn E2E model’s 0.75 RTF (§VII-A). On turn-taking, SCoT-Full generates overlapping speech in 64.5% of blocks with the best overlap recall (71.5%) among all systems, close to the 57.5% overlap rate observed in real Eval2000 conversations, while all turn-based systems (cascaded and E2E alike) produce zero overlap by construction (Table 2). On UTMOS audio quality, SCoT-Response reaches 2.46, higher than the Duplex E2E baseline’s 1.95 and higher than any other duplex or E2E system evaluated, though still well below the 4.19 UTMOS of a dedicated single-speaker TTS system (LJSpeech VITS) used as a ceiling reference (Table 4). SCoT-ASR, the transcript-only variant, underperforms the plain Duplex E2E baseline on nearly every metric, including perplexity (432.7 vs. 313.8) and emotion-alignment rank (6.54, worst of all systems tested), showing that ASR-only intermediate reasoning is not sufficient on its own.
Novelty Assessment
The core novelty is architectural: adapting turn-by-turn CoT reasoning to a streaming, blockwise duplex setting via CTC forced alignment of intermediate ASR and text targets, something the authors position as unexplored by prior duplex systems (e.g., OMNI-Flatten) that lack this alignment-driven coupling between speech and text reasoning. The contribution is incremental in the sense that both duplex time-multiplexing and turn-by-turn CoT training exist separately in prior work; SCoT’s contribution is the specific mechanism (frame-level forced alignment) that lets CoT be applied within short, fixed-duration blocks rather than full utterances, plus a systematic ablation across three CoT variants that quantifies which reasoning path matters most. The evaluation is reasonably thorough for a single-domain study (semantic quality, turn-taking, emotion consistency, audio quality, latency, and a block-size ablation), but it is confined to English telephone-conversation data (Switchboard + Fisher) at a 1.7B backbone scale, considerably smaller and narrower than industrial duplex systems such as Moshi.
Field Significance
Moderate — the paper demonstrates that intermediate CoT-style reasoning, previously validated only for turn-by-turn spoken dialogue, can be adapted into a streaming duplex framework using forced alignment, narrowing the semantic-coherence gap between duplex and turn-by-turn systems without sacrificing low latency or the ability to produce overlapping speech. Its value is mainly as a systematic characterization of trade-offs between reasoning-path choice, latency, turn-taking naturalness, and audio quality within a single controlled setup, rather than a new architectural paradigm.
Claims
- supports: Incorporating chain-of-thought intermediate reasoning targets into a streaming duplex spoken dialogue system substantially improves semantic coherence over a duplex baseline without intermediate reasoning.
Evidence: SCoT-Response reduces perplexity from 313.8 (Duplex E2E) to 42.3 and raises ROUGE-1 from 14.4 to 27.9 on Eval2000, exceeding the ROUGE/METEOR scores of a ground-truth-transcript text LLM baseline. (§VII-A, Table 1)
- complicates: Predicting only an intermediate ASR transcript is not, by itself, a sufficient reasoning target for improving duplex dialogue coherence.
Evidence: SCoT-ASR (transcript-only intermediate target) underperforms the plain Duplex E2E baseline on perplexity (432.7 vs. 313.8) and has the worst emotion-alignment rank (6.54) of all systems tested. (§VII-A, §VII-C, Table 1, Table 3)
- supports: Blockwise streaming CoT duplex systems can approach the latency of purely reactive duplex systems while achieving much higher semantic coherence than latency-costly turn-by-turn CoT alternatives.
Evidence: SCoT-Response achieves an RTF of 0.56, below a standard turn-by-turn E2E model’s 0.75, while scoring 3.24 on an LLM judge versus 2.20 for Duplex E2E; the turn-by-turn CoT model scores higher (4.02) but at RTF 2.95. (§VII-A, Table 1)
- complicates: Shrinking the processing block size to reduce interaction latency in streaming duplex CoT systems degrades semantic coherence.
Evidence: Reducing block size from 2s to 1s for SCoT-Full lowers RTF from 0.85 to 0.30 but raises perplexity from 55.2 to 161.2 and drops ROUGE-L from 20.3 to 7.06. (§VII-E, Table 5)
- refines: Combining both ASR-transcript and text-response intermediate targets in a duplex CoT system trades a small amount of peak semantic/audio quality for more natural overlapping turn-taking behavior.
Evidence: SCoT-Response outperforms SCoT-Full on ROUGE/METEOR/perplexity and UTMOS (2.46 vs. 2.19), but SCoT-Full achieves higher overlap recall (71.5% vs. 66.8%) relative to the 57.5% natural overlap rate observed in Eval2000. (§VII, Table 1, Table 2, Table 4)
Limitations and Open Questions
Inference in both the multi-turn CoT baseline and all duplex/SCoT evaluations uses ground-truth context from previous turns or blocks rather than the system's own prior outputs, so the reported semantic-quality and latency results do not reflect fully autoregressive, error-accumulating multi-turn deployment.
Beyond this, absolute audio quality remains modest across all E2E and duplex systems (UTMOS 1.9-2.5) compared to a dedicated single-speaker TTS ceiling (4.19), indicating a substantial synthesis-quality gap that CoT reasoning narrows but does not close. Evaluation relies on automatic metrics (ROUGE, METEOR, perplexity, UTMOS) and an LLM judge rather than human listening tests, and is confined to English telephone conversations (Switchboard and Fisher) at a 1.7B-parameter backbone, leaving open how the approach scales to larger models, other languages, or more diverse conversational domains. The authors also note adaptive block sizing and deeper analysis of how different CoT paths affect performance across utterance types as open directions.
Wiki Connections
- Spoken Language Model — builds directly on a pretrained speech-text LM (ESPnet-SpeechLM, initialized from SmolLM2) that consumes external user speech in a genuine full-duplex dialogue setting.
- Speech-to-Speech — the SCoT framework generates spoken responses directly from streaming spoken input without an explicit turn-taking or VAD stage.
- Streaming TTS — introduces a blockwise streaming generation scheme with an explicit latency/coherence trade-off (block-size ablation) for real-time spoken response generation.
- Neural Audio Codec — represents speech using concatenated ESPnet-Codec and XEUS tokens as the multi-stream input/output representation for the speech-text LM.
- Self-Supervised Speech — uses XEUS self-supervised speech tokens, concatenated frame-by-frame with codec tokens, as a core part of the model’s audio tokenization.
- Moshi — cited as the dual-channel duplex precedent whose training-data and compute requirements motivate this paper’s single-channel time-multiplexing design choice.
- OmniFlatten — directly contrasted as a prior duplex system lacking the CTC-forced-alignment-driven coupling between intermediate text targets and speech responses that SCoT introduces.
- ESPnet-SpeechLM — supplies the pretrained speech-text LM backbone that SCoT initializes from and fine-tunes.
- ESPnet-SDS — the toolkit used to implement and evaluate the spoken dialogue system experiments.
- LSLM — cited among prior duplex (“listen while speaking”) architectures that motivate moving beyond VAD-gated turn-by-turn systems.
- emotion2vec — used to extract emotion vectors for computing the Emotion Rank speaking-style-consistency metric.
- Landscape of Spoken Language Models Survey — cited as background framing for the speech-text LM paradigm underlying this paper’s backbone.
- SpeechGPT — cited as prior end-to-end speech-text LM work motivating the shift away from cascaded spoken dialogue pipelines.