arXiv · 2025 · Preprint
Li et al. (Northwestern Polytechnical University; Huawei Technologies) · → Paper · Demo: ✗ · Code: ✓
Easy Turn is an open-source, bimodal (acoustic + linguistic) turn-taking detection model for full-duplex spoken dialogue systems, released together with a 1,145-hour training corpus and a manually annotated four-state test set.
Problem
Full-duplex spoken dialogue systems need to decide continuously whether to speak, keep listening, or stay silent, which requires a robust turn-taking detector. The two existing open-source options each cover only part of the problem: TEN Turn Detection uses a 7B-parameter LLM for semantic analysis but has no direct speech input, so it depends on an upstream ASR model that can lose acoustic cues (e.g. prosodic markers of hesitation); Smart Turn V2 takes speech directly but relies on a single linear classifier over Wav2Vec2 features, giving it little capacity to model semantic content. Neither model distinguishes backchannel utterances or explicit stop/pause requests, both of which matter for natural full-duplex interaction. The alternative strategy, fine-tuning an LLM backbone directly for full-duplex capability (as in Moshi, Freeze-Omni, GLM-4-Voice), avoids a separate detector but requires large amounts of full-duplex dialogue data that remain scarce and poorly documented in open-source form.
Method
Easy Turn frames turn-taking detection as a sequence generation problem over four dialogue-turn states: complete, incomplete, backchannel, and wait. The architecture follows the Qwen-Audio design pattern: a Whisper-Medium audio encoder (two 1D convolutional layers plus 24 transformer layers) feeds into an adaptor module (three 1D convolutional layers and four transformer layers) that bridges acoustic features to the input space of a Qwen2.5-0.5B-Instruct LLM. Rather than predicting the turn-state label directly from either modality alone, the model adopts an “ASR+Turn-Detection” paradigm: the LLM first generates the ASR transcription of the input speech and then, conditioned on both the acoustic features and its own generated transcription, predicts the turn-state label. Natural-language instruction prompts (five candidate templates generated by DeepSeek V3, one sampled per training example) guide the LLM toward this two-part output.
Training proceeds in two supervised stages. Stage one performs modality alignment: the audio encoder and adaptor are trained on 23,000 hours of ASR data (Aishell-1, Aishell-2, WenetSpeech, plus internal data) while the LLM stays frozen. Stage two fully fine-tunes the encoder, adaptor, and LLM jointly on the paper’s own Easy Turn trainset, a 1,145-hour dataset spanning all four turn states, built from both real conversational speech (MagicData-RAMC, cross-annotated with Qwen2.5-32B-Instruct and TEN Turn Detection) and synthetic speech (texts generated by DeepSeek V3/Qwen2.5-72B, synthesized with CosyVoice 2 using Emilia as the reference speech source, and filtered by zero-WER ASR verification with Paraformer). At inference, decoding is greedy (no sampling, temperature 1.0) on a single RTX 4090 GPU.
Key Results
On the paper’s own Easy Turn testset (300 complete, 300 incomplete, 100 backchannel, 100 wait samples, balanced 1:1 real/synthetic), Easy Turn reaches 96.33% accuracy on complete, 97.67% on incomplete, 91% on backchannel, and 98% on wait, outperforming both open-source baselines on every state (Table 2). TEN Turn Detection (using Paraformer as an ASR frontend) reaches 86.67%/89.3%/-/91% on the states it supports (it has no backchannel category), but at a much heavier footprint: 7,220 MB of parameters and 15,419 MB of GPU memory versus Easy Turn’s 850 MB and 2,559 MB. Smart Turn V2 is far lighter (95 MB, 27 ms latency) but far less accurate: 78.67% on complete and only 62% on incomplete, and it supports only two of the four states. An ablation (Table 3) isolates the contribution of the design choices: removing the ASR+Turn-Detection paradigm (predicting the state directly, without first generating a transcription) drops average accuracy from 95.75% to 87.88%; using acoustic-only input (fine-tuned Whisper + linear classifier) or linguistic-only input (fine-tuned Qwen2.5-0.5B on transcripts alone) both land around 85-86%, each roughly 10 points below the full bimodal model.
All comparisons are run on the authors’ own testset; no independent, third-party benchmark comparison is reported.
Novelty Assessment
The contribution is primarily engineering integration and dataset construction rather than architectural novelty. The encoder-adaptor-LLM architecture directly follows the Qwen-Audio design, and the “ASR+Turn-Detection” paradigm of generating a transcription before the label is adopted from prior work rather than introduced here. What is genuinely new is the combination applied specifically to turn-taking detection with four states (including backchannel and wait, which neither open-source baseline supports), and the release of a sizeable, purpose-built training corpus and matched test set at a point where full-duplex training data is otherwise scarce and undocumented in open-source form. The empirical gains over TEN Turn Detection and Smart Turn V2 are real and fairly measured on a controlled, internally-built benchmark, but the comparison set is limited to two prior open-source systems and does not include the fine-tuned end-to-end full-duplex LLMs (Moshi, Freeze-Omni, GLM-4-Voice) that the paper positions as the main alternative strategy.
Limitations and Open Questions
Warning
All reported accuracy numbers come from the authors’ own Easy Turn testset, built and annotated by the same team that constructed the training data and designed the model; there is no evaluation on an independent, externally curated benchmark.
The training data is heavily imbalanced across states: complete and incomplete together account for over 1,100 of the 1,145 hours, while backchannel and wait contribute only 10 and 23 hours respectively, which may limit robustness on the rarer classes despite their strong reported accuracy. The corpus is built primarily from Mandarin sources (MagicData-RAMC, Aishell-1/2, WenetSpeech), so generalization to other languages is untested. Inference latency (263 ms) is higher than Smart Turn V2’s (27 ms), a trade-off the paper frames as acceptable but does not stress-test under realistic full-duplex timing constraints. The qualitative deployment examples (§4.4) are drawn from informal internal use rather than a systematic user study.
Claims
- supports: Fusing acoustic and linguistic modality signals in a single turn-taking or dialogue-state classifier substantially improves accuracy over either modality used alone.
Evidence: Ablation shows the bimodal model reaches 95.75% average accuracy across four turn states, versus 85.5% for an acoustic-only classifier (fine-tuned Whisper + linear layer) and 86.25% for a linguistic-only classifier (fine-tuned Qwen2.5-0.5B on transcripts alone) (§4.3, Table 3).
- supports: Generating an explicit intermediate ASR transcription before predicting a dialogue-turn label, rather than predicting the label directly from acoustic features, improves detection accuracy in a speech-conditioned LLM.
Evidence: Removing the ASR+Turn-Detection paradigm while keeping the same architecture (“Easy Turn only-state”) drops average accuracy from 95.75% to 87.88% (§4.3, Table 3).
- complicates: Lightweight, latency-optimized turn-taking classifiers trade off accuracy against efficiency, and the gap widens specifically for detecting incomplete (as opposed to complete) utterances.
Evidence: Smart Turn V2 (95 MB, 27 ms latency) reaches 78.67% accuracy on complete but only 62% on incomplete, and provides no backchannel or wait detection at all, versus 96.33%/97.67% for the proposed model (§4.2, Table 2).
- complicates: Semantic-only turn-taking detectors built on large LLM backbones require a separate upstream ASR module, adding parameter count and memory overhead without necessarily surpassing the accuracy of a compact, natively bimodal alternative.
Evidence: TEN Turn Detection (7B-parameter LLM plus a Paraformer ASR frontend, 7,220 MB parameters, 15,419 MB memory, 204 ms latency) is outperformed on every measured state by an 850 MB, 2,559 MB bimodal model, despite the latter’s slightly higher latency (263 ms) (§4.2, Table 2).
Wiki Connections
- Spoken Language Model — Easy Turn adapts a compact LLM (Qwen2.5-0.5B) to consume an external speech signal through a Whisper audio encoder and adaptor, following the same encoder-adaptor-LLM pattern used elsewhere for speech-conditioned language models.
- Speech-to-Speech — Easy Turn is designed as a drop-in turn-taking module for full-duplex spoken dialogue systems, the same application setting targeted by end-to-end full-duplex systems such as Moshi and GLM-4-Voice that the paper cites as an alternative strategy.
- Evaluation Metrics — the paper releases a manually annotated, four-state Easy Turn testset, giving the field a reproducible benchmark for turn-taking detection instead of relying on closed or single-vendor evaluation setups.
- Full-Duplex-Bench — cited as a benchmark for evaluating turn-taking behavior in full-duplex spoken dialogue models, situating Easy Turn’s own model-level test set within the broader push toward standardized full-duplex evaluation.