arXiv · 2025 · Preprint
Cui et al. (CUHK / Huawei Technologies) · → Paper · Demo: ✓ · Code: ?
TurnGuide introduces a turn-level text-speech interleaving strategy for end-to-end full-duplex speech language models that segments assistant speech into dialogue turns and prepends turn-level text tokens, substantially improving semantic coherence while preserving natural turn-taking behaviour.
Problem
End-to-end full-duplex speech language models (FD-SLMs) process both speaker channels simultaneously and can handle interruptions, backchannels, and overlapping speech — but their semantic output is markedly weaker than text-only LLMs. The degradation stems from two compounding difficulties: the extreme length of raw speech token sequences erodes language model coherence, and high-quality double-channel conversational data is scarce. The natural remedy — injecting text tokens to guide speech generation — is difficult because FD-SLMs require precise time alignment between user and assistant audio streams. Inserting text at the wrong moment (too early, causing hallucination without full user context; too late, undermining guidance) or at the wrong granularity (too long, causing the assistant to overshoot turn boundaries; too short, fragmenting semantics) breaks both timing and content coherence.
Method
TurnGuide fine-tunes GLM-4-Voice, an existing speech-text aligned language model, on the Fisher telephone conversation corpus using a two-stage pipeline. First, a dynamic turn segmentation and alignment framework converts raw double-channel audio into aligned text-speech pairs: Voice Activity Detection (via pyannote) identifies speech segments, which are merged into Inter-Pausal Units (IPUs) by grouping segments with pauses under 0.5 seconds; Whisper medium then provides word-level timestamps that are used to assign transcript words to their corresponding IPUs, yielding time-aligned (text, speech) turn pairs.
Second, a text-guided full-duplex dialogue modelling framework trains the model with two interleaved sequence structures. Channel-wise interleaving alternates user and assistant speech in 5-token chunks (400 ms at GLM-4-Voice’s 12.5 Hz frame rate), with a speaker-ID token prefixing each chunk. Text-speech interleaving then prepends text chunks to the first speech chunk of each assistant turn, where each text chunk of 5 tokens is placed at the timestamp of its first word’s onset. An end-of-turn token marks turn boundaries and an end-of-chunk token separates internal text chunks, forcing the model to learn turn structure. The 5:5 text-to-speech chunk ratio ensures text generation precedes speech generation within each turn. Weighting the text loss more heavily than the speech loss (2:1 or 3:1 ratio) consistently improves semantic quality.
Key Results
Semantic quality is assessed using a GPT-4o-based scoring protocol (GPT-score, 0–10 after rescaling from 0–3.3 ground-truth range), validated against human MOS in a separate listening test. On the Fisher test set under the conditional generation setting (assistant channel responding to user audio), TurnGuide achieves an overall GPT-score of 7.27 versus 5.91 for the Speech Chunk Interleaving (SCI) baseline and 5.82 for Moshi — a 24% gain over SCI. The 2:1 and 3:1 text-loss variants reach 7.70 and 7.79 respectively, matching or exceeding the upper bound obtained when ground-truth Fisher transcripts are used as text guidance (7.39 at 1:1 loss).
Human listening tests corroborate the automated scores: TurnGuide achieves M-MOS 3.26 and N-MOS 3.23 against Moshi’s 2.69/2.75 and SCI’s 2.24/2.47, with ground truth at 4.25/4.29 (Table 6).
On Full-Duplex-Bench — which evaluates fine-grained turn-taking events including smooth turn-taking, user interruption, pause handling, and backchanneling — TurnGuide (2:1 loss, temperature 1.3) outperforms dGSLM and Moshi on nearly all metrics, though it lags on smooth turn-taking TOR and user-interruption latency compared to Moshi (Table 3).
Ablations in Table 4 confirm that both text insertion timing and length are material: shifting insertion 4 seconds early causes the largest degradation (GPT-score 5.61 vs. 6.76 at nominal timing), while too-long or too-short text chunks degrade by a smaller but still significant margin.
Note
GPT-score comparisons against Moshi use a Fisher-fine-tuned reimplementation of Moshi’s training strategy (Moshi TS) for the baseline, not the original Moshi checkpoint, which was trained on broader data. The original Moshi checkpoint is reported separately and is not directly comparable.
Novelty Assessment
The turn segmentation idea is principled and fills a genuine gap: prior token-level text-speech interleaving (Moshi’s approach) fragments semantics so severely that it provides minimal benefit. Aggregating text by IPU turns and aligning at turn boundaries, rather than word boundaries, is a meaningful design choice backed by clear ablation evidence.
That said, the components are assembled from existing tools — GLM-4-Voice as backbone, pyannote for VAD, Whisper for ASR — and the training recipe is a fine-tuning of an off-the-shelf SLM on Fisher. The contribution is primarily a training data construction framework and a sequence formatting strategy, not a new model architecture. Evaluation is restricted to Fisher (telephone conversations only), which limits generalisability claims. The GPT-score metric, though validated by human judgements in 25/30 pairwise comparisons, is a proxy and not a standard MOS protocol.
Field Significance
Moderate — TurnGuide demonstrates that turn-level text-speech interleaving substantially closes the semantic quality gap between end-to-end FD-SLMs and text-based dialogue systems without degrading turn-taking behaviour, providing useful evidence for how text guidance should be scoped in full-duplex speech modelling. The approach is transferable to other e2e FD-SLM backbones, though its validation on a single telephone conversation corpus limits confidence in broad generalisability.
Claims
- Aggregating text tokens at the dialogue-turn level provides stronger semantic guidance for full-duplex speech generation than word-level token-by-token text insertion. (§2.2, §4.2.1, Table 2)
- The timing of text insertion into full-duplex dialogue sequences is more sensitive to error than the length of inserted text, with mistimed insertions causing larger semantic degradation than incorrectly sized text chunks. (§4.2.3, Table 4)
- Increasing the training loss weight of text tokens relative to speech tokens in a text-speech interleaved model improves semantic quality without sacrificing turn-taking naturalness. (§4.2.1, Table 2)
- Fine-grained turn-taking benchmarks are more informative than corpus-level statistical correlations for evaluating full-duplex spoken dialogue models. (§4.2.2)
- GPT-based automated semantic evaluation of spoken dialogue aligns closely with human preference judgements when score differences exceed one point, but reliability degrades for near-tied comparisons. (Appendix A, Table 5)
Limitations and Open Questions
Warning
All results are on the Fisher telephone conversation corpus only. Fisher’s conversational style and acoustic conditions (telephone, English, spontaneous) are narrow, and the paper provides no evidence that TurnGuide generalises to other languages, speaking styles, or domains.
The model has not undergone safety alignment (RLHF or equivalent), which the authors acknowledge is a prerequisite for deployment. The theoretical first-package latency is 1.05 seconds — workable for some interactive applications but not near real-time; the authors note the vocoder imposes the dominant cost and that an optimised decoder would reduce this.
The evaluation uses GPT-4o as both the semantic evaluator and implicitly as an oracle for dialogue quality, which introduces a potential circularity if the trained model’s outputs are biased toward patterns GPT-4o scores favourably. The corpus-level Pearson correlation analysis (Table 7) shows TurnGuide is at parity with baselines on turn-taking statistics but does not establish whether the improvement in GPT-score translates to user preference in live interaction.
Wiki Connections
Core concepts: spoken-language-model, speech-to-speech, streaming-tts, evaluation-metrics.
In-corpus reference: 2503.04721 (Full-Duplex-Bench, the fine-grained turn-taking benchmark used for evaluation in §4.2.2).