arXiv · 2026 · Preprint

Chi-Yuan Hsiao et al. (National Taiwan University / ASUS Open Cloud Infrastructure Software Center / NVIDIA) · → Paper · Demo: ? · Code: ?

Reinforcement learning on a full-duplex speech LM is restricted to a binary speak-or-stay-silent policy, obtained by summing padding and non-padding text logits, so that turn-taking can be optimized with GRPO without the repetition collapse that raw-token GRPO causes.

Problem

Full-duplex speech LMs such as Moshi must decide at every frame whether to speak, stay silent, backchannel, or yield to a user who barges in. Recent work applies reinforcement learning (typically GRPO) with timing rewards to tune these behaviors, but the reward is applied to the full text-token policy. The authors argue that this forces one optimization signal to shape both conversational timing and lexical content: under latency rewards, standard GRPO becomes over-eager, talks over pauses and backchannels, and degrades into repetitive, semantically empty speech. Plain supervised fine-tuning on conversational data does not fix timing either and also damages content quality.

Method

ASPIRin (Action Space Projection for Interactivity-Optimized Reinforcement Learning) keeps the base model’s architecture and changes only the space in which the RL objective is defined. In Moshi-style models the inner monologue text stream signals timing: padding tokens mean silence, non-padding tokens mean speech. The text vocabulary is partitioned into a padding set and a non-padding set, and each generated token is mapped to a binary action state (inactive silence vs. active speech). The projected state logit for each class is the sum of the raw logits of the tokens in that set, and a two-way softmax over these state logits defines a coarse state policy. This projected policy replaces the token policy inside the GRPO objective (importance ratio, KL penalty to a reference model’s projected policy), so gradients push probability mass between “speak” and “do not speak” without directly rewarding or penalizing specific words (§2.1).

Overview of the ASPIRin framework. (a) Action Space Projection & State Policy Optimization: The fine-grained text vocabulary is decoupled into a coarse-grained binary state (Active Speech vs. Inactive Silence) by grouping and summing non-padding and padding logits. This projected state policy is then explicitly optimized. (b) Rule-Based Rewards: The state policy is guided by continuous temporal constraints that penalize user interruption and excessive response latency. This explicit decoupling allows ASPIRin to master conversational timing without compromising semantic generation.

Rewards are rule-based rather than learned. User voice activity intervals come from ASR timestamps (parakeet-tdt-0.6b-v3), and the model’s state sequence is segmented into utterances. An Interruption Score gives the fraction of model utterances whose overlap with the user is below a tolerance (1.0 s), and a Response Score gives the fraction whose latency after the last user utterance is below a limit (1.0 s). The sequence reward is the product of the two, normalized across the group to give advantages (§2.2).

The base model is Moshi, used with a 3-second prompt delay heuristic in all experiments. Training uses a 43-hour in-house corpus of consented, anonymized dual-channel conversations (about 1,300 two-minute clips, filtered to at least 50% active speech), LoRA (r = 256) on all linear layers with fully trained temporal transformer embeddings, 8 V100 GPUs, 3 epochs, learning rate 1e-5, GRPO group size G = 2, and KL weight 0.001 (§3.1). Model size and codec are not restated in the paper.

Key Results

All results are on Full-Duplex-Bench (pause handling, backchanneling, smooth turn-taking, user interruption) against three baselines: Moshi with a 3-second prompt delay, standard SFT on the same data, and standard GRPO on the raw token policy (Table 1).

  • Prompt-delay heuristic: adding a 3-second prompt delay to Moshi cuts pause-handling and backchannel takeover rates (TOR) from about 0.98 to 1.0 down to 0.44 to 0.50 and raises the user-interruption GPT-4o score from 0.765 to 3.894, at the cost of higher interruption latency (0.257 s to 1.159 s). This delayed Moshi is the main baseline.
  • Standard SFT worsens nearly every dimension: backchannel TOR rises to 0.927, interruption TOR falls to 0.625, and the GPT-4o score collapses to 0.44.
  • Standard GRPO is the most eager system: best turn-taking TOR (0.857) and lowest turn-taking and interruption latencies (0.153 s, 0.614 s), but pause-handling TOR rises to 0.642/0.704 and backchannel TOR to 0.709, and the GPT-4o score drops to 3.247.
  • ASPIRin lowers backchannel TOR to 0.364 (best of all systems) and Candor pause TOR to 0.486, raises turn-taking TOR to 0.765 and interruption TOR to 0.941, and cuts interruption latency to 0.992 s relative to delayed Moshi, with a GPT-4o score of 3.734 (0.16 below delayed Moshi). It is not uniformly better than the delayed Moshi baseline: synthetic pause-handling TOR is slightly worse (0.482 vs. 0.467), backchannel JSD is worse (0.752 vs. 0.705), and turn-taking latency is higher (0.273 s vs. 0.161 s).
  • Repetition: relative to standard GRPO, ASPIRin reduces seq-rep-2 from 0.117 to 0.054 and seq-rep-3 from 0.072 to 0.029 (both over 50%), seq-rep-1 from 0.303 to 0.202, and Self-BLEU from 0.369 to 0.343 (Table 3). Qualitative interruption examples show standard GRPO looping on fillers (“you know”) and SFT producing word salad (Table 2).
  • Reward dynamics: both RL variants increase total reward, but the interruption component of standard GRPO oscillates and trends downward while ASPIRin’s stays stable, so total reward alone is a misleading stopping criterion for raw-token GRPO (§4.2, Figure 2).

Semantic quality is judged by GPT-4o and n-gram statistics on ASR transcripts; there is no human listening test, and Table 3 does not include the Moshi or SFT baselines.

Novelty Assessment

The architectural contribution is narrow but genuine: redefining the GRPO policy over a projected binary action space (summed logits over padding vs. non-padding tokens) is a new training objective for full-duplex speech LMs, and it neatly exploits the fact that Moshi-style inner-monologue padding already encodes timing. The reward design (overlap and latency thresholds from ASR timestamps) is a straightforward rule set, and the RL algorithm is standard GRPO. The empirical case is mainly that ASPIRin avoids the failure modes of raw-token GRPO and SFT; against the strong delayed-Moshi baseline the gains are modest and mixed. Evidence comes from a single base model, a small in-house dataset, a group size of two, and no variance or significance reporting.

Field Significance

Moderate — The paper gives a clear, mechanistic demonstration that applying timing rewards to the full token policy of a full-duplex speech LM can trade semantic quality for eagerness, and that restricting the RL action space to the speak/silence decision keeps content quality close to the base model. It offers a simple, reusable recipe for interaction-timing alignment in models whose text stream uses padding to mark silence, although its gains over a well-tuned heuristic baseline are small.

Claims

  • supports: Optimizing turn-taking rewards directly on the full token policy of a full-duplex speech LM degrades semantic coherence and induces repetitive generation.

    Evidence: Raw-token GRPO with latency/interruption rewards on Moshi drops the user-interruption GPT-4o score from 3.894 to 3.247 and yields seq-rep-2 of 0.117 and Self-BLEU of 0.369, with qualitative outputs looping on fillers. (§4.1, §4.3, Table 1, Table 2, Table 3)

  • supports: Decoupling when to speak from what to say, by restricting reinforcement learning to a binary speak/silence action space, preserves content quality while improving interaction timing in full-duplex speech LMs.

    Evidence: GRPO over a projected state policy (summed padding vs. non-padding logits) reduces seq-rep-2 to 0.054 and seq-rep-3 to 0.029 versus standard GRPO, keeps the GPT-4o score at 3.734 (vs. 3.894 for the base), and lowers backchannel TOR to 0.364 while raising interruption TOR to 0.941. (§2.1, §4.1, §4.3, Table 1, Table 3)

  • complicates: Supervised fine-tuning on natural conversational data does not by itself teach full-duplex speech LMs appropriate turn-taking behavior and can damage response quality.

    Evidence: SFT of Moshi on 43 hours of dual-channel conversation raises backchannel TOR from 0.436 to 0.927, lowers interruption TOR from 0.901 to 0.625, and collapses the GPT-4o score to 0.44. (§4.1, Table 1, Table 2)

  • complicates: In full-duplex dialogue, reward schemes that favor responsiveness trade off against yielding behavior, so gains in turn-taking latency often come with more talking over pauses and backchannels.

    Evidence: Standard GRPO achieves the lowest turn-taking and interruption latencies but the worst pause-handling and backchannel TOR among the RL systems; its interruption score oscillates and declines during training even as total reward rises. ASPIRin’s balanced profile comes with higher turn-taking latency than delayed Moshi (0.273 s vs. 0.161 s). (§4.1, §4.2, Table 1, Figure 2)

  • refines: Simple inference-time delays can be a strong baseline for full-duplex interaction timing, against which trained interaction policies show only modest additional gains.

    Evidence: A 3-second prompt delay alone reduces Moshi’s pause and backchannel TOR by roughly 0.5 and raises the interruption GPT-4o score from 0.765 to 3.894; ASPIRin’s further changes over this baseline are within about 0.02 to 0.07 TOR on most dimensions and slightly worse on synthetic pause handling. (§4.1, Table 1)

Limitations and Open Questions

Warning

All results come from a single base model (Moshi), a 43-hour in-house training set that is not released, and a GRPO group size of two, with no variance, seeds, or significance tests reported; semantic quality is judged only by GPT-4o and n-gram statistics, with no human evaluation.

  • Gains over the delayed-Moshi baseline are small and not uniform: synthetic pause-handling TOR, backchannel JSD, and turn-taking latency are worse than the baseline.
  • The repetition analysis (Table 3) compares only ASPIRin and standard GRPO, so it does not show whether ASPIRin matches the base model’s repetition level.
  • The projection relies on padding tokens in a text stream marking silence; applicability to full-duplex models without such a stream is not discussed.
  • Rewards depend on ASR-derived timestamps and fixed 1-second thresholds; sensitivity to these choices is not studied.
  • The binary action space cannot distinguish backchannels from full responses; the authors propose multi-class or hierarchical action spaces as future work (§5).

Wiki Connections

  • Spoken Language Model — Aligns the interaction timing of a full-duplex speech LM that consumes a live user audio stream while generating its own speech.
  • Speech-to-Speech — Targets end-to-end spoken dialogue, optimizing turn-taking, backchanneling, pause handling, and interruption behavior.
  • RLHF Speech — Applies GRPO with rule-based timing rewards and shows that restricting the policy’s action space avoids reward-hacking collapse seen with raw-token GRPO.
  • Moshi — Serves as the base model and, with a 3-second prompt delay, the main baseline that ASPIRin fine-tunes.
  • Full-Duplex-Bench — Provides the evaluation benchmark and metrics (TOR, latency, GPT-4o score) used for all interactivity results.
  • Full-Duplex-Bench v1.5 — Cited as part of the evaluation line of work on overlap handling that motivates precise temporal dynamics.
  • Full-Duplex-Bench-v2 — Cited among full-duplex systems and evaluation frameworks motivating the problem.
  • Align-SLM — Prior RL alignment of speech LMs from AI feedback that ASPIRin positions itself against as a raw-token policy approach.
  • RLAIF for Spoken Dialogue — Related RL-based optimization of conversational quality in spoken dialogue systems, cited as part of the raw-token RL paradigm.
  • LSLM — Earlier listen-while-speaking full-duplex model cited as background for the full-duplex paradigm.
  • Efficient and Direct Duplex Modeling — Another full-duplex speech-to-speech model cited as background whose text stream similarly guides interaction timing.