ACL · 2025 · Conference
Chiba et al. (NTT / Nagoya University) · → Paper · Demo: ? · Code: ?
The first method to incorporate dialogue state tracking into a full-duplex spoken dialogue model, enabling task-oriented speech interaction without breaking the end-to-end audio stream.
Problem
Full-duplex spoken dialogue models can naturally handle turn-taking phenomena such as interruptions and backchannels, but they lack mechanisms for language understanding and dialogue management. This makes them ill-suited for task-oriented dialogue systems that must query external databases and track user intent across turns. Moshi represents the state of the art in full-duplex speech interaction but has no capacity for structured dialogue state inference or controlled response generation from retrieved data.
Method
The paper proposes incorporating dialogue state tracking (DST) into J-Moshi, the Japanese adaptation of Moshi, by repurposing its existing text stream. Moshi’s architecture consists of two Transformers: the Temporal Transformer (a 7B LLM backbone operating along the time axis, which autoregressively outputs a context vector per time step) and the Depth Transformer (which converts that context vector into multi-codebook audio tokens for both system and user channels). In normal operation, the Temporal Transformer also outputs system text tokens via a text-linear layer.
The proposed method inserts special delimiter tokens into the text stream to mark belief state segments. When the model generates a <bs> (begin belief state) token, the Depth Transformer is deactivated and the Temporal Transformer performs DST, estimating slot-value pairs until a </bs> token is produced. External information (database search results) is then injected into the text stream before audio generation resumes. DST thus runs entirely within the 7B language model backbone, leveraging its text understanding capability without architectural modifications.
Training requires stereo spoken dialogue data, which the authors synthesize from JMultiWOZ, a Japanese text-based task-oriented dialogue corpus. Text dialogues are first converted to conversational style using GPT-4o-mini (adding fillers, backchannels, and natural interjections), then synthesized to speech using OuteTTS (a zero-shot multilingual TTS system) with a single fixed speaker for the system channel and per-dialogue consistent voices for user turns. Dialogue states are tokenized and inserted at the corresponding positions in the token sequence; audio tokens within DST segments are masked from the loss calculation. The model is fully fine-tuned with AdamW on 12 NVIDIA V100 32 GB GPUs.
Key Results
Evaluated on the JMultiWOZ test set (300 dialogues), the proposed method achieves JGA 65.69 and Slot F1 97.78 with 100% of training data (Table 1). Both metrics improve monotonically as training data volume increases from 25% to 100%. Text-based upper bounds reveal a meaningful gap: T5-base reaches JGA 74.07 and T5-large reaches JGA 84.45, with the proposed method falling about 19 JGA points below T5-large. The disparity is even larger for response generation, where the proposed method achieves BLEU 0.129 and BERTScore 0.689 against T5-large’s BLEU 0.364 and BERTScore 0.830. DST initiation timing averaged 4.9 seconds MAE from the reference. The authors attribute the response generation gap to insufficient training data and to audio quality issues from the multilingual TTS system used for data synthesis, which introduced recognition errors during time-aligned tokenization.
Novelty Assessment
The primary contribution is engineering integration: DST is implemented by repurposing Moshi’s existing text stream with special delimiter tokens and fine-tuning on synthesized spoken data. No new architecture is introduced; the Temporal and Depth Transformers are used without structural changes. The data synthesis pipeline (GPT-4o-mini dialogue conversion, OuteTTS speech generation, time-aligned tokenization) is procedurally novel but assembles existing tools. The value lies in demonstrating that full-duplex spoken dialogue models can be extended to task-oriented settings without breaking the audio-in/audio-out paradigm, and in exposing concretely the performance gap that remains relative to text-based systems.
Field Significance
Moderate — This paper provides a concrete baseline for task-oriented full-duplex dialogue in Japanese, demonstrating that the 7B LLM backbone in Moshi can handle belief state estimation reasonably well within its text stream. The response generation gap relative to text-based systems reveals that bridging text-audio modality differences in task-directed spoken interaction remains a major open challenge for full-duplex spoken conversational agents.
Claims
-
supports: A full-duplex speech language model can perform structured dialogue state tracking when DST is executed entirely through the model’s text stream using explicit delimiter tokens.
Evidence: J-Moshi-ext fine-tuned on synthesized JMultiWOZ data achieves JGA 65.69 and Slot F1 97.78 on the JMultiWOZ test set, with both metrics improving monotonically as training data volume increases from 25% to 100%. (§3, §5.3, Table 1)
-
complicates: Extending full-duplex speech models to task-oriented dialogue does not close the gap with text-based systems on response generation quality.
Evidence: The proposed method reaches BLEU 0.129 and BERTScore 0.689, against text-based upper bounds of BLEU 0.364 and BERTScore 0.830 (T5-large), with the gap attributed to insufficient training data and to the difficulty of aligning text and audio modalities during response generation. (§5.3, Table 1)
-
complicates: TTS data quality limitations compound into downstream training noise when synthesized speech is used to construct full-duplex dialogue training corpora.
Evidence: The multilingual OuteTTS model used for Japanese speech synthesis generated speech with reduced clarity, causing recognition errors during time-annotated tokenization that hindered the model’s learning of coherent text-audio token sequences for response generation. (§5.3)
-
supports: DST timing estimation is feasible within a full-duplex spoken dialogue model but introduces substantial prediction error tied to the model’s speech perception accuracy.
Evidence: DST initiation timing (defined as the time of
<bs>token generation) achieved a MAE of 4.9 seconds, with discrepancies attributed to the difficulty of accurately predicting user utterance boundaries from the model’s audio token representations alone. (§5.1, §5.3)
Limitations and Open Questions
Warning
All experiments are conducted in Japanese using J-Moshi, which was trained on less Japanese data than the original English Moshi. The authors explicitly note that performance on English equivalents (MultiWOZ, SpokenWOZ) could be substantially higher, and cross-lingual generalisability is untested.
Evaluation is fully automatic (JGA, Slot F1, BLEU, BERTScore); no subjective listening tests or user studies are reported, leaving response naturalness, latency under real deployment conditions, and the effect of DST pauses on conversational flow unassessed. The experiment covers only the domains included in JMultiWOZ (travel planning), and extension to diverse or open-domain settings is planned but not yet evaluated. Future work includes experiments with the original Moshi on SpokenWOZ and DSTC11 datasets and application to additional dialogue domains.
Wiki Connections
- Spoken Language Model — This paper extends Moshi, a full-duplex spoken language model based on discrete audio tokens, by adding dialogue state tracking capability within its text stream, enabling task-oriented dialogue without abandoning end-to-end audio processing.
- Speech-to-Speech — The paper targets full-duplex audio-in/audio-out interaction where user and system speech channels are processed and generated simultaneously, extending this paradigm to task-oriented dialogue settings.
- Neural Audio Codec — The system relies on the Mimi codec underlying Moshi to represent both user and system audio as discrete multi-codebook token streams that the Depth Transformer generates autoregressively.
- Autoregressive Codec TTS — J-Moshi-ext, the base model, autoregressively generates multi-codebook audio tokens using a Depth Transformer conditioned on temporal context vectors from the 7B LLM, a codec-native autoregressive generation paradigm.
- Full-Duplex-Bench — Benchmarking work for evaluating full-duplex spoken dialogue models on turn-taking dynamics, cited as a related evaluation framework alongside this paper’s contribution.