IWSDS · 2026 · Workshop

Takao Obi et al. (Equmenopolis / Institute of Science Tokyo / Waseda University) · → Paper · Demo: ? · Code: ?

Fine-tunes the full-duplex spoken dialogue model Moshi on proficiency-labeled second-language learner interviews and tests, via real-time sessions against an automated interviewer agent, whether the resulting CEFR-conditioned models reproduce the timing and fluency patterns of human learners at each proficiency level.

Problem

Full-duplex spoken dialogue models promise human-like conversational behavior, such as overlap handling, barge-ins, and rapid turn adjustment, without explicit turn segmentation. However, empirical work testing whether these models actually reproduce fine-grained human interactional behavior is still nascent: most full-duplex research emphasizes architecture and overlap-handling benchmarks rather than whether generated dialogue matches the statistical structure of real human conversation at a given behavioral profile. This gap is acute for language-proficiency assessment, where a system that could reliably emulate a learner at a given CEFR level would reduce the human effort required for large-scale automated evaluation, but no prior work has tested whether a full-duplex model can be conditioned on proficiency level and reproduce the corresponding dialogue features.

Method

The system adapts Moshi, a full-duplex speech-text foundation model, to second-language (L2) learner interview dialogues. Moshi consists of Mimi, a neural audio codec built from a SEANet autoencoder and a residual vector quantizer that discretizes 24 kHz audio into 8 RVQ codebooks at 12.5 Hz, and RQ-Transformer, a Temporal Transformer that models token sequences at 12.5 Hz combined with a Depth Transformer that models audio tokens along the depth dimension, with a one-step delay on acoustic tokens for stability.

The authors fine-tuned three separate copies of Moshi, one per aggregated CEFR proficiency level (A/B/C), on a corpus of 24,320 L2 learner interview dialogues (2,451 collected hours, 1,936 extracted hours) gathered over three academic years by InteLLA, an existing spoken dialogue system for eliciting spontaneous learner speech. Each dialogue carries an automatically assigned CEFR label from InteLLA, aggregated into three coarse levels. Training used the public moshi-finetune recipe for three epochs per level on 8 NVIDIA H200 GPUs (OneCycleLR, max learning rate 2e-6, batch size 8, 100-second input length, weight decay 0.1), with PAD-token losses down-weighted by 50% and a 100:1 semantic-to-acoustic token loss ratio, matching the original Moshi recipe. Training segments were restricted to spans from the end of a system utterance to the end of the following user utterance, rather than full interviews, because full-interview training produced excessive generation silence at inference.

At inference, the fine-tuned models were connected to InteLLA over WebRTC (end-to-end latency ≤200 ms) to run real-time interview sessions, with sampling temperature fixed at 0.95 to increase the models’ propensity to speak. Four single-topic interview scenarios, drawn from InteLLA’s A-level topic pool, were used to limit topic-driven variance, and 400 sessions were run per level (1,200 total), with each session started from a fresh model instance.

Key Results

The study compares model-generated dialogue against real human interviews (400 randomly sampled per level, drawn from pools of 3,117/5,579/778 dialogues for A/B/C) along two axes: level-wise trend direction (via Kruskal-Wallis and pairwise Mann-Whitney U tests with Holm-Bonferroni correction, plus Cliff’s delta effect size) and distributional similarity (bootstrap confidence intervals of the model-human mean/median difference ∆ and the one-dimensional Wasserstein distance W1, judged against strict/partial reproduction criteria).

For trend direction, articulation rate, pause ratio, and speech rate matched the human-observed direction across all three level-pair comparisons (A-B, B-C, A-C). Mid-clause pause ratio, end-clause pause ratio, mean length of run, and disfluency ratio matched in most but not all pairs, with weakest agreement specifically at the A-B boundary. Reaction time trend direction was reversed between levels A and B relative to humans, and the model’s response frequency showed no significant level effect at all (the only metric without a significant Kruskal-Wallis result in the model sessions), unlike the human data where response frequency differed significantly by level.

For distributional comparison, partial CI overlap with the human variability band was observed for the mean/median difference (∆) on several metric-level combinations (all levels for response frequency and pause ratio and disfluency ratio; a subset of levels for other metrics), but no overlap was observed on the Wasserstein distance (W1) for any metric, indicating that even where central tendencies partially aligned, the full generated distributions remained measurably different from human distributions. A baseline of the original, non-fine-tuned Moshi run through the same procedure showed larger |∆| and W1 than the fine-tuned, CEFR-conditioned models across several metrics and levels, indicating the proficiency fine-tuning helped rather than the WebRTC/InteLLA pipeline alone.

Novelty Assessment

The paper introduces no new architecture or training method; it fine-tunes an unmodified Moshi using the publicly released moshi-finetune recipe and connects it to an existing interviewer agent (InteLLA) over standard real-time audio infrastructure. The contribution is empirical and applied: a systematic, statistically grounded test of whether an existing full-duplex dialogue model, once conditioned on a proficiency label, reproduces the dialogue-level behavioral signature of humans at that same proficiency level. The rigor of the evaluation methodology (bootstrap CIs, dual trend/distribution criteria, an unconditioned-Moshi ablation baseline) is the paper’s main technical contribution, alongside the novel application framing of full-duplex models as proficiency-conditioned learner emulators for L2 assessment.

Field Significance

moderate — this paper contributes a first systematic empirical test of whether full-duplex dialogue models, once fine-tuned on proficiency-labeled data, can reproduce human interactional behavior at matching proficiency levels, and proposes a concrete new application (learner emulation for automated L2 proficiency assessment) for this model class. It also surfaces a specific, generalizable failure mode of full-duplex models fine-tuned this way: partial trend reproduction without full distributional match, and a persistent bias toward silence inherited from the base architecture. The evaluation is confined to a single proprietary corpus and interviewer system, and the model itself is an off-the-shelf fine-tune rather than a new method, which limits how far the findings generalize to other full-duplex architectures or dialogue domains.

Claims

  • supports: Full-duplex spoken dialogue models, once fine-tuned on behavior-labeled human dialogue corpora, can reproduce coarse level-wise trends in conversational timing and fluency observed in matching human data, even without any explicit disentanglement of the conditioning attribute from content or acoustics.

    Evidence: Articulation rate, pause ratio, and speech rate matched the human-observed pairwise trend direction across all three CEFR level comparisons (A-B, B-C, A-C) in the fine-tuned models, out of nine dialogue metrics evaluated. (§4.3, Table 4)

  • complicates: Matching the level-wise trend direction of a behavioral metric does not imply that a generative model’s output distribution matches the human distribution for that metric; the two properties can diverge sharply.

    Evidence: Bootstrap 95% CIs of the model-human mean/median difference (∆) showed partial overlap with the human variability band for several metric-level combinations, but CIs of the one-dimensional Wasserstein distance (W1) showed no overlap for any metric, meaning the full generated distributions remained distinguishable from human ones even where central tendencies partially aligned. (§4.4, Figure 3, Figure 4)

  • complicates: Full-duplex dialogue models can retain a systemic bias toward excessive silence and delayed response onset inherited from the base architecture, which persists after task-specific fine-tuning and requires separate heuristic mitigation (segment-level training extraction, elevated sampling temperature, single-topic scenario constraints) rather than being resolved by conditioning alone.

    Evidence: Despite training-segment restriction and a 0.95 sampling temperature intended to suppress silence, the model’s reaction-time trend direction between levels A and B was the reverse of the human direction, and the model’s response frequency showed no significant Kruskal-Wallis difference across levels, unlike human data. (§4.3, §5.3, Table 4)

  • refines: When fine-tuning separate conditioned models per category from an imbalanced corpus, categories with substantially less training data show weaker reproduction of fine-grained distinctions from their nearest neighboring category, even when the model still reproduces trends against more distant categories.

    Evidence: The CEFR level-A model was fine-tuned on roughly one-third the extracted hours of the level-B model (367h vs. 1,132h); of the metrics with any significant human A-B difference, the A-B level pair showed the weakest trend agreement of the three pairwise comparisons across nearly all fluency metrics. (§6.3, Table 1, Table 4)

Limitations and Open Questions

The training and inference procedures rely on several heuristic choices, most notably restricting fine-tuning to short utterance-end-to-utterance-end segments (rather than full dialogues) and fixing the sampling temperature at 0.95, both introduced specifically to suppress excessive generation silence. The authors acknowledge these choices were tuned empirically rather than derived from a principled procedure, and that they may bias the very timing distributions the study is trying to measure.

The evaluation metrics (reaction time, response frequency, and a fixed set of fluency indicators) do not cover overlap or backchannel behavior, which the authors note are central to full-duplex interaction but outside this study’s scope. Training data was substantially imbalanced across CEFR levels (level B had roughly three times the extracted hours of level A), confounding proficiency effects with data-volume effects. The evaluation is also context-specific: all sessions used single-topic interview scenarios from a single interviewer system (InteLLA) drawn from a single institution’s proprietary corpus, so it remains untested whether the reported reproduction (partial, level-wise) generalizes to other dialogue domains, interviewer systems, or full-duplex model families beyond Moshi.

Wiki Connections

  • Spoken Language Model — fine-tunes Moshi, an autoregressive speech-text foundation model, and evaluates the fidelity of its generated dialogue behavior against human data.
  • Speech-to-Speech — the evaluated system is a full-duplex, real-time speech-to-speech dialogue pipeline connecting a fine-tuned Moshi model with an interviewer agent over WebRTC.
  • Neural Audio Codec — relies on Moshi’s Mimi codec (SEANet encoder-decoder plus residual vector quantization at 12.5 Hz) as the underlying audio representation for both input and generated speech.
  • Evaluation Metrics — contributes a bootstrap-based dual-criterion methodology (trend-direction agreement plus distributional CI overlap) for judging whether a generative dialogue model reproduces a target human behavioral profile.
  • Moshi — fine-tunes this full-duplex speech-text foundation model directly, using its unmodified Mimi codec and RQ-Transformer architecture as the base for all proficiency-conditioned models.
  • Full-Duplex-Bench v1.5 — a contemporaneous, complementary evaluation effort for full-duplex spoken dialogue models, focused on overlap handling rather than proficiency-conditioned reproduction of human dialogue features.
  • Neural Codec Language Models are Zero-Shot TTS Synthesizers — cited as background establishing that neural codec language models can achieve zero-shot speaker and style transfer and long-range prosody, motivating the codec-token modeling approach underlying Moshi.
  • SoundStorm — cited as background on parallel, bidirectional decoding of codec tokens as an alternative direction within the speech foundation model line that Moshi’s full-duplex approach builds on.