IWSDS · 2026 · Workshop
Yuto Abe et al. (Waseda University) · → Paper · Demo: ? · Code: ?
Quantifies how the interactional style (turn-taking timing) of Japanese spoken dialogue corpora shapes the behavior of a Moshi-based full-duplex dialogue model fine-tuned on them via a curriculum strategy.
Problem
Full-duplex spoken dialogue models such as Moshi, J-Moshi, FreezeOmni, and SyncLLM can listen and speak simultaneously, enabling low-latency turn exchange. Prior work on scaling and improving such models has mostly targeted acoustic and linguistic quality, but an effective dialogue system must also reproduce task-dependent interactional dynamics, such as conversational tempo and turn-taking pattern, that differ sharply between casual chat and formal consultation. Existing corpus-level analyses of turn-taking (pauses, gaps, overlaps) tend to rely on small or text-derived datasets, and curricula used to fine-tune large speech dialogue models are rarely designed with explicit attention to a corpus’s turn-taking profile. This leaves open whether, and how, a training corpus’s interactional characteristics actually transfer into the fine-tuned model’s generated dialogue behavior.
Method
The paper analyzes four Japanese spoken dialogue corpora spanning a continuum from large, noisy web-derived conversation to smaller, high-quality task-specific recordings: J-Chat (web-derived podcast and YouTube subsets), J-Chat-Clean (a quality-filtered subset of J-Chat), CSJ (formal, well-structured spontaneous speech), and Tabidachi (travel-agency consultation dialogues with asymmetric, system-led turns). Turn-taking style is quantified with four timing-based indicators, inter-pausal units (IPU), pause, gap, and overlap, computed over 20-second windows from voice-activity-detector (Silero VAD) segments. Acoustic quality is scored with the NISQA model (predicted MOS), and linguistic/semantic quality is scored with an LLM-as-a-Judge framework rating coherence, naturalness, relevance, instruction-following, turn-taking, and overall quality on a ten-point scale.
Using these corpus profiles, the authors fine-tune the Moshi full-duplex architecture, which encodes stereo dialogue audio and generates time-synchronized multi-channel speech in real time, with a three-stage curriculum: pre-training on the large, noisy J-Chat corpus to acquire general conversational structure, intermediate fine-tuning on either the cleaner J-Chat-Clean or the formal CSJ to improve robustness and temporal alignment, and final fine-tuning on the task-specific, high-quality Tabidachi corpus to refine travel-consultation interaction patterns. Four curriculum variants (Models 1-4) combine these corpora in different orders and combinations. Evaluation uses a dialogue-continuation setup: each model receives a 10-second audio prompt from a held-out Tabidachi split and generates the next 20 seconds of dialogue, which is then scored with the same acoustic, linguistic, and timing metrics used to characterize the source corpora.
Key Results
Corpus-level analysis (Tables 1-2) shows the four corpora occupy distinct positions on both acoustic-quality and interactional-style axes independently of each other: CSJ attains the highest acoustic naturalness (NISQA 3.14) and semantic naturalness (LLM-as-a-Judge NAT 6.96) but is not the most overlap-rich; J-Chat (Podcast) shows the most frequent overlaps (5.06 per 20s) consistent with spontaneous chat; Tabidachi shows the fewest overlaps (1.16) and the largest inter-speaker IPU asymmetry (5.06 vs. 2.12), consistent with one-sided, task-oriented turns; and web-derived J-Chat (YouTube) has the lowest acoustic quality (NISQA 1.94) despite scale.
In the fine-tuning experiment (Table 3), Model 1 (Tabidachi only) achieves the highest acoustic quality (NISQA 3.12), while Model 2 (CSJ + Tabidachi) scores best on LLM-as-a-Judge Coherence (4.50), Relevance (3.73), Instruction-Following (2.86), and Overall (4.20). Model 4, which mixes three stylistically different corpora (J-Chat-Clean + CSJ + Tabidachi), does not surpass the simpler two-corpus curricula on any linguistic metric. On generated-dialogue turn-taking (Table 4), Model 3 (J-Chat-Clean + Tabidachi) produces the longest, most frequent gaps (2.32 per 20s; 6.32s cumulative) and fewest overlaps (0.66; 0.27s cumulative) relative to Model 1, showing that the source corpus’s turn-taking profile carries through into the fine-tuned model’s generated behavior rather than washing out during fine-tuning.
Novelty Assessment
The contribution is empirical rather than architectural: the paper introduces no new model, conditioning mechanism, or training objective, instead fine-tuning the existing Moshi architecture with a standard curriculum-learning recipe. Its novelty lies in systematically quantifying corpus-level turn-taking style with four timing indicators across corpora that differ in domain and recording quality, and in demonstrating empirically that these interactional properties, not just corpus size or acoustic cleanliness, propagate into a fine-tuned full-duplex dialogue model’s generated behavior. This is best read as a data-curation and curriculum-design study for full-duplex spoken dialogue models rather than a new modeling technique.
Field Significance
moderate — The paper provides a concrete, transferable methodology (four timing-based turn-taking indicators plus NISQA/LLM-as-a-Judge scoring) for characterizing spoken dialogue corpora and linking those characteristics to curriculum design choices for full-duplex dialogue model fine-tuning. Its scope is narrow (Japanese-language corpora, a single base architecture, four curriculum variants), but the finding that turn-taking profile transfers from training corpus to generated dialogue behavior, and that naively combining stylistically mismatched high-quality corpora does not compound their benefits, is directly actionable for anyone assembling fine-tuning curricula for full-duplex spoken dialogue systems.
Claims
- supports: Corpus-level turn-taking properties (overlap frequency, gap duration, pause structure) measurably shape the interactional behavior a fine-tuned full-duplex dialogue model reproduces, so training-data selection functions as a lever for controlling dialogue style, not only for improving quality or scale.
Evidence: Fine-tuning on J-Chat-Clean + Tabidachi (Model 3) yielded the fewest overlaps (0.66 per 20s) and longest gaps (6.32s cumulative) in generated dialogues, tracking the calmer, less-overlapping profile already measured in those source corpora, while Model 1 (Tabidachi only) showed more overlaps (0.98) and shorter gaps (4.94s). (§3.3, Table 4)
- complicates: Combining multiple high-quality but stylistically mismatched corpora in a fine-tuning curriculum does not straightforwardly compound their individual behavioral benefits and can instead slightly degrade linguistic coherence relative to simpler two-corpus curricula.
Evidence: Model 4 (J-Chat-Clean + CSJ + Tabidachi) did not surpass the two-corpus curricula (Models 2 and 3) on LLM-as-a-Judge Overall (4.09 vs. 4.20 / 4.11) or Coherence (4.20 vs. 4.50 / 4.39), which the authors attribute to redundancy and over-regularization from mixing stylistically mismatched high-quality data. (§3.2, Table 3)
- supports: Automated acoustic-quality and interactional-timing metrics can separate spoken dialogue corpora along axes, recording/acoustic quality versus conversational turn-taking style, that vary independently and are conflated when corpora are characterized only by size or domain label.
Evidence: CSJ attained the highest NISQA acoustic-quality score (3.14) among the four corpora while showing far fewer overlaps (2.30 per 20s) than the chat-style J-Chat Podcast subset (5.06), demonstrating that acoustic cleanliness and interactional style are separable corpus properties. (§2.3, Tables 1-2)
- complicates: Fine-tuning a full-duplex dialogue model on clean, domain-matched, low-overlap data alone can improve perceived orderliness and turn separation while reducing conversational spontaneity.
Evidence: Model 3, fine-tuned on J-Chat-Clean + Tabidachi, produced the most orderly generated turn-taking (fewest overlaps, most/longest gaps) but the authors report this suppresses mutual overlaps, which they identify as “a hallmark of lively conversation.” (§3.3)
Limitations and Open Questions
All quality judgments in this study come from automated proxies, NISQA for acoustic quality and an LLM-as-a-Judge for linguistic/semantic quality, with no human listening test or human rater validation reported. The reliability of the reported naturalness and coherence differences therefore depends entirely on how well these automated proxies track human judgment for full-duplex Japanese dialogue speech, which the paper does not verify.
The study is limited to Japanese-language corpora and a single base architecture (Moshi), so it is unclear whether the observed transfer of corpus turn-taking style into fine-tuned model behavior generalizes to other languages or full-duplex architectures. Only four curriculum variants were tested, all ending in fine-tuning on the same target corpus (Tabidachi), and the dialogue-continuation evaluation uses a single held-out split from that same corpus, limiting the diversity of dialogue styles the fine-tuned models are actually tested on. The paper also does not report model size or codec details for the fine-tuned Moshi variant, and does not test whether corpus turn-taking effects persist under longer fine-tuning or larger held-out evaluation sets.
Wiki Connections
- Spoken Language Model — studies how training-corpus interactional properties shape the fine-tuned behavior of a full-duplex speech language model that listens and speaks concurrently on live dialogue audio.
- Speech-to-Speech — targets the dialogue sub-paradigm of speech-to-speech interaction, evaluating turn-taking and conversational continuation rather than translation or codec-level transforms.
- Streaming TTS — the fine-tuned Moshi-based model generates time-synchronized speech in real time, and the paper’s timing metrics (gap, overlap) directly characterize this streaming turn-exchange behavior.
- Evaluation Metrics — introduces a corpus-level turn-taking characterization scheme (IPU, pause, gap, overlap) alongside NISQA and LLM-as-a-Judge scoring as a methodology for evaluating spoken dialogue corpora and models.
- Moshi — fine-tunes the Moshi full-duplex architecture directly, studying how training-corpus properties shape its dialogue behavior rather than proposing a new architecture.
- Towards a Japanese Full-duplex Spoken Dialogue System — closely related prior work from an overlapping author group that built a Japanese full-duplex Moshi variant (J-Moshi); this paper extends that line by systematically analyzing how corpus turn-taking properties shape curriculum fine-tuning outcomes.
- Freeze-Omni — cited as a comparable full-duplex speech-to-speech dialogue architecture with a frozen LLM backbone, contextualizing Moshi within the broader family of full-duplex dialogue systems this paper builds on.