Interspeech · 2025 · Conference
Ohashi et al. (Nagoya University) · → Paper · Demo: ? · Code: ?
J-Moshi adapts Moshi’s full-duplex dialogue architecture to Japanese through two-stage training on 69,000 hours of monophonic speech and 344 hours of stereo dialogue, augmented with 602 hours of TTS-synthesized dialogue, establishing the first publicly available full-duplex spoken dialogue system for Japanese.
Problem
Full-duplex spoken dialogue systems capable of modeling simultaneous bidirectional features such as speech overlaps and backchannels have been developed almost exclusively in English. Existing Japanese dialogue systems follow half-duplex conventions, requiring a speaker to finish before the system responds. This gap leaves Japanese-specific conversational phenomena, such as frequent backchannels and speech overlaps, unaddressed by current spoken conversational agent technology.
Method
The paper builds J-Moshi by adapting Moshi’s two-component architecture to Japanese. The base architecture consists of the Mimi neural audio codec, which encodes 24kHz speech into discrete tokens at 12.5Hz using eight codebook layers (one semantic layer plus seven acoustic layers), and the RQ-Transformer, a 7B-parameter autoregressive language model built on Llama 2 7B that jointly models text tokens (inner monologue) and audio tokens for both the system and the user in parallel streams.
Because Moshi’s original text tokenizer covers only English, the first adaptation step replaces it with a SentencePiece model from Japanese GPT-2 (32,000 vocabulary) and reinitializes the associated embedding tables and output projection in RQ-Transformer. Mimi’s parameters are frozen throughout, as early experiments showed it can encode and resynthesize Japanese speech without additional training.
Training proceeds in two stages. In pre-training, the model trains for one epoch on approximately 60,000 hours of J-CHAT, a large Japanese spoken dialogue corpus collected from YouTube and podcasts. Because J-CHAT is monophonic, speaker diarization is applied to construct artificial two-channel stereo audio. Text token timestamps are obtained with WhisperX and PAD tokens fill timesteps without text assignment; the resulting data contains 3 billion text tokens, 88% of which are PAD tokens. Training runs on 128 NVIDIA V100 GPUs using ZeRO-3 parallelism, requiring 36 hours.
Fine-tuning uses 344 hours of genuine stereo Japanese spoken dialogue from five corpora spanning telephone conversations, Zoom meetings, casual dialogue, and consultation scenarios. This stage models natural turn-taking, including speech overlaps and backchannels absent from the diarized pre-training data. The fine-tuned model is called J-Moshi.
A second variant, J-Moshi-ext, incorporates 602 hours of synthetic stereo dialogue generated by a multi-stream TTS system. Four Japanese text dialogue corpora are rewritten using an LLM to introduce spoken language expressions, then synthesized with 10 different random seeds per dialogue; the sample with the lowest WER (overall 24.6%) is selected. Fine-tuning on the combined 946-hour dataset (344 hours real plus 602 hours synthetic) is expected to increase dialogue diversity and improve language capability.
Key Results
Human evaluation on 50 randomly selected 20-second dialogue continuations per model uses a 5-point scale for naturalness and meaningfulness. J-Moshi achieves naturalness 2.67 and meaningfulness 2.19, outperforming dGSLM (naturalness 2.44, meaningfulness 1.76) on both dimensions. J-Moshi-ext further improves meaningfulness to 2.30, confirming that synthetic dialogue augmentation benefits language capability without degrading naturalness. Both models fall well below the Mimi re-synthesis ceiling (naturalness 3.90, meaningfulness 3.92), indicating the RQ-Transformer component has significant room for improvement. Ground-truth scores of approximately 4.45 on both dimensions confirm a large quality gap to natural speech (Table 2).
Automatic evaluation via language model perplexity on ASR transcripts of generated speech shows J-Moshi with temperature 0.8 improves PPL by approximately 100 points over dGSLM. Re-synthesis by Mimi alone degrades PPL minimally from ground truth, confirming the codec can be applied to Japanese without retraining.
Comparison of turn-taking statistics against English Moshi (Table 3) shows J-Moshi generates 53.2 IPUs per minute with 5.0 seconds of overlap per minute, versus English Moshi’s 35.1 IPUs and 1.2 seconds of overlap. Japanese ground-truth shows 8.1 seconds of overlap per minute, consistent with known characteristics of Japanese conversation. This suggests the model has internalized language-specific turn-taking norms from the training data.
Novelty Assessment
The contribution is engineering integration: Moshi’s architecture and training recipe are applied to Japanese without structural modification. The vocabulary replacement, two-stage training pipeline, and TTS augmentation strategy all follow Defossez et al.’s original Moshi design. The genuine contribution is a working Japanese baseline, the public release of J-Moshi, and a comparative analysis of English versus Japanese turn-taking behavior in a full-duplex system. The findings on Mimi’s cross-lingual robustness and the high PAD token ratio in Japanese data (88% vs 65% in English) are practical insights for future language adaptation work.
Field Significance
Moderate — J-Moshi provides the first public baseline for full-duplex spoken dialogue in Japanese, demonstrating that the Moshi architecture can be adapted to non-English languages without modifying the codec. The comparative analysis of turn-taking statistics across Japanese and English introduces a replicable evaluation methodology for cross-lingual full-duplex dialogue research.
Claims
-
supports: Large-scale monolingual pre-training followed by stereo dialogue fine-tuning enables spoken dialogue models to acquire language-specific conversational behaviors.
Evidence: J-Moshi, trained on J-CHAT and stereo Japanese dialogue, exhibits more speech overlaps (5.0s/min) and more IPUs (53.2/min) than English Moshi (1.2s overlap, 35.1 IPUs), consistent with Japanese conversational norms. (§5, Table 3)
-
supports: Synthetic spoken dialogue generated by multi-stream TTS improves language capability in full-duplex dialogue models when added to fine-tuning data.
Evidence: J-Moshi-ext (trained with 602 hours of TTS-synthesized dialogue added) achieves meaningfulness 2.30 versus J-Moshi’s 2.19, a statistically distinguishable improvement, with no degradation in naturalness. (§4.3, Table 2)
-
complicates: Neural audio codecs pre-trained on one language can transfer to another with minimal acoustic degradation, but the autoregressive language model component requires substantial retraining to achieve acceptable dialogue quality.
Evidence: Mimi re-synthesis of Japanese speech degrades by approximately 0.5 MOS from ground truth, while J-Moshi (with RQ-Transformer adapted) degrades by more than 1 MOS, identifying the language model as the primary quality bottleneck. (§4.3, Table 2)
-
complicates: Morphological density differences across languages affect full-duplex dialogue model training dynamics: languages with higher phoneme-to-token ratios produce sparser text-to-audio token alignments that may require adjusted training objectives.
Evidence: Japanese data preprocessing results in 88% PAD tokens in text sequences versus 65% for English in Moshi, reflecting that kanji characters encode more phonemes per token, and the authors flag this as a design consideration for future Japanese-specific training objectives. (§5)
Limitations and Open Questions
Both J-Moshi and J-Moshi-ext score above 1 MOS below the Mimi re-synthesis ceiling, indicating the autoregressive RQ-Transformer component is a major quality bottleneck in Japanese. Mimi itself degrades by approximately 0.5 MOS from ground truth when applied to Japanese without adaptation, suggesting codec fine-tuning for Japanese will be necessary for production-quality systems.
The comparison with English Moshi in Table 3 is not conducted under identical experimental conditions (different test sets, possibly different prompt lengths), so turn-taking statistics should be interpreted as indicative rather than rigorously controlled. The 24.6% overall WER of the TTS-synthesized augmentation data introduces noise, and the effect of this noise on specific error categories is not analyzed. The paper does not evaluate spoken dialogue content quality beyond naturalness and meaningfulness, leaving turn-taking appropriateness and response coherence unmeasured.
Wiki Connections
- Spoken Language Model — J-Moshi adapts the RQ-Transformer, a 7B-parameter LLM-based architecture, to Japanese spoken dialogue, demonstrating that LLM-native full-duplex dialogue models can be extended to non-English languages through vocabulary replacement and staged fine-tuning.
- Autoregressive Codec TTS — The system generates discrete speech tokens autoregressively via RQ-Transformer, following the same codec-based generation approach as the original Moshi.
- Neural Audio Codec — Mimi encodes and decodes Japanese speech at 12.5Hz with minimal degradation even without Japanese-specific retraining, providing evidence that neural codecs can transfer across languages.
- Speech-to-Speech — J-Moshi is a full-duplex speech-to-speech dialogue system that models both the system and user speech streams simultaneously in separate channels.
- Streaming TTS — The system operates in real-time, generating speech tokens at 12.5Hz while continuously processing the user’s audio stream, enabling simultaneous bidirectional spoken interaction.