arXiv · 2024 · Preprint
Défossez et al. (Kyutai) · → Paper · Demo: ✓ · Code: ✓
Moshi is the first real-time full-duplex spoken dialogue system that eliminates the cascaded ASR-LM-TTS pipeline by casting conversation as speech-to-speech generation with parallel streams for both speakers, achieving a theoretical latency of 160ms through a hierarchical multi-stream architecture and a novel “Inner Monologue” mechanism that generates text tokens as a prefix to audio tokens.
Problem
Existing spoken dialogue systems (Gemini, ChatGPT voice mode) are pipeline composites: voice activity detection feeds an ASR model, whose output enters an LLM, whose text output drives a TTS. This cascade imposes multiple latency penalties — typically several seconds end-to-end — and discards all paralinguistic information at each text bottleneck. Furthermore, all such systems model dialogue as a strict sequence of speaker turns, which cannot represent overlap, interruptions, or backchanneling that are constitutive of natural spoken conversation. The only prior full-duplex model (dGSLM) addressed multi-stream modeling but lacked text LLM knowledge, ran offline only, and modeled semantic tokens without acoustic reconstruction.
Method
Moshi integrates three purpose-built components into a single autoregressive generation system.
Helium is a 7B-parameter text language model trained from scratch on 2.1T tokens of filtered English text. Its architecture follows modern LLM conventions (RMSNorm, RoPE, FlashAttention, SiLU-gated MLPs, SentencePiece tokenizer with 32k vocabulary). Helium provides the Temporal Transformer backbone for Moshi.
Mimi is a causal neural audio codec operating at 24kHz with a frame rate of 12.5Hz and a bitrate of 1.1kbps. It extends the SoundStream/EnCodec encoder-decoder architecture with two Transformer modules in the bottleneck (8 layers, 8 heads, causal masking) and a “split residual vector quantizer”: one plain VQ trained with WavLM distillation captures semantic information, while a parallel 7-level RVQ handles acoustic reconstruction. This split avoids the information-theoretic tension between phonetic discriminability and acoustic fidelity that plagues single-tower distillation approaches like SpeechTokenizer. Training with adversarial losses only (discarding reconstruction losses) substantially boosts perceptual quality (MUSHRA 81.0 vs. 58.8 with mixed losses), while the causal architecture makes Mimi the only streaming semantic-acoustic codec at this quality level.
The generative architecture uses an RQ-Transformer pairing a large Temporal Transformer (initialized from Helium, 7B parameters, 4096 context) with a small Depth Transformer (6 layers, 1024 dimensions) that predicts the K=17 tokens per timestep in sequence — text token, Moshi’s semantic token, Moshi’s 7 acoustic tokens, user’s semantic token, user’s 7 acoustic tokens. The multi-stream design models both sides of the conversation as parallel autoregressive token streams with no explicit turn boundary; Moshi generates silence tokens when the user is speaking.
Inner Monologue inserts time-aligned text tokens as a prefix to the semantic token at each frame. By adjusting the delay between text and audio tokens, the same model architecture can be configured as a full dialogue agent (text coincident with audio), a streaming TTS (text ahead of audio), or a streaming ASR (text behind audio). This single-hyper-parameter switching requires no architectural changes or retraining. Ablations show Inner Monologue is the single largest contributor to linguistic quality, improving transcript length from 602 to 1920 characters and NLL from 3.65 to 2.77 (§5.3, Table 6).
Training proceeds in four stages: (1) Helium text pre-training; (2) audio pre-training on 7M hours of unsupervised audio (single-stream); (3) multi-stream post-training using diarized unsupervised audio then the Fisher telephone conversation dataset; (4) instruction fine-tuning on 20k+ hours of synthetic speech generated from LLM-written transcripts, TTS-synthesized with a consistent target voice.
Key Results
Moshi achieves 4.7% WER on LibriSpeech test-clean in streaming TTS mode, outperforming VALL-E (5.9%) while requiring only 2 seconds of lookahead versus VALL-E’s full-sequence access (§5.7). Its streaming ASR yields 5.7% WER, within 2 points of a dedicated Streaming FastConformer (3.6%).
On spoken question answering, Moshi with Inner Monologue scores 26.6 / 62.3 / 22.8 on Web Questions / LlaMA-Questions / TriviaQA (spoken), compared to 6.5 / 21.6 / 14.8 for SpeechGPT and 6.1 / 22.9 / — for Spectron (§5.5, Table 8). Inner Monologue nearly triples accuracy relative to the same model without it (e.g., Web Questions: 9.2 without vs. 26.6 with), at negligible inference cost (17 tokens per step instead of 16).
On audio language modeling benchmarks (sWUGGY, sBLIMP, Spoken StoryCloze), Moshi matches or outperforms prior work in all three evaluation categories: audio-only cold start, audio-only warm start, and text-and-audio warm start (§5.4, Table 7). MMLU for Moshi after instruct tuning reaches 49.7, 12 points higher than Spirit-LM.
Mimi achieves MUSHRA 81.0 with adversarial-only training at 1.1kbps and 12.5Hz, outperforming SpeechTokenizer at 4.0kbps (MUSHRA 74.3) and SemantiCodec (MUSHRA 64.8), while operating causally and at 4x lower frame rate (§5.2, Table 4).
Note
The spoken QA benchmarks are evaluated on synthesized audio versions of text datasets, and the test sets used for TTS/ASR comparison are not identical across baselines. Moshi’s turn-taking statistics (Table 9) are evaluated on generated dialogues seeded from Fisher, not from live interactions, which may underrepresent real conversational dynamics.
Novelty Assessment
The contribution is architecturally significant on multiple fronts simultaneously. The multi-stream autoregressive formulation with no explicit turn boundary is genuinely novel — prior full-duplex systems either required turn segmentation or sacrificed acoustic quality. Inner Monologue as a framework for unifying dialogue, TTS, and ASR under a single delay parameter is a conceptual contribution that bridges the speech-text modality gap more cleanly than Chain-of-Modality (which forces full text generation before audio) or parallel text-speech prediction (which degrades quality). Mimi’s split-RVQ design resolves a known tension in distillation-based codecs between semantic and acoustic objectives, and its adversarial-only training result is a counterintuitive finding with practical implications. The work is also honest about what Moshi does not solve: safety scores fall below industry instruction-tuned models, spoken factual QA degrades noticeably versus the text-only Helium baseline (particularly on TriviaQA), and signal-based watermarking fails against codec compression.
Field Significance
Important
Foundational — Moshi establishes the multi-stream full-duplex speech-text LLM as a viable architecture, demonstrating that natural conversation dynamics (overlap, interruptions, backchanneling) can be modeled end-to-end without cascaded pipelines, and that a single delay hyper-parameter is sufficient to unify dialogue, TTS, and ASR under one model.
Claims
- Eliminating the text bottleneck in spoken dialogue requires modeling acoustic tokens jointly with semantic tokens in a single generative model, as purely semantic approaches cannot capture paralinguistic information or generate in arbitrary voices. (§3.4, §5.4)
- Predicting time-aligned text tokens as a per-frame prefix to audio tokens substantially improves the linguistic quality and factual accuracy of speech generated by audio language models, with minimal inference overhead. (§3.4.4, §5.3, Table 6)
- Modeling conversation as parallel autoregressive streams for each speaker, without explicit turn boundaries, enables full-duplex spoken interaction and allows training on naturally overlapping speech. (§3.4.3, §5.6, Table 9)
- Adversarial-only training of neural audio codecs substantially improves subjectively rated audio quality relative to mixed reconstruction-adversarial objectives, despite degrading objective metrics such as VisQOL. (§3.3, §5.2, Table 4)
- Standard objective audio quality metrics (VisQOL, MOSNet) are unreliable proxies for perceived quality when the training objective changes, making human evaluation indispensable for codec comparison. (§5.2, §5.8)
Limitations and Open Questions
Warning
Moshi’s spoken factual question answering performance lags substantially behind its Helium text baseline, particularly on multi-sentence or syntactically complex questions (TriviaQA: 22.8 vs. 56.4 for text-only Helium). This indicates that audio training causes significant forgetting of factual knowledge, and the instruct fine-tuning data does not cover the syntactic diversity needed to recover it.
Signal-based watermarking (Audioseal) is ineffective against codec compression: Mimi’s own lossy coding removes the watermark to below detection threshold. The generative watermarking alternatives explored in §6.4 are blocked by the non-idempotence of audio codecs, leaving no robust content attribution mechanism available at release.
The instruction fine-tuning pipeline relies heavily on synthetic TTS-generated speech for both conversation transcripts and user voice diversity. This introduces a distribution mismatch with real conversational speech that likely limits robustness to unusual acoustic conditions and speaking styles. The paper notes this but leaves more realistic instruct data collection as future work.
Quantization below 4-bit precision causes noticeable audio artifacts (repetitive generation, noisy voice) that current automatic metrics fail to detect, requiring entropy-spectrum analysis as a surrogate. This underscores a general gap in speech quality evaluation tooling for generative dialogue models.
Wiki Connections
Core architecture concepts: spoken-language-model, speech-to-speech, neural-codec, autoregressive-codec-tts, streaming-tts, self-supervised-speech
In-corpus papers cited by this work: 2305.09636, 2312.11805, 2403.03100, 1711.05101, 2402.05755, 2306.12925, 2002.05202, 2302.13971, 2307.09288, 1609.03499, 2301.02111, 2310.00704
Cited by: 2411.13577 (WavChat survey, Moshi as primary reference for full-duplex spoken dialogue) · 2504.18425 (Kimi-Audio, Moshi as architectural reference for SCA design)
2411.19842 — Scaling Transformers for Low-Bitrate High-Quality Speech Coding 2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction 2507.16632 — Step-Audio 2 Technical Report