arXiv · 2025 · Preprint
Zeghidour et al. (Kyutai) · → Paper · Demo: ✓ · Code: ✓
Delayed Streams Modeling (DSM) is a unified framework for streaming sequence-to-sequence learning that aligns multimodal token streams to a shared framerate and introduces configurable delays between them, enabling a single decoder-only model to perform both streaming ASR and streaming TTS at latencies as low as 150ms while remaining competitive with offline baselines.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
DSM addresses a fundamental tension in autoregressive speech models: prefix-based seq2seq formulations require the full input before producing any output, which prevents real-time streaming. The paper resolves this by aligning text and audio to the same framerate using the Mimi neural codec (12.5 Hz) and shifting one stream by a learnable delay, so a standard decoder-only Transformer can consume and generate both streams synchronously. For TTS specifically, DSM introduces an action output stream that controls when new words are fed to the model, a lookahead second text stream to help the model plan word boundaries, and speaker conditioning via cross-attention over reference audio embeddings — enabling long-form streaming synthesis of both monologues and multi-speaker dialogues. The delay parameter can be conditioned at inference time without retraining, giving practitioners explicit control over the latency/quality trade-off, which is directly applicable to spoken conversational agents and low-latency voice applications.
Wiki Connections
- streaming-tts — DSM provides a principled architectural solution for streaming TTS at sub-400ms latency by pre-aligning audio and text streams and introducing configurable inter-stream delays.
- autoregressive-codec-tts — DSM is an autoregressive codec-based TTS system built on Mimi RVQ tokens, extending the multistream decoder paradigm for streaming generation.
- neural-codec — DSM relies on the Mimi codec (12.5 Hz, 32 RVQ codebooks with cardinality 2048) as its audio representation backbone for both ASR and TTS.
- spoken-language-model — DSM’s multistream decoder architecture directly extends the design introduced in Moshi for real-time spoken dialogue, applying it to ASR and TTS as symmetric tasks.
- 2410.00037 (Moshi) — DSM explicitly extends the multistream decoder architecture of Moshi (Défossez et al., 2024), generalising it from spoken dialogue to ASR and TTS with delay conditioning.
- 2406.02430 (Seed-TTS) — cited as a representative autoregressive TTS baseline in the streaming seq2seq landscape that DSM addresses.
- 2410.06885 (F5-TTS) — compared directly in TTS experiments as a non-streaming flow-matching baseline with different latency and quality trade-offs.
- 2301.02111 (VALL-E) — cited as a foundational autoregressive codec TTS system whose offline prefix-based formulation DSM’s streaming framework supersedes.
- 2306.12925 (AudioPaLM) — cited as a multimodal decoder-only model performing bidirectional ASR/TTS, motivating DSM’s symmetric treatment of the two tasks under a streaming constraint.
- 2207.12598 (Classifier-Free Guidance) — DSM-TTS uses CFG over both speaker conditioning and text inputs, with a fine-tuning stage to distil CFG into a conditioning tensor for safe open-source release.
- 2505.17589 — cited in the experimental context as a related streaming speech system.
- 2312.05187 (SeamlessStreaming) — used as a streaming ASR baseline in Table 1, with DSM-ASR achieving substantially lower WER.
- 2106.06909 — cited as a streaming ASR system in the related work on seq2seq learning.
- 2503.01743 (Phi-4 Multimodal) — included as a non-streaming ASR baseline on the OpenASR Leaderboard for comparison with DSM-ASR.