arXiv · 2026 · Preprint
Tanyu Chen et al. (FlashLabs) · → Paper · Demo: ✗ · Code: ✓
An open-source, real-time end-to-end spoken dialogue model that combines sub-second streaming latency with personalized voice cloning, decoupling coarse acoustic-code generation from fine-grained residual refinement to keep both dimensions fast.
Problem
End-to-end spoken dialogue systems built on speech tokenizers and neural codecs let LLMs operate directly on discrete speech, avoiding the latency and paralinguistic-information loss of cascaded ASR-LLM-TTS pipelines. However, existing systems trade off between two goals: models optimized for real-time streaming dialogue (e.g., Moshi-style architectures) sacrifice fine-grained speaker control, while models capable of high-fidelity voice cloning (e.g., neural codec language models like VALL-E, or the CosyVoice series) lack real-time streaming with consistent voice identity preserved across multi-turn conversations. No prior open-source system had combined both low-latency streaming interaction and high-fidelity personalized voice cloning in one end-to-end model.
Method
Chroma 1.0 is composed of four modules. The Reasoner, built on the Thinker module from Qwen2.5-Omni and the Qwen2-Audio encoding pipeline, performs multimodal (text + audio) understanding and produces text embeddings and multimodal hidden states via cross-modal attention and Time-aligned Multimodal RoPE; it is frozen during Chroma’s own training and acts purely as a semantic/prosodic feature provider. The Backbone, a 1B-parameter LLaMA-architecture variant, autoregressively generates the coarse (first-layer) acoustic code at each frame, conditioned on the Reasoner’s text embeddings and hidden states plus a reference-audio/reference-transcript speaker prompt encoded via CSM-1B and prepended to the input sequence for voice cloning. To support streaming, text tokens and coarse audio codes are interleaved at a fixed 1:2 ratio, letting the Backbone generate audio in parallel with the Reasoner’s incremental text output rather than waiting for a complete text sequence.
The Decoder (~100M parameters, also a LLaMA variant) is a separate, lightweight module that generates the remaining N-1 = 7 residual-quantization codebook levels for each frame. Unlike the Backbone, it is frame-synchronous: it conditions only on the Backbone’s current-timestep hidden state and coarse code, not on the full text/audio history, which keeps its per-frame computation cheap regardless of conversation length. The Codec Decoder, following the Mimi vocoder’s causal-CNN design (from Moshi), reconstructs the full 8-codebook RVQ sequence into a continuous waveform with strict causal ordering to support streaming. Training is two-stage: Backbone and Decoder are jointly trained with a 0.5 loss-weight balance first, then the Backbone is frozen and the Decoder is fine-tuned alone (loss weight 1) to sharpen fine-grained acoustic detail. Training data comes from a synthetic pipeline: an LLM generates text responses to user queries, which are then synthesized into speech matching a reference speaker’s timbre via a TTS system, producing paired speech-dialogue training targets.
Key Results
On zero-shot voice cloning (CommonVoice, English, WavLM-based speaker-similarity embeddings per the SEED-TTS-EVAL protocol), Chroma achieves SIM = 0.81, exceeding the human baseline (0.73) by a relative 10.96% and outperforming all compared systems: F5-TTS (0.64), FireRedTTS-2 (0.66), Step-Audio-TTS (0.66), CosyVoice 3 (0.72), and even Seed-TTS (0.76), the only other system to exceed the human baseline. In head-to-head CMOS testing against ElevenLabs, ElevenLabs was preferred for naturalness (57.2% vs. 24.4%, 18.3% deuce) but the speaker-similarity preference gap was narrow (ElevenLabs 42.4% vs. Chroma 40.6%, 17.0% deuce). A follow-up experiment comparing ElevenLabs output against ground-truth reference recordings found listeners preferred the synthesized audio 92% of the time, indicating subjective naturalness preference does not track speaker-similarity fidelity.
Latency: end-to-end TTFT of 146.87ms and RTF of 0.43 (2.3x faster than real-time) on a 38.8-second generated response, with the frame-synchronous Decoder averaging 17.56ms per frame across its 7 refinement levels. On URO-Bench’s basic track (understanding, reasoning, oral conversation), Chroma (4B parameters, with voice cloning) scores second-best overall among compared systems behind GLM-4-Voice (9B, no voice cloning), and ahead of LLaMA-Omni (8B), Freeze-Omni (7B), and the 0.5B Mini-Omni/Mini-Omni2/SLAM-Omni models; Chroma is the only model in the comparison with personalized voice-cloning capability.
Novelty Assessment
The architectural contribution is a decoupling strategy: splitting acoustic generation into a long-context, cross-modally-conditioned Backbone (coarse code, autoregressive over the full conversation) and a short-context, frame-local Decoder (residual codes, conditioned only on the current frame), which the paper argues lets fine-grained refinement scale independently of conversation length. This is a genuine architectural design choice, though it builds heavily on existing components (Qwen2.5-Omni’s Thinker as Reasoner, Moshi’s Mimi vocoder as Codec Decoder, CSM-1B for speaker-prompt encoding) rather than introducing new generative mechanisms. The paper’s most notable empirical claim, that a real-time streaming system can match or exceed dedicated non-streaming zero-shot TTS systems on speaker similarity, is a meaningful empirical contribution, though the CMOS naturalness/speaker-similarity comparison against ElevenLabs is based on a small sample (30 comparative pairs, 12 sessions).
Field Significance
Field Significance
high — This paper demonstrates that real-time streaming and high-fidelity personalized voice cloning, previously treated as competing design goals in end-to-end spoken dialogue systems, can be combined in a single open-source, reproducible model, and provides a concrete architectural decoupling strategy (long-context Backbone vs. frame-local Decoder) other systems can adopt or compare against.
Write only what this paper itself demonstrates.
Claims
- supports: End-to-end spoken dialogue systems can preserve speaker identity in voice cloning to a degree competitive with, or exceeding, dedicated zero-shot TTS systems and human baseline recordings, without sacrificing real-time streaming.
Evidence: Chroma achieves SIM = 0.81 on CommonVoice, exceeding the human baseline (0.73) and all five compared TTS systems (F5-TTS, Seed-TTS, FireRedTTS-2, Step-Audio-TTS, CosyVoice 3), while maintaining RTF = 0.43 (§4.2, Table 1).
- complicates: Subjective naturalness preference in comparative listening tests does not track speaker-similarity preference, complicating the interpretation of CMOS-style speaker-similarity results as a pure measure of voice-cloning fidelity.
Evidence: Listeners preferred synthesized ElevenLabs audio over actual ground-truth reference recordings 92% of the time, while Chroma’s speaker-similarity preference (SCMOS 40.6%) was close to ElevenLabs’s (42.4%) despite a much larger naturalness gap (NCMOS 24.4% vs. 57.2%) (§4.3, Table 2, Table 3).
- supports: Decoupling coarse acoustic-code generation (long-context, cross-modally conditioned) from residual-codebook refinement (frame-local, short-context) into separate autoregressive modules reduces per-frame latency in streaming speech generation.
Evidence: The frame-synchronous Decoder, conditioned only on the current Backbone timestep rather than full conversation history, generates 7 residual RVQ levels in 17.56ms average latency per frame, contributing to an overall TTFT of 146.87ms and RTF of 0.43 (§3.3, §4.4, Table 4).
- complicates: Optimizing an end-to-end spoken dialogue model for personalized voice-cloning fidelity can come at a measurable cost to reasoning and language-understanding performance relative to similarly- or larger-scale dialogue models that do not support voice cloning.
Evidence: Chroma (4B, with voice cloning) trails GLM-4-Voice (9B, no voice cloning) on most reasoning and understanding sub-tasks in URO-Bench, e.g. GSM8K (22.74 vs. 30.93) and TruthfulQA (51.69 vs. 59.28) (§4.5, Table 5).
Limitations and Open Questions
Chroma's Reasoner supports multilingual (Chinese and English) input understanding, but the system generates speech output only in English; cross-lingual voice cloning (input and output languages differing while preserving speaker identity) is not implemented or evaluated, materially restricting the "personalized voice AI" claim to English-output use cases.
The system does not incorporate RLHF or DPO post-training, which the authors note could improve dialogue quality and instruction-following. The architecture does not support batch processing, so all latency figures are measured at concurrency 1; throughput under concurrent load is not reported. The CMOS comparison against ElevenLabs uses a small evaluation set (30 comparative samples across 12 sessions per dimension), limiting the statistical power of the naturalness and speaker-similarity preference findings. The paper’s ethical-considerations section acknowledges voice-cloning misuse risks (impersonation, fraud, non-consensual synthesis) and recommends safeguards such as consent requirements, detection mechanisms, and watermarking, but does not implement or evaluate any of them in the released system.
Wiki Connections
- Spoken Language Models — proposes a decoupled Backbone/Decoder architecture for a Qwen2.5-Omni-Reasoner-based end-to-end spoken dialogue model, evaluated against contemporary open-source SCA systems on reasoning, understanding, and oral-conversation tasks.
- Zero-Shot Text-to-Speech — demonstrates zero-shot speaker-similarity performance from a few seconds of reference audio that exceeds the human baseline and several dedicated zero-shot TTS systems, within a streaming dialogue system rather than an offline TTS model.
- Streaming Text-to-Speech — introduces a 1:2 interleaved text-audio token schedule and a frame-synchronous residual-codebook decoder specifically to achieve sub-second end-to-end latency (TTFT 146.87ms, RTF 0.43) in a streaming dialogue setting.
- Speaker Adaptation — conditions generation on a CSM-1B-encoded reference-audio/reference-transcript speaker prompt prepended to the Backbone’s input sequence, maintaining consistent speaker identity across multi-turn conversations.
- Qwen2.5-Omni — supplies the Thinker module and cross-modal attention/TM-RoPE design that Chroma’s Reasoner is directly built on.
- Moshi — supplies the Mimi vocoder design that Chroma’s Codec Decoder follows, and is discussed as a real-time streaming baseline that sacrifices fine-grained speaker control.
- GLM-4-Voice — the strongest compared baseline on URO-Bench reasoning and understanding tasks, at more than double Chroma’s parameter count and without voice-cloning capability.
- Freeze-Omni — a compared end-to-end spoken dialogue baseline on URO-Bench, outperformed by Chroma on most dimensions despite Chroma’s added voice-cloning capability.
- SLAM-Omni — a compared timbre-controllable voice-interaction baseline on URO-Bench, substantially outperformed by Chroma across all evaluated dimensions.
- F5-TTS — a zero-shot TTS baseline in the Table 1 speaker-similarity comparison, outperformed by Chroma despite operating without Chroma’s real-time streaming constraint.