arXiv · 2026 · Preprint

Feng Yin et al. (VUI Labs) · → Paper · Demo: ? · Code: ?

Masked discrete diffusion over an RVQ codec token grid, adapted from a 0.6B text LLM and pretrained on 1M hours of four-language speech, yields one TTS lineage with two operating points: a fully parallel offline model and a block-causal streaming model with 41.6 ms local first-block latency.

Problem

Most current large-scale TTS systems are autoregressive codec language models. The authors list three structural costs of left-to-right decoding. Latency grows with utterance length and token rate. Sampling errors are frozen into the prefix and show up as skipped or repeated words. A residual vector quantization (RVQ) token grid has no natural order along its codebook axis, so AR systems need delay patterns, depth Transformers or multi-token heads to impose one.

Masked generative models such as SoundStorm and MaskGCT already decode the RVQ grid in parallel, but as task-specific models at moderate scale. Recent diffusion-LLM TTS work (LLaDA-TTS, DiffuSpeech, OmniVoice) runs at smaller data scales or targets language breadth. Block-diffusion streaming TTS so far (Chatterbox-Flash) converts a pretrained AR checkpoint rather than building on diffusion pretraining. The report targets three gaps: streaming block diffusion backed by large-scale diffusion pretraining, a controlled comparison of fully parallel and block-autoregressive decoding under matched tokenizer, data and capacity, and production-level quality on major languages.

Method

Tokenizer. Luna-Codec is a causal RVQ codec trained in-house on 400K hours of zh/en/ja/ko speech. It encodes 24 kHz audio at 25 Hz with Q = 8 codebooks of 2048 entries (2.2 kbps). The first codebook is distilled toward frame-level WavLM features, so the depth axis runs roughly from linguistic to acoustic content. Training uses reconstruction, commitment and adversarial (multi-period and multi-resolution STFT discriminator) losses with quantizer dropout. The report gives no reconstruction numbers for the codec.

Backbone and objective. A single 0.6B Transformer, initialized from Qwen3-0.6B, models the token grid with absorbing-state masked diffusion (authors’ term: diffusion language model). Text (Qwen3 BPE), an optional clean acoustic prompt grid and the partially masked target grid form one sequence. The 8 codebook embeddings of each frame are summed into one position, so the sequence runs at 25 Hz rather than 200 tokens/s, and 8 parallel heads predict every masked position. Training masks each grid position independently with a per-utterance noise level, with no constraint along time or depth. Emotion is set by an utterance-level control token and non-verbal vocalizations (NVVs, e.g. laughs, sighs, coughs) by inline text tokens; no separate style encoder is used.

Figure 1: Overview of Luna-TTS Family. (a) Luna-Codec, a causal RVQ codec (25 Hz, Q = 8 codebooks) whose first codebook is semantically anchored by WavLM distillation. (b) The shared diffusion LM backbone: text, prompt token grid, and partially masked target token grid form one sequence; per-frame codebook embeddings are summed at the input, and Q parallel heads predict all masked positions in parallel. (c) The two decoding regimes: Luna-TTS generates the full token grid in S parallel refinement steps with an external duration predictor; Luna-TTS Realtime decodes block by block with KV caching and per-block denoising, streaming audio as each block commits. Bottom: the progressive adaptation lineage from a pretrained Qwen3 LLM through Luna-TTS to Luna-TTS Realtime.

Progressive adaptation. The causal mask of Qwen3-0.6B is replaced by full bidirectional attention, and training continues with the masked-diffusion loss. The schedule has four stages: foundation pretraining on the ~1M-hour mixture (~100B packed speech-text tokens, ~1.1 weighted epochs), annealing on a ~100K-hour high-quality subset, expressive continual pretraining on emotion- and NVV-annotated data mixed with neutral speech, and block adaptation (~20K steps). Expressive labels come from Gemini 3.1 Pro Preview and other audio-language models, cross-checked, with disagreements routed to human review.

Luna-TTS (offline). This variant generates the whole grid in S = 32 refinement steps. It uses MaskGIT-style confidence-based unmasking with a cosine schedule, annealed Gumbel noise and classifier-free guidance on text. Because the grid length T must be known in advance, a second Qwen3-0.6B model, fine-tuned with bidirectional attention, predicts a quantized duration for each text token; the per-token durations are summed to give T. Zero-shot cloning and speech editing are both infilling: the prompt is an unmasked region of the grid, and editing re-masks a span of an existing utterance.

Luna-TTS Realtime (streaming). This variant is initialized from Luna-TTS and adapted with the block-diffusion objective of Arriola et al. It uses a block-causal mask and independent per-block noise levels. Generation is autoregressive over 32-frame (1.28 s) blocks with KV caching, while all frames and codebooks inside a block are denoised in parallel. Each committed block is immutable and goes straight to the causal codec decoder. A learned end-of-speech decision replaces the duration predictor.

RL post-training. GRPO is applied to the masked-diffusion model. Importance ratios are computed per token decision actually committed along the realized denoising trajectory, under the same sampling transformations (top-k, temperature, CFG) used at rollout. The unmasking positions are treated as given and are not optimized. A PPO-style clipped surrogate is applied per token and normalized per utterance. The reward is ordinal within each prompt group: frozen ASR WER takes priority and speaker similarity breaks ties. All reported Luna-TTS Family zero-shot rows are post-RL.

Key Results

Zero-shot cloning, Seed-TTS-Eval (Table 8). Luna-TTS scores 0.73 CER / 79.7 SIM on test-zh and 1.49 WER / 76.8 SIM on test-en. Each value is the best in its column across all 13 rows. The next-best entries are 0.83 CER (MiniMax-Speech), 79.6 zh SIM (Seed-TTS), 1.54 WER (Qwen-Audio-3.0-TTS) and 76.2 en SIM (Seed-TTS and Qwen-Audio-3.0-TTS). The zh SIM lead over Seed-TTS is 0.1 points. Only Qwen3-TTS-12Hz-1.7B-Base was re-run by the authors; all other baseline rows are quoted from their reports and may differ in evaluation details.

In-the-wild cloning, CV3-Eval (Table 9). Luna-TTS has the lowest error rate in four of seven columns: zh 3.17 (Qwen3-TTS 3.19), en 3.18 (Qwen3-TTS 3.92), hard-zh 6.90 and hard-en 6.18 (Qwen-Audio-3.0-TTS 7.44 and 6.71). Qwen-Audio-3.0-TTS keeps the best ja (4.78 vs. 5.00), ko (4.30 vs. 5.93) and four-language average (4.17 vs. 4.32). Korean, with 6.9% of the training data, is Luna-TTS’s weakest subset. Here Qwen3-TTS and OmniVoice were re-run under the same protocol; the other rows are quoted.

Streaming cost. Relative to Luna-TTS, Luna-TTS Realtime adds +0.35 CER on test-zh (1.08 vs. 0.73) and +0.32 WER on test-en (1.81 vs. 1.49). It loses 2.8 and 3.4 SIM points, and its CV3 average rises from 4.32 to 4.95. The gap widens on the hard CV3 subsets: 12.56 vs. 6.90 (hard-zh) and 13.98 vs. 6.18 (hard-en). The authors attribute this to irrevocable block commits and errors in the learned end-of-speech decision. They offer this as a plausible explanation and do not test it with an ablation.

Serving speed (Tables 5-7). On H20 GPUs at batch 1, median of 12 warmed runs with network excluded:

  • Luna-TTS at 16 steps returns the full waveform in 216.0 ms (end-to-end RTF 0.0211).
  • Luna-TTS Realtime at 8 steps with CFG split across 2 GPUs commits its first 1.28 s block in 41.6 ms and finishes a 10.6 s utterance in 254.0 ms (RTF 0.0240).
  • Parallel CFG cuts full-response latency by 46.1% at 16 steps and 44.5% at 8 steps.

In Table 6, 0.0211 is the lowest RTF and 0.0240 the second lowest. In Table 7, 41.6 ms is the lowest first-output latency. Both tables compare against figures published by other systems on different hardware and under different protocols.

Expressive control vs. three commercial systems (Tables 10-13). The comparison is against MiniMax Speech 2.8 HD, ElevenLabs Eleven v3 and Fish Audio S2.1 Pro.

  • NVV controllability (Table 10): Luna-TTS has the lowest PCER (39.95%) and the highest recall (66.76%) and F1 (72.52%). ElevenLabs has the highest precision (87.45% vs. 79.38%).
  • Gemini-rated NVV evaluation (Table 11): Luna-TTS leads overall naturalness (3.65), quality (4.18) and expression (3.75). It trails on NVV Accuracy (3.60 vs. MiniMax 3.95) and NVV Perceptual Effect (3.63 vs. ElevenLabs 3.75).
  • Emotion (Tables 12-13): Luna-TTS has the highest overall E-Sim (0.558 vs. 0.552), Gemini-rated Expression Quality (4.54) and human E-MOS (3.90 vs. 3.82). It is third on Gemini Emotion Match (2.63; MiniMax 2.70, ElevenLabs 2.67) and second on human N-MOS (4.14 vs. ElevenLabs 4.18).

The number of human listeners is not reported, and the emotion test sets differ by system because each system is scored on its own supported emotions.

Dedicated voices (Tables 14-15). Luna-TTS was fine-tuned on three internal voices and compared in an internal English pairwise arena against three preset voices per commercial system. It has the highest median Bradley-Terry Elo (1548.47), with a 57.36% overall win rate. Gemini 3.1 Flash TTS is at 1546.13 with an overlapping 90% interval. The authors state that the difference is not significant, and the head-to-head result against Gemini is 117-113.

Novelty Assessment

The base Luna-TTS recipe draws on existing work. Masked diffusion over the full RVQ grid with unrestricted random masking follows OmniVoice. Converting an AR text LLM into a masked-diffusion speech model follows LLaDA-TTS. MaskGIT-style decoding and summed per-frame codebook embeddings are standard. For Luna-TTS itself, the new elements are scale (1M hours) and the token-level LLM duration predictor for four writing systems.

Two contributions are more clearly new:

  • Streaming model. Luna-TTS Realtime is a block-diffusion streaming TTS obtained by continual training from a diffusion-pretrained parent rather than from an AR checkpoint. Paired with Luna-TTS, it gives a matched-tokenizer, matched-data, matched-backbone comparison of fully parallel and block-causal decoding.
  • RL objective. The GRPO formulation for masked speech-token diffusion computes ratios over the token decisions committed along the realized denoising trajectory, under the same effective sampling policy. This differs from the one-step mean-field surrogate used in prior dLLM RL.

The report includes no ablation that isolates either contribution. There is no pre-RL vs. post-RL table, no codec reconstruction evaluation, and no duration-predictor comparison. The parallel-versus-block comparison varies only the decoding regime, which is its point. Some headline baseline numbers are quoted rather than re-run.

Field Significance

Tip

High — The report shows that a 0.6B masked-diffusion TTS adapted from a text LLM can match or exceed reported AR codec-LM and flow-matching results on Seed-TTS-Eval and on the Mandarin and English CV3-Eval subsets. One lineage serves both offline and streaming operating points. It also provides a matched comparison of fully parallel and block-causal diffusion decoding, quantifying how much streaming costs (small on regular text, large on hard in-the-wild text). It shows that GRPO can be defined over masked-diffusion denoising trajectories.

Claims

  • supports: Masked discrete diffusion over a multi-codebook acoustic token grid, initialized from a pretrained text LLM and scaled to large multilingual data, can reach content accuracy and speaker similarity at or above those reported by autoregressive codec-LM and flow-matching zero-shot TTS systems.

    Evidence: The 0.6B Luna-TTS, trained on ~1M hours of four-language speech, records 0.73 CER / 79.7 SIM (test-zh) and 1.49 WER / 76.8 SIM (test-en) on Seed-TTS-Eval. Each is the best value in its column against 11 systems including Seed-TTS, CosyVoice 3, MiniMax-Speech, Qwen3-TTS and F5-TTS. Most baseline rows are quoted from their own reports. (§6.1, Table 8)

  • complicates: Block-causal streaming decoding preserves most zero-shot quality on regular text, but the gap to fully parallel decoding grows sharply on long, irregular text with degraded prompts. Irrevocable block commits and learned stopping replace global refinement and an explicit length.

    Evidence: With tokenizer, data and backbone matched, Luna-TTS Realtime trails Luna-TTS by about 0.3 CER/WER on Seed-TTS-Eval and by 0.63 on the CV3-Eval four-language average (4.95 vs. 4.32). On CV3 hard-zh and hard-en it roughly doubles the error rate (12.56 vs. 6.90; 13.98 vs. 6.18). (§6.1, Tables 8-9)

  • supports: Group-relative policy optimization can be applied to masked-diffusion speech generation by computing importance ratios over the token decisions committed along the realized denoising trajectory, rather than over a left-to-right factorization.

    Evidence: Luna-TTS defines per-token ratios under the same top-k/temperature/CFG policy used at rollout, uses a clipped surrogate normalized per utterance, and uses an ordinal reward that ranks ASR WER first and speaker similarity second. Reported zero-shot results are post-RL, but no pre-RL comparison is given. (§4.1, §4.2, §6.1)

  • supports: Parallel refinement within fixed-size blocks lets a diffusion TTS deliver streaming audio with first-block latency well below the block duration on a single-request serving path.

    Evidence: Luna-TTS Realtime at 8 steps with CFG split across two H20 GPUs commits its first 1.28 s block in 41.6 ms and finishes a 10.6 s utterance in 254.0 ms (RTF 0.0240). With sequential CFG on one GPU, first-block latency is 59.6 ms. (§5.2, Table 5)

  • complicates: Low-resource languages within a multilingual TTS training mixture can remain the weakest even at million-hour total scale.

    Evidence: Korean (6.9% of training data) is Luna-TTS’s highest-error CV3-Eval language subset at 5.93 CER. That is above Qwen-Audio-3.0-TTS (4.30) and Qwen3-TTS (4.63), while Luna-TTS has the lowest zh and en error rates in the comparison. The authors attribute this partly to the small Korean share. (§6.1, Table 9; §7)

Limitations and Open Questions

  • Missing ablations. The report has no ablation of the RL stage, the WavLM-distilled first codebook, the AR-LLM initialization or the duration predictor. The size of each component’s contribution is therefore unknown.
  • Codec quality is not measured. Luna-Codec is described as “in line with” recent LLM-TTS tokenizers, but no reconstruction metrics are reported.
  • Mixed baseline provenance. Many Seed-TTS-Eval and CV3-Eval baseline rows are quoted from other reports, and the latency tables mix hardware and measurement boundaries. The “lowest RTF” and “lowest first-output latency” rankings are therefore indicative, not controlled.
  • Expressive evaluation. Much of the expressive comparison relies on a single LLM judge (Gemini 3.1 Pro Preview). The human listening test does not report rater count. Emotion and NVV scores are averaged over each system’s own supported categories, so the systems are not scored on identical inventories.
  • Dedicated-voice arena. The arena is internal and English-only, and its Luna-TTS voices were fine-tuned on internal recordings. It does not separate Luna-TTS from Gemini 3.1 Flash TTS statistically.
  • Design limits named by the authors. Coverage is four languages. Korean is weakest. Luna-TTS depends on an external duration predictor. Luna-TTS Realtime uses a fixed 1.28 s block and degrades most on hard in-the-wild text.
  • Unclear release. The report does not say whether code or weights are released.

Wiki Connections

  • Diffusion TTS — Provides a large-scale masked discrete diffusion TTS over RVQ codec tokens, in both a fully parallel and a block-diffusion form.
  • Zero-Shot TTS — Performs prompt-based cloning of unseen speakers as grid infilling, evaluated on Seed-TTS-Eval and CV3-Eval.
  • Streaming TTS — Luna-TTS Realtime streams 1.28 s blocks with KV caching, with measured first-block latency and RTF on H20 GPUs.
  • Multilingual TTS — Trained on Mandarin, English, Japanese and Korean and evaluated on all four CV3-Eval language subsets, with Korean weakest.
  • Emotion Synthesis — Adds utterance-level emotion tokens and inline non-verbal vocalization tokens, evaluated with E-Sim, an LLM judge and human E-MOS against commercial systems.
  • RLHF Speech — Applies GRPO with trajectory-level importance ratios to a masked-diffusion TTS model, using an ordinal WER-then-SIM reward.
  • OmniVoice — Supplies the unrestricted random-masking recipe that Luna-TTS follows, and serves as a re-evaluated baseline on CV3-Eval and an RTF comparison point.
  • LLaDA-TTS — Prior conversion of a compact AR text LLM into a masked-diffusion speech model, which Luna-TTS scales to 1M hours and four languages.
  • MaskGCT — Earlier masked generative codec-token TTS that the report positions as task-specific, and a quoted Seed-TTS-Eval baseline.
  • Qwen3-TTS — Main same-protocol AR baseline (12Hz-1.7B-Base, re-evaluated on Seed-TTS-Eval and CV3-Eval), and the multi-token-prediction design Luna-TTS argues against.
  • Qwen-Audio-3.0-TTS — Quoted baseline that keeps the best Japanese, Korean and four-language-average CV3-Eval results over Luna-TTS.
  • Seed-TTS — Defines the Seed-TTS-Eval benchmark and is the closest quoted competitor on speaker similarity.
  • DiSTAR — Prior block-level AR-plus-masked-diffusion speech generator, which Luna-TTS Realtime contrasts with by building on diffusion pretraining.
  • NV-Bench — Supplies the Multilingual-NVASR protocol used for the objective NVV controllability evaluation.