arXiv · 2026 · Preprint
Jianbo Ma et al. (Dolby Laboratories / Canva Research) · → Paper · Demo: ? · Code: ?
Chain-of-Details (CoD) moves coarse-to-fine codec-token TTS from the quantizer axis to the time axis: a single shared masked-token transformer generates speech tokens at progressively higher frame rates, each level conditioned on the one below.
Problem
Multi-stage token-based TTS systems (SPEAR-TTS, VALL-E, MaskGCT, NaturalSpeech 3) generate speech coarse-to-fine, but “coarse” in these systems means semantic tokens or the first residual quantizer layer, always at the full token rate. None of them model temporal granularity explicitly: the first stage must commit to frame-level timing at the final resolution. Single-stage masked audio token modeling (KD-NARSIS, via knowledge distillation) avoids the cascade but trails two-stage models in intelligibility. The paper asks whether a temporal hierarchy, borrowed from next-scale prediction in image generation (VAR), gives a cheaper and more accurate generation path.
Method
The system follows the masked audio token modeling (MATM) recipe of MaskGIT: speech is tokenized by a pretrained DAC codec (44.1 kHz, 9 RVQ codebooks, 86.13 Hz token rate, 8 kbps), and a bidirectional Llama-style transformer is trained to predict randomly masked tokens with a cosine masking schedule and cross-entropy loss. The authors call this non-autoregressive masked generative modeling; within each level all tokens are decoded in parallel over iterative confidence-based unmasking steps.
CoD splits generation into L temporal levels with increasing token rates (f1 < f2 < … < fL). In the main configuration (L = 3), the coarser levels are obtained by decimating the first-quantizer token sequence by factors of 4 (21.53 Hz) and 2 (43.07 Hz), while the finest level holds the full 9-codebook RVQ stack at 86.13 Hz. Level 1 is predicted from text and speaker conditions alone; each higher level is predicted conditioned on the completed output of the previous level. One transformer and one codebook are shared across all levels, so the level index is sampled at training time (probabilities 0.2, 0.3, 0.5 from coarsest to finest) and a single model learns every stage. To make higher levels robust to errors in their coarse conditioning, 10% of the previous-level tokens are replaced with random vocabulary indices during training. Classifier-free guidance uses 10% condition dropout, with the transcript and previous-level tokens replaced by learned embeddings.
Conditioning comes from three frozen pretrained components: a SoundChoice G2P model (SpeechBrain) for phoneme indices, a lightweight 6-layer transformer duration predictor that estimates only total utterance length (not per-phoneme durations), and a WeSpeaker speaker encoder whose embedding enters the decoder through adaptive LayerNorm (DiT-style). Phonemes are concatenated with a fully masked token sequence at level 1, and the authors observe that this coarsest level performs phonetic planning implicitly, so no explicit phoneme-level duration predictor is needed.

Inference runs 20 unmasking steps per level, with CFG scale decaying linearly from 3 to 0.75 and Gaussian logit noise (variance decaying from 3.0 to 0) for diversity. Two variants are trained for 400K steps at batch size 256: CoD-Base (12 layers, hidden size 1024, 263M parameters, LibriTTS-clean, 245 h) and CoD-Large (24 layers, 503M parameters, LibriTTS-clean plus an MLS English subset filtered with Brouhaha to high SNR and C50, about 3,297 h). The paper also tests an alternative tokenizer that adds L-1 extra quantizers to DAC, each encoding a decimated version of the finer level’s representation, with shared or independent codebooks.
Key Results
All reported results are WER; the ASR model used for transcription is not stated.
- LibriSpeech test-clean, 4-10 s (Table I): CoD-Base reaches 3.09% WER with 263M parameters on 245 h of LibriTTS-clean, versus 3.6% for an in-house NAR 2-stage model (476M, same data), 5.9% for KD-NARSIS (249M, same data), 4.0% for StyleTTS 2, 5.9% for VALL-E (370M, 60k h) and 7.1% for YourTTS. CoD-Large reaches 2.81%, against 2.4% for DAC resynthesis and 2.2% for ground truth.
- Seed-TTS test set (Table III): CoD-Base scores 2.89% and CoD-Large 2.73% WER, slightly behind MaskGCT at 2.62% (1B parameters, 100k h of Emilia). DAC resynthesis scores 2.60% here, so CoD-Large sits close to its codec ceiling.
- Number of temporal levels (Table IV, LibriTTS test-clean): WER (4-10 s / all lengths) falls from 4.64 / 7.67 with one level to 4.00 / 5.19 with two and 3.78 / 4.88 with three. The largest gain is from one to two levels, and the gap is wider on the unrestricted-length set. Four levels were not tried because the coarsest sequence would become shorter than the phoneme sequence for most utterances.
- Coarse token type (Table V): decimated first-layer RVQ tokens (3.78 / 4.88) beat HuBERT tokens as a coarse level (4.62 / 5.61, two levels) and clearly beat the purpose-built hierarchical tokenizer, whether with independent (5.81 / 7.03) or shared codebooks (7.99 / 8.48).
Comparison fairness is mixed. The NAR 2-stage and KD-NARSIS comparisons share training data and appear to share the MATM framework, which makes them the most informative rows; the other Table I baselines differ in data, scale and evaluation provenance, and the paper does not state whether baseline numbers were re-measured or copied from prior work. The paper states that MOS was collected, but no MOS or speaker similarity results appear anywhere in the paper.
Novelty Assessment
The architectural idea is genuine but modest in scope. Coarse-to-fine cascades and MaskGIT-style decoding for speech tokens are established; the new element is making the hierarchy temporal (decimated token rates) rather than semantic or quantizer-depth, with one shared network serving every level. This is a direct transfer of VAR’s next-scale prediction from image resolution to speech frame rate, adapted to use masked iterative decoding at each scale. The observation that the coarsest level removes the need for a phoneme-level duration predictor is interesting but reported qualitatively, without alignment analysis. Evidence is thin: intelligibility only, two English read-speech test sets, and a small ablation on a third. The hierarchical tokenizer, the more original component, underperforms simple decimation and is left as future work.
Field Significance
Moderate. The paper provides a controlled data point that temporal coarse-to-fine decomposition can replace semantic-then-acoustic staging in masked codec-token TTS, matching a two-stage baseline’s intelligibility with roughly half the parameters on the same 245 h of data. Without naturalness, speaker similarity or speed measurements it remains a WER-only proof of concept rather than evidence of an overall quality advantage.
Claims
- supports: Generating codec tokens coarse-to-fine along the time axis, with each finer frame rate conditioned on the coarser one, improves intelligibility over single-resolution masked token generation.
Evidence: With the same base model, WER on LibriTTS test-clean drops from 4.64% (one level) to 4.00% (two levels) and 3.78% (three levels) on 4-10 s utterances, and from 7.67% to 4.88% on all lengths. (§IV-G, Table IV)
- supports: A single network with shared parameters can serve every stage of a multi-stage coarse-to-fine speech token generator, reducing model size relative to cascades of separate stage models.
Evidence: CoD-Base (263M, one shared decoder) obtains 3.09% WER on LibriSpeech test-clean versus 3.6% for a 476M NAR two-stage model trained on the same LibriTTS-clean data. (§IV-E, Table I)
- refines: For coarse temporal levels, decimating the first residual-quantizer layer of an existing acoustic codec is a better intermediate representation than tokens from a separately learned temporal-hierarchy quantizer.
Evidence: Decimated first-layer DAC tokens give 3.78% / 4.88% WER, HuBERT coarse tokens 4.62% / 5.61%, and added hierarchical quantizers 5.81% / 7.03% (independent codebooks) or 7.99% / 8.48% (shared codebook); the authors attribute the gap to the learned coarse tokens lacking a direct mapping to the final acoustic tokens, or to non-convergence. (§IV-A, §IV-G, Table V)
- complicates: Parameter-efficient token TTS trained on a few hundred to a few thousand hours remains slightly behind much larger, large-data multi-stage systems on out-of-domain intelligibility benchmarks.
Evidence: On the Seed-TTS test set, CoD-Base (263M, 245 h) scores 2.89% and CoD-Large (503M, 3,297 h) 2.73% WER, against 2.62% for MaskGCT (1B, 100k h). (§IV-F, Table III)
Limitations and Open Questions
Warning
The evaluation is WER-only. The paper says MOS was collected but reports no MOS, and it reports no speaker similarity, prosody or inference-speed results, so its claims of more natural speech and efficient synthesis are not backed by measurements.
- The ASR system used for WER is not specified, and it is unclear whether Table I baseline numbers were re-measured under the same protocol or taken from prior papers.
- The claim that level 1 performs phonetic planning without a phoneme duration predictor is not supported by any alignment or duration analysis. The system still depends on a separate total-length predictor and a pretrained G2P model.
- The number of levels is capped at three, with no explanation of how the scheme would extend beyond the phoneme-rate floor of roughly 21 Hz. The authors leave the optimal number of levels to future work.
- The base and large models differ in more than size: Table II marks the phoneme tokenizer as used only for Base. This confound is not discussed. Table II also lists Large at 502M parameters while Tables I and III list 503M.
- Experiments are English read speech only (LibriTTS, MLS, LibriSpeech). The speaker encoder is fed reference audio, but cloning of unseen speakers is never evaluated.
- No code, checkpoints or demo page are mentioned.
Wiki Connections
- Autoregressive Codec TTS — A masked (MaskGIT-style) codec-token generator in the SoundStorm and MaskGCT line, which this concept tracks as the parallel alternative to causal codec LMs; CoD’s contribution is a temporal coarse-to-fine schedule for that generator.
- Neural Audio Codec — Generates DAC RVQ tokens, builds coarse temporal levels by decimating the first quantizer, and tests a modified DAC with extra decimated quantizers as a temporal-hierarchy tokenizer.
- MaskGCT — The main large-scale masked generative baseline on the Seed-TTS test set; CoD replaces MaskGCT’s semantic-then-acoustic three-model cascade with temporal levels in one shared model.
- VALL-E — A Table I baseline and an example of quantizer-depth coarse-to-fine generation (AR first layer, NAR remaining layers), which CoD contrasts with its temporal hierarchy.
- NaturalSpeech 3 — Cited as a multi-stage system that cascades models over factorized attribute tokens rather than over temporal resolution.
- AudioLM — The origin of the semantic-to-acoustic coarse-to-fine progression that CoD extends to the time axis.
- Seed-TTS — Provides one of the two main evaluation sets.
- LibriTTS — The training corpus for CoD-Base and the ablation test set.
- Parler-TTS — The Llama-style decoder backbone and the Brouhaha-based data filtering follow this work.