arXiv · 2026 · Preprint

Vadim Popov et al. (Huawei Noah’s Ark Lab / National Research University Higher School of Economics / National University of Science and Technology MISIS) · → Paper · Demo: ? · Code: ?

Argues, through a path-measure KL result, a local-optimality theorem and numerical checks, that the fixed hypercube-grid geometry of FSQ codebooks suits continuous diffusion over discrete tokens, then replaces the LLM text-to-token stage of CosyVoice2 with a roughly 45M-parameter continuous diffusion model over the same FSQ speech tokens.

Problem

Continuous Diffusion for Categorical Data (CDCD) is a hybrid between continuous and discrete diffusion: forward and reverse processes run in a continuous space of token embeddings, while the network predicts token probabilities and is trained with cross-entropy. It has received less attention than absorbing (masked) discrete diffusion, partly because the embedding space is usually learned jointly with the diffusion model, which the paper notes can be unstable (for example, embedding collapse). No prior work had asked which fixed embedding geometry is best for CDCD in its original formulation. On the speech side, systems such as CosyVoice 2 generate FSQ speech tokens autoregressively with a 0.5B LLM, while diffusion and flow-matching TTS systems (F5-TTS, E2 TTS) mostly generate mel-spectrograms; the few diffusion models over speech tokens (NaturalSpeech 3) use discrete diffusion. CDCD had not been applied to TTS.

Method

Theory (§4). Lemma 4.1 derives the reverse-time SDE of a diffusion bridge conditioned on a start point and endpoint . Statement 4.2 then shows, via Girsanov’s theorem, that the KL divergence between the path measures of two reverse bridges that start at the same prior sample and end at data points and equals . So, for a fixed noise schedule, how distinguishable two tokens’ generative trajectories are scales with the squared distance between their embeddings. Neither result needs the data to be discrete. The authors’ intuition, which they state as a suggestion rather than prove, is that a smaller KL makes it harder to train a model that tells the trajectories apart. This motivates the average nearest-neighbour distance of a codebook as a worst-case quality criterion. Theorem 4.3 proves that base-2 () and base-3 () FSQ codebooks are locally optimal for among codebooks of the same size with entries bounded by 1 in norm: any small perturbation of a codebook vector lowers . The paper also states a “Best Accuracy Hypothesis”: for an optimally trained CDCD with equal token priors, FSQ codebooks maximise top-1 token prediction accuracy at every diffusion time. This is proven only for 1-dimensional latents (Appendix D) and supported numerically for higher dimensions. The authors note that top-1 accuracy is not the same as generation quality, since sampling uses the full probability-weighted embedding average.

CDCD-TTS (§5.2, Appendix E). The starting point is CosyVoice 2, which has three stages: an LLM text-to-token stage, a flow-matching token-to-mel stage, and a HiFi-GAN vocoder, with base-3, 8-dimensional FSQ speech tokens ( tokens at 25 Hz) from an ASR-trained tokenizer. CDCD-TTS replaces only the LLM stage. The tokenizer, the flow-matching stage (which receives the CAM++ speaker embedding and the reference mel) and the vocoder are kept unchanged. The CDCD backbone uses the F5-TTS architecture: character-level text through 4 ConvNeXt v2 blocks, concatenated with clean reference-token latents and noisy target latents, then 8 DiT blocks (adaLN-zero, 8 heads, inner dimension 512, RoPE), and finally a softmax over the 6561 token classes. It has about 45M parameters and a 1024-token context (about 41 s). Zero-shot cloning works the way F5-TTS does it: reference tokens and reference text are prepended as an infill-style prompt. Because generation is non-autoregressive, a non-neural statistical duration predictor sets the output length. It learns per-character average token counts and rescales them by the reference utterance’s speaking rate. Sampling solves the probability-flow ODE with DDIM in 5 to 25 steps, using text classifier-free guidance at weight 0.5 and a prior of instead of ; the narrower prior was chosen on a development set.

Codebook variants. Four CDCD models are trained on 65k hours of English (LibriLight, GigaSpeech, English Emilia):

  • FSQ-original: the true FSQ embeddings.
  • FSQ-perturb: each 0 coordinate is shifted to , which keeps the same convex hull. It stands in for learned single-codebook VQ.
  • FSQ-permute: the FSQ embeddings with token IDs randomly reassigned.
  • RVQ: predicts the first EnCodec RVQ layer (128-dimensional embeddings). The remaining 7 layers come from the VALL-E X non-autoregressive module, and batch size and learning rate are halved.

The three FSQ variants share architecture and training (1.5M steps, 8 V100 GPUs).

Key Results

Numerical checks (§5.1). Among 10,000 random bounded codebooks, the best square-root average nearest-neighbour distance stays below the FSQ value. For base 2 the FSQ value is 2, against random-codebook maxima of 1.993, 1.984, 1.941 and 1.818 for 2, 3, 4 and 8 dimensions. For base 3 the FSQ value is 1, against 0.969, 0.913, 0.871 and 0.825. Monte-Carlo accuracy estimates over 1000 random codebooks per setting also stay below FSQ. For base 2 the FSQ accuracies are 38.179%, 23.580% and 14.573% in 2, 3 and 4 dimensions, against best random values of 37.389%, 22.816% and 13.789%. For base 3 they are 17.052%, 7.044% and 2.889%, against 16.659%, 6.739% and 2.731%. On a toy unconditional task (2-dimensional latents, 4-token sequences, 5 seeds), the average log-KL to the ground-truth distribution is better with FSQ embeddings than with slightly perturbed ones: vs. for base 2 and vs. for base 3 (lower is better). These experiments are random-search and toy-scale evidence, not proofs of global optimality.

TTS on Seed-TTS test-en (Table 1). Each sentence is synthesised 4 times and medians are reported. WER and speaker similarity (WavLM-based SIM) come from the official Seed-TTS tool, the paper’s “MOS” column is UTMOS (an automatic predictor, not human ratings), and EMO is the emotion2vec label agreement between reference and output.

  • FSQ-original vs. FSQ-perturb: FSQ-original is ahead on all four metrics at every step count (5, 8, 12, 25). The WER gap shrinks as steps increase: 2.39% vs. 3.10% at 5 steps, 2.10% vs. 2.32% at 8, 2.05% vs. 2.14% at 12, and 2.00% vs. 2.03% at 25. The UTMOS gap shrinks in the same way, from 4.093 vs. 3.834 at 5 steps to 4.119 vs. 4.118 at 25. SIM differences are 0.005 to 0.007 throughout.
  • Badly matched token geometry: at 25 steps, FSQ-permute reaches 15.4% WER, 0.588 SIM and 3.631 UTMOS, and RVQ reaches 21.3% WER, 0.382 SIM and 2.932 UTMOS. The paper takes this as evidence that the FSQ grid alone is not enough: token IDs must be mapped to embeddings by autoencoder training. The RVQ result is confounded by the 128- vs. 8-dimensional embedding size, the halved batch and learning rate, and the borrowed NAR stage for the remaining layers, which the paper partly acknowledges.
  • vs. CosyVoice 2 (same tokenizer, flow-matching stage and vocoder): FSQ-original at 25 steps has lower WER (2.00% vs. 2.57%) and slightly higher UTMOS (4.119 vs. 4.077) and EMO (72.7% vs. 72.2%). SIM is essentially tied (0.653 vs. 0.652; 0.654 at 5 and 8 steps). The 5-step model has lower WER than CosyVoice 2 (2.39%) but trails it on EMO (71.7% vs. 72.2%). The paper describes the intelligibility and naturalness advantage as “a large margin”; on these numbers that fits WER (about 22% relative), while the UTMOS difference is small. The paper’s claim of better “zero-shot capabilities” is not backed by a meaningful SIM gap.
  • Reference systems: F5-TTS (1.83% WER, 0.665 SIM) and CosyVoice 3 (1.68% WER, 0.695 SIM) have lower WER and higher SIM than every CDCD-TTS variant. CDCD-TTS has higher UTMOS (4.119 vs. 3.754 and 3.937), and its EMO at 25 steps ties CosyVoice 3 (72.7%). The paper marks these two rows as given “as a reference” and does not say whether they were run under the same 4-sample median protocol.
  • Efficiency (§5.2, text only): the paper reports the DiT backbone as 10x smaller than the 0.5B LLM and 5 to 10x faster depending on reference length, with a whole-system real-time factor of about 0.2 to 0.3 on a GPU at 25 steps. No table, hardware details or per-module latency breakdown are given.

Novelty Assessment

The theoretical part is the more original contribution. Linking the squared distance between token embeddings to KL divergence between reverse-diffusion path measures follows directly from Girsanov’s theorem, but using it to rank fixed codebook geometries for CDCD is new to this paper, as is the local-optimality proof for FSQ under nearest-neighbour distance. The claims are scoped carefully: global optimality and the Best Accuracy Hypothesis beyond one dimension rest on random search, and the KL-to-training-difficulty link is presented as intuition. The TTS system is an engineering integration. CDCD (an existing framework) is placed into the token-generation slot of CosyVoice 2, using F5-TTS’s DiT architecture with a softmax head and a simple statistical duration model, and every other module is inherited. The paper presents it as the first CDCD-based TTS model. Its controlled ablation varies the embedding geometry, not the architecture, and that geometry comparison (original vs. perturbed vs. permuted FSQ, plus RVQ) is what ties the theory to speech.

Field Significance

Moderate. The paper gives a theory-motivated argument, backed by a controlled speech experiment, that the fixed grid geometry of FSQ tokens suits continuous diffusion over discrete speech tokens. It also shows that a roughly 45M-parameter non-autoregressive token generator can match or exceed a 0.5B autoregressive LLM in the same CosyVoice 2 pipeline on intelligibility and UTMOS, without a SIM gain. Evidence comes from one tokenizer, one English test set and automatic metrics only, so it reads as a promising data point for diffusion over speech tokens, not a settled result. The paper also appears in the ICML 2026 proceedings.

Claims

  • supports: For continuous diffusion over discrete token embeddings, how distinguishable two tokens’ generative trajectories are is set by the squared distance between their embeddings, which motivates codebooks that maximise nearest-neighbour separation within a bounded region.

    Evidence: Statement 4.2 gives the KL divergence between reverse diffusion-bridge path measures ending at and as times a schedule-dependent integral, and Theorem 4.3 proves base-2 and base-3 FSQ codebooks are locally optimal for average nearest-neighbour distance under an bound; random search over 10,000 codebooks finds none exceeding the FSQ value. (§4.1, Statement 4.2, Theorem 4.3; §5.1)

  • supports: Non-autoregressive diffusion over discrete speech tokens can replace an autoregressive LLM token generator in a cascaded TTS pipeline at a fraction of its size while improving intelligibility.

    Evidence: With the CosyVoice 2 FSQ tokenizer, flow-matching stage and vocoder unchanged, a ~45M-parameter CDCD backbone at 25 steps gives 2.00% WER, 0.653 SIM and 4.119 UTMOS vs. 2.57%, 0.652 and 4.077 for CosyVoice 2’s 0.5B LLM on Seed-TTS test-en. (§5.2, Table 1)

  • refines: The benefit of a well-structured token embedding geometry for diffusion-based token generation is largest at few sampling steps and narrows as the step count grows.

    Evidence: FSQ-original beats FSQ-perturb on every metric at every step count, but the WER gap falls from 2.39% vs. 3.10% at 5 steps to 2.00% vs. 2.03% at 25, and UTMOS from 4.093 vs. 3.834 to 4.119 vs. 4.118. (§5.2, Table 1)

  • complicates: A favourable embedding geometry is not enough for diffusion over speech tokens; the assignment of token identities to embeddings must come from tokenizer training.

    Evidence: FSQ-permute, which uses the exact FSQ embeddings with randomly reassigned token IDs, reaches 15.4% WER, 0.588 SIM and 3.631 UTMOS at 25 steps, against 2.00%, 0.653 and 4.119 for FSQ-original. (§5.2, Table 1)

  • complicates: Gains from replacing an autoregressive token generator with a diffusion one do not automatically carry over to speaker similarity in zero-shot cloning when the downstream acoustic stage is unchanged.

    Evidence: Every CDCD-TTS variant has SIM between 0.647 and 0.654, essentially equal to CosyVoice 2 (0.652) and below the F5-TTS (0.665) and CosyVoice 3 (0.695) reference rows, even though the paper claims better zero-shot capability. (§5.2, Table 1)

Limitations and Open Questions

All speech evidence uses automatic metrics on a single English test set (Seed-TTS test-en): the "MOS" column is UTMOS and there is no human listening test. Only one FSQ tokenizer (CosyVoice 2's base-3, 8-dimensional codebook) is tested, and the "optimality" results hold only under the paper's assumptions (bounded codebooks, equal token priors, an optimally trained model for the accuracy hypothesis).

  • The Best Accuracy Hypothesis is proven only in one dimension, and global optimality of FSQ for nearest-neighbour distance is supported by random search, not proof. The theory assumes equal token priors, which real speech-token distributions do not satisfy; the paper leaves the unequal case to future work.
  • The link from path-measure KL to training difficulty is offered as intuition, and the paper says top-1 prediction accuracy does not imply generation quality.
  • The RVQ comparison changes several things at once: embedding dimensionality (128 vs. 8), batch size, learning rate and the hierarchical NAR stage for the remaining layers. Its very poor result therefore cannot be attributed to codebook geometry alone.
  • F5-TTS and CosyVoice 3 rows are labelled “as a reference” and may not follow the same 4-sample median protocol. Both beat CDCD-TTS on WER and SIM.
  • Speed and real-time-factor figures are reported in text only, without a table, hardware specification or variance.
  • Generation length comes from a statistical, character-level duration model fit to the reference speaking rate; its effect on prosody and on long or atypical inputs is not evaluated.
  • Code and checkpoint release is promised (GitHub repository named in §1) but was not confirmed at the time of the paper. No demo page is mentioned.

Wiki Connections

  • Diffusion TTS — Introduces a continuous diffusion model over discrete speech-token embeddings (CDCD) as the text-to-token stage of a TTS pipeline, with a theoretical argument for which token geometry suits it.
  • Zero-Shot TTS — CDCD-TTS clones unseen speakers from an infilled audio prompt and is evaluated on Seed-TTS test-en, matching but not exceeding its autoregressive counterpart on speaker similarity.
  • CosyVoice 2 — Host pipeline whose FSQ tokenizer, flow-matching stage and vocoder are reused unchanged, and whose 0.5B LLM token generator is the main baseline that CDCD-TTS replaces.
  • F5-TTS — Source of the DiT backbone architecture and infill-style prompting adapted for CDCD, and a reference system with lower WER and higher SIM.
  • CosyVoice 3 — Reference system with the lowest WER and highest SIM in Table 1, ahead of all CDCD-TTS variants on both.
  • Finite Scalar Quantization — Quantization scheme whose codebook geometry the paper proves locally optimal for CDCD under nearest-neighbour distance.
  • Seed-TTS — Provides the test-en evaluation set and the official WER and SIM tooling.
  • EnCodec — RVQ tokenizer used for the alternative-codebook baseline, which performs far worse than FSQ tokens under CDCD.
  • VALL-E X — Source of the non-autoregressive module that generates the remaining RVQ layers for the RVQ baseline.
  • NaturalSpeech 3 — Prior diffusion model over speech tokens that uses discrete rather than continuous-embedding diffusion, contrasted in related work.