arXiv · 2025 · Preprint
Yanuka et al. · → Paper · Demo: ? · Code: ?
Introduces Principled Coarse-Graining (PCG), a speculative-decoding acceptance rule that verifies draft tokens at the level of overlapping Acoustic Similarity Groups rather than exact token identity, raising acceptance rates for autoregressive speech-token generation while retaining an exact sampling guarantee at the group level.
Problem
Speculative decoding speeds up autoregressive generation by having a small draft model propose tokens that a larger target model verifies, provably preserving the target’s output distribution when acceptance requires an exact token match. For speech tokenizers, this exactness requirement is a poor fit: many discrete acoustic tokens are perceptually interchangeable, so a “wrong” draft token can still correspond to audio that sounds identical to what the target model would have produced. Standard speculative decoding rejects these perceptually valid proposals anyway, driving acceptance rates down and eroding the speedups the technique is meant to deliver. Prior speech-specific relaxations either restrict sampling to small top-k pools (reducing output diversity) or add an ad hoc constant bias to the acceptance ratio (trading accuracy for speed without any distributional guarantee, risking erroneous acceptances).
Method
PCG replaces token-level exact matching with acceptance over Acoustic Similarity Groups (ASGs). For each token in the target model’s vocabulary, an ASG is defined as the set of tokens whose embeddings (from the target model’s own embedding table) exceed a cosine-similarity threshold θ with it; the authors find θ ∈ [0.38, 0.45] works best in their setting (§2.1). Because ASGs are constructed from pairwise thresholding, they overlap: a token can belong to several groups simultaneously, which preserves smooth acoustic neighborhoods but prevents naive probability aggregation across groups.
To handle this overlap, PCG splits each token’s probability mass across every group that contains it (equal-split by default), producing proper group-level distributions P_c and Q_c for the draft and target models respectively (§2.2, Eq. 2–5). At each decoding step, a draft token is sampled as usual, then a group label is drawn from among the groups containing that token using the same per-token weights, so the induced group-level proposal exactly matches the coarse-grained draft distribution. Acceptance then applies the standard speculative-decoding ratio, but computed on this group random variable rather than on the raw token. The paper proves that, under this coupling, the sequence of accepted groups is an exact sample from the target model’s coarse-grained distribution at every step (§2.2, Proposition). In the reject branch, a group-level residual is sampled via a thinning procedure that avoids enumerating all groups, and the paper reports the expected number of thinning trials never exceeds 3 in their experiments. Upon acceptance, the original draft token is retained as the group’s representative (rather than resampling a token from the group) to preserve KV-cache validity, trading exact token-level output for computational efficiency while keeping the acoustic (group-level) distribution exact.
The method is evaluated as a drop-in modification to an existing autoregressive TTS pipeline: LLaSA-8B (LLaSA), built on a LLaMA-8B backbone and tokenized with X-Codec2 using FSQ-style discretization over a 65,536-entry codebook. The draft model is a 3-layer subset of the LLaSA target, initialized from the target’s own parameters and trained (in prior work) on LibriHeavy (50,000 hours of read English); the draft was not further optimized with knowledge distillation. Inference uses temperature 0.8, a speculation lookahead of 3 draft tokens, and offline-precomputed ASGs (with average group size up to ~140 tokens, adding negligible O(|G|) overhead relative to the O(n) vocabulary softmax).
Key Results
On 500 LibriTTS test-clean utterances in a cross-sentence zero-shot voice-cloning setup, standard speculative decoding (Target+SD) yields almost no speedup (0.98×) while achieving the best WER/CER (11.1/5.5). SSD, a prior speech-specific relaxation that adds a constant bias to the acceptance ratio, raises throughput to 1.4× but degrades WER/CER to 18.5/11.6 and naturalness MOS (reported as NMOS in the paper) to 3.78. At the same 1.4× speedup, PCG reduces WER/CER to 13.8/7.8, matches the exact-decoding speaker similarity (Sim-O 43.7 vs. 43.7), and improves NMOS to 4.09 relative to SSD; the WER improvement over SSD is reported as statistically significant (p = 0.039) (§4, Table 1).
An ablation constructing groups from Mel-spectrogram similarity instead of cosine similarity in the target model’s embedding space gives higher WER and CER (13.9/7.9 vs. 12.6/6.6) at matched speaker similarity, indicating the gains stem from structure in the model’s own embedding space rather than raw acoustic proximity (§4, Table 3). A separate stress test replaces target-generated tokens with a uniformly sampled alternative from their ASG: at a relaxed threshold swapping 91.4% of tokens, WER increases by only 0.007 and speaker similarity drops by only 0.027 relative to unmodified target output, supporting the premise that ASG membership captures genuine perceptual interchangeability (§4, Table 2). A lookahead ablation (Figure 4) finds 3 speculated tokens gives the best speedup for SD with PCG.
Novelty Assessment
The architectural components involved (speculative decoding, embedding-space similarity, rejection sampling) are all pre-existing; the contribution is a new acceptance/rejection scheme for the group-level random variable, together with a proof that it yields an exact sample from the target’s coarse-grained distribution. This is a genuine algorithmic novelty relative to prior speech speculative-decoding relaxations, which either restrict sampling (Viterbi search, tolerance-based validation) or bias acceptance heuristically (SSD) without any exactness guarantee at any level of granularity. The contribution is narrowly scoped to the decoding/inference stage; it does not modify or retrain the underlying TTS architecture, and is demonstrated on a single target-draft pair (LLaSA-8B family) and a single language (English, LibriTTS).
Field Significance
moderate — This paper contributes a theoretically grounded, general acceptance rule for speculative decoding over discrete speech tokens, addressing a genuine efficiency bottleneck for on-device or latency-constrained autoregressive speech LLMs. Its scope is a single inference-time technique validated on one target-draft model pair and one benchmark, so its significance rests on the generality of the coarse-graining idea rather than on breadth of validation already demonstrated in the paper.
Claims
- supports: Coarse-grained, similarity-based acceptance criteria can substantially raise speculative-decoding acceptance and throughput for discrete acoustic-token generation without abandoning an exact distributional guarantee.
Evidence: PCG defines overlap-aware Acoustic Similarity Groups from cosine similarity in the target model’s embedding space and proves the accepted group sequence is an exact sample from the target’s group-level distribution; at a matched 1.4× speedup it reduces WER/CER from 18.5/11.6 (SSD baseline) to 13.8/7.8 while matching speaker similarity. (§2.2, Table 1)
- complicates: Heuristic acceptance-relaxation methods that bias the speculative acceptance ratio to trade accuracy for speed can degrade output quality without offering any formal distributional guarantee at any level of granularity.
Evidence: The SSD baseline, which adds a constant bias term to the acceptance ratio, reaches the same 1.4× speedup as PCG but with higher WER/CER (18.5/11.6) and lower naturalness MOS (3.78 vs. 4.09), while providing no exactness guarantee. (§3.2, Table 1)
- supports: Similarity in a codec model’s own learned token-embedding space is a more faithful perceptual proxy for acoustic interchangeability than raw spectral (Mel) similarity.
Evidence: Constructing acceptance groups from Mel-spectrogram similarity instead of embedding cosine similarity yields higher WER and CER (13.9/7.9 vs. 12.6/6.6) at matched speaker similarity, for the same underlying method. (§4, Table 3)
- refines: Discrete acoustic tokens produced by neural codec language models often form large equivalence classes that can be substituted for one another with only minor perceptual cost, refining how much token-level precision is actually needed for faithful speech reconstruction.
Evidence: Randomly replacing 91.4% of target-generated tokens with alternatives from their Acoustic Similarity Group changes WER by only +0.007 and speaker similarity by only -0.027 relative to unmodified target output. (§4, Table 2)
Limitations and Open Questions
The evaluation is confined to a single target-draft model pair (LLaSA-8B with a 3-layer draft derived from it) and a single codec (X-Codec2), on English read speech (LibriTTS test-clean) in a zero-shot voice-cloning setup; generality to other codecs, languages, or speech-LM architectures with different embedding geometries is not tested. The human perceptual evaluation is small in scale (four raters, 85 samples per method), which limits the precision of the NMOS comparisons. The draft model itself was not optimized with knowledge distillation, and the authors note all methods (including baselines) should improve with a stronger draft, so the reported speedups may not reflect the ceiling achievable with a better-trained draft model. The similarity threshold θ controlling group granularity was tuned empirically per experiment rather than derived analytically, and larger, more permissive groups increase per-step group-membership computation (average group size up to ~140 tokens in the reported setting).
Wiki Connections
- Autoregressive Codec TTS — applies to and accelerates inference for autoregressive codec-token TTS systems, using LLaSA-8B as the target model in its experiments.
- Zero-Shot TTS — evaluates its decoding method in a cross-sentence zero-shot speech-cloning setup, requiring the group-level acceptance criterion to preserve speaker similarity under zero-shot cloning.
- Neural Audio Codec — builds its Acoustic Similarity Groups directly from the token embedding space of the X-Codec2 neural codec used by the target model.
- Subjective Evaluation — reports a naturalness MOS listening study (four human raters) to validate that group-level acceptance preserves perceived speech quality alongside automatic metrics.
- LLaSA — uses LLaSA-8B as the target model and X-Codec2 tokenizer for all experiments, making PCG a drop-in acceleration technique for this system.
- Analyzing and Mitigating Inconsistency in Discrete Speech Tokens for Neural Codec Language Models — cited as evidence that discrete codec tokens exhibit acoustic/semantic interchangeability, the core assumption motivating Acoustic Similarity Groups.
- AudioLM — cited to motivate embedding-space similarity as a perceptual proxy, given AudioLM’s demonstration of rich semantic structure in learned audio representations.