arXiv · 2026 · Preprint
Yejin Lee et al. (Sungkyunkwan University / University of Seoul) · → Paper · Demo: ? · Code: ?
TLDR retrofits a frozen, pretrained codec-token AR-TTS backbone (CosyVoice3) to run at a patch rate instead of a token rate: a learned compressor folds every k speech tokens into one patch embedding, the LoRA-adapted backbone predicts the next patch, and a small speaker-conditioned extractor decodes the k tokens inside it. At k=4 this gives a 1.8× RTF speedup and cuts the global KV cache to 0.26× while keeping SeedTTS-EN WER within 0.01 points of the backbone.
Problem
Codec-token AR-TTS systems run one forward pass of the large backbone per generated speech token, and their KV cache grows with the token count. Multi-token prediction and speculative decoding reduce the cost per accepted token, but the backbone still predicts at token timesteps and still caches one state per token. Prior patch-level speech generation (DiTAR) built a new continuous-latent pipeline from scratch. The paper asks whether an existing discrete-token AR-TTS stack can move to patch-level global modeling without replacing its tokenizer, text frontend, backbone or vocoder.
Method
The target speech-token sequence (CosyVoice3 supervised tokens, 25 tokens/s, vocabulary 6,561) is split into N = ⌈T/k⌉ fixed-length patches. Padding in the last patch is excluded from the loss. There are three components:
- Token-to-patch compressor (trained from scratch). Each patch starts as the mean-pooled, RMSNorm-ed embedding of its tokens. One block then refines it: a local Transformer layer (windowed causal self-attention, window 16, RoPE, SwiGLU) followed by masked cross-attention, where each patch query attends only to its own tokens (§3.2, §A.1, Table 7).
- Patch-level Transformer. The CosyVoice3 AR backbone (Qwen2-0.5B) is kept frozen and adapted with LoRA (rank 64, alpha 64). It takes the non-speech prefix (SOS, text, task tokens) followed by the patch embeddings, and its KV cache is updated once per patch. During training the patch sequence is shifted, so the context that predicts patch i+1 never sees that patch’s own embedding, which would leak the target (§3.3, §A.2).
- Speaker-conditioned patch-to-token extractor (4 layers, trained from scratch). A speaker embedding from the reference speech (WavLM with an ECAPA-TDNN head) is projected and concatenated with the global patch context. Each layer projects the result into 4 cross-attention slots, and the extractor then decodes the patch’s tokens autoregressively with causal self-attention inside the patch (§3.4, §A.3).

All trainable modules (136.2M of 648M parameters) are trained with teacher-forced token-level cross-entropy on 585 h of LibriTTS. The training schedule is AdamW, 500k steps and an effective batch size of 64. At inference the reference-prompt tokens are also compressed into patches before they enter the backbone. Without this step the prompt reaches the backbone at a rate k times too high, which effectively stretches it and degrades continuation. Generation alternates between one backbone step and one local extractor decode until EOS. The concatenated tokens then go to CosyVoice3’s unchanged token-to-waveform stage, which the paper calls the vocoder (§3.5).
Key Results
Zero-shot quality on SeedTTS-EN (Table 1):
| System | WER ↓ | SIM ↑ | RTF ↓ |
|---|---|---|---|
| CosyVoice3 (backbone) | 2.02 | 0.691 | 0.605 |
| TLDR k=4 | 2.03 | 0.684 | 0.336 |
| TLDR k=6 | 2.10 | 0.686 | 0.278 |
| TLDR k=8 | 2.49 | 0.688 | 0.248 |
- Speed and memory. At k=4 the RTF falls from 0.605 to 0.336, a 1.80× speedup (baseline RTF divided by TLDR RTF). The global KV cache at 12 s of output is 0.26× the backbone’s (Table 6). End-to-end peak GPU memory at batch 64 is also lower across 4-20 s outputs, but the paper gives that result only as a plot (Figure 3). Every TLDR variant has a lower RTF than every baseline with a reported RTF in Table 1: Spark-TTS 0.952, FireRedTTS-2 1.225, IndexTTS2 0.823, CosyVoice2 0.457 and CosyVoice3 0.605. The other baselines have no RTF entry.
- Quality on SeedTTS-EN. TLDR k=4 does not have the lowest WER. FireRedTTS-2 (1.95), Spark-TTS (1.98), F5-TTS (2.00) and CosyVoice3 (2.02) are all lower than its 2.03. Its SIM of 0.684 is below Seed-TTS (0.762), CosyVoice3 (0.691), MaskGCT (0.714), E2 TTS (0.71) and IndexTTS2 (0.706).
- Quality on LibriSpeech-PC (Table 2). The gap to the backbone is wider here. TLDR k=4 reaches WER 2.15, SIM 0.71 and UTMOS 4.24, against CosyVoice3’s 1.95, 0.718 and 4.28. CosyVoice2 also has lower WER (2.05) and higher UTMOS (4.38).
- Patch size. Increasing k mainly trades WER for speed. SIM stays between 0.684 and 0.688 across k=4 to 8, while WER rises from 2.03 to 2.49 (Table 1, Figure 2).
- Subjective evaluation (Table 3). With 25 crowdsourced English raters, TLDR k=4 scores SMOS 3.953 against CosyVoice3’s 3.739, CMOS +0.19 in TLDR’s favour, and wins 53.9% of A/B preferences. No confidence intervals or significance tests are reported. The higher SMOS runs against the slightly lower objective SIM.
- Independent speaker evaluator (Table 10). The main SIM metric uses the same WavLM-ECAPA model that conditions the extractor. With an independent WeSpeaker model, SIM is 0.7619 ± 0.067 for TLDR against 0.7699 ± 0.073 for CosyVoice3.
Ablations, all at k=4 on SeedTTS-EN:
- No LoRA (frozen backbone used as-is): WER rises to 7.66 while SIM is unchanged (0.687) (Table 4).
- No speaker conditioning in the extractor: WER improves to 1.77, but SIM drops to 0.580 (Table 5).
- Parallel decoding inside the patch: replacing the AR extractor with 3 MTP heads gives 4.85 / 0.631 (WER / SIM), and a NAR extractor gives 3.79 / 0.573, both at the same 0.26× KV ratio (Table 6).
- Mean pooling instead of the cross-attention compressor: WER rises from 2.03 to 2.66 with identical SIM (Table 9).
Text-speech cross-attention maps keep their monotonic alignment at patch resolution (Figure 4).
Fairness. Most baseline numbers are copied from the CosyVoice3 repository. F5-TTS and E2 TTS were re-run under matched decoding, and the DiTAR numbers come from its own paper (§4.1). The “0.6k LibriTTS” dataset label in Tables 1 and 2 describes the adaptation data only: the frozen backbone inherits CosyVoice3’s pretraining on the 1000k-hour multilingual data.
Novelty Assessment
Global/local patch decomposition is borrowed from byte-level language modeling (MEGABYTE, BLT). DiTAR already applied patch-level AR to speech, but over continuous latents and trained from scratch. TLDR’s new element is the retrofit. A frozen discrete-token AR-TTS backbone is converted to patch-rate operation using three pieces: a masked cross-attention compressor, LoRA, and a speaker-conditioned AR extractor. Prompt patchification is also necessary, because without it the prompt is fed at the wrong rate. Each piece is ablated (Tables 4, 5, 6, 9), and the comparison against MTP and NAR extraction inside the same patch structure is the most informative of these ablations. It separates the gain from shortening the global sequence from the gain from parallelising local decoding.
The evidence base is narrow. There is one backbone, one English adaptation set and English-only evaluation, and no comparison against other acceleration methods applied to the same backbone, such as speculative decoding or VocalNet-style MTP on CosyVoice3. The speedup is measured at batch size 1 on a fixed 100-token generation, about 4 s of audio.
Field Significance
Moderate. The paper shows that the global update rate of a pretrained discrete-token AR-TTS backbone can be cut by a factor of 4 with LoRA and two small modules, without retraining the tokenizer or the pipeline around the backbone, while nearly matching the backbone’s SeedTTS-EN quality. Its controlled ablation suggests that the speed and memory gains come from running the global model at the patch rate, while autoregressive decoding inside each patch still matters for intelligibility and speaker similarity. Both findings come from a single backbone.
Claims
- supports: Grouping consecutive discrete speech tokens into patches lets a pretrained AR-TTS backbone run at a lower global update rate, cutting latency and KV-cache memory with little loss in intelligibility.
Evidence: At k=4, TLDR’s RTF is 0.336 against CosyVoice3’s 0.605, and its global KV cache at 12 s is 0.26× the backbone’s. SeedTTS-EN WER is 2.03 against 2.02. (§4.2, Table 1, Table 6, Figure 3)
- complicates: Patch size in patch-level speech-token modeling trades content accuracy against speed, while speaker similarity stays roughly constant.
Evidence: Going from k=4 to k=8 lowers RTF from 0.336 to 0.248 and raises SeedTTS-EN WER from 2.03 to 2.49, while SIM stays at 0.684 to 0.688. (§4.2, Table 1, Figure 2)
- refines: When the global speech-token sequence is shortened, decoding the tokens inside each chunk autoregressively matters more for intelligibility and speaker similarity than parallel decoding inside the chunk.
Evidence: Holding the KV ratio at 0.26×, the AR extractor reaches WER 2.03 and SIM 0.684. Three MTP heads give 4.85 and 0.631, and a NAR extractor gives 3.79 and 0.573. (§5.3, Table 6)
- supports: A frozen token-level speech LM can be adapted to compressed input representations through low-rank adapters, but it does not transfer to such inputs without adaptation.
Evidence: Removing LoRA and training only the compressor and extractor raises SeedTTS-EN WER from 2.03 to 7.66, with SIM essentially unchanged (0.687 against 0.684). (§5.1, Table 4)
- complicates: Compressing speech tokens for global modeling weakens speaker identity, which then has to be restored by explicit speaker conditioning at the cost of some intelligibility.
Evidence: Removing the speaker embedding from the extractor lowers SIM sharply (0.684 to 0.580) but improves WER (2.03 to 1.77). (§5.2, Table 5)
Limitations and Open Questions
Warning
All results come from a single backbone (CosyVoice3), adapted on 585 h of English LibriTTS and evaluated only on English. There is no head-to-head comparison with speculative decoding or multi-token prediction applied to the same backbone, so the speedup is shown only against the unmodified backbone and against other systems’ RTFs.
- The patch size k is fixed. The authors suggest adaptive patching to match speech’s uneven information density as future work (§Limitations).
- On LibriSpeech-PC the quality gap to the backbone is wider (WER 2.15 against 1.95), which the headline SeedTTS-EN comparison does not show (Table 2).
- The subjective test (25 English raters) reports no confidence intervals or significance testing. Its SMOS advantage contrasts with a slightly lower objective SIM under both evaluators (Table 3, Table 10).
- RTF is measured at batch size 1 for a fixed 100-token generation. Throughput under batched serving is argued from KV-cache size but not measured directly (§4.1, §4.2).
- Streaming generation and long-form synthesis are not evaluated. Code and demo availability are not stated.
Wiki Connections
- Autoregressive Codec TTS — Reformulates the AR model over CosyVoice3 supervised speech tokens to predict patches of k tokens, reducing global decoding steps by a factor of k.
- Zero-Shot TTS — Evaluated on prompt-based cloning of unseen speakers on SeedTTS-EN and LibriSpeech-PC, with prompt patchification required for correct continuation.
- Subjective Evaluation — Reports crowdsourced SMOS, CMOS and A/B preference against the backbone, with 25 English raters.
- Self-Supervised Speech — The patch-to-token extractor is conditioned on a WavLM-based (ECAPA-TDNN head) speaker embedding, which the ablation shows is needed to preserve speaker similarity.
- CosyVoice 3 — The frozen backbone, tokenizer, text frontend and token-to-waveform stage that TLDR retrofits, and its primary baseline.
- DiTAR — Prior patch-based AR speech generation over continuous latents trained from scratch, which TLDR contrasts with its discrete-token retrofit and compares against on LibriSpeech-PC.
- Speech Speculative Decoding — Token-level AR-TTS acceleration that TLDR positions itself against, because it keeps one prediction and one KV entry per token.
- VocalNet — Multi-token prediction for speech LMs; TLDR’s MTP-inside-the-patch ablation shows this style of parallel decoding degrades WER and SIM in its setting.
- CosyVoice 2 — AR baseline in both zero-shot tables, with lower LibriSpeech-PC WER and higher UTMOS than TLDR k=4.
- F5-TTS — Flow-matching NAR baseline re-run under matched decoding, slightly lower SeedTTS-EN WER than TLDR k=4.
- FireRedTTS-2 — AR baseline with the lowest SeedTTS-EN WER in Table 1 but a much higher reported RTF.
- LibriTTS — The 585-hour adaptation corpus used to train TLDR’s compressor, extractor and LoRA adapters.