arXiv · 2025 · Preprint
Tan Dat Nguyen et al. (KAIST, 42dot Inc.) · → Paper · Demo: ✓ · Code: ?
Introduces a structured pruning and distillation framework that halves the transformer depth of large autoregressive LLM-TTS backbones while preserving near-parity naturalness and speaker similarity.
Problem
Autoregressive LLM-TTS systems built on large text-LLM backbones (CosyVoice, VALL-E, CLaM-TTS, RALL-E, LLaSA) achieve strong controllability and zero-shot generalization, but they inherit the parameter counts, memory footprint, and slow autoregressive decoding of the text-only LLMs they are built on. Model compression techniques such as pruning, distillation, and quantization are well studied for text LLMs, but a systematic compression methodology that specifically preserves prosody, naturalness, and long-context autoregressive coherence for LLM-TTS had not been established.
Method
SPADE compresses an LLM-TTS backbone in two stages. First, it identifies which transformer layers can be safely removed using a proposed WER-based layer importance index (WLI): each layer is masked out in turn and the resulting increase in word error rate on a held-out subset is measured, directly tying the pruning criterion to intelligibility rather than to internal activation similarity. The authors show that the commonly used cosine-based layer importance criterion from text-LLM pruning (comparing a layer’s input and output latents) does not align with which layers actually matter for TTS quality, motivating the WER-based alternative. Layers with the lowest WLI are pruned; the analysis finds that the earliest, middle, and final layers consistently matter most across backbones, while many interior layers contribute little.
Second, the pruned student model is recovered via multi-level knowledge distillation from the original, un-pruned model acting as teacher. The training objective combines a cross-entropy supervised loss with four distillation terms: a Skew-KL-based logit alignment loss, and mean-squared-error alignment losses on intermediate latents, attention maps, and the embedding output. To bridge the discontinuity introduced by removed layers, distillation targets for latent and attention alignment are dynamically selected: the student’s layer aligns to the teacher’s last layer before the next retained layer, rather than to the corresponding teacher layer by index. A weighting factor (set to 0.25) balances the supervised and distillation components, favoring supervised guidance. No additional parameters (e.g. adapters) are introduced; distillation only heals the retained weights.
SPADE is evaluated on two representative, publicly available LLM-TTS backbones: CosyVoice 2 (24-layer backbone, 0.63B params) and LLaSA-1B (16-layer, 1.7B params, codec-based). Pruned variants are fine-tuned on a small fraction of data (25% of LibriTTS for CosyVoice 2, 25% of LibriHeavy for LLaSA), corresponding to under 5% of the original pretraining corpus size, on 4×A6000 GPUs.
Key Results
For CosyVoice 2, pruning to 12 layers (halving depth) cuts parameters by 39.7%, improves real-time factor by 42.6%, and reduces VRAM usage by 14%, while degrading Seed-TTS WER by only 0.68 (2.03→2.71) and NMOS by 0.13 (3.71→3.58); speaker similarity and UTMOS are essentially unchanged. A more aggressive 9-layer variant reduces parameters by 49.2% and improves RTF by 45.9%, but WER on Seed-TTS rises further to 3.09, showing that intelligibility degrades faster than perceptual quality metrics as pruning becomes more aggressive. For LLaSA, halving depth to 8 layers reduces parameters by 23.5% and improves RTF by 29.3% (1.41× speed-up) with 20% VRAM reduction, but shows a larger relative quality drop than CosyVoice 2 (speaker similarity −0.045, UTMOS −0.04), which the authors attribute to LLaSA’s layers having more uniformly high WLI values (less redundancy to exploit).
An ablation on CosyVoice 2 (LibriTTS test-clean) shows that replacing WLI-based pruning with cosine-based pruning increases WER from 1.59 to 1.74 and CER from 0.54 to 0.61, confirming that the WER-grounded criterion better preserves intelligibility than the activation-similarity criterion borrowed from text-LLM pruning. Removing the dynamic distillation target selection (aligning only to the corresponding teacher layer by index) also degrades performance (WER 1.59→1.65, CER 0.54→0.58).
Novelty Assessment
The pruning-then-distillation recipe itself is standard practice in text-LLM compression; SPADE’s contribution is adapting this recipe to LLM-TTS with two domain-specific design choices: a WER-grounded layer importance criterion in place of the cosine-similarity criterion used for text LLMs, and a dynamic (rather than static, index-matched) target selection scheme for latent and attention distillation across pruned layer segments. The ablations directly isolate the value of both choices, which is a genuine, if incremental, methodological contribution rather than a new architecture. The evaluation is limited to two backbones and English-only test sets, and the largest efficiency gains (9-layer CosyVoice 2, LLaSA) come with a WER trade-off that is left largely unquantified beyond the reported numbers.
Field Significance
Moderate — the paper is the first to frame model compression for LLM-TTS as a systematic problem distinct from text-LLM compression, and provides evidence that a WER-grounded pruning criterion outperforms the cosine-based criterion carried over from the text domain. Its contribution is primarily an engineering adaptation of established compression techniques (structured pruning, knowledge distillation) validated with a targeted ablation, rather than a new architecture or training paradigm.
Claims
- supports: Many transformer layers in autoregressive LLM-TTS backbones contribute little to synthesis quality and can be removed with minimal loss in naturalness and speaker similarity.
Evidence: Halving CosyVoice 2’s depth to 12 layers (39.7% fewer parameters) increases Seed-TTS WER by only 0.68 and decreases NMOS by 0.13, while speaker similarity and UTMOS remain essentially unchanged. (§4.1, Table 1)
- refines: Layer-importance criteria developed for pruning text-only LLMs do not transfer directly to speech generation backbones; intelligibility-grounded criteria are needed to identify prunable layers correctly.
Evidence: Cosine-based layer importance (input/output latent similarity) diverges from WER-based importance in TTS backbones, and substituting it for the proposed WER-based criterion increases WER from 1.59 to 1.74 and CER from 0.54 to 0.61 on LibriTTS test-clean. (§2.1, §4.2, Table 2)
- supports: Knowledge distillation can recover most of the performance lost from aggressive layer pruning in speech-generation LLMs using only a small fraction of the original pretraining data.
Evidence: Fine-tuning pruned variants required under 5% of the original pretraining data (25% of LibriTTS for CosyVoice 2, an upper-bounded 12.5% of LibriHeavy for LLaSA) yet speaker similarity and UTMOS changed by at most 0.045 and 0.04 respectively relative to the uncompressed backbones. (§3, §4.1, Table 1b)
- complicates: Robustness to layer pruning varies substantially across LLM-TTS backbones depending on how redundant their transformer layers are, so a single pruning ratio does not generalize uniformly.
Evidence: LLaSA showed a larger relative quality drop after 50% layer pruning (speaker similarity −0.045, UTMOS −0.04) than CosyVoice 2 at the same pruning ratio, attributed to LLaSA’s WLI values being more uniformly high across layers, indicating less exploitable redundancy. (§4.1)
Limitations and Open Questions
The evaluation covers only two backbones (CosyVoice 2 and LLaSA-1B) and English-only test sets (LibriTTS test-clean, Seed-TTS test-en), so it is untested whether the WER-based pruning criterion and dynamic distillation scheme generalize to other LLM-TTS architectures, multilingual settings, or streaming inference. The more aggressive pruning configurations (9-layer CosyVoice 2) trade a larger, unquantified increase in WER for additional speed and memory gains, and the paper does not characterize where this trade-off becomes unacceptable for deployment. Computing WLI itself requires running WER evaluation over a data subset for each candidate layer removal, adding an upfront cost to the pruning procedure that is not reported in wall-clock terms.
Wiki Connections
- Autoregressive Codec TTS — SPADE targets exactly this architecture family (CosyVoice 2, LLaSA), proposing a compression methodology specific to its transformer-decoder structure.
- Zero-Shot TTS — both backbones are evaluated on zero-shot benchmarks, and the paper shows that zero-shot generalization and speaker similarity survive substantial layer pruning.
- Neural Audio Codec — LLaSA’s backbone operates on speech codec tokens, and SPADE’s pruning and distillation must preserve the codec-token prediction behavior of the teacher model.
- Subjective Evaluation — naturalness is validated with a listening test (20 listeners, 50 samples per model) in addition to automatic metrics, to confirm perceptual parity after compression.
- CosyVoice 2 — one of the two backbones SPADE compresses; used as the primary evaluation target with the most complete ablation results.
- LLaSA — the second backbone SPADE compresses, used to test whether the pruning and distillation framework generalizes beyond a single architecture.
- VALL-E — cited as an early representative LLM-TTS system establishing the neural codec language modeling paradigm that SPADE’s target backbones extend.
- RALL-E — cited as a representative LLM-TTS system in the same family of codec-language-model backbones motivating the need for compression.
- CosyVoice — the predecessor system to CosyVoice 2, establishing the supervised semantic token approach that the pruned backbone builds on.
- Seed-TTS — its evaluation set is used as one of the two zero-shot benchmarks (Seed-TTS eval, test-en) for measuring compression quality.