arXiv · 2026 · Preprint
Tan Dat Nguyen et al. (KAIST, Chung-Ang University) · → Paper · Demo: ✓ · Code: ?
Converts a pretrained attention-based autoregressive TTS backbone into a hybrid Mamba-Transformer model via structural weight transfer and multi-level distillation, cutting inference memory by up to 34% with minimal loss in speech fidelity.
Problem
LLM-based TTS systems built on autoregressive Transformer backbones achieve near-human naturalness but inherit quadratic time and memory complexity in self-attention, which produces high latency and large KV-cache footprints in long-form scenarios such as podcasts, audiobooks, and streaming dialogue agents. Linear-time state-space alternatives like Mamba avoid this bottleneck but tend to underperform Transformers on tasks requiring strong global context and in-context reasoning. Hybrid Mamba-Transformer designs that interleave both primitives have shown promise for general language modeling, but within TTS the only prior attempt (Zonos-v0.1) relies on expensive pretraining from scratch and has not released its architectural details, leaving the hybrid design space for speech synthesis largely unexplored.
Method
MamTra converts a pretrained attention-based autoregressive TTS backbone (CosyVoice 2) into a hybrid Mamba-Transformer model by replacing a subset of its Transformer blocks with Mamba layers, avoiding training from scratch. The conversion rests on a structural equivalence: recasting causal self-attention (with the softmax removed) as a linear recurrence reveals that the key-value product corresponds to the SSM’s input update, so the pretrained query, key, and value projection weights can be directly transferred to initialize the Mamba layers’ output, input-gating, and input projections respectively. The paper systematically explores where to place the retained Transformer layers across four design-space categories: interleaved (periodic Transformer layers at block start/end), contiguous (grouped at the front, middle, back, or in a sandwich pattern), and data-driven (selecting layers to replace by cosine-similarity or WER-based importance), evaluated across Transformer:Mamba ratios from 1:1 to an aggressive 1:11.
Because removing the softmax nonlinearity alters the attention dynamics and reduces expressive capacity, structural weight transfer alone is insufficient to preserve teacher performance. MamTra therefore applies a multi-level distillation objective combining ground-truth cross-entropy supervision, a skew KL-divergence loss between teacher and student output logits, and a mean-squared-error constraint on token embeddings, jointly aligning the student’s representations with the teacher’s semantic-acoustic space. The hybrid student is fine-tuned from the CosyVoice 2 backbone (0.5B parameters, 24 layers) on LibriTTS using less than 2% of the original English training data.
Key Results
Complexity analysis shows the hybrid architecture achieves sub-quadratic compute and sub-linear KV-cache growth, since scaling is dominated only by the retained Transformer layers; this is empirically confirmed by near-constant Mamba state size alongside attenuated cache growth relative to sequence length. In practice, the MamTra 1:1 configuration matches the CosyVoice 2 teacher’s perceptual metrics on Seed-TTS-eval test-en (NMOS 3.66 vs. 3.68, UTMOS 4.16 vs. 4.15, SSIM 0.72 vs. 0.66) with only a 0.25% absolute WER increase (2.28% vs. 2.03%), while reducing average inference-time GPU memory by up to 34% relative to CosyVoice 2 and 17% relative to the Zonos-v0.1 hybrid baseline, and cutting per-token FLOPs by up to 1.4×10^11 at a 2,048-token context length. Under a length-stressed test condition spanning 1-62 words, the WER-importance-selected 1:5 configuration (WER 2.28%) outperforms the more conservative contiguous 1:3 configuration (WER 2.99%) and matches the 1:1 baseline while consistently surpassing Zonos-v0.1. At the most aggressive 1:11 ratio, however, WER rises to 3.99-4.08%, falling below the Zonos-v0.1 baseline and marking the upper bound of the efficiency-quality trade-off. Ablations after 15 training epochs show that removing any one of the three distillation loss terms degrades WER from 3.48% to between 5.63% and 6.70% (and CER from 2.86% to between 3.26% and 4.32%), while UTMOS and SSIM remain nearly unchanged, indicating the losses primarily rescue linguistic accuracy rather than perceptual quality.
Novelty Assessment
The core technical building block, converting attention layers into SSM layers via a structural weight-transfer equivalence, is adapted from prior general-purpose LLM-to-hybrid distillation work rather than invented in this paper. What is new is the systematic application and empirical characterization of this technique specifically for autoregressive TTS backbones: MamTra is presented as the first systematic study mapping the hybrid design space (interleaved, contiguous, data-driven layer placement across multiple Transformer:Mamba ratios) for speech synthesis, together with a TTS-specific multi-level distillation recipe that recovers teacher fidelity from under 2% of the original training data. The contribution is primarily architectural-adaptation and training-recipe engineering rather than a fundamentally new modeling primitive, and it is evaluated only against one backbone family (CosyVoice 2) and one prior hybrid TTS baseline whose own architecture is not fully disclosed.
Field Significance
moderate — MamTra provides a concrete, data-efficient recipe for converting an existing attention-based TTS backbone into a memory-efficient Mamba-Transformer hybrid without retraining from scratch, along with a systematic characterization of where hybridization helps and where it breaks down. It demonstrates that inference memory can be reduced substantially with a small absolute WER cost, offering a practical efficiency lever for long-context TTS deployment, but the underlying weight-transfer and distillation technique is imported from general LLM compression work rather than originating in this paper, and the evaluation is confined to a single backbone and English-only data.
Claims
- supports: Attention layers in an autoregressive TTS backbone can be selectively replaced with linear-time SSM layers via structural weight transfer, retaining most of the teacher’s speech quality and fidelity while sharply reducing inference memory and compute.
Evidence: The MamTra 1:1 configuration matches the CosyVoice 2 teacher’s NMOS, UTMOS, and SSIM within a 0.25% absolute WER increase while reducing average inference-time memory by 34% and per-token FLOPs by up to 1.4×10^11 at a 2,048-token context. (§4.1, §4.2, Table 3, Figure 3)
- complicates: Hybrid attention-SSM TTS backbones trade off intelligibility for efficiency as the proportion of SSM layers increases, with degradation becoming severe beyond a critical replacement ratio.
Evidence: At the most aggressive 1:11 Mamba:Transformer ratio, WER rises to 3.99-4.08%, dropping below the Zonos-v0.1 hybrid baseline and marking the upper bound of the observed efficiency-quality trade-off. (§4.2, Table 3)
- supports: For attention-to-SSM layer replacement, an importance-based layer-selection strategy generalizes better under variable-length or long-context conditions than simply retaining more Transformer layers in fixed positions.
Evidence: Under a length-stressed test set spanning 1-62 words, the WER-selection 1:5 configuration achieves lower WER (2.28%) than the more conservative contiguous 1:3 configuration (2.99%) despite retaining fewer Transformer layers. (§4.3)
- supports: Multi-level knowledge distillation combining ground-truth supervision, logit-matching, and embedding alignment is necessary to recover a distilled hybrid model’s linguistic accuracy after replacing attention with SSM layers, more so than for perceptual quality.
Evidence: Ablating any one of the cross-entropy, logit-KL, or embedding-MSE loss terms substantially degrades WER and CER after 15 training epochs, while UTMOS and SSIM remain almost unchanged across all ablations. (§4.4, Table 4)
Limitations and Open Questions
The evaluation is confined to a single backbone family (CosyVoice 2, 0.5B parameters), a single language (English, via LibriTTS and Seed-TTS-eval test-en), and a single prior hybrid baseline (Zonos-v0.1) whose own architectural details are not publicly disclosed, so the generality of the reported trade-offs across other TTS backbones, model scales, and languages remains untested.
The subjective naturalness evaluation (NMOS) is based on 50 sampled utterances per model rated by 15 listeners, a modest scale for statistical robustness. The paper also does not report which neural codec underlies the CosyVoice 2 backbone’s token representation, limiting traceability of MamTra’s output representation. Finally, the most aggressive hybridization ratio (1:11) shows that the efficiency-quality trade-off has a hard ceiling within this design space, and the paper does not explore combining its hybrid architecture with complementary efficiency techniques such as speculative decoding or KV-cache compression.
Wiki Connections
- Autoregressive Codec TTS — MamTra modifies the attention backbone of an existing autoregressive codec-token TTS system (CosyVoice 2) rather than proposing a new codec or generation paradigm, targeting the inference efficiency of this model family.
- Zero-Shot TTS — the system is evaluated on the Seed-TTS-eval zero-shot cloning benchmark using audio-prompt conditioning, and results confirm that hybridization preserves the backbone’s zero-shot speaker-similarity performance.
- Subjective Evaluation — reports a naturalness MOS listening test with 15 independent human raters scoring 50 sampled utterances per model, alongside objective metrics.
- CosyVoice 2 — MamTra’s backbone and teacher model; the paper directly fine-tunes and distills from this pretrained architecture rather than training from scratch.
- DiSTAR — cited as a related autoregressive-diffusion hybrid representation for speech generation, situating MamTra among recent hybrid-architecture TTS approaches.
- SPADE — prior efficiency-focused work by overlapping authors on structured pruning and adaptive distillation for LLM-based TTS, addressing a related but distinct axis of backbone compression.
- Seed-TTS — provides the Seed-TTS-eval test-en benchmark used as MamTra’s primary evaluation setting.
- CosyVoice — predecessor to the CosyVoice 2 backbone MamTra builds on, cited as part of the supervised-semantic-token TTS lineage.
- VALL-E — cited as foundational prior work establishing the neural-codec-language-model paradigm for zero-shot TTS that motivates the attention-heavy backbones MamTra seeks to make more efficient.