arXiv · 2026 · Preprint
Haibin Wu et al. (Meta) · → Paper · Demo: ? · Code: ?
Introduces T-Mimi, a purely transformer-based replacement for the Mimi neural audio codec’s hybrid transformer-convolution decoder, cutting on-device 80ms-chunk decoding latency from 42.1ms to 4.4ms on a Samsung Galaxy S22 with no significant quality loss, and identifies through quantization-aware-training ablations that the decoder layers closest to the waveform output must be preserved at full precision to maintain audio fidelity.
Problem
Neural audio codec features (from EnCodec, DAC, Mimi) have largely displaced mel-spectrograms as the intermediate representation in modern TTS pipelines, with the codec’s own decoder serving as the final waveform-generation stage. Mimi is a particularly strong choice for streaming TTS: it disentangles audio into separate semantic and acoustic token streams (enabling independent control over content and speaking style), reconstructs high-quality audio at a low 12.5Hz frame rate, and supports streaming encode/decode. But Mimi’s decoder relies on de-convolution (transposed convolution) layers for upsampling, and while these are parameter-efficient, they are poorly optimized in mobile inference frameworks like XNNPACK, which handle transformer operations far more efficiently than de-convolution. This makes Mimi’s decoder a latency bottleneck that stands between the codec’s quality advantages and genuine real-time, on-device TTS deployment on phones.
Method
T-Mimi replaces Mimi’s hybrid decoder (8 transformer layers followed by de-convolution upsampling layers) with a purely transformer-and-linear alternative, inspired by the authors’ own prior TS3-Codec architecture: the de-convolution stage is replaced by 4 additional transformer layers (using fixed-window streaming self-attention to preserve streaming capability) and 2 linear layers that perform the final upsampling, with output waveform segments concatenated directly rather than using overlap-and-add. Total parameter count is held constant relative to the original Mimi decoder. An ablation found a deeper 12-layer configuration outperforms a wider 8-layer one, attributed to the 12-layer setting being able to reuse the original 8 pretrained Mimi transformer layers as initialization, a warm-start advantage the 8-layer configuration cannot access as cleanly. Training uses a composite objective (weighted multi-scale mel-spectrogram L1 reconstruction loss, least-squares GAN loss and feature-matching loss from a Multi-Scale STFT Discriminator, plus a smaller general L1 term) with the T-Mimi encoder kept frozen throughout; a two-stage schedule first trains on the full composite loss to convergence, then fine-tunes using only the feature-matching loss to further improve subjective perceptual quality. A silence-augmentation trick (prepending/appending pure silence to 10% of training clips) was added specifically to suppress a low-level noise artifact the model otherwise produced during silent regions.
For deployment efficiency, the paper applies quantization-aware training (QAT) using TorchAO, testing 4-bit group-wise and 8-bit per-channel weight quantization alongside 8-bit dynamic activation quantization. A systematic sweep varies which of the decoder’s final layers are kept at full precision versus quantized, testing the hypothesis that layers closer to the waveform output are more sensitive to precision loss (since they are responsible for the final high-fidelity reconstruction of a 24kHz signal). This sweep identifies that preserving only the final two transformer layers and two linear layers at full precision, while quantizing the rest of the decoder to 8-bit, is the best point on the quality/storage/latency trade-off curve, and this selective-precision scheme is then carried into a full QAT training run.
Key Results
In blind CMOS (Comparative MOS) listening tests (200 audio pairs, 10 raters each) comparing the non-quantized, full-precision T-Mimi decoder against a parameter-matched Mimi baseline fine-tuned on the same 5-million-hour in-house corpus, the two systems are statistically indistinguishable (T-Mimi wins by +2.32%, with a 95% confidence interval crossing zero), establishing that the architectural swap from de-convolution to transformer-and-linear upsampling does not cost perceptual quality. On real hardware (Samsung Galaxy S22, measuring average latency to generate an 80ms audio chunk as part of a full TTS system), T-Mimi requires only 4.4ms versus 42.1ms for the de-convolution-based baseline (with a 5-frame CNN context window) or 18.0ms for a reduced-context-window (2-frame) variant of that baseline, a 9.6x speedup over the standard baseline. The quantization-sensitivity ablation confirms the paper’s central hypothesis directly: PESQ rises monotonically from 2.32 (uniform 4-bit) through several intermediate mixed-precision configurations to 3.04 as increasingly many of the final layers are kept at full precision rather than 8-bit, while storage grows correspondingly from 20.4MB to 78.2MB; the selected configuration (final two transformer layers plus two linear layers at full precision, rest at 8-bit) reaches a fully-trained PESQ of 3.16, close to the non-quantized model’s 3.21, while cutting storage from 163.2MB to 68.7MB, a 58% reduction. A separate architecture ablation shows expanding from 8 to 12 transformer layers yields the largest single quality jump across all metrics (PESQ, STOI, SI-SDR), with further depth (16 layers) or width (3072-dim linear layers) providing progressively smaller marginal gains, motivating the paper’s choice of a 12-layer, 2048-dim configuration as the practical operating point.
Novelty Assessment
The contribution is a targeted engineering adaptation rather than a newly invented architecture class: the core idea of a purely transformer-based, convolution-free codec decoder was already established by the authors’ own prior TS3-Codec work, and this paper’s specific contribution is applying that design to modify the widely-used Mimi codec’s decoder and validating it with real on-device latency measurements rather than only simulated FLOP counts. The quantization-sensitivity finding, that decoder layers closest to the waveform output require disproportionately higher precision, is a genuinely useful, mechanism-grounded empirical result (validated through a systematic layer-precision sweep, not asserted from intuition alone) that plausibly generalizes to other convolution-based neural codec decoders being adapted for on-device deployment, as the authors themselves note. The 9.6x latency reduction is a concrete, hardware-validated practical result with direct deployment relevance, though it should be read alongside the paper’s own caveat that the CNN baseline could not be quantization-aware-trained due to tooling constraints, making the exact latency comparison not fully apples-to-apples.
Field Significance
high — this paper demonstrates, with real on-device benchmarking rather than only simulated efficiency estimates, that a widely-adopted streaming neural codec's convolutional decoder bottleneck can be eliminated via a transformer-only replacement at no measurable perceptual cost, and its quantization-sensitivity finding (waveform-adjacent layers need disproportionately higher precision) offers a concrete, generalizable principle for compressing other convolution-based codec decoders for edge deployment, directly addressing a practical barrier to real-time on-device TTS that the codec-features-for-TTS paradigm had not previously confronted at the hardware level.
Claims
- supports: Replacing the deconvolution-based upsampling stage of a hybrid transformer-convolution neural audio codec decoder with additional transformer and linear layers, matched in total parameter count, can achieve comparable audio quality while dramatically improving on-device inference latency, since deconvolution layers are poorly optimized in mobile inference frameworks despite their parameter efficiency.
Evidence: T-Mimi-32-bit is statistically indistinguishable from a parameter-matched fine-tuned Mimi baseline in blind CMOS listening tests (95% CI crossing zero), while cutting measured on-device 80ms-chunk decoding latency on a Samsung Galaxy S22 from 42.1ms to 4.4ms, a 9.6x speedup. (§4.2.1, §4.2.3, Tables 1, 3)
- supports: In a neural audio codec decoder, sensitivity to quantization increases specifically with proximity to the final waveform output, such that the last few layers require substantially higher numerical precision than earlier layers to preserve audio quality.
Evidence: A systematic sweep of mixed-precision QAT configurations shows PESQ improving monotonically (2.32 → 3.04 at 50k steps) as more of the final layers are preserved at full precision while earlier layers stay at 8-bit, with the best quality/storage trade-off requiring full precision only in the final two transformer and two linear layers. (§4.2.2, Table 2)
- complicates: Applying post-training quantization to a convolution-based neural codec decoder not specifically designed or trained for quantization is substantially more damaging than applying quantization-aware training to a comparably-sized transformer-based alternative, complicating direct latency comparisons between convolution- and transformer-based codec decoders under compressed deployment.
Evidence: The paper explicitly notes the deconvolution-based CNN-Mimi baseline was evaluated without quantization-aware training due to CNN-specific QAT library constraints, since quantizing a model not optimized for it is known to disproportionately degrade generative models, meaning the reported baseline latency reflects an unquantized model against T-Mimi’s QAT-optimized one. (§4.2.3)
- supports: Increasing depth in a purely transformer-based codec decoder yields more consistent quality improvements than increasing per-layer width, up to a point of diminishing returns.
Evidence: Expanding the decoder from 8 to 12 transformer layers produces substantial gains across all reconstruction-quality metrics (PESQ, STOI, SI-SDR), while increasing linear-layer width from 2048 to 3072 at fixed depth yields only modest gains, and further deepening to 16 layers gives progressively smaller marginal improvement than the 8-to-12 jump. (§4.2.4, Table 4)
Limitations and Open Questions
The headline 9.6x latency reduction compares a quantization-aware-trained T-Mimi against a CNN-Mimi baseline that was not quantization-aware-trained, because CNN-specific QAT tooling constraints prevented it; the paper itself flags that applying quantization to a model not designed for it is known to disproportionately hurt generative-model performance, so the reported latency gap may partly reflect this asymmetry rather than the architectural change alone.
On-device latency is measured on a single device (Samsung Galaxy S22); generalization of the 9.6x speedup to other mobile chipsets or inference frameworks besides XNNPACK is not directly tested. The paper frames its methodology as generalizable to other convolution-based neural audio codecs beyond Mimi, but this generalization claim is not empirically validated on any codec other than Mimi within the paper itself.
Wiki Connections
- Neural Audio Codec — modifies the Mimi codec’s decoder architecture, replacing deconvolution-based upsampling with a purely transformer-and-linear design to eliminate an on-device latency bottleneck while preserving Mimi’s semantic/acoustic dual-stream tokenization.
- Streaming TTS — targets real-time, on-device TTS deployment, preserving Mimi’s streaming decode capability (via fixed-window streaming self-attention) while reducing per-chunk decoding latency to enable genuine real-time responsiveness on mobile hardware.
- TS3-Codec — supplies the purely transformer-based, convolution-free codec decoder design that T-Mimi’s architecture is directly inspired by and adapts specifically to the Mimi codec.
- Moshi — the original source of the Mimi codec, whose decoder architecture this paper modifies; Mimi’s semantic/acoustic token disentanglement and streaming capability are inherited unchanged by T-Mimi.
- EnCodec — cited as one of the neural audio codecs (alongside DAC and Mimi) that have displaced mel-spectrograms as the standard intermediate representation in modern TTS pipelines.
- VALL-E — cited as an example of state-of-the-art TTS systems that leverage neural codec (EnCodec) features as their acoustic representation, part of the broader paradigm this paper’s decoder-efficiency work supports.
- AudioLM — cited as an example system using SoundStream tokens for codec-feature-based audio generation, part of the motivating landscape for codec-decoder efficiency.
- Spark-TTS — cited as an example LLM-based TTS system built on BiCodec features, representative of the range of acoustic modeling techniques that ultimately rely on a codec decoder as the final waveform-generation stage.
- MaskGCT — cited as an example of modern codec-feature-based zero-shot TTS systems that depend on a high-quality, efficient codec decoder for final audio synthesis.