arXiv · 2025 · Preprint
Dong, Wang, Han et al. (Hunan University and Beijing Xiaomi Mobile Software Co., Ltd) · → Paper · Demo: ? · Code: ?
A low-bitrate (1.5 kbps) neural speech codec that resolves the acoustic-fidelity-versus-semantic-richness trade-off by splitting quantization into two specialized, high-utilization stages: a frozen external semantic codebook that anchors linguistic content, and a SimVQ-based single-layer quantizer that captures the remaining acoustic residual.
Problem
Neural speech codecs built on multi-layer Residual Vector Quantization (RVQ), such as Encodec and DAC, degrade sharply once the bitrate budget is constrained to around 1.5 kbps, and their multi-stream token output complicates downstream autoregressive speech-language-model integration. Single-codebook codecs like WavTokenizer simplify this integration but are optimized purely for signal reconstruction, leaving their tokens semantically shallow. Existing approaches to inject semantics, either distilling from an external SSL teacher or learning semantics endogenously, both tend to require retaining a complex multi-layer RVQ backend to preserve reconstruction quality, conflicting with the low-bitrate goal, or add substantial training complexity. Underlying all of this is a more basic inefficiency: standard vector quantization updates only the nearest codebook entry per step, causing codebook collapse that caps the representational capacity of any codec built on it.
Method
SACodec is a GAN-based, end-to-end codec with an Encodec-style convolutional-LSTM encoder (24kHz input, 320x downsampling to a 75Hz frame rate) and a ConvNeXt-Attention decoder that reconstructs the waveform via inverse STFT. Its core contribution is an asymmetric dual quantizer applied to the encoder’s latent representation. The first stage, Semantic Anchoring, quantizes the latent against a codebook derived from a fixed, pre-computed set of 1,000 mHuBERT cluster centroids; rather than looking up this frozen codebook directly (which collapses to near-zero utilization), a lightweight learned linear projector first transforms the entire frozen codebook into an adapted, effective codebook in the encoder’s latent space, and nearest-neighbor lookup happens against this adapted version. This “codebook-space projection” strategy is what achieves near-100% codebook utilization in practice. The acoustic residual left after subtracting the semantic embedding, which the paper identifies as carrying paralinguistic information like speaker timbre and prosody, is then quantized by a second, single-layer Residual Activation module using SimVQ: rather than learning a residual codebook directly, SimVQ reparameterizes it as a frozen, randomly-initialized coefficient matrix multiplied by a single learnable linear basis, so only the shared basis is updated during training, which propagates gradient updates globally across the entire effective codebook rather than only to the single nearest entry, again achieving full codebook activation. The two quantized embeddings are summed and passed to the decoder. The generator is trained end-to-end against an ensemble of multi-period and multi-band multi-scale STFT discriminators, with a composite loss combining multi-scale mel reconstruction, adversarial, feature-matching, and two separately-weighted commitment losses for the semantic and residual branches (with the semantic branch weighted more heavily to enforce alignment with the fixed external mHuBERT space).
Key Results
At 1.5 kbps on LibriTTS test-clean, SACodec achieves a UTMOS of 4.0373, more than 2.5x higher than Encodec (1.5551) and roughly 2x higher than DAC (1.9152) at the same bitrate, and it surpasses the more aggressively-compressed 0.9 kbps WavTokenizer (3.9687). On the noisy LibriTTS test-other set, SACodec’s UTMOS (3.4786) is nearly identical to ground truth (3.483) and exceeds several higher-bitrate baselines, including 6 kbps DAC (3.1338), 4 kbps SpeechTokenizer (3.2851), and 4.8 kbps FACodec (2.930). On the out-of-domain LJSpeech set, SACodec (UTMOS 3.9912) remains competitive with 6 kbps DAC (4.0415), indicating the improvements generalize beyond the training corpus. A MUSHRA subjective listening test on LibriTTS test-clean places SACodec’s median score (96.8) in the same perceptual tier as ground truth (97.5), far above 1.5 kbps DAC (median 49.0) and above 6 kbps DAC (median 87.4). On the ARCH semantic-classification benchmark, SACodec (mean accuracy 0.4809 compressed-domain, 0.6311 reconstruction-domain) outperforms the speech-only-trained WavTokenizer on both dimensions and remains competitive with the multi-domain-trained DAC, while SpeechTokenizer, despite scoring highest in the compressed domain (0.5099), loses its lead after reconstruction, which the authors attribute to its decoder failing to fully preserve token semantics through generation. Training is reported as over 6x faster per epoch than WavTokenizer on the same 585-hour LibriTTS corpus, and SACodec reaches these results without the massive proprietary training corpora used by Encodec, DAC, and FACodec.
Novelty Assessment
Both underlying quantization ingredients, semantic distillation from a frozen SSL teacher and SimVQ’s reparameterized global-update quantization, exist in prior work; SACodec’s contribution is combining them into an asymmetric, two-stage architecture that assigns each mechanism to the information stream it is best suited for (a fixed external codebook for semantics, a globally-updated learnable codebook for the acoustic residual) within a single end-to-end GAN framework. The codebook-space projection technique for adapting a frozen external codebook (rather than looking it up directly) is a specific, ablation-validated design choice the paper attributes its high semantic-codebook utilization to. The ARCH reconstruction-domain evaluation, measuring whether semantic information survives the full encode-decode pipeline rather than only the raw compressed tokens, is presented as a methodological addition the authors introduce themselves to surface a real failure mode (SpeechTokenizer’s semantic-quality drop after reconstruction) that compressed-domain-only evaluation would have missed.
Field Significance
high — SACodec offers a genuinely novel and clearly ablated solution to a real, previously-unresolved trade-off (semantic richness versus acoustic fidelity at low bitrate) using an architecturally simple, single-stage asymmetric quantizer, validated with both comprehensive objective/subjective reconstruction metrics and a dedicated semantic-classification benchmark across in-domain, noisy, and out-of-domain conditions.
Because low-bitrate, semantically-rich, single-sequence tokens are precisely what modern speech language models and TTS systems built on discrete codecs need, SACodec’s asymmetric dual-quantizer design (and specifically its codebook-space projection trick for using frozen external codebooks without collapse) offers a reusable template other codec designs could adopt, though the paper’s own limitations note that direct integration into a downstream SLM or TTS system, rather than token-level proxy evaluation alone, is still needed for full validation.
Claims
- supports: A frozen, externally pre-trained semantic codebook can be quantized against efficiently, with near-full codebook utilization, if the codebook itself is first transformed into the encoder’s latent space by a lightweight learned projector, rather than looked up directly in its original space.
Evidence: A naive direct lookup between the encoder and the fixed mHuBERT codebook yields near-zero (5%) codebook utilization and a UTMOS of 3.65, while the proposed codebook-space projection achieves nearly 100% utilization and a UTMOS of 4.04 on the same setup. (§Ablation Study, Figure 3)
- supports: Splitting quantization into a dedicated semantic stage (anchored to external linguistic knowledge) and a dedicated acoustic-residual stage (using a globally-updated, collapse-resistant quantizer) allows a low-bitrate codec to achieve reconstruction quality and semantic richness simultaneously, where prior single-mechanism approaches had to trade one off against the other.
Evidence: At 1.5 kbps, SACodec achieves both a UTMOS of 4.0373 (comparable to or exceeding 6 kbps RVQ-based baselines) and an ARCH reconstruction-domain semantic accuracy of 0.6311 (on par with 6 kbps DAC), while removing either quantizer stage in ablation causes either a 30% collapse in semantic accuracy (removing the semantic anchor) or a 12.8% drop in PESQ (removing the residual activator). (§Ablation Study, Table 3)
- complicates: A codec’s semantic token quality measured on raw compressed tokens does not guarantee the same semantic quality survives after the codec’s own reconstruction pipeline, so compressed-domain-only semantic evaluation can overstate a codec’s practical semantic usefulness.
Evidence: SpeechTokenizer achieves the highest compressed-domain ARCH accuracy (0.5099) among compared codecs but drops behind SACodec after reconstruction (0.6013 vs. SACodec’s 0.6311), a discrepancy the authors attribute to its decoder not fully preserving the intrinsic semantics of its own tokens. (§Results and Analysis, Table 2)
- complicates: Optimizing a residual quantizer purely for semantic accuracy, by removing its dedicated acoustic-detail-recovery component, can locally improve a semantic metric while substantially degrading the reconstruction quality that downstream applications also depend on.
Evidence: Removing the Residual Activation module entirely (“semantic-only” configuration) yields the single highest compressed-domain semantic accuracy (0.5065) in the ablation study, but causes reconstruction PESQ to drop by 12.8% relative to the full model. (§Ablation Study, Table 3)
Limitations and Open Questions
The paper’s own discussion section identifies its evaluation as confined to English, leaving cross-lingual robustness untested. All evaluation is token-level (reconstruction metrics and downstream classification accuracy on the ARCH benchmark) rather than validated through direct integration into a downstream speech language model or TTS system, which the authors explicitly flag as necessary for ultimate validation of the tokens’ practical usefulness. The semantic codebook size (1,000 entries) and residual codebook size (1,024 entries) were fixed by the specific publicly-available mHuBERT centroids used; the authors note scaling the semantic codebook further, and exploring model compression for on-device deployment, as open future directions rather than settled questions.
Wiki Connections
- Neural Codec — introduces an asymmetric dual-quantizer architecture combining a frozen, externally-anchored semantic codebook with a globally-updated SimVQ residual quantizer to resolve the semantic-versus-acoustic-fidelity trade-off at low bitrate.
- Self-Supervised Speech Representations — anchors its semantic quantization stage to a frozen codebook derived from mHuBERT cluster centroids, injecting self-supervised linguistic priors directly into the codec’s discrete tokens.
- Subjective Evaluation — validates reconstruction quality with a real human-rater MUSHRA listening test, showing near-ground-truth median scores at 1.5 kbps.
- Encodec — used as a primary RVQ-based neural codec baseline across all reconstruction-quality and semantic-representation evaluations.
- SpeechTokenizer — used as a semantic-distillation-based codec baseline, whose compressed-domain semantic lead over SACodec does not survive reconstruction, a discrepancy SACodec’s evaluation design specifically surfaces.
- WavTokenizer — the single-codebook codec paradigm this work builds on and compares against; SACodec’s training loss weights follow WavTokenizer’s standard practice, and WavTokenizer’s speech-only variant serves as the primary single-codebook baseline throughout.
- NaturalSpeech 3 — its FACodec quantizer serves as the disentanglement-focused neural codec baseline across all reconstruction and semantic evaluations.
- LibriTTS — the corpus SACodec is trained on (585 hours) and the primary in-domain (test-clean, test-other) evaluation benchmark.