arXiv · 2025 · Preprint
Chary et al. · → Paper · Demo: ? · Code: ?
A neural speech codec that replaces residual vector quantization (RVQ) stacks with a single-stage 2D patch quantization over mel spectrograms, achieving competitive perceptual quality at 7.5 kbits/s with a simpler, streaming-friendly architecture.
Problem
Most competitive neural speech codecs — SoundStream, EnCodec, DAC — rely on residual vector quantization (RVQ) stacks, which progressively refine quantization across multiple codebook levels. While effective for quality, RVQ stacks increase architectural complexity and memory use. This paper asks whether a single-stage, non-residual quantization scheme can achieve comparable perceptual quality and intelligibility, particularly in low-latency streaming settings where simpler architectures are preferable.
Method
The system operates on 16 kHz waveforms converted to 80-band mel spectrograms (512-sample STFT window, 128-sample hop). The core of the codec is a VQ-VAE that treats the mel spectrogram as a 2D image: a convolutional encoder downsamples by 4x, and a patchification layer tiles the result into non-overlapping 4×4 patches along the time and frequency axes. Each patch is projected and quantized against a single shared codebook of K=4096 entries (12 bits/index). The resulting discrete grid has shape (T/4) × 20, yielding a bitrate of 7.5 kbits/s (625 tokens/second × 12 bits/token).
Training proceeds in two stages. The VQ-VAE is first trained for 150,000 steps with reconstruction (L1 + LPIPS) and VQ commitment losses, with a PatchGAN discriminator introduced after 20,000 steps for late adversarial fine-tuning. A dynamic gradient-norm weighting stabilises the adversarial objective. A HiFi-GAN vocoder is then trained separately for 1,000,000 steps on mel spectrograms reconstructed by the trained VQ-VAE rather than on clean ground-truth spectrograms — this explicit exposure to codec artefacts is the key design choice for improving vocoder robustness.
Key Results
Evaluated on a 1,000-utterance held-out test set drawn from a multilingual corpus, the proposed codec is compared against DAC (24kHz/8kbits/s, 16kHz/8kbits/s), EnCodec (22kHz at 12 and 6 kbits/s), and SNAC (24kHz/0.98 kbits/s) using PESQ, STOI, ViSQOL, MCD (mel-cepstral distance), and RTF.
At the most comparable operating point — EnCodec at 6 kbits/s — the proposed codec (7.5 kbits/s, 16kHz) achieves PESQ 2.70 vs. 2.24 and STOI 0.844 vs. 0.921, meaning better PESQ but lower STOI than EnCodec 6kbits/s. Against DAC at 16kHz/8kbits/s (the closest sampling rate match), DAC is superior on all metrics: PESQ 2.92 vs. 2.70, STOI 0.874 vs. 0.844, ViSQOL 4.01 vs. 2.82. MCD for the proposed system (133.53) is substantially higher than all baselines, indicating greater spectral distortion. The RTF of 0.013 on an RTX 4090 is real-time capable.
Note
Baselines span different sampling rates (16–24 kHz) and bitrates (0.98–12 kbits/s). MCD comparisons across different sampling rates may not reflect perceptual quality faithfully. No subjective (MOS) evaluation is included. The training dataset is proprietary and undescribed, limiting reproducibility.
Novelty Assessment
The core idea — treating a mel spectrogram as a 2D image and applying spatial patch quantization without residual stacks — is a genuine departure from the RVQ paradigm that dominates the field. The architectural insight is clear and clearly motivated. However, the contribution is primarily a proof-of-concept at small scale (16kHz, single test set, proprietary training data). The VQ-VAE and HiFi-GAN components are individually established; the novelty lies in their combination and the 2D patchification design. The codec-artefact-conditioned vocoder training is a sensible engineering choice but not novel in principle. Results show competitive but not superior performance: the system trails DAC on most metrics at comparable bitrate. The evaluation is objective-only, lacks ablations on patchification design choices, and uses a proprietary dataset that precludes external validation.
Field Significance
Low — This paper offers a conceptually distinct alternative to RVQ-based codecs, demonstrating that single-stage patch quantization on mel spectrograms can produce intelligible and perceptually adequate speech at practical bitrates. Its primary value is as an open baseline for exploring non-residual codec designs, not as a performance advance over existing systems.
Claims
- Single-stage vector quantization without residual refinement can achieve intelligibility and perceptual quality competitive with multi-level RVQ codecs at comparable bitrates. (§5.2, Table 1)
- Training a HiFi-GAN vocoder on codec-reconstructed spectrograms rather than clean references improves robustness to codec artefacts. (§4.2)
- Patchwise quantization of mel spectrograms produces a 2D discrete token grid compatible with low-latency streaming at practical bitrates without architectural complexity. (§3, §5.4)
- Objective metrics such as MCD can diverge from perceptual quality metrics like PESQ when comparing codecs across different sampling rates and architectures. (§5.2, Table 1)
Limitations and Open Questions
Warning
The training dataset is described only as a “multilingual speech corpus” without further detail — speaker count, language distribution, total hours, and data sources are unreported. This makes the results impossible to reproduce and limits the generalisability of claims about performance on “diverse” speech.
No subjective evaluation (MOS or MUSHRA) is included; all comparisons are objective-only, which is particularly limiting for a perceptual task like codec quality assessment. The paper does not ablate the patch size (4×4 is the only configuration tested) or codebook size. MCD comparisons between 16kHz and 24kHz systems may be confounded by the sampling rate difference. The codec’s real-world performance at varying packet-loss rates or noisy input conditions is untested. Future directions flagged by the authors include reducing the frequency-axis dimensionality toward a 1D token sequence, joint codec-vocoder training, and downstream use with autoregressive language models.
Wiki Connections
Concept pages most relevant to this paper: neural-codec, gan-vocoder, evaluation-metrics.