arXiv · 2026 · Preprint
Leyan Yang et al. (Nanjing University / Horizon Robotics) · → Paper · Demo: ✓ · Code: ?
Proposes VoCodec, a causal, time-frequency-domain neural speech codec that jointly targets low computational complexity (349.29M MACs/s), low latency (30ms), and low bitrate (down to 1 kbps) for real-time communication, submitted to the 2025 Low-Resource Audio Codec (LRAC) Challenge.
Problem
Recent neural speech codecs achieve high-fidelity reconstruction at extremely low bitrates, but the models that reach this quality tend to be computationally heavy and non-causal, which makes them unsuitable for real-time communication. Real-world transmission scenarios add a further complication: background noise and reverberation degrade codec performance, yet most codec research treats compression and enhancement as separate problems. The 2025 Low-Resource Audio Codec (LRAC) Challenge frames this as a joint-constraint problem, requiring competitive reconstruction quality under simultaneous limits on computational complexity, bitrate, and latency (Track 1), and additionally under noisy and reverberant conditions (Track 2). This paper targets that joint constraint directly rather than optimizing bitrate or quality in isolation.
Method
VoCodec builds its encoder and decoder on the Vocos backbone, but the key design decision is to operate the codec directly in the time-frequency (T-F) domain rather than the time domain used by most contemporary codecs (e.g. DAC, BigCodec, WavTokenizer). The input waveform is transformed via STFT into a complex spectrum, from which log-magnitude and phase are extracted, concatenated, and projected to a lower-dimensional feature. This choice serves two purposes: speech has pronounced harmonic structure in the T-F domain, and the STFT/inverse-STFT pair performs downsampling and upsampling in a single step, avoiding the extra computation and latency of learned up/downsampling layers.
The encoder stacks ConvNeXt blocks (following the WavTokenizer variant of Vocos) followed by an attention module built from ResNet blocks and a self-attention layer; all convolutions use causal zero-padding and the self-attention layer uses masked attention so the model never accesses future frames. Quantization uses Residual Vector Quantization (RVQ) with 6 layers of 1024 codewords each, applying the factorized, L2-normalized codebook lookup introduced in DAC (8-D low-dimensional code lookup via a fully connected layer, converting Euclidean to cosine distance for the codebook). At a 100Hz encoder frame rate this gives 1 kbps per RVQ layer, up to 6 kbps total. The decoder mirrors the encoder but drops transposed convolutions (no learned upsampling is needed) and removes the inverted bottleneck in the ConvNeXt block and uses grouped convolution in the ResNet block specifically to keep receiver-side compute under 150M MACs/s; the network outputs complex spectral coefficients that are converted back to a waveform via inverse STFT. Training uses a multi-scale STFT discriminator only (no multi-scale or multi-period waveform discriminators), with a generator loss combining multi-scale mel-spectrogram reconstruction loss, adversarial and feature-matching losses, and the standard VQ-VAE codebook/commitment losses.
To address real-world noise and reverberation (Track 2), the authors cascade UL-UNAS, a separately published lightweight speech-enhancement network based on time-frequency masking, at the front end of VoCodec: the input passes through UL-UNAS to produce an enhanced spectrum, which is then fed into VoCodec’s normal pipeline. UL-UNAS and VoCodec are pretrained independently (UL-UNAS with SI-SNR and power-compressed spectrum losses), then UL-UNAS is frozen and VoCodec is fine-tuned on top of it to absorb the nonlinear distortions the enhancement stage introduces. The final VoCodec model has 3.47M parameters, a computational complexity of 349.29M MACs/s (144.82M MACs/s of which is receiver-side) and 30ms latency (10ms STFT window buffering plus 20ms inverse-STFT algorithmic latency); the full cascade with UL-UNAS scales to 5.34M parameters, 1.28G MACs/s, and 50ms latency (adding 20ms of UL-UNAS lookahead).
Key Results
The official LRAC 2025 Challenge evaluation used five objective metrics (ScoreQ-ref, UTMOS, Sheet-SSQA, PESQ, Audiobox AE-CE) plus organizer-run subjective listening tests (MUSHRA on clean speech, DMOS/MOS on real-world noisy and reverberant conditions, and diagnostic rhyme test intelligibility scores). VoCodec ranked fourth on Track 1 of the challenge and achieved the highest MUSHRA score among evaluated systems on the clean speech test set. On the blind Track 1 test set, VoCodec’s MUSHRA score at 6 kbps was 89.19, compared to 74.28 for the official baseline, and at 1 kbps VoCodec scored 65.2 against a 17.92 baseline. On Track 2 (noisy/reverberant), the SE+VoCodec cascade improved MUSHRA at 6 kbps from a 60.06 baseline to 75.96, and improved MOS scores on both real-world noise and reverberation conditions relative to the baseline. The authors also report that even though no noise or reverberation was injected during VoCodec’s own training, the model still reconstructs mildly noisy or reverberant speech at 6 kbps with high quality, which they attribute to strong generalization of the base codec.
Novelty Assessment
VoCodec’s individual components are largely drawn from prior work: the Vocos backbone, DAC’s factorized/L2-normalized RVQ quantizer, ConvNeXt and ResNet blocks, and UL-UNAS as an off-the-shelf enhancement front end. The genuine design contribution is the combination and constraint-driven adaptation of these pieces: choosing to operate the codec in the time-frequency domain (rather than the time domain most low-bitrate codecs use) specifically to cut computation and latency, enforcing strict causality throughout, and asymmetrically constraining receiver-side compute below 150M MACs/s by removing the ConvNeXt inverted bottleneck and switching to grouped convolution in the decoder. This is best characterized as an engineering-integration contribution targeted at a specific resource-constrained deployment setting (the LRAC challenge’s joint complexity/bitrate/latency constraints) rather than a new codec architecture family.
Field Significance
moderate — This paper is a competitive challenge submission that demonstrates a concrete design point for real-time, low-complexity, low-bitrate speech coding, showing that operating a codec in the time-frequency domain can outperform a time-domain baseline under matched compute and latency budgets, and that cascading a frozen lightweight enhancement network with fine-tuning can add robustness to noise and reverberation without redesigning the codec. Its evidence is confirmatory of the design choice within one challenge’s specific benchmark rather than a demonstration of general superiority.
Claims
- supports: Operating a neural speech codec directly in the time-frequency domain, rather than the time domain, can reduce computational complexity and latency while remaining competitive on subjective reconstruction quality.
Evidence: VoCodec computes STFT-domain complex spectral features and reconstructs via inverse STFT, reaching 349.29M MACs/s total compute (144.82M MACs/s receiver-side) and 30ms latency while outperforming the challenge’s baseline codec on MUSHRA at both 1 kbps and 6 kbps on the Track 1 blind test set. (§2.1, §3.2, Table 3)
- supports: Cascading a frozen, independently-pretrained speech-enhancement front end with a low-bitrate codec, followed by fine-tuning of the codec alone, can improve robustness to real-world noise and reverberation without retraining the enhancement model.
Evidence: The SE + VoCodec cascade (UL-UNAS frozen, VoCodec fine-tuned) improves MUSHRA at 6 kbps from a 60.06 baseline to 75.96 and improves MOS on both real-world noise and reverberation conditions on the Track 2 blind test set. (§2.3, Table 4)
- complicates: Extremely low-bitrate transmission (around 1 kbps) remains substantially harder to preserve losslessly than moderate bitrates, even for causal, low-complexity codec designs.
Evidence: VoCodec’s own MUSHRA score drops from 89.19 at 6 kbps to 65.2 at 1 kbps on the same clean-speech blind test set, and the authors state explicitly that distortionless transmission at 1 kbps or lower “remains a challenging task” for future work. (§5, Table 3)
Limitations and Open Questions
The paper reports that distortionless transmission of real-world speech at extremely low bitrates (1 kbps or below) remains unresolved, and this is reflected in VoCodec’s own results, where 1 kbps scores trail 6 kbps scores by a wide margin across MUSHRA, DMOS, and DRT metrics on both tracks. The training corpus is described only as “a large-scale dataset provided by the challenge organizers” following the LRAC baseline’s cleaning and preprocessing procedure; its composition and scale are not disclosed in the paper itself, which limits reproducibility outside the challenge. VoCodec ranked fourth on Track 1 overall despite topping the clean-speech MUSHRA metric, indicating the model does not dominate every official evaluation criterion. Objective metric results (ScoreQ-ref, UTMOS, Sheet-SSQA, PESQ, Audiobox AE-CE) are reported per bitrate and condition, but the corresponding table in the source PDF has parsing inconsistencies that make some individual per-condition values ambiguous; only cleanly resolvable metrics were carried into this page’s frontmatter.
Wiki Connections
- Neural Audio Codec — proposes a causal, time-frequency-domain RVQ codec design that trades off computational complexity, latency, and bitrate for real-time communication, building on Vocos and DAC-style quantization.
- Subjective Evaluation — reports organizer-run MUSHRA, DMOS/MOS, and diagnostic rhyme test listening evaluations on blind test sets as part of the 2025 LRAC Challenge.
- Evaluation Metrics — evaluates the proposed codec against official LRAC Challenge objective metrics (ScoreQ-ref, UTMOS, Sheet-SSQA, PESQ, Audiobox AE-CE) alongside subjective listening tests.