arXiv · 2025 · Preprint
Yusuf Ziya Isik et al. (Cisco Systems (Collaboration AI)) · → Paper · Demo: ✗ · Code: ✓
Presents the official convolutional RVQ codec baselines for the 2025 Low-Resource Audio Codec (LRAC) Challenge, covering a transparency track (1-6 kbps, ≤30 ms latency) and a joint denoising/dereverberation-plus-coding track (≤50 ms latency), both trained end-to-end with adversarial and reconstruction losses.
Problem
Neural audio codecs are increasingly deployed in bandwidth- and compute-constrained real-time settings (VoIP, hearing aids, embedded devices), but published codec research rarely operates under simultaneous hard limits on bitrate, latency, and FLOPs, and rarely reports robustness to the noise and reverberation present in everyday recording conditions. The 2025 LRAC Challenge was created to standardize comparison of low-resource codecs under these joint constraints, but a new challenge needs reference baseline systems: implementations that demonstrably meet the constraints, expose a public training recipe, and give participants a starting point and a performance floor to beat. Without an official baseline, competing submissions cannot be reliably calibrated against a common reference architecture and training procedure.
Method
Both baselines are convolutional encoder-decoder codecs with a mid-network Residual Vector Quantizer (RVQ), adapted from the SoundStream and EnCodec designs already implemented in ESPnet. All layers use weight-normalized convolutions; residual sub-blocks use dilated convolutions with ELU nonlinearities and skip connections. Both tracks operate at 24 kHz and share a 10 ms frame rate (240x downsampling stride), a 6-layer RVQ with 1,024 codewords per layer (10 bits/frame, giving 1-6 kbps via random quantizer dropout during training), and Euclidean-distance codeword selection with straight-through gradients and an EMA-updated codebook.
The Track 1 (transparency) encoder uses four downsampling blocks (strides 3, 4, 4, 5) reaching a 160-dimensional embedding, with no training-time data augmentation; it is trained solely to reconstruct clean speech under mild noise/reverberation. Total system latency is 20 ms (10 ms buffering + 10 ms algorithmic) at 691.35 MFLOPS overall, within the challenge’s 700 MFLOPS / 30 ms budget for Track 1. The Track 2 (enhancement) encoder adds a fifth downsampling block (strides 2, 2, 3, 4, 5) reaching a 320-dimensional embedding and is trained with on-the-fly reverberation (p=0.5, from real recorded RIRs) and additive noise (p=0.8, SNR sampled uniformly in [-5, 30] dB) via ESPnet’s EnhPreprocessor, targeting a clean reference that retains early reflections. Track 2 totals 40 ms latency at 2546.2 MFLOPS, within its 50 ms / 2600 MFLOPS budget.
Training combines a multi-scale mel-spectrogram reconstruction loss (six window/mel-bin configurations), an adversarial loss using EnCodec-style multi-scale complex-STFT discriminators with hinge loss, a feature-matching loss on discriminator intermediate activations, and a codebook commitment loss (loss weights 10/5/1/2 respectively). Training data is drawn from public corpora (LibriTTS, VCTK, EARS, Librivox/DNS5, MLS, GLOBE) filtered by estimated SNR, reverberation, and bandwidth, then stratified by speaker gender, identity, and duration, retaining 702.7 of 1,641.7 original hours (42.8%). Noise data is curated and balanced using CLAP-based classification against an AudioSet-derived ontology. Models are trained with RAdam (lr 3e-4, decay 0.998/step) for 1,150 (Track 1) or 1,325 (Track 2) epochs across 6 or 8 GPUs, with checkpoints selected by lowest validation multi-scale mel-spectrogram loss.
Key Results
Both baselines are evaluated on the challenge’s open (non-blind) development test set using five objective metrics: SHEET_SSQA, SCOREQ_Ref, Audiobox_AE_CE, UTMOS, and PESQ, each reported separately for clean, noisy, and reverberant conditions at 1 kbps and 6 kbps. For Track 1 at 6 kbps under clean conditions, UTMOS reaches 3.23 and PESQ 2.67; under reverberant conditions at 6 kbps these fall to 1.32 UTMOS and 1.18 PESQ. Track 2 (which is explicitly trained to denoise and dereverberate) improves reverberant-condition scores at 6 kbps to 1.79 UTMOS and 1.29 PESQ, and improves Audiobox_AE_CE to 4.25 versus Track 1’s 3.43 under the same condition, consistent with its enhancement objective. At 1 kbps, both tracks show substantially degraded objective scores across all conditions (e.g., Track 1 UTMOS drops from 3.23 to 1.44 clean), reflecting the difficulty of the ultra-low-bitrate operating point. There is no baseline from prior published work to compare against, since this is the first edition of the challenge; the reported numbers establish the reference floor for challenge participants rather than a comparison against external systems.
Novelty Assessment
The contribution is engineering integration, not architectural novelty: both baselines directly adapt the SoundStream/EnCodec convolutional-RVQ-GAN design already available in ESPnet, with capacity and stride configurations tuned to meet the challenge’s explicit FLOPs and latency budgets. The paper’s value is in exposing a working, budget-compliant reference implementation, a public training recipe, and curated public-data splits, not in proposing a new coding or enhancement method. The data curation and noise-balancing pipeline (CLAP-based noise classification, quality-based speech filtering) is a useful practical contribution but is also assembled from existing tools rather than new methodology.
Field Significance
Low — this paper establishes the official reference baselines for a newly launched community challenge on low-resource, latency- and compute-constrained neural speech coding. It does not propose new architecture or training techniques; its role is to give challenge participants a working, budget-compliant starting point and a performance floor, and to document a concrete recipe for curating public speech and noise data under closed-set constraints.
Claims
- supports: Residual-vector-quantized convolutional codecs trained end-to-end with adversarial and multi-scale mel-spectrogram losses can be configured to satisfy joint sub-1 kbps-to-6 kbps bitrate, sub-50 ms latency, and sub-3000 MFLOPS compute budgets simultaneously.
Evidence: The Track 1 baseline totals 691.35 MFLOPS and 20 ms latency (within a 700 MFLOPS / 30 ms budget) and the Track 2 baseline totals 2546.2 MFLOPS and 40 ms latency (within a 2600 MFLOPS / 50 ms budget), both operating across the full 1-6 kbps range via quantizer dropout. (§3.1, §3.2, Table 2, Table 3)
- complicates: Joint denoising-and-dereverberation (“enhancement”) codecs require substantially more compute than transparency-only codecs even when the coding architecture and bitrate range are otherwise matched.
Evidence: The Track 2 (enhancement) baseline requires 2546.2 MFLOPS overall versus 691.35 MFLOPS for the Track 1 (transparency-only) baseline, roughly 3.7x more compute, despite sharing the same 6-layer, 1,024-codeword RVQ design and 1-6 kbps target range. (§3.1, §3.2, Table 2, Table 3)
- complicates: Automatic objective codec-quality metrics can disagree substantially with one another at low bitrates, complicating single-metric benchmarking of low-resource codecs.
Evidence: At 1 kbps under clean conditions, SCOREQ_Ref reports 1.15/1.01 (Track 1/2, near floor) while Audiobox_AE_CE reports 3.9/3.96 for the same systems and conditions in the same evaluation table. (§4, Table 4, Table 5)
- supports: Filtering public speech corpora by estimated signal quality (SNR, reverberation, bandwidth) before training a low-resource codec substantially reduces the usable data volume relative to the source corpora.
Evidence: Quality- and diversity-based curation across LibriTTS, VCTK, EARS, Librivox (DNS5), MLS, and GLOBE retains 702.7 of 1,641.7 total hours (42.8% overall retention), ranging from 24.2% (LibriTTS) to 100% (EARS) by dataset. (§2, Table 1)
Limitations and Open Questions
Warning
All reported results (Table 4, Table 5) are automatic objective metrics on the challenge’s open development test set; the paper reports no human listening-test (MOS/subjective) results, and the blind test set used for official challenge scoring had not yet been released at the time of writing, so these baseline numbers may not reflect final challenge-condition performance.
Checkpoint selection relies solely on validation multi-scale mel-spectrogram loss; the authors themselves note this prioritizes implementation simplicity over correlation with subjective quality, and suggest combining objective metrics that correlate better with listening tests as a likely improvement. The Track 2 validation set uses on-the-fly (rather than fixed, offline) noise and reverberation augmentation, which the authors acknowledge may increase variance in validation loss estimates. No ablations isolate the individual contributions of the loss terms, the data curation pipeline, or the quantizer-dropout bitrate-scalability scheme.
Wiki Connections
- Neural Audio Codec — provides two openly documented, budget-constrained RVQ codec baselines (SoundStream/EnCodec-derived) as reference implementations for a public low-resource coding challenge.
- GAN Vocoder — trains both baseline codecs with EnCodec-style multi-scale complex-STFT discriminators, hinge loss, and feature-matching loss, following the established GAN-based codec/vocoder training recipe.
- Evaluation Metrics — reports a five-metric objective evaluation protocol (SHEET_SSQA, SCOREQ_Ref, Audiobox_AE_CE, UTMOS, PESQ) across clean, noisy, and reverberant conditions at two bitrates, illustrating how such metrics can disagree at low bitrates.