Speech Generation Research Wiki

Low-Resource Audio Codec (LRAC): 2025 Challenge Description

8 min read

arXiv · 2025 · Preprint

Kamil Wojcicki et al. (Cisco Systems (Collaboration AI)) · → Paper · Demo: ✗ · Code: ✓

Introduces the 2025 Low-Resource Audio Codec (LRAC) Challenge, defining two tracks with joint bitrate, compute, and latency constraints, a standardized curated training corpus, two open-source baseline systems, and a large-scale crowdsourced evaluation protocol for benchmarking speech codecs under real-world deployment conditions.

Problem

Neural audio codecs such as SoundStream, EnCodec, and Descript Audio Codec demonstrate that learned analysis-synthesis pipelines can reach high perceptual quality at very low bitrates, and subsequent systems such as BigCodec push this further. However, these gains typically come with increased compute and memory demands that make deployment on power-, memory-, and latency-constrained edge devices difficult. Prior benchmarking efforts, such as the Codec-SUPERB Challenge, evaluate perceptual quality and information preservation without imposing explicit compute or latency restrictions, so the field lacks a standardized way to compare codecs under the joint constraints that real-time, low-resource deployment actually imposes. Robustness to everyday acoustic degradations (background noise, reverberation) and the integration of speech enhancement directly into the coding pipeline are likewise largely unaddressed by existing codec benchmarks.

Method

This paper is a challenge description, not a modeling paper: it specifies the tracks, constraints, data, and evaluation protocol of the 2025 LRAC Challenge rather than proposing a new codec architecture.

The challenge defines two tracks. Track 1 targets transparency codecs that minimize perceptual degradation on largely clean speech (mild noise/reverb only), under tight constraints (≤700 MFLOP/s total compute, ≤30 ms total latency, ≤1/≤6 kbps ultralow/low bitrate modes). Track 2 targets codecs that actively integrate speech enhancement (denoising and dereverberation) with compression, under an expanded budget (≤2600 MFLOP/s, ≤50 ms latency) and a broader range of real-world noise and reverberation intensities. Both tracks require a single system and a single decoder to support both bitrate modes, mono 24 kHz audio, and constant-bitrate coding only; latency and compute are computed analytically from model architecture rather than measured at runtime, using formulas the organizers supply.

For training data, the organizers curated approximately 702.7 hours of clean speech from LibriTTS, VCTK, EARS, Librivox (DNS5), MLS (French/German/Spanish), and GLOBE V2, applying dataset-specific thresholds on SNR, reverberation, bandwidth, sample rate, and duration, along with speaker-disjointness checks and speaker/gender balancing across splits. Noise and room-impulse-response resources for augmentation (drawn from subsets of Audio Set, FreeSound, WHAM!, FSD50K, Free Music Archive, Motus, and OpenSLR 28) were curated by classifying noise recordings with CLAP and downsampling over-represented categories, reducing an initial 518-hour pool to 335 hours after removing speech-contaminated clips.

Evaluation runs in two phases. During development, participants receive continuous feedback from a set of objective metrics selected for their reported correlation with listening tests on codecs under clean conditions (SCOREQ in non-matching-reference mode, UTMOS, SHEET-SSQA, and Audiobox Aesthetics Content Enjoyment, alongside PESQ as a widely used but weaker-correlating reference). For the final test-phase ranking, objective metrics are dropped entirely in favor of large-scale crowdsourced listening tests on Prolific, because objective metrics were judged unreliable for generative-model outputs. The listening-test battery combines MUSHRA-1S (a single-condition variant of MUSHRA chosen for scalability without the range-equalizing and context biases of ACR/MUSHRA) for clean-speech quality, Degradation Category Rating (DCR) for robustness to light noise, reverberation, and overlapping talkers (Track 1), Absolute Category Rating (ACR) for denoising/dereverberation performance on real-world recordings (Track 2), and the Diagnostic Rhyme Test (DRT), adapted to crowdsourcing, for intelligibility in both tracks. Per-track scores are normalized to [0, 100] and combined via track- and bitrate-mode-specific weights.

Two open-source baseline systems (convolutional encoder-decoder architectures with a six-layer, 1,024-codeword residual vector quantizer, trained adversarially with mel-spectrogram, commitment, and feature-matching losses) are provided as ESPnet-toolkit reference implementations for each track; their architecture and training recipe are described in full in the companion baseline-systems paper rather than here.

Key Results

As a challenge report rather than a system paper, this paper does not present its own codec performance metrics; the test phase drew six unique Track 1 submissions and nine unique Track 2 submissions (excluding the two baselines), and the resulting crowdsourced rankings and participant system-description reports were published on the challenge website. The paper explicitly defers a comprehensive analysis of the submitted systems and their comparative performance to future work.

Novelty Assessment

The contribution here is a benchmark and dataset design, not an architectural one: no new codec is proposed in this paper. Its novelty lies in combining, for the first time in a public codec challenge, joint bitrate/compute/latency constraints with an explicit, separately-scored track for codecs that integrate speech enhancement, and in pairing that with a large-scale crowdsourced evaluation battery (MUSHRA-1S plus DCR/ACR/DRT) rather than relying on objective proxies for the deciding ranking. This distinguishes it from prior codec benchmarking efforts such as Codec-SUPERB, which optimizes for perceptual quality and information preservation without hard compute or latency budgets. The curated 702.7-hour training corpus and the paired noise/RIR augmentation pipeline are also a concrete, reusable resource, though they are themselves assembled from existing public datasets rather than newly collected.

Field Significance

Moderate — this paper establishes a new benchmark that jointly constrains codec bitrate, compute, and latency alongside robustness to real-world noise, reverberation, and integrated speech enhancement, and pairs it with a standardized curated training corpus and a crowdsourced, listening-test-based evaluation protocol. Its contribution is a benchmark and resource rather than a modeling advance, and its long-term significance will depend on the corpus and evaluation protocol seeing use beyond this first edition, which the paper itself explicitly plans for through future editions.

Claims

  • supports: Benchmarking speech codecs under joint constraints on bitrate, compute, and latency exposes design trade-offs that are invisible when bitrate or perceptual quality is optimized in isolation.

    Evidence: The challenge specifies simultaneous per-track ceilings on receive-side and total compute (≤300/700 MFLOP/s for Track 1, ≤600/2600 MFLOP/s for Track 2), total latency (≤30/50 ms), and bitrate (≤1/≤6 kbps ultralow/low modes), with a single system and decoder required to satisfy both bitrate modes at once. (§2.1, Table 1)

  • complicates: Objective codec-quality metrics can be unreliable for ranking systems trained with generative methods, making them unsuitable as a sole or final arbiter of codec quality.

    Evidence: The organizers used objective metrics (SCOREQ, UTMOS, SHEET-SSQA, AE-CE, PESQ) only for development-phase feedback and explicitly excluded them from the final test-phase ranking, stating they “may exhibit limited reliability for models trained with generative methods,” relying instead on crowdsourced MUSHRA-1S, DCR, ACR, and DRT listening tests. (§5, §5.1)

  • supports: Integrating speech enhancement (denoising and dereverberation) directly into the codec pipeline, rather than treating it as an upstream front-end, is a viable and separately evaluable design direction for low-bitrate speech coding.

    Evidence: Track 2 is dedicated to codecs that jointly denoise, dereverberate, and compress under an expanded compute/latency budget, with dedicated ACR-based subjective tests for dereverberation and denoising performance on real-world recordings. (§2.1, §5.2.3, Table 4)

  • complicates: Curated multi-corpus training sets for speech coding benchmarks can still be skewed toward read speech and a single dominant language, limiting how representative the resulting benchmark is of real-world speech diversity.

    Evidence: Read-speech sources (LibriTTS, VCTK, Librivox/DNS5, MLS) account for about 61% of the 702.7-hour curated training corpus, which is predominantly English with only partial French/German/Spanish coverage from MLS. (§3.1.1, Table 3)

Limitations and Open Questions

Warning

Compliance with the challenge’s bitrate, compute, and latency constraints was based on the honor system, with latency and complexity computed analytically from each team’s own architecture description rather than independently measured or audited by the organizers. This limits how directly cross-system comparisons in the challenge results can be trusted without independent verification.

The first edition deliberately scopes down to a subset of core speech-coding advancements: the authors note that extensions to general audio coding, network resilience, memory footprint, and wider system-integration considerations are left to future editions. The paper also defers any comprehensive analysis of the submitted systems’ relative performance to future work, so this description paper alone does not yet indicate which design choices among submissions were most effective. Finally, the training and test data remain predominantly English and read-speech in character, which may limit how the benchmark’s conclusions generalize to more spontaneous or multilingual speech conditions.

Wiki Connections

Backlinks

Graph View