arXiv · 2025 · Preprint
Hyeongju Kim et al. (Supertone Inc.) · → Paper · Demo: ✗ · Code: ✗
Replaces the greedy per-layer nearest-code search used in residual vector quantization (RVQ) encoding with a beam-search encoder that reduces quantization error and improves reconstruction quality on pre-trained neural audio codecs, without any retraining.
Problem
RVQ-based neural audio codecs (SoundStream, EnCodec, DAC, HiFi-Codec) quantize a continuous latent into a sequence of discrete codes by greedily minimizing the residual at each quantization layer in turn: layer 1 picks the closest code to the input, layer 2 picks the closest code to the resulting residual, and so on. Because each step commits irrevocably before later layers are considered, this greedy encoding is not guaranteed to find the code sequence that minimizes the overall reconstruction error, even though the decoder and codebooks themselves are fixed and unchanged. The paper argues that this suboptimality is a property of the encoding algorithm, not of the trained codec, and is therefore fixable at test time without touching model weights.
Method
The paper reformulates RVQ encoding as a search problem and replaces greedy per-layer selection with beam search. At layer 1, the top-B closest codes to the input are kept as B parallel candidate paths (beam size B). At each subsequent layer i, for every one of the B candidate paths the algorithm computes the residual, finds the top-k closest codes in codebook i, forms B×k new candidate extensions, scores each by its cumulative quantization error, and prunes back down to the top B. After the final layer, the single lowest-error candidate in the beam is returned as the encoded output. Conventional greedy RVQ encoding is the special case B = k = 1. The algorithm requires no retraining or fine-tuning of the codec: it is applied directly to two off-the-shelf pre-trained checkpoints, EnCodec (24 kHz, 6 kbps, trained on mixed speech/noisy-speech/music/general audio) and HiFi-Codec (24 kHz, trained on English/Chinese speech, using its group-residual GRVQ codebooks). Because the inner loop over B candidates and k codebook lookups is embarrassingly parallel, the authors also implement a GPU-parallelized version of the beam-search step to avoid the naive linear-in-B latency blowup of a sequential CPU implementation.
Key Results
On LibriTTS test-clean with EnCodec at 6 kbps, increasing beam size from B=1 (greedy) to B=16 reduces average L2 quantization error from 5.096 to 4.625, and improves PESQ from 2.726 to 2.850, STOI from 0.942 to 0.948, NISQA from 3.491 to 3.588, and SI-SNR from 4.239 dB to 4.649 dB (Table 1, Table 3). HiFi-Codec shows smaller but consistent gains over the same beam-size range (PESQ 3.013→3.034, quantization error 22.29→21.81). The improvement holds outside speech: on MUSDB music and an internal non-verbal-vocalization set, mel-spectrogram distance and SI-SNR both improve monotonically as B increases (Table 2). Gains persist across EnCodec bit-rates from 3 to 24 kbps (4 to 32 codebooks), including at 24 kbps where PESQ still improves from 3.670 to 3.691 despite 32 quantization layers already being in use (Table 4). On latency, a naive sequential implementation increases CPU inference time by 285% from B=1 to B=16 (43.53 ms → 167.6 ms for a 5-second clip), while the GPU-parallelized implementation increases only from 6.780 ms to 7.366 ms (about 9%) over the same range (Table 5).
Novelty Assessment
The core idea, beam search as a drop-in replacement for greedy decoding, is a well-established technique elsewhere (e.g. sequence generation), and the paper’s contribution is its careful application to RVQ codec encoding, framed explicitly as generalizing conventional RVQ (B=k=1 is a special case). This is a genuine, if narrow, architectural/algorithmic contribution: it changes the inference procedure of an existing model class rather than combining pre-existing systems, and it is training-free, meaning it can in principle be applied post hoc to any RVQ-based codec checkpoint. The evaluation is reasonably broad (two codec architectures, three audio domains, four bit-rates, both CPU and GPU latency), but all evaluation is intrinsic to the codec (reconstruction fidelity metrics); the paper does not measure whether the reduced quantization error translates into any benefit for downstream tasks that consume the resulting codes, such as codec-token language modeling or TTS.
Field Significance
Moderate — the paper identifies and fixes a genuine inefficiency in the standard RVQ encoding procedure used across most neural audio codecs, and demonstrates the fix is training-free and applicable to already-deployed pre-trained models. Its contribution is a targeted inference-time improvement to an existing, widely-used building block rather than a new codec design or a new downstream capability; the gains reported are consistent but modest in absolute terms.
Claims
- supports: Greedy per-layer code selection in residual vector quantization leaves quantization error on the table that can be recovered purely at test time, without retraining the codec.
Evidence: Beam-search encoding (B=16) lowers average L2 quantization error on LibriTTS from 5.096 to 4.625 for EnCodec and from 22.29 to 21.81 for HiFi-Codec relative to greedy (B=1) encoding of the same pre-trained checkpoints. (§5.2, Table 3)
- supports: Reductions in RVQ quantization error obtained by a better search strategy translate into measurable gains on standard reconstruction-quality metrics, not just the raw error term.
Evidence: Increasing beam size from 1 to 16 improves PESQ, STOI, NISQA, and SI-SNR simultaneously for both EnCodec and HiFi-Codec on LibriTTS test-clean. (§5.3, Table 1)
- refines: Adding more RVQ codebooks (higher bit-rate) does not by itself eliminate the suboptimality of greedy encoding; the search-strategy gap persists even at high codebook counts.
Evidence: At 24 kbps with 32 codebooks, EnCodec’s greedy encoding still trails beam-search encoding (PESQ 3.670 vs. 3.691, NISQA 3.992 vs. 4.001), a gap of comparable relative size to the one seen at 6 kbps with 8 codebooks. (§5.5, Table 4)
- complicates: A naive implementation of wider-search codec encoding introduces a latency cost that scales with search width, and only a parallel hardware-aware implementation avoids this trade-off.
Evidence: Sequential CPU beam-search encoding increases inference time by 285% from B=1 to B=16 (43.53 ms to 167.6 ms per 5-second clip), while a GPU-parallelized implementation of the same algorithm increases by only about 9% (6.780 ms to 7.366 ms) over the same range. (§5.6, Table 5)
Limitations and Open Questions
The evaluation is confined to two pre-trained codec checkpoints (EnCodec, HiFi-Codec); the paper does not test the algorithm on more recent RVQ-GAN codec designs (e.g. DAC-style codecs) or on codecs with substantially larger codebook sizes, where the exponential O(S^L) exhaustive-search cost the method is designed to avoid is even more pronounced. Beam size B and the per-step candidate width k are always set equal in the experiments, so their individual contributions to the error reduction are not disentangled. All reported gains are measured on intrinsic reconstruction-quality metrics (PESQ, STOI, NISQA, SI-SNR, mel distance); the paper does not evaluate whether the improved codec reconstructions change downstream outcomes for systems that consume the codes, such as word error rate in a codec-token TTS or speech language model pipeline. The non-verbal vocalization test set is an internal, non-public dataset, limiting reproducibility of that portion of the evaluation.
Wiki Connections
- Neural Audio Codec — proposes a training-free, beam-search encoding algorithm that improves the reconstruction fidelity of existing RVQ-based neural audio codecs at test time.
- Autoregressive Codec TTS — targets the RVQ codecs (EnCodec, HiFi-Codec) that serve as the discrete tokenizers underlying autoregressive codec-based TTS systems, though this paper itself evaluates only reconstruction quality, not downstream TTS performance.
- Spoken Language Model — improves the fidelity of the RVQ codec representations that speech language models typically consume or predict as discrete tokens, without modifying or evaluating any language model component.