arXiv · 2026 · Preprint
Samir Sadok et al. (Inria, Univ. Grenoble Alpes, CNRS, LJK / GIPSA-lab) · → Paper · Demo: ? · Code: ?
Introduces shape-gain decomposition, a classical speech/audio coding technique, into the neural audio codec pipeline by normalizing energy out of the encoder’s input signal and transmitting it as a separately quantized scalar, improving bitrate-distortion performance and robustness to input gain variation.
Problem
Neural audio codecs (NACs) such as SoundStream, EnCodec, DAC, and SpeechTokenizer encode a signal’s short-term energy (gain) and normalized structure (shape) jointly within a single latent embedding space. The paper’s empirical analysis shows this entangles amplitude and content: two waveforms that differ only by a global gain factor are mapped to embedding vectors that are not merely rescaled but also rotated in direction, and this directional shift is enough to change the codeword selected by the quantizer for a large fraction of frames. Classical speech/audio codecs (CELP, AMR, Opus) avoided this problem decades ago through shape-gain decomposition, which separates a frame’s magnitude from its normalized direction before quantization. According to the authors, no prior NAC has incorporated this principle, leaving modern codecs both compute-inefficient (redundant codewords needed to cover the same shape at different loudness levels) and fragile to signal level changes not seen during training.
Method
The paper proceeds in two stages. First (§2), the authors quantify the problem: they feed gain-scaled copies of LibriSpeech speech (from -12 dB to +12 dB) through three off-the-shelf NACs (DAC, SpeechTokenizer, BigCodec) and measure how embedding norm, cosine similarity to the 0 dB reference, and discrete code stability change as a function of input gain. All three codecs show substantial directional drift in the embedding space and high sensitivity of codeword allocation, with 55-85% of frames receiving a different codeword under only a ±4 dB gain shift.
Second (§3), they propose The Equalizer, a general methodology applicable to any NAC. Each short-term frame of the input waveform is normalized to a fixed energy before being sent to the NAC encoder: the frame gain is computed as its L2 norm, the frame is divided by that gain to produce a normalized shape vector, and normalized frames are recombined via overlap-add (OLA) synthesis into a full “equalized” waveform with constant energy profile. This equalized waveform, not the raw signal, is what the NAC actually encodes. The temporal gain envelope is quantized separately with a non-linear scalar (mu-law) quantizer, exploiting the logarithmic nature of loudness perception. At the decoder, the reconstructed equalized waveform is de-equalized by multiplying each frame by its corresponding quantized gain and re-synthesized via OLA, restoring the original signal energy. In the reference implementation, the equalization and de-equalization steps are external pre/post-processing modules that operate on the full waveform rather than on the encoder’s internal input vectors, making the approach applicable to any NAC without needing access to its internals.
The underlying NAC is an EnCodec-based encoder-decoder with a convolutional front end whose original two-layer LSTM is replaced with a two-layer Bidirectional LSTM (following SpeechTokenizer), and residual vector quantization (RVQ) with 8 stages at 50 vectors/second. The NAC is retrained from scratch on equalized (i.e., energy-normalized) training data so that the encoder only has to represent shape.
Key Results
Models were trained on LibriSpeech-100 (16 kHz) and evaluated on the LibriSpeech test-clean partition (2,620 utterances, ~5.4 hours) using PESQ, STOI, and SI-SDR, with an ablated baseline that shares the identical architecture, training data, and bitrate configuration but skips the shape-gain decomposition step. Under gain variation, the ablated baseline shows the expected bell-shaped degradation peaking at 0 dB (e.g., a 6 dB gain shift drops PESQ from 2.61 to 2.45 at codebook size C=1024), while the Equalizer maintains near-invariant performance across the full ±12 dB range (§4.2, Figure 3).
Critically, the Equalizer also outperforms the baseline at 0 dB, i.e., with no train/test gain mismatch at all: at C=1024, PESQ=2.75 vs. 2.61, STOI=0.93 vs. 0.92, and SI-SDR=6.98 dB vs. 6.31 dB (§4.2). When results are expressed as a function of total bitrate (including the extra scalar-quantized gain bitrate, e.g., +0.4 kbps for 8-bit mu-law gain at 50 Hz), the Equalizer consistently dominates the baseline: at matched bitrate, SI-SDR reaches 5.46 dB vs. 4.40 dB at 3.2 kbps, and PESQ reaches 2.65 vs. 2.25 at 3.6 kbps (§4.2, Figure 4). The Equalizer at 3.2 kbps (C=128) is reported to roughly match the baseline at 4 kbps (C=1024), corresponding to an 8x reduction in codebook size and search complexity for comparable quality. The method also outperforms SpeechTokenizer (trained on 960h, nearly 10x more data than the Equalizer’s 100h) on all three metrics at comparable bitrates. A separate ablation varying RVQ depth (Nq in {1,2,4,6,8,16} at fixed C=1024) shows the Equalizer’s advantage is largest at low depths, where the baseline’s limited codebook capacity must simultaneously encode amplitude and shape (§4.2, Figure 5).
Novelty Assessment
The core idea, shape-gain decomposition, is not new to speech/audio coding; it underlies decades-old classical codecs such as CELP, AMR, and Opus. What is new is the systematic transplantation of this principle into the NAC framework, applied as pre/post-processing around an otherwise standard EnCodec-style encoder-decoder, together with an empirical demonstration that this entanglement is a real and previously undocumented problem in modern NACs. The contribution is primarily architectural/methodological: the decomposition mechanism itself, the retraining protocol on normalized data, and the ablation isolating its effect from confounds (identical architecture, data, and bitrate budget between baseline and proposed method) are the paper’s genuine additions. The underlying codec backbone (EnCodec + BiLSTM, RVQ) is not itself new. The authors are explicit that they are not trying to beat state-of-the-art NAC quality in absolute terms, but to demonstrate a general, architecture-agnostic technique.
Field Significance
moderate — This paper surfaces and names a previously undocumented failure mode of neural audio codecs (gain-shape entanglement causing codeword instability under signal level shifts) and provides a clean, well-ablated fix that is explicitly designed to be portable to any NAC. The validation is limited to a single backbone (EnCodec-derived) trained on a modest amount of clean read speech (100 hours), so the generality claim rests on the diagnostic analysis in Section 2 (run on DAC, SpeechTokenizer, and BigCodec) rather than on retraining the fix into those other architectures.
Claims
- supports: In neural audio codecs, a non-linear encoder can entangle a signal’s amplitude and structural content within a single embedding vector, making the encoder’s output direction sensitive to gain rather than purely representing normalized content.
Evidence: Across DAC, SpeechTokenizer, and BigCodec, cosine similarity between the embedding of a reference signal and its gain-scaled version drops substantially as gain deviates from 0 dB, with a fast exponential decay for DAC and a slower but deeper decrease for BigCodec (0.5 at -12 dB, 0.4 at +12 dB). (§2, Figure 1)
- supports: Explicitly decomposing an input signal into a separately quantized gain and a normalized shape vector before a codec’s encoder reduces vector-quantization codebook redundancy and improves rate-distortion performance relative to jointly encoding gain and shape.
Evidence: At matched bitrate, the shape-gain decomposed model achieves higher SI-SDR, PESQ, and STOI than an architecturally identical baseline without decomposition across all four tested codebook sizes (C=1024, 512, 256, 128), e.g. SI-SDR of 5.46 dB vs. 4.40 dB at 3.2 kbps. (§4.2, Figure 4)
- complicates: Codeword allocation in residual vector quantization can be highly sensitive to small input-level perturbations even when the underlying acoustic content is unchanged, which is not captured by evaluating codecs only at a single reference gain level.
Evidence: A gain variation of only ±4 dB was found to change the assigned codeword for 55-85% of frames (depending on the codec) in three off-the-shelf NACs tested with no gain-normalization mechanism. (§2, Figure 1, bottom)
- refines: The benefit of separating a signal’s gain from its structural representation before quantization, established for classical vector-quantization-based speech/audio codecs, also transfers to codecs whose encoder is a trained non-linear neural network rather than a fixed linear transform.
Evidence: The same shape-gain decomposition principle used in CELP, AMR, and Opus (linear transform plus VQ) is applied ahead of a trainable, non-linear EnCodec-style encoder and yields consistent bitrate-distortion gains after retraining on normalized data. (§3.1, §4.2)
Limitations and Open Questions
The evaluation is confined to a single codec backbone (an EnCodec variant with a BiLSTM modification) trained on 100 hours of clean, single-domain read speech (LibriSpeech-100); the diagnostic analysis of gain sensitivity in Section 2 covers three other NACs (DAC, SpeechTokenizer, BigCodec), but the proposed fix itself is not retrained into those architectures, so the portability claim is not directly demonstrated end-to-end. The equalization and de-equalization steps are implemented as external, non-streaming modules operating on the reconstructed waveform rather than on the actual internal vectors seen by the encoder/decoder; the authors note this is a proof-of-concept simplification and that a streaming, internally-integrated version is feasible but untested. The method is also validated on speech only, not the broader music/general-audio domain that some NACs target, and its interaction with more sophisticated quantizer designs (beyond plain RVQ) is left to future work.
Wiki Connections
- Neural Audio Codec — proposes a general shape-gain decomposition mechanism, applied as pre/post-processing around an EnCodec-style encoder-decoder, that improves bitrate-distortion performance and gain robustness relative to jointly encoding amplitude and structure in the codec’s latent space.