Soft Disentanglement in Frequency Bands for Neural Audio Codecs

arXiv · 2025 · Preprint

Ginies et al. (Télécom Paris) · → Paper · Demo: ? · Code: ?

Introduces a two-branch cascade neural audio codec in which a higher-sample-rate branch encodes only the residual left after a lower-sample-rate branch reconstructs the shared low-frequency band, producing frequency-disentangled discrete tokens without hard spectral sub-band splitting.

Problem

Neural audio codecs built on encoder-decoder-quantizer architectures compress audio effectively but produce features that carry little semantic structure, since nothing in the training objective encourages interpretable representations. Prior efforts to add structure have relied on task- and data-specific disentanglement: separating content, pitch, and speaker identity in speech synthesis; isolating speech from background noise or other sources in mixtures; or splitting pitch and amplitude for singing voice conversion. These methods require specialized supervision signals tied to the target task. The one existing generic, task-agnostic approach (Luo et al.’s GULL) splits the input spectrogram into hard frequency sub-bands that must be reconstructed independently and then recombined, which risks discontinuity artifacts at the band boundaries. This paper asks whether a codec can achieve frequency disentanglement without hard splitting or task-specific supervision.

Method

The proposed codec replicates the Descript Audio Codec (DAC) architecture, a convolutional encoder-decoder with a Residual Vector Quantizer (RVQ), independently for two branches operating at different native sampling rates: 16 kHz (covering the [0-8 kHz] band) and 32 kHz (covering [0-16 kHz]). Rather than splitting a shared spectrogram into disjoint sub-bands, the model exploits the link between sampling rate and spectral content directly in the time domain. The 16 kHz branch encodes and reconstructs its input signal directly. The 32 kHz branch instead receives the residual between the true 32 kHz signal and the upsampled 16 kHz reconstruction, and its own decoded output is summed with the upsampled low-band reconstruction to produce the final 32 kHz output.

Because nothing forces the 32 kHz branch to encode only content above 8 kHz, the separation between branches is “soft”: the higher-rate branch can still contribute corrective information to the shared low-frequency region, which the authors argue reduces artifacts at the crossover frequency relative to hard sub-band splitting. Each branch is scaled to keep the ratio between sampling rate and compression rate constant, and uses four RVQ codebooks, giving a 2 kbps bitrate for the 16 kHz branch and 4 kbps for the 32 kHz branch (both at compression rate 128). Training proceeds in cascade: the 16 kHz branch is trained first and frozen, then the 32 kHz branch is trained on its residual input, and finally all weights are jointly fine-tuned. Each branch has its own discriminator and is trained with DAC’s standard loss suite (adversarial generative loss, feature-matching loss, multi-scale mel loss, codebook loss, commitment loss); during fine-tuning, the reconstruction-related losses (generative, feature-matching, mel) are averaged across branches while the non-overlapping codebook and commitment losses are summed. A DAC baseline is retrained from scratch at matching sampling rates, compression rates, and bitrates for a controlled comparison. Model size is not reported.

Key Results

On held-out MUSDB18/Jamendo test tracks, the proposed model improves on the matched DAC baseline across every reconstruction metric at both operating points: mel loss 0.95 vs. 0.90 (baseline 1.08) at 16 kHz and 0.80 vs. 0.90 at 32 kHz, with corresponding gains in STFT loss, waveform loss, SI-SDR, and ViSQOL (Table I). A 13-participant MUSHRA listening test confirms the direction of these gains: the proposed model scores 53±20 (16 kHz) and 66±19 (32 kHz) against a reference of 95±9/96±5, versus 31±18/49±20 for the retrained DAC baseline and 37±25/38±24 for a 3.5 kHz low-pass degraded anchor (Table II).

To verify disentanglement directly, the authors compute SDR between the 32 kHz branch’s output and the true signal, split by frequency band: 5.85 dB in the [8-16 kHz] band it is intended to cover, versus 3.80 dB in the [0-8 kHz] band it also touches (Table III). SDR measured in a narrow window around the 8 kHz crossover is nearly identical to the overall SDR (5.61 dB vs. 5.67 dB), indicating the soft decomposition does not introduce a discontinuity artifact at the band boundary. In a toy bandwidth-inpainting experiment, feeding the 32 kHz branch a residual signal containing no information above 8 kHz still yields a small positive SDR (Table IV) in the missing high band, and the resulting reconstruction improves mel loss (2.04 vs. 2.56), STFT loss (5.37 vs. 8.65), and SI-SDR (6.22 dB vs. 3.75 dB) relative to simply upsampling the 16 kHz output (Table V).

Novelty Assessment

The core contribution is compositional rather than a new loss function or quantization scheme: both branches reuse DAC’s encoder-decoder-RVQ architecture and its full adversarial/mel/codebook loss suite unmodified. What is new is the cascade-residual connection between branches operating at different native sampling rates, which replaces hard spectrogram sub-band splitting with a softer, artifact-avoiding decomposition. The authors explicitly frame this as a two-branch “proof of concept” extensible to more bands, and describe their own gains as “modest” and “marginal.” All experiments are conducted on music (MUSDB18, Jamendo) even though the paper motivates its approach by citing speech-domain disentanglement work (content/speaker/prosody separation, codec-based zero-shot TTS); no speech-domain validation is provided. The MUSHRA test uses a small, non-expert (mostly audio-researcher) panel of 13 listeners.

Field Significance

Low — this paper proposes a genuinely different mechanism for frequency-based disentanglement in neural codecs (cascade residual branches across sampling rates) rather than hard spectral sub-band splitting, and provides direct SDR evidence that the mechanism avoids boundary artifacts. Its scope is narrow: two branches, music-only data, and reconstruction gains the authors themselves characterize as modest, so it reads as an early proof of concept rather than a result that reshapes codec design practice.

Claims

• supports: A cascade of codec branches operating at different native sampling rates, where each higher-rate branch encodes only the residual left by lower-rate branches, can achieve frequency disentanglement without the reconstruction-and-recombination artifacts of hard spectral sub-band splitting.

  Evidence: The 32 kHz branch’s output correlates far more strongly with the true signal in its target [8-16 kHz] band (5.85 dB SDR) than in the [0-8 kHz] band it also touches (3.80 dB), and SDR measured near the 8 kHz crossover matches the overall SDR (5.61 dB vs. 5.67 dB), indicating no discontinuity artifact at the band boundary. (§V, Table III)

• complicates: Frequency-cascade disentanglement in neural codecs can improve reconstruction and perceptual quality over a matched non-disentangled baseline, but absolute perceptual quality still trails the reference signal by a wide margin.

  Evidence: MUSHRA scores for the proposed model reach only 53±20 (16 kHz) and 66±19 (32 kHz) against a reference of 95±9/96±5, an improvement over the retrained DAC baseline (31±18/49±20) but far from reference quality. (§IV.D, Table II)

• supports: A codec trained with residual, band-limited conditioning between branches can extrapolate plausible high-frequency content from low-frequency-only input, offering a route to bandwidth extension without task-specific supervision.

  Evidence: Given a residual input containing no information above 8 kHz, the 32 kHz branch still achieves a small positive SDR in the missing [8-16 kHz] band and improves overall reconstruction metrics (mel loss 2.04 vs. 2.56, SI-SDR 6.22 dB vs. 3.75 dB) relative to naive upsampling of the low-rate reconstruction. (§VI, Table IV, Table V)

• complicates: Generic, task-agnostic frequency-disentanglement methods for neural codecs are often validated only on music, leaving their applicability to the speech-specific disentanglement goals (content, speaker, prosody separation) that motivate them empirically untested.

  Evidence: All training and evaluation data are drawn from MUSDB18 and Jamendo (music only), even though the paper’s introduction and related work motivate the approach by referencing speech disentanglement and codec-based zero-shot TTS methods. (§IV.A)

Limitations and Open Questions

Warning

No code, model checkpoints, or audio samples are released, and the method is validated only as a two-branch proof of concept on music data (MUSDB18, Jamendo); reproducibility and generalization to speech or to additional frequency bands remain unverified.

The MUSHRA evaluation uses only 13 non-expert (mostly audio-researcher) listeners across two small sets of six excerpts, a modest sample for perceptual claims. The bandwidth-inpainting experiment is explicitly described as a “toy” demonstration without comparison to dedicated bandwidth-extension baselines. Model size is not reported, and all loss functions and discriminator design are inherited unmodified from DAC, so the contribution rests entirely on the cascade-residual composition rather than any new training objective.

Wiki Connections

• Neural Audio Codec — extends the discrete RVQ-based codec paradigm with a cascade of branches at different native sampling rates to achieve frequency disentanglement without hard sub-band splitting.
• GAN Vocoder — trains each codec branch with the same adversarial generator/discriminator and feature-matching loss design used in GAN-based codec and vocoder training.
• Subjective Evaluation — validates the perceptual quality gains of the proposed codec with a MUSHRA listening test against a retrained DAC baseline and a low-pass degraded anchor.
• VALL-E — cited as an example of the broader applications discrete codec representations enable, including codec-based zero-shot speech synthesis.
• NaturalSpeech 3 — cited as prior task-specific disentanglement work (prosody, context, and acoustic detail via dedicated VQ modules) that this paper’s generic, frequency-based approach contrasts with.