arXiv · 2026 · Preprint
Grötschla et al. (Amazon AGI) · → Paper · Demo: ✗ · Code: ?
Introduces a single neural audio codec that natively encodes and decodes mono, stereo, and 5.1 surround audio with one shared encoder-decoder parametrization and codebook set, instead of training separate codecs per channel configuration.
Problem
Neural audio codecs such as EnCodec and DAC are built around fixed channel architectures: a model trained for mono audio cannot process stereo or surround content, and vice versa. Applications that must handle multiple channel counts in the same pipeline (speech synthesis at mono, music at stereo, film audio at 5.1 surround) are forced to either train and maintain a separate codec per configuration, each with its own incompatible latent space, or pad every input up to the maximum channel count the application might see, which wastes compute on the common case of fewer channels. Prior multi-channel extensions have been limited to two channels and typically just widen the input/output convolutions of a mono design while processing all channels jointly through the rest of the network, which does not generalize to a variable number of channels or preserve a shared representation across configurations.
Method
VCNAC extends the DAC encoder-decoder architecture with a variable-channel processing pipeline built around three stages: (1) each input channel is processed independently through parallel, weight-shared convolutional streams; (2) the per-channel streams are fused into a single unified bottleneck representation before residual vector quantization (RVQ); and (3) the quantized representation is split back out into the number of channels required at decode time. Because streams are only instantiated for channels that actually exist in a given example, the model avoids the padding overhead of prior fixed-channel designs. Channel identity is preserved with learnable positional embeddings (initialized as small-magnitude orthogonal vectors) added at both the pre-fusion and post-split stages, since without them the fused representation would have no way to distinguish which channel a given stream belongs to.
To let the parallel channel streams exchange information before they are collapsed into the shared bottleneck, VCNAC inserts lightweight Transformer Audio AutoEncoder (TAAE) attention blocks that operate on temporally interleaved channel sequences, so a single attention layer can capture both temporal structure and cross-channel dependencies within a local sliding window.
The model operates on 48kHz audio through five convolutional layers (strides 2, 4, 5, 6, 8) for 1920x downsampling to a 25Hz frame rate, quantizing a 16-dimensional latent with 26 RVQ codebooks (16384 entries in the first codebook, 4096 in the rest) for a total bitrate of 7.85 kbit/s regardless of channel count. Training combines multi-scale mel-spectrogram reconstruction losses and DAC-style multi-scale discriminators (adversarial training) with channel-compatibility losses: for stereo, a mid/side decomposition (mid = L+R, side = L-R) applies reconstruction losses to the monophonic downmix and the spatial-difference channel separately; for surround, the same mid/side treatment is extended to front and rear channel pairs, and an ITU-R BS.775-4 standardized downmix is used to apply an additional reconstruction loss ensuring that a 6-channel encoding still sounds correct when rendered down to fewer channels. Because real 5.1 surround training data is scarce, the authors build a synthetic mixing framework that combines mono speech (populating the center channel, drawn from LibriTTS and LibriVox), stereo music and sound effects (front channels), and secondary stereo material with cross-channel bleed (rear channels), with a synthesized low-frequency-effects channel from filtered summed content.
Key Results
On single-channel speech reconstruction (LibriTTS test set), VCNAC reaches PESQ 4.16 and SI-SDR 11.3 dB at 7.9 kbit/s, outperforming DAC (PESQ 3.94, SI-SDR 10.2 dB at 8 kbit/s), EnCodec (PESQ 3.39, SI-SDR 6.8 dB at 12 kbit/s), Opus (PESQ 3.93 at 12 kbit/s), and SNAC (PESQ 2.25 at 0.98 kbit/s), while also beating its own ablations: a concatenation-based fusion baseline without split channel streams (PESQ 3.45) and a variant without cross-channel attention (PESQ 3.07). The same ordering holds on stereo music (FMA-small): VCNAC obtains the best SI-SDR (8.6 dB) and lowest mel/STFT distances of the compared codecs at 7.9 kbit/s. For 5.1 surround audio (four Creative Commons films), VCNAC achieves the best front-channel reconstruction quality among the tested codecs except for spatial-cue preservation, where EnCodec (the only other neural codec with native stereo support) is slightly better; VCNAC matches the perceptual quality of SNAC at roughly half its bitrate. A MUSHRA listening test with 10 audio-technology professionals, covering stereo music, front/rear surround channels, and stereo downmixes of surround content, confirms good perceptual quality across all three content types, including for the downmixed condition relevant to playback on limited-speaker devices.
Novelty Assessment
The core contribution is architectural: a parallel-stream, weight-shared encoder/decoder with a fuse-then-split bottleneck and cross-channel attention that lets one model natively handle a variable number of channels with a single shared codebook, rather than training separate per-configuration codecs or padding to a fixed maximum. The training recipe (mid/side and ITU-standard downmix losses for channel compatibility, synthetic surround data generation) is a necessary companion contribution given the scarcity of real multichannel training data, but is incremental relative to established DAC-style codec training practices. The paper’s own ablations (concatenation-fusion baseline, no-attention baseline) support that both the split-stream design and the cross-channel attention mechanism, not just the loss changes, are responsible for the reported gains.
Field Significance
moderate — VCNAC demonstrates that a single neural codec can natively span mono speech through 5.1 surround with one shared latent space and codebook set, which is directly relevant to any application (including speech-generation pipelines) that must move between channel configurations without swapping codecs. Its speech-specific evaluation is limited to reconstruction quality on LibriTTS; the paper does not train or evaluate a downstream generative language model on VCNAC’s codes, so the claimed benefit of unified codebooks for such models remains a stated motivation rather than a demonstrated result in this paper.
Claims
- supports: A neural audio codec can share a single encoder-decoder parametrization and codebook set across mono, stereo, and surround channel configurations, matching or exceeding fixed-channel codecs without training a separate model per configuration.
Evidence: VCNAC uses one checkpoint to reconstruct mono speech (PESQ 4.16), stereo music (SI-SDR 8.6 dB), and 5.1 surround content, outperforming fixed-channel baselines (DAC, EnCodec, Opus, SNAC) at a lower or comparable bitrate of 7.9 kbit/s across all three settings. (§IV.C, Tables I-III)
- supports: Explicit cross-channel information exchange before quantization is necessary for high-fidelity joint coding of multiple audio channels, beyond what independent per-channel processing with a shared bottleneck alone provides.
Evidence: Removing the inter-channel attention mechanism while keeping the split-stream fuse/split design drops speech PESQ from 4.16 to 3.07 and stereo SI-SDR from 8.6 to 8.4 dB, an ablation isolating the attention module’s contribution. (§IV.C, Table I, Table II)
- supports: Applying reconstruction losses on standardized downmixed representations (e.g., mid/side, ITU-R multichannel-to-stereo downmix) during training helps a codec preserve perceptual quality when multichannel content is rendered down to fewer output channels at inference.
Evidence: The MUSHRA study reports good perceptual quality for stereo downmixes of encoded 5.1 surround content, a condition the training explicitly targets via ITU-R BS.775-4 downmix losses. (§III.B, §IV.C, Figure 2)
- complicates: Synthetic surround training data built from simulated channel mixing can introduce loudness and level relationships between channels that do not match real multichannel content, degrading reconstruction accuracy specifically for the channels most affected by the mismatch.
Evidence: The authors attribute weak rear- and center-channel SI-SDR/SI-SNR values (e.g., VCNAC center-channel SI-SDR of -1.59 dB) to imperfect loudness balance in their synthetic surround simulation relative to the real Creative Commons movie test set, noting the simulated rear channels were comparatively louder than in the realistic test data. (§IV.C, Table III)
Limitations and Open Questions
The 5.1 surround training data is entirely synthetic, generated by randomly mixing mono speech, stereo music, and sound effects into a simulated 6-channel layout; the paper reports that loudness and level relationships in this simulation do not fully match the real movie test data used for evaluation, and identifies this as a likely source of the weaker center- and rear-channel reconstruction results.
The paper does not evaluate VCNAC’s codes in an actual downstream generative model (speech or audio language model); the claim that a unified codebook benefits such training is motivation, not a demonstrated result. Code availability is not stated, and audio samples are marked as “to be added for camera ready” rather than available at submission time, so neither can currently be verified. The MUSHRA listening test uses a small panel (10 audio-technology professionals) and only 12 short (7-second) samples per condition, which limits the statistical strength of the subjective quality claims. The LFE (low-frequency effects) channel evaluation is noted by the authors as unreliable due to its extremely sparse activity in the test data.
Wiki Connections
- Neural Audio Codec — proposes a variable-channel codec architecture that generalizes RVQ-based neural coding (as used in EnCodec, DAC, SNAC) from fixed mono/stereo designs to a single model spanning mono through 5.1 surround.
- GAN Vocoder — trains its encoder-decoder with DAC-style multi-scale adversarial discriminators alongside multi-scale mel reconstruction losses, following established GAN-based neural codec training practice.
- Subjective Evaluation — validates reconstruction quality with a MUSHRA listening test conducted by audio-technology professionals across music, surround, and downmixed content conditions.
- SpectroStream — both papers target general-purpose (not speech-only) neural audio coding; VCNAC specifically targets variable channel-count compatibility within one model, while SpectroStream focuses on versatile bitrate/quality scaling for general audio.