arXiv · 2025 · Preprint
Philipp Grundhuber et al. (Fraunhofer IIS / International Audio Laboratories Erlangen) · → Paper · Demo: ✓ · Code: ?
Extends a disentangled neural audio codec to an EnCodec backbone that separates speech content from acoustic-environment (room) characteristics, enabling “acoustic teleportation” — transplanting the reverberation profile of one recording onto the speech content of another.
Problem
Neural audio codecs typically encode speech into a single entangled latent space that mixes content, speaker identity, and acoustic environment (room reverberation) information, making it difficult to manipulate any one factor independently. Prior work on disentangling acoustic environment from speech content, based on SoundStream, achieved only limited output quality: dereverberated and acoustically transformed signals suffered from audible artifacts. This paper asks whether swapping the underlying codec architecture and extending the training task set can close that quality gap while preserving the disentanglement property.
Method
The system is built on an EnCodec-based neural audio codec operating at 16kHz with an encoder hop length of 320. The encoder output (128 dimensions per frame) is split into two 64-dimensional feature maps: one intended to capture speech content and one to capture acoustic (room) characteristics. Each stream is quantized independently by its own residual vector quantizer (RVQ) with its own codebook (size 1024), producing separate token streams for content and acoustics. This is a direct architectural extension of a prior disentanglement approach originally built on SoundStream, applied here to EnCodec’s convolutional encoder-decoder with adversarial (GAN) training instead.
Disentanglement is induced purely through the training task design rather than an explicit disentanglement loss. Training combines five tasks organized into four categories: clean reconstruction (no embedding manipulation), reverb reconstruction (no embedding manipulation), dereverberation (acoustic tokens zeroed out), and two acoustic-teleportation variants — swapping acoustic embeddings between recordings of the same underlying source (same-source, AT-SS) or between recordings from different sources (different-source, AT-DS). Reverberant speech is synthesized by convolving anechoic DNS5 utterances with room impulse responses (RIRs) from GWAsmall, using a balanced two-RIR-per-sample scheme (one low-RT60, one high-RT60 room) to make the teleportation task explicit. Training follows FunCodec’s recipe (60 epochs, 8×A100 GPUs), with the reconstruction and multi-spectral loss weights reduced from 1.0 to 0.1 to rebalance the adversarial discriminator against the added task complexity.
At inference, dereverberation is performed by zeroing the acoustic tokens, and acoustic teleportation is performed by encoding two recordings, extracting their respective acoustic embeddings, and decoding each speech stream with the other’s acoustic embedding.
Key Results
The proposed models outperform the prior disentanglement baseline (Omran et al., built on SoundStream) on non-intrusive ScoreQ for acoustic teleportation: the best quantized configuration (trained on the “Omran taskset” of reverb reconstruction, dereverberation, and different-source teleportation) reaches ScoreQ NR 3.03, versus 2.44 for the baseline, at a comparable bitrate (Table 2). Non-quantized task-specific reference networks establish upper bounds: clean reconstruction reaches ScoreQ NR 4.37 / ViSQOL 4.41, while dereverberation is markedly harder (ScoreQ NR 3.75, ViSQOL 2.67), reflecting the difficulty of removing reverberation compared to simply reconstructing it.
Ablating the training task set shows a trade-off: training on acoustic-teleportation tasks only maximizes teleportation quality (ScoreQ NR 2.95, ViSQOL 3.02) but degrades clean reconstruction (ScoreQ NR 3.96 vs. 4.14 for the all-tasks model), while training on all five tasks together yields moderate but non-specialized performance across the board. Quantizer count shows diminishing returns beyond 8 quantizers (ScoreQ NR 3.82 at N=8 vs. 3.88 at N=16), and a persistent quality gap remains between quantized and non-quantized models even at N=16 (3.88 vs. 4.12), indicating the discrete bottleneck itself limits disentangled representation quality. Temporal downsampling of the acoustic embedding, tested as a way to reduce its bitrate, significantly degrades ViSQOL across all tasks even at a 2× downsampling factor (p < 0.01, two-sample t-tests). The learned acoustic embeddings correlate strongly with RT60 (Pearson correlation up to 0.93 in some configurations), and t-SNE clustering shows acoustic embeddings group by room while speech embeddings group by speaker, evidencing effective (though imperfect) disentanglement.
Novelty Assessment
The core disentanglement idea, splitting a codec’s latent space into content and acoustic-environment partitions, is not new; it is taken directly from Omran et al.’s SoundStream-based approach. This paper’s contribution is to re-implement that idea on the EnCodec architecture and to systematically ablate the training task combination and embedding downsampling factor, which the original work did not do. The resulting quality gains (ScoreQ NR improving from 2.44 to 3.03) are best characterized as an engineering-integration result: a stronger codec backbone plus a more thorough training recipe, rather than a structurally new disentanglement mechanism. The systematic ablations (task-set composition, quantizer count, downsampling factor, RT60 correlation) constitute the paper’s most useful empirical contribution, providing design guidance for future disentangled-codec work even though the underlying architecture is not novel.
Field Significance
Low — the paper provides a focused empirical study of how training-task composition and quantization choices affect the quality of environment/content disentanglement in a neural codec, and demonstrates that even modest temporal downsampling of a specialized embedding stream measurably hurts reconstruction quality. Its scope is narrow (room-acoustic transfer, single architecture, simulated reverberation only), and it advances an existing method rather than introducing a new one.
Claims
- supports: Partitioning a neural codec’s latent space into task-specific streams, combined with training on reconstruction and swap-based tasks, can separate speech content from acoustic-environment information without an explicit disentanglement loss.
Evidence: t-SNE clustering of extracted embeddings shows acoustic embeddings cluster by room (speaker-invariant) while speech embeddings cluster by speaker (room-invariant), confirmed across 10-room and 10-speaker evaluation sets. (§4.5, Figure 4)
- complicates: Reducing the temporal resolution of a specialized latent stream to save bitrate degrades codec reconstruction quality even at small compression factors.
Evidence: Two-sample t-tests show that downsampling the acoustic embedding by a factor of 2 already produces a statistically significant drop in ViSQOL across all evaluation tasks (p < 0.01), with degradation increasing monotonically up to a factor of 120. (§4.3, Figure 1)
- complicates: Training a single codec on multiple competing reconstruction and manipulation objectives trades off peak performance on any individual task against balanced performance across tasks.
Evidence: A model trained only on acoustic-teleportation tasks achieves the best teleportation ScoreQ NR (2.95) but the worst clean-reconstruction ScoreQ NR (3.96) among the ablated configurations, while the all-tasks model is moderate but not best on any single task. (§4.1, Table 2)
- refines: Vector quantization of disentangled latent streams introduces a residual quality gap relative to continuous (non-quantized) representations that does not close by adding more quantizers.
Evidence: Increasing quantizers from 8 to 16 yields only a marginal ScoreQ NR gain (3.82 to 3.88), while the non-quantized reference model still scores 4.12, a gap that persists at the highest tested bitrate. (§4.1, Table 2)
- complicates: Transferring acoustic-environment characteristics between recordings via swapped codec embeddings degrades in quality as the acoustic mismatch between source and target environments grows.
Evidence: Teleportation quality (measured by ScoreQ NR) correlates negatively with the absolute RT60 difference between the source and target rooms (Pearson correlation -0.61), a relationship that holds across all ablated model configurations. (§4.4, Figure 3)
Limitations and Open Questions
Warning
Evaluation is restricted to English read speech and simulated room impulse responses with mean RT60 under 1.2s, with no background noise and no real-world recordings tested; the authors state this limits generalizability to real-world acoustic environments.
Beyond the evaluated domain, the authors note that teleportation quality degrades substantially once the RT60 difference between source and target rooms exceeds 0.8s, indicating a fundamental limit on how large an acoustic transformation the model can perform reliably. The paper also does not evaluate whether speaker identity is fully preserved under acoustic teleportation (only qualitative t-SNE evidence of speaker-embedding independence is given), and does not test multilingual speech or non-speech audio content, which the authors flag as future work.
Wiki Connections
- Neural Audio Codec — extends an EnCodec-based codec with a partitioned dual-RVQ latent space that separates speech content from acoustic-environment tokens.
- Disentanglement — trains a codec to disentangle speech content from room-acoustic characteristics using task-based supervision (reconstruction, dereverberation, and embedding-swap objectives) rather than an explicit disentanglement loss.