arXiv · 2026 · Preprint

Seetharaman et al. (Adobe Research) · → Paper · Demo: ✗ · Code: ✗

Replaces the hand-tuned KL weight in continuous audio VAEs with a squared-error loss that regresses the KL term to a target value, so each VAE is trained at a chosen theoretical bitrate and can be placed on the same rate-distortion plot as discrete codecs; sweeping that bitrate then reveals which latent space works best for downstream latent diffusion (text-to-sound and zero-shot TTS).

Problem

Latent diffusion pipelines for audio depend on a continuous autoencoder whose regularization strength is set by a KL weight λ. Large λ over-regularizes the latent and caps reconstruction quality; small λ gives sharp, perturbation-sensitive latents that are hard for the generative model to predict. Because λ has no direct interpretation, it is tuned by hand, and in practice most audio VAEs end up regularized mainly through latent dimensionality. Discrete neural codecs, by contrast, have a well-studied rate-distortion picture because bitrate follows directly from codebook size and count. The paper (accepted at ICASSP 2026) aims to give continuous VAEs an equivalent, controllable notion of rate.

Method

The core observation is that the expected KL term of a Gaussian VAE with an N(0, I) prior is the coding cost (in nats) of encoding posterior samples using the prior as codebook. Multiplying by the latent frame rate S and dividing by log 2 gives a theoretical bitrate, so a target bitrate B maps to a target KL value KL_target = B · log 2 / S. Training then minimizes (KL − KL_target)² instead of λ · KL, with both quantities normalized by latent dimension D because regressing the raw high-magnitude KL was hard to optimize. For B = 7.68 kbps, S = 40 Hz and D = 128 the target is about 276. The authors relate this to free bits, which only lower-bounds per-dimension KL, and report that direct regression to the target worked better in practice (no table is given for that comparison). The target-KL loss is itself weighted (λ = 1, 2, 10), and different weights produce different adherence to the target, so measured bitrates scatter around the targets.

The autoencoder (DAC-VAE) reuses the fully convolutional encoder-decoder and GAN training recipe of DAC (Descript Audio Codec), replacing the residual vector quantizer with a Gaussian bottleneck. It operates on 48 kHz audio with a 40 Hz latent. Three modifications are added: an 80-bin mel-spectrogram projection added to the encoder output (for faster convergence), the CQT discriminator from BigVGAN v2 in place of the multi-band spectrogram discriminator, and “passthrough” training in which a fraction of batches bypass the bottleneck as a plain autoencoder (25% is used for all main models). DAC-VAEs are trained with KL targets of 0, 80, 160, 320, 640 and 1280 for 250k steps at batch 128 on an internal proprietary and licensed mix of speech, music and sound effects. A continuous variant of SpectroStream (targets 150 and 400) and a discrete DAC are also trained on the same data; EnCodec and the Stable Audio Open VAE are evaluated from public checkpoints.

Two downstream latent diffusion models use v-prediction with a shifted cosine noise schedule, and only the VAE varies across runs. The TTS model follows DiTTo-TTS: a 740M-parameter decoder-only diffusion transformer (d_model 1536, 12 heads) that takes T5-Large text embeddings, IPA phonemes, a speaker prompt latent and the noised latent as one concatenated sequence. It is trained for 300k steps on English subsets of CommonVoice, LibriVox and Emilia-YODAS (Emilia). The text-to-sound model is a 1B-parameter, 24-layer transformer that cross-attends to T5-XXL embeddings and is trained for 400k steps on proprietary sound-effects data.

Key Results

Reconstruction (Figure 1, AudioSet eval subset). Plotting mel-loss reconstruction error against KL/bitrate puts DAC-VAE on the lower envelope across the full range. Its error drops from about 1.9 at the lowest rate to about 0.56 at roughly 74 kbps (values read from the plot, not tabulated). Discrete RVQ models (DAC, EnCodec), the Stable Audio VAE and continuous SpectroStream all sit above the DAC-VAE curve at similar bitrates. SpectroStream comes close at low bitrate but does not keep pace at higher rates, and discrete DAC improves less with bitrate than its continuous counterpart. One caveat: the discrete DAC uses quantizer dropout to cover several bitrates with a single model, while each DAC-VAE point is a separately trained model. Reconstruction is reported only on general audio (AudioSet); there is no speech-only reconstruction test set or speech-specific reconstruction metric.

Ablation (Table 1, target ~20 kbps). The CQT discriminator plus 25% passthrough lowers mel distance from 0.626 (base) to 0.604 at a nearly identical measured bitrate (19.54 vs 19.69 kbps). Passthrough alone at 50% gives no gain (0.627).

Text-to-sound (Table 2, 250 Adobe Audition SFX prompts, FLAM-based metrics). The 11.56 kbps VAE (KL 200.39) is best on every metric: text-audio similarity 70.67, KAD 1.70, FAD 0.11 (FAD ties with the 7.65 kbps model). Quality is worse at both lower and higher rates. For example, the 74.10 kbps VAE gets similarity 66.84 and KAD 2.16, and the 19.69 kbps VAE has the worst KAD at 2.28. The result is an interior optimum, not a monotonic trend. The Stable Audio Open VAE (4.74 kbps), which comes with a different generator pipeline, scores 68.38 / 2.13 / 0.13.

TTS (Table 3, Seed-TTS test-en). WER (Whisper large-v3) ranges from 1.61 to 1.98 and WavLM speaker similarity from 0.66 to 0.68 across the five VAEs. The lowest-bitrate VAE (7.65 kbps) ties for the best WER (1.61) and, with the 11.56 kbps VAE, for the best similarity (0.68). The 74.10 kbps VAE matches that WER (1.61), but the authors’ informal listening found its output less natural and more monotonous; no MOS or other human rating is reported. The mid-range 19.69 kbps VAE is worst on WER (1.98). The authors attribute the non-monotonic pattern partly to the TTS model copying information from less-regularized prompt latents. Across both tasks they recommend about 11.56 kbps as a reasonable starting point, although on the TTS table alone the 7.65 kbps VAE is at least as good.

Novelty Assessment

The architectural change is a small one. DAC-VAE is DAC with the quantizer swapped for a Gaussian bottleneck, plus off-the-shelf tweaks (a BigVGAN v2 discriminator and passthrough training from prior work). The new idea is the target-KL objective together with the reading of VAE KL as a theoretical bitrate. That reading puts continuous and discrete audio autoencoders on one rate-distortion axis and replaces the uninterpretable λ with a target expressed in kbps. The rate-distortion view of VAEs is established in the general ML literature, which the paper cites, so the contribution is applying it to audio VAEs and making it a training target, not new theory. The evidence is limited by the 5-page format. Reconstruction is a single mel-loss metric on one general-audio set. Downstream results cover one model size per task, with no variance estimates and no human evaluation. The free-bits comparison is described but not shown, and the “bitrate” is a theoretical proxy, not an actual entropy-coded rate.

Field Significance

Moderate. Target-KL regularization gives speech and audio latent diffusion work a way to choose and report VAE regularization as a target bitrate, and to compare continuous latents directly against discrete codecs. The downstream sweeps show that the bitrate that reconstructs best is not the one that generates best, and that the best bitrate differs between text-to-sound and TTS. The TTS evidence is thin: WER and speaker-similarity differences across bitrates are small, and naturalness is not measured.

Claims

  • supports: Continuous latent bottlenecks scale reconstruction quality with bitrate more effectively than residual vector quantization in otherwise matched audio autoencoders.

    Evidence: On AudioSet, the Gaussian-bottleneck DAC-VAE lies below discrete DAC and EnCodec on the mel-loss vs bitrate curve at every rate tested. The authors note that discrete DAC uses quantizer dropout for multiple rates while each VAE is trained separately. (§4, Figure 1)

  • refines: Better autoencoder reconstruction does not translate into better latent diffusion generation; downstream quality peaks at an intermediate latent bitrate.

    Evidence: For text-to-sound, the 11.56 kbps VAE is best on text-audio similarity (70.67) and KAD (1.70), while both the 7.65 kbps and 74.10 kbps VAEs score worse, even though reconstruction error falls steadily with bitrate. (§4, Table 2, Figure 1)

  • complicates: The latent bitrate that suits zero-shot TTS can differ from the one that suits general audio generation, because less-regularized prompt latents let the generator copy information from the prompt.

    Evidence: On Seed-TTS test-en, the lowest-bitrate VAE (7.65 kbps) ties for the best WER (1.61) and, with the 11.56 kbps VAE, for the best speaker similarity (0.68). A high-bitrate VAE (74.10 kbps) matches that WER but sounded less natural and more monotonous on informal inspection, which the authors link to prompt-latent copying. (§4, Table 3)

  • complicates: WER and speaker similarity alone can miss naturalness and prosody differences between TTS systems built on different latent spaces.

    Evidence: The authors report that a high-bitrate VAE with low WER produced monotonous speech that the automatic metrics did not capture, and call for metrics that better reflect prosody and naturalness. (§4)

  • supports: Regressing the VAE KL term to a target value makes the latent bitrate a controllable training hyperparameter, replacing manual tuning of the KL weight.

    Evidence: With targets of 80 to 1280 (normalized), measured KL lands near the target (e.g. 132.63, 200.39, 341.26, 642.35, 1284.21), with the spread depending on the target-KL loss weight λ ∈ {1, 2, 10}. (§2, §3.1, Table 2)

Limitations and Open Questions

  • Reconstruction is evaluated only with mel loss on the AudioSet evaluation subset, which mixes speech, music and sound effects. There is no speech-only reconstruction test, and no PESQ, STOI, ViSQOL or listening test, so speech-specific codec quality is not directly measured.
  • TTS evidence consists of one model size and one English test set, with no human ratings. The between-bitrate differences in WER (1.61 to 1.98) and speaker similarity (0.66 to 0.68) are small, and no variance is reported. The naturalness concern rests on informal inspection.
  • The bitrate is a theoretical proxy derived from KL, not the size of an actually entropy-coded bitstream, so the comparison with discrete codecs mixes a realized rate with a bound.
  • The VAE and text-to-sound training data are proprietary, and no code or checkpoints are released, which limits reproducibility.
  • The paper says regressing directly to a target KL works better than free bits but gives no quantitative comparison.
  • The paper leaves open how model size, latent frame rate and latent dimension interact with a fixed bitrate budget, and how target-KL combines with semantic-alignment regularizers. The authors list these as future work.

Wiki Connections

  • Neural Audio Codec — Places continuous VAEs and discrete RVQ codecs on one rate-distortion curve and proposes DAC-VAE, a continuous-bottleneck variant of DAC.
  • Diffusion TTS — Trains a DiTTo-TTS-style latent diffusion transformer on each target-bitrate VAE to study how latent regularization affects TTS.
  • Zero-Shot TTS — Its TTS model is prompted with speaker latents and evaluated on Seed-TTS test-en, and prompt-latent copying is raised as a confound for high-bitrate latents.
  • GAN Vocoder — DAC-VAE keeps DAC’s adversarial waveform-decoder training and swaps in the BigVGAN v2 CQT discriminator.
  • EnCodec — Evaluated from its public checkpoint as a discrete baseline on the rate-distortion plot, where it falls above the DAC-VAE curve.
  • SpectroStream — Retrained as a continuous VAE at two KL targets; it comes close to DAC-VAE at low bitrate but scales worse.
  • DiTTo-TTS — Its diffusion-transformer TTS design is the template for the downstream TTS model, with phonemes and T5 embeddings in place of ByT5.
  • BigVGAN — Source of the CQT discriminator that replaces DAC’s multi-band spectrogram discriminator in the ablation.
  • Seed-TTS — Its English test set is the TTS evaluation benchmark.
  • Emilia — The English Emilia-YODAS subset is part of the TTS training data.