arXiv · 2025 · Preprint
Andong Li et al. · → Paper · Demo: ✓ · Code: ✓
Reframes the neural vocoder task as an audio restoration problem, using a Schrödinger Bridge to transform a degraded range-space surrogate of the Mel-spectrum directly into the target spectrum instead of denoising from Gaussian noise, and pairs this with a subband-aware convolutional network to reach state-of-the-art vocoding quality in as few as 4 (or even 1, distilled) sampling steps.
Problem
Diffusion-based neural vocoders (DDPM- and flow-matching-based) push waveform reconstruction quality higher than GAN vocoders, but they inherit a performance-inference dilemma: high fidelity requires a relatively large number of iterative sampling steps (typically ≥5), and achieving competitive quality at few steps usually forces larger models and more compute. The paper attributes this to two design choices shared by essentially all existing diffusion vocoders. First, they follow a noise-to-data (N2D) paradigm, starting the reverse process from pure Gaussian noise and using the Mel-spectrum only as a side condition, even though reconstructing a target from unstructured noise is intrinsically harder than necessary. Second, their network backbones (e.g. NCSN++, borrowed from image generation) largely ignore the fact that time-frequency (T-F) audio spectra have strong, exploitable hierarchical structure across frequency subbands, instead relying on generic, compute-heavy structures for implicit modeling. The paper sets out to address both: shorten the generative trajectory by starting closer to the target, and build a network that directly exploits acoustic T-F priors.
Method
The authors first formalize the vocoder task through range-null decomposition (RND), a classical signal-restoration lens: given a linear degradation model, any signal can be split into a range-space (RS) component recoverable from the observation and a null-space (NS) component that must be generated. Because the Mel-spectrum is produced from the linear spectrum by a linear compression matrix (the Mel filterbank) plus loss of phase, the authors show it can be treated as noise-free linear degradation, and derive a “range-space spectral” (RSS) surrogate of the target complex spectrum by applying the pseudo-inverse of the Mel filterbank and zero-initializing phase. A rank-based analysis (§III-A3, Table I) compares this Mel-derived degradation to four other common acoustic degradations (noise, reverberation, low-pass filtering, audio codec compression) on the VoiceBank+DEMAND benchmark, and shows the Mel-spectrum’s rank-reduction behavior is most similar to low-pass filtering and audio-codec compression: this motivates treating vocoding as a specialized case of audio restoration rather than pure generation from noise.
Building on this framing, the paper adopts a Schrödinger Bridge (ScB) diffusion formulation in which the RSS surrogate and the target complex spectrum are the two boundary distributions of a stochastic forward-reverse trajectory (a data-to-data, D2D, paradigm), rather than a Gaussian prior and a target (N2D). Because the two endpoints are already acoustically close, the generative trajectory is substantially shorter than in conventional diffusion vocoders. Three noise schedules (gmax, variance-preserving, variance-exploding) and both SDE and ODE reverse samplers are supported.
For the data-prediction network, the paper replaces the image-derived NCSN++ backbone used in its own IJCAI 2025 precursor with a new subband-aware convolutional diffusion network, BCD, designed around three modules: a Convolutional-Style Subband-Division (CSBD) block that splits the T-F input unevenly across frequency regions (finer resolution at low/mid frequencies where harmonic content concentrates); a Large-Kernel Convolutional Attention Module (LKCAM), a stack of blocks that replace self-attention with large-kernel depthwise convolutions (modulated by a lightweight AdaLN-SOLA timestep conditioning) to model inter-frame and inter-band dependencies efficiently; and a Convolutional-Style Subband-Merge (CSBM) block that reconstructs the full-band spectrum from the processed subbands.
Training combines a data-prediction L2 loss on the target spectrum, a multi-resolution Mel L1 loss, and adversarial plus feature-matching losses using multi-period and multi-resolution STFT discriminators, matching the standard GAN-vocoder recipe. Inference iterates the reverse SDE/ODE update rule starting from the RSS surrogate (rather than Gaussian noise) and converts the final spectrum to a waveform via inverse STFT.
For low-latency deployment, the paper distills a single-step student model, BridgeVoC*, from a pretrained multi-step teacher via a deterministic ODE mapping. A naive L2 distillation loss on the student/teacher real-imaginary spectral outputs produces severe spectral distortion, traced to phase-wrapping ambiguity in the complex representation. To address this, the paper proposes an omnidirectional distillation loss that applies nine fixed 3×3 directional convolution kernels to the phase component before computing the distillation loss, generalizing prior group-delay/instantaneous-frequency phase-optimization tricks to all neighboring T-F bins. This is combined with Mel, adversarial, feature-matching, and a bijective consistency-preservation loss (round-trip mappings through student and teacher in both directions) that further stabilizes semantic content.
The BCD network has 7.65M trainable parameters (versus 16.8M/64.8M for two NCSN++ variants used for comparison), and the system operates directly on Mel/STFT representations without a neural audio codec.
Key Results
On LibriTTS (Table IX) with 4 sampling steps, BridgeVoC reaches PESQ 4.427, UTMOS 3.746, M-STFT 0.708, MCD 1.534, VISQOL 4.963, and WER 6.64, outperforming or matching strong flow-matching baselines evaluated at more steps (PeriodWave, 16 steps: PESQ 4.240; RFWave, 10 steps: PESQ 4.251) and GAN vocoders (BigVGAN, 112M params: PESQ 3.991), while using far fewer parameters (7.65M vs. 112.39M for BigVGAN) and substantially less compute (171.68 GMACs vs. up to ~4987 GMACs for PeriodWave). WER is the one metric where BridgeVoC does not lead, which the authors interpret as evidence that acoustic-reconstruction fidelity and ASR-measured semantic preservation are not perfectly correlated. Similar trends hold on LJSpeech (Table X: PESQ 4.385, UTMOS 4.302).
Ablations (Table VI) show the BCD network matches or exceeds the larger 64.8M-parameter NCSN++ variant while using only 4.38% of its MACs at NFE=4, confirming the value of subband-prior modeling over a generic image-diffusion backbone. Comparing diffusion schemes directly (Fig. 15), the Schrödinger Bridge outperforms DDPM, standard SDE diffusion, ResShift, and Rectified Flow Matching (RFM) under matched settings, and a D2D variant of RFM (started from the RSS surrogate rather than noise) outperforms the standard N2D RFM, especially at low step counts, supporting the generality of the restoration/D2D framing beyond the paper’s own architecture.
On out-of-distribution data (Table XI: EARS and AISHELL-3, unseen at training time), BridgeVoC leads by a wide margin on M-STFT, PESQ, and MCD, though it is not always best on UTMOS or CER, and the authors flag a large UTMOS gap between English and Mandarin OOD sets as a limitation of UTMOS for cross-lingual quality prediction.
The single-step distilled variant, BridgeVoC* (NFE=1), reaches PESQ 4.351 and UTMOS 3.635, close to the 4-step teacher and clearly ahead of the single-step distilled version of the strongest flow-matching baseline, WaveFM* (PESQ 3.559), while running at 13.91×/128× real time on CPU/GPU respectively. Subjective evaluation (24 trained listeners, similarity-MOS and AB preference tests on out-of-distribution AISHELL-3 and MUSDB18 audio) shows consistent human preference for BridgeVoC over both GAN- and flow-matching-based baselines; notably, WaveFM scores competitively on objective metrics but receives markedly lower listener ratings, attributed by participants to audible high-frequency artifacts not penalized by the objective metrics used.
Novelty Assessment
The central conceptual move, casting Mel-to-waveform vocoding as a restoration problem via range-null decomposition and using rank analysis to justify treating the Mel-spectrum like a degraded (low-pass/codec-compressed) signal, is a genuinely distinct way of motivating diffusion vocoder design compared to prior N2D diffusion and flow-matching vocoders. However, this specific data-to-data Schrödinger Bridge formulation originates in the authors’ own prior IJCAI 2025 conference paper (cited as a precursor); this arXiv submission is an explicit journal-style extension. The two substantive new contributions in this version are the BCD subband-aware network, which replaces an image-derived NCSN++ backbone with a modality-appropriate design and cuts compute by over 95% at matched or better quality, and the single-step distillation scheme, whose omnidirectional phase-distillation loss addresses a concrete training failure mode (phase-wrapping ambiguity in complex-spectrum distillation) rather than being a routine engineering addition. The paper also strengthens the generality of its restoration framing by showing it improves other diffusion families (ResShift, conditional flow matching) when adapted to the D2D setting, rather than resting the claim solely on its own architecture. Overall the contribution is best characterized as architectural (new network design plus a new distillation loss), built on top of a training-recipe-level reframing whose origin predates this specific paper.
Field Significance
High
High — the restoration/D2D reframing offers a reusable principle for diffusion-based vocoders beyond this paper’s own architecture (validated on ResShift and conditional flow matching), and the accompanying subband network demonstrates a substantial compute reduction without sacrificing quality, giving the broader vocoder literature both a new problem framing and a concrete efficient-architecture template to build on.
This paper provides confirmatory-and-extending evidence in the diffusion-vocoder line of work: it demonstrates that shortening the generative trajectory by starting from a structurally-informed source (rather than Gaussian noise) is a viable general strategy, and that pairing this with a T-F-domain-appropriate network materially improves the quality-efficiency trade-off relative to both GAN and flow-matching vocoders evaluated at comparable or higher step counts.
Claims
- supports: Framing an acoustic-feature-to-waveform generation problem as data-to-data restoration, where the reverse diffusion trajectory begins from a structurally-related surrogate of the target rather than Gaussian noise, shortens the required sampling trajectory and improves quality at low step counts.
Evidence: The proposed Schrödinger Bridge formulation, conditioned on a range-space spectral surrogate of the target Mel-spectrum instead of noise, matches or exceeds GAN and flow-matching baselines evaluated at 10-16 steps using only 4 (and competitively even 2) diffusion steps. (§V-B, Table IX; Fig. 13)
- supports: Matching a diffusion model’s internal network architecture to the frequency-domain structure of audio, via uneven subband decomposition concentrated at low/mid frequencies and large-kernel convolutional attention in place of self-attention, can substantially cut compute relative to image-derived backbones without sacrificing quality.
Evidence: Replacing an NCSN++ backbone (borrowed from image diffusion) with the proposed subband-aware BCD network reduces MACs by 95.6% at matched inference steps while achieving equal or better PESQ, UTMOS, and M-STFT. (§V-A, Table VI)
- complicates: Naive feature-space distillation losses for single-step diffusion-based spectral generation can fail specifically due to phase-wrapping ambiguity in complex spectral representations, rather than due to distillation as a strategy in general.
Evidence: A standard L2 distillation loss on real/imaginary student-teacher outputs produces severe spectral distortion (PESQ 2.614); replacing it with an omnidirectional phase-aware distillation loss that applies fixed directional convolution kernels to the phase before comparison raises PESQ to 4.182 in the corresponding ablation. (§III-F, §V-D, Table XIII, Id1→Id2)
- complicates: Objective spectral-reconstruction metrics for vocoders do not reliably predict subjective listener preference, particularly for high-frequency artifacts that fall outside the sensitivity of commonly used metrics.
Evidence: WaveFM attains competitive objective scores on LibriTTS and out-of-distribution sets but receives markedly lower similarity-MOS ratings than BridgeVoC and other baselines in listening tests, which participants attributed to audible high-frequency current-noise artifacts not penalized by the objective metrics used. (§V-B, Table XII discussion)
Limitations and Open Questions
The restoration framing depends on a linear degradation assumption that does not hold for the acoustic representations increasingly used elsewhere in the field.
The authors state explicitly that the range-space spectral surrogate is only cleanly derivable because the Mel-spectrum is a linear (log-compressed) transform of the target spectrum; this does not extend to the nonlinear latent features produced by variational autoencoders or to discrete tokens from neural audio codecs, both of which are increasingly common conditioning/target representations in recent TTS and codec-based systems. Extending the restoration framework to these nonlinear or discrete representations is left as future work. (§VI, Limitations)
Beyond this, WER is the one objective metric on which BridgeVoC does not lead its baselines, indicating a gap between the paper’s acoustic-fidelity gains and downstream ASR-measured intelligibility. The out-of-distribution evaluation also surfaces a large UTMOS score gap between English (EARS) and Mandarin (AISHELL-3) test sets, which the authors attribute to a limitation of UTMOS itself for cross-lingual quality prediction rather than to the vocoder. The single-step distilled variant (BridgeVoC*) trails the multi-step teacher on nearly every metric, so the quality-latency trade-off is not fully eliminated, only shifted. Finally, because this paper is an extension of the authors’ own IJCAI 2025 conference paper, the core Schrödinger Bridge restoration idea itself was already established prior to this specific submission; the incremental contributions here are the BCD network and the distillation scheme.
Wiki Connections
- Diffusion TTS — recasts the neural vocoder task as data-to-data restoration within a Schrödinger Bridge diffusion framework instead of standard noise-to-data denoising, and shows the same restoration framing also improves other diffusion families (ResShift, conditional flow matching) when adapted to start from a structured source.
- GAN Vocoder — targets the same Mel-spectrogram-to-waveform vocoding problem addressed by GAN vocoders, directly benchmarking against HiFi-GAN, BigVGAN, and Vocos and reporting consistent quality and efficiency gains at matched or fewer parameters.
- Subjective Evaluation — backs its objective results with similarity-MOS and AB preference listening tests on out-of-distribution audio, surfacing a case (WaveFM) where objective metrics and human ratings diverge.
- HiFi-GAN — used as a GAN-based vocoder baseline across all benchmark tables; BridgeVoC is consistently reported ahead on quality metrics at comparable or lower inference cost.
- BigVGAN — the strongest GAN-based baseline in the comparison; BridgeVoC matches or exceeds it on most objective metrics while using a fraction of its parameters (7.65M vs. 112.39M).
- Vocos — compared as an efficient time-frequency-domain GAN vocoder baseline; BridgeVoC reports higher PESQ and UTMOS at comparable inference speed.
- PeriodWave — a flow-matching diffusion vocoder baseline requiring 16 function evaluations; BridgeVoC matches or exceeds its quality using only 4 steps.
- WaveFM — the closest prior single-step-distilled flow-matching vocoder baseline; both BridgeVoC and its distilled variant BridgeVoC* are reported ahead of WaveFM and WaveFM* in objective metrics and AB preference tests.