NAACL · 2025 · Conference
Kumar et al. (University of Maryland, College Park) · → Paper · Demo: ? · Code: ✓
ProSE decouples diffusion from the speech enhancement output by using a latent diffusion model solely to generate a compact prior, which then guides a deterministic transformer-based regression model, achieving state-of-the-art results across matched and mismatched noise conditions while requiring only 2 diffusion steps at inference.
Problem
Applying denoising diffusion probabilistic models (DDPMs) directly to speech enhancement (SE) runs into two fundamental obstacles. First, SE requires that the output closely match the original clean signal, whereas the multi-step denoising process in DDPMs tends to introduce misaligned details and artifacts not present in the ground-truth speech. Second, conventional diffusion-based SE methods operate over the full mel-spectrogram or waveform, demanding 50 or more inference steps and high computational cost. Prior strategies — either interpolating between clean and noisy speech in the diffusion process, or using noisy speech as an auxiliary condition — partially address one issue but not both simultaneously.
Method
ProSE (diffusion-based Priors for Speech Enhancement) introduces a two-stage training architecture that separates the roles of the diffusion model and the regression model. The core idea is that the LDM does not produce enhanced speech directly; instead, it generates a compact latent prior that conditions a U-Net-shaped transformer, which performs the actual mapping from noisy to clean speech.
Stage 1. Both clean and noisy mel-spectrograms are first encoded by a convolutional VAE (compression ratio r=4). These VAE latents are concatenated and fed into a convolutional latent encoder (LE_S1) to produce a highly compact prior z_S1 of shape N×C’ (N=16, C’=256). This prior is integrated into the SE Transformer via multi-scale cross-attention at each encoder and decoder block, with multi-scale copies obtained by downsampling z_S1. The latent encoder and SE Transformer are trained jointly end-to-end with an L1 mel-spectrogram reconstruction loss.
Stage 2. The compact prior z_S1 from stage 1 serves as the starting point for the LDM’s forward diffusion process. A second latent encoder (LE_S2) encodes the noisy speech into a conditioning latent z_S2, and the LDM learns to reverse the diffusion of z_S1, conditioned on z_S2. The final training objective combines the SE reconstruction loss with a direct L1 diffusion loss between predicted and actual priors. Crucially, T=2 diffusion steps suffice at both training and inference time because the LDM operates in this very compact latent space rather than over the full signal.
At inference, the pipeline starts from Gaussian noise, runs the LDM conditioned on z_S2 for 2 steps to produce a prior, injects it into the SE Transformer via cross-attention, and recovers clean speech through the VAE decoder followed by a HiFi-GAN vocoder trained on VoiceBank-DEMAND.
Key Results
On the VoiceBank-DEMAND benchmark (Table 1), ProSE reaches PESQ 2.87 and STOI 94.10% using only 2 diffusion steps, outperforming the strongest diffusion-based baseline DOSE (PESQ 2.54, STOI 93.52%) which requires 50 steps. The margin on TIMIT+MUSAN (a synthetically noisier corpus) is larger: ProSE PESQ 2.28 vs. DOSE 2.02. On CHiME-4, a real-world mismatched evaluation where models are trained on VoiceBank-DEMAND, ProSE achieves STOI 88.18% vs. 85.42% for DOSE (Table 2), demonstrating stronger generalization to unseen noise conditions. Computational efficiency is favorable: ProSE requires 150.78 GFLOPs vs. 174.25 for DOSE and 289.58 for DR-DiffuSE (Table 3). Subjective evaluations (Table 4) with 20 raters confirm the advantage: MOS 4.45 vs. 3.70 (DOSE) in matched conditions, and MOS 3.45 vs. 2.95 (DOSE) in mismatched conditions.
The ablation comparing ProSE against the bare SE Transformer (no LDM prior, single-stage training) shows a substantial drop in all metrics without the prior, confirming that the LDM conditioning is the key performance driver rather than the transformer architecture alone.
Novelty Assessment
The contribution is architectural: separating the generative role of diffusion (prior generation in a compact latent space) from the discriminative role of a regression model (direct noisy-to-clean mapping). This framing neatly sidesteps the artifact problem that plagues direct diffusion-based enhancement while retaining the distributional modeling strength of DDPMs. The individual components — VAE, latent diffusion, U-Net transformer, HiFi-GAN — are all established. The novelty lies in the specific 2-stage coupling protocol and the insight that a 2-step LDM over a 16x256 latent can guide a large transformer without the full iterative cost of waveform-space diffusion. The baseline set is limited to diffusion-only SE methods; comparisons against the strongest deterministic SE models (e.g., Demucs, MANNER) are absent, which partially limits the scope of the “state-of-the-art” claim.
Field Significance
Moderate — ProSE demonstrates that diffusion models can serve as efficient prior generators for a regression backbone rather than end-to-end enhancement systems, offering a practical route to lower inference cost without sacrificing quality in speech enhancement. The architectural pattern (compact latent prior from a fast LDM conditioning a deterministic model) is transferable to other conditional speech generation tasks where output faithfulness to a reference signal is important.
Claims
-
supports: Applying diffusion in a compact latent space rather than over the full signal substantially reduces the number of inference steps required without degrading output quality.
Evidence: ProSE achieves state-of-the-art PESQ and STOI on VoiceBank-DEMAND and TIMIT+MUSAN using only T=2 diffusion steps, outperforming baselines that require 6–50 steps. (§6, Table 3)
-
supports: Separating prior generation by a diffusion model from regression-based reconstruction reduces artifacts caused by direct multi-step denoising in speech enhancement.
Evidence: Waveform and mel-spectrogram comparisons (Figures 2 and 3) show that DOSE introduces extraneous misaligned details absent in clean speech, whereas ProSE does not, because the regression model performs the actual mapping. (§6, Figure 3)
-
complicates: Diffusion-based speech enhancement methods do not uniformly generalize to real-world noise conditions unseen during training.
Evidence: Several baselines (DiffuSE, CDiffuSE, SGMSE, DR-DiffuSE, DOSE) show markedly lower subjective MOS in mismatched CHiME-4 conditions (Table 4), suggesting condition-collapse or domain shift; ProSE is more robust but still incurs a gap from matched to mismatched conditions. (§5, Table 4)
-
refines: The computational advantage of latent diffusion over waveform-space diffusion holds for speech enhancement tasks, not only for speech synthesis.
Evidence: ProSE uses fewer FLOPs (150.78G) than most diffusion-based SE baselines while achieving higher PESQ, attributable to running diffusion on a compact N=16, C’=256 latent rather than over the full mel-spectrogram or waveform. (§6, Table 3)
Limitations and Open Questions
Warning
Comparisons are restricted to diffusion-based SE baselines; no deterministic discriminative methods (e.g., Demucs, MANNER) are included in the main evaluation tables. The “state-of-the-art” claim therefore applies specifically within the diffusion SE category.
The two-stage training pipeline adds substantial complexity: stage 1 requires 300K iterations, and stage 2 repeats this, doubling training cost. The authors acknowledge that simplifying this into a single-stage procedure is open future work. Additionally, the HiFi-GAN vocoder and VAE encoder introduce their own reconstruction losses that propagate to final quality, an overhead the paper notes without quantifying. All experiments use speech sampled at 16 kHz; applicability to higher-fidelity conditions is untested.
Wiki Connections
- Evaluation Metrics — The paper evaluates using PESQ, STOI, CSIG, CBAK, COVL, and MOS, providing a multi-dimensional view of enhancement quality across objective and subjective dimensions.
- Subjective Evaluation — ProSE conducts both naturalness MOS and consistency MOS tests with 20 raters, showing larger gains in subjective quality (MOS 4.45 vs. 3.70 for DOSE) than in objective metrics, illustrating that objective metrics can understate perceptual improvements.