arXiv · 2025 · Preprint
Welker et al. (University of Hamburg) · → Paper · Demo: ? · Code: ✓
MelFlow achieves real-time streaming mel vocoding using generative flow matching with a 48 ms total latency, outperforming HiFi-GAN on intrusive quality metrics (PESQ, SI-SDR) while operating within the streaming constraint on a consumer laptop GPU.
Problem
Mel vocoding (spectrogram-to-waveform inversion) remains a critical component of two-stage TTS pipelines, yet virtually all high-quality vocoders, including GAN-based systems such as HiFi-GAN, operate in non-streaming batch mode. Prior work on streamable vocoding has been limited to GAN-based approaches, and the only alternative streaming method for diffusion-based models (the Diffusion Buffer) requires a minimum algorithmic latency of roughly 340 ms due to its coupling of physical and diffusion time. No prior work had demonstrated diffusion or flow matching models applied to mel spectrogram inversion in a streaming setting.
Method
MelFlow adapts the authors’ prior diffusion-based STFT phase retrieval system (DiffPhase) to mel vocoding by replacing the corruption model. Rather than using the magnitude spectrogram directly, the system defines the corrupted input as the per-frame lossy Mel compression followed by pseudoinverse decompression (following FreeV), applied to the STFT magnitude. The target is the full complex STFT. An interpolating flow matching process (following FlowDec) is then trained to recover the clean STFT from this corrupted input, generating phase along with enhanced magnitudes. During inference the model uses N=5 equidistant Euler steps of the ODE solver.
The streaming inference is enabled by a novel N-buffer caching scheme. Instead of one set of per-layer rolling buffers (as in standard causal convolutional streaming), the scheme maintains N independent collections of buffers, one for each flow step. Frame t is processed sequentially through all N DNN calls using the cached activations from past frames. This requires the same total computation as offline processing and produces bit-identical results to batch inference (up to floating-point precision), eliminating any train-test mismatch. STFT window duration is 32 ms (512-point Hann window) with 16 ms hop, giving an algorithmic latency of 32 ms and total latency of 48 ms.
The model is a 27.9M-parameter custom causal DNN based on the NCSN++ U-Net architecture. Key modifications include replacing all convolutions with causal (zero-padded) convolutions, performing down- and upsampling only along the frequency axis, using dilated convolutions along time in place of temporal striding, removing attention layers, and substituting standard GroupNorm with a sub-band grouped BatchNorm. Training uses the EARS-WHAM v2 dataset (~87 hours of clean speech downsampled to 16 kHz), the SOAP optimizer with cosine decay, and a flow matching noise level of 0.25.
Key Results
On EARS-WHAM v2, MelFlow (N=5, streaming) achieves PESQ 4.12 and SI-SDR -8.8 dB, substantially better than the 16 kHz SpeechBrain HiFi-GAN at PESQ 2.99 and SI-SDR -29.9 dB. WER (7.2% vs 7.3%) is essentially equal. On LibriTTS, MelFlow achieves PESQ 3.97 and SI-SDR -14.5 dB vs HiFi-GAN at 3.03 and -25.8 dB, and improves WER from 5.1% to 3.5%. Non-intrusive metrics (DistillMOS, WVMOS, NISQA) are competitive across both test sets.
However, MelFlow is weaker than HiFi-GAN in spectral distance metrics: on EARS-WHAM v2, log-spectral distance (LSD) is 1.0 vs 0.77, and mel-cepstral distance (MCD) is 3.28 vs 2.41. The non-streaming N=25 MelFlow variant recovers this gap (LSD 0.7, MCD 1.7), suggesting these spectral metrics are sensitive to the step-count budget. On the GPU tested (NVIDIA RTX 4080 Laptop), each DNN call costs 2.71 ms, allowing up to N=5 steps within the 16 ms frame shift budget.
Vocos (24 kHz, not streaming) achieves PESQ 3.67 on LibriTTS, below the streaming MelFlow’s 3.97, which is notable given MelFlow operates at 16 kHz and under a streaming constraint.
Novelty Assessment
The core architectural contribution is the N-independent-buffer streaming inference scheme, which extends standard causal convolutional caching to multi-step flow or diffusion models without additional latency, redundant computation, or train-test mismatch. This is a principled formalization that prior streamable vocoder work had not explored. The mel vocoding corruption model (pseudoinverse initialization from FreeV, flow matching from FlowDec, causal DNN from DiffPhase) is an integration of existing ideas rather than a new design, but the streaming extension is not merely engineering: the insight that N independent buffer sets can run diffusion steps in serial per frame without algorithmic overhead is genuinely novel. The paper also provides the first public code and checkpoint for streamable mel vocoding.
The evaluation is limited to a single output sample rate (16 kHz) and a relatively small training set (~87h). The MCD and LSD results are weaker than HiFi-GAN at N=5, reflecting the step-count constraint on spectral detail.
Field Significance
Moderate — MelFlow demonstrates that flow matching can support real-time streaming mel vocoding at TTS-compatible latencies, providing both a working system and a principled framework for streaming diffusion/flow inference. The zero train-test mismatch property is practically valuable for deploying trained models in streaming contexts. The result extends generative flow matching beyond its established batch-mode applications into the latency-constrained streaming setting.
Claims
-
supports: Flow matching vocoders can match or exceed GAN vocoders in intrusive perceptual quality metrics even under a streaming constraint that reduces inference steps.
Evidence: MelFlow at N=5 (streaming) achieves PESQ 4.12 vs HiFi-GAN’s 2.99 and SI-SDR -8.8 dB vs -29.9 dB on EARS-WHAM v2, while maintaining RTF < 1 on a consumer GPU. (§5, Table 1)
-
complicates: Low step-count budgets in streaming flow/diffusion inference degrade spectral fidelity even when perceptual quality remains competitive.
Evidence: MelFlow at N=5 shows LSD 1.0 and MCD 3.28 on EARS-WHAM v2, worse than HiFi-GAN (LSD 0.77, MCD 2.41), whereas the non-streaming N=25 variant recovers to LSD 0.7 and MCD 1.7. (§5, Table 1)
-
supports: Streaming inference for multi-step flow and diffusion models can be achieved without train-test mismatch by maintaining N independent sets of per-layer causal cache buffers.
Evidence: The N-buffer scheme produces outputs identical to offline batch processing (up to floating-point precision) because cached past activations never change, enabling standard parallel training alongside streaming deployment. (§3.2)
-
complicates: High algorithmic latency is a significant barrier for diffusion-buffer streaming approaches, limiting practical streaming to methods that avoid coupling diffusion time to physical time.
Evidence: The Diffusion Buffer achieves sub-second latency but requires at least 20 frames per buffer, giving a minimum effective algorithmic latency of ~340 ms at 16 ms hop size, which MelFlow avoids by using the per-layer cache scheme. (§2.2)
Limitations and Open Questions
MelFlow operates at 16 kHz only; extension to 24 or 44.1 kHz is not evaluated and the step-count constraint tightens at higher sample rates. Spectral detail metrics (LSD, MCD) at N=5 are weaker than HiFi-GAN, suggesting a quality-latency tradeoff that few-step distillation might address (noted as future work). The training dataset is relatively small (~87 hours); behavior on highly diverse or noisy speech is untested. Comparison vocoders operating at 22-24 kHz are evaluated after downsampling to 16 kHz, introducing a resampling artifact that may affect fairness.
Wiki Connections
- Flow Matching — MelFlow applies interpolating flow matching as its generative process, following FlowDec’s formulation, extending flow matching for the first time to streaming mel vocoding.
- Streaming TTS — the paper’s central contribution is enabling real-time (RTF < 1, 48 ms total latency) streaming mel vocoding, a prerequisite for low-latency conversational TTS pipelines.
- GAN Vocoder — HiFi-GAN, Vocos, Parallel WaveGAN, and StyleMelGAN are used as baselines; MelFlow is shown to outperform them on PESQ and SI-SDR despite operating under a streaming constraint.
- Evaluation Metrics — the paper employs a comprehensive nine-metric evaluation suite covering intrusive metrics (PESQ, ESTOI, SI-SDR, MCD, LSD), ASR-based intelligibility (WER), and three non-intrusive estimators (DistillMOS, WVMOS, NISQA).