arXiv · 2026 · Preprint
Takuhiro Kaneko et al. (NTT, Inc.) · → Paper · Demo: ✓ · Code: ?
Proposes MeanVoiceFlow, a one-step nonparallel voice conversion model trained from scratch with mean flows, avoiding the pretraining or distillation stages required by prior one-step diffusion-based VC systems.
Problem
Diffusion and flow-matching voice conversion (VC) systems (author term: diffusion/flow-matching VC) achieve strong speech quality and speaker similarity but require iterative multi-step inference, which is slow. Prior work has reduced diffusion-based VC to one step via knowledge distillation, often combined with adversarial training, but this approach has two costs: it requires training both a multi-step teacher and a one-step student (higher training cost), and adversarial training is prone to divergence, typically needing a pretrained feature extractor (e.g., a pretrained vocoder) for stabilization. The paper asks whether one-step nonparallel VC can be trained from scratch, without a teacher model or distillation, while matching the quality of multi-step and distillation-based one-step baselines.
Method
MeanVoiceFlow replaces the instantaneous velocity field used in conventional flow matching with the average velocity of mean flows (author term: mean flows), an inference-efficient generative formulation. Conventional flow matching learns a velocity field v(z_t, t) that must be numerically integrated over discretized time steps (e.g., with an Euler solver), which introduces discretization error that grows as the number of steps shrinks toward one. Mean flows instead learn u(z_t, r, t), the average velocity between two time points r and t, which directly encodes the displacement needed for a jump between those times; in the one-step case, sampling reduces to a single subtraction z_0 = z_1 - u_θ(z_1, 0, 1).
For conditional generation, the average velocity is extended to u(z_t, r, t, s, c), where s is a speaker embedding from a pretrained speaker encoder and c is a content (linguistic bottleneck) embedding from a pretrained bottleneck feature extractor, both fixed off-the-shelf components rather than jointly trained. Log-mel spectrograms serve as the generation target, later vocoded to waveform with a pretrained HiFi-GAN.
Training the average velocity requires computing its target via a Jacobian-vector product involving the derivative of the network’s own output, which is unreliable early in training and can destabilize optimization. The paper addresses this with a structural margin reconstruction loss applied as a zero-input constraint: rather than regularizing all outputs with a direct element-wise loss (which the authors show causes over-smoothing, i.e., statistical averaging, similar to VAE posterior collapse), the constraint (1) uses an SSIM-based structural loss instead of an element-wise loss, (2) applies a margin so already-good samples are not penalized, and (3) is restricted to zero-input samples only (the center of the prior distribution). The total objective sums the mean flow loss and this margin-weighted zero-input reconstruction loss.
The paper also introduces conditional diffused-input training. Prior diffusion-based VC feeds a mixture of noise and diffused source data as input only at inference (to help preserve linguistic content), creating a training-inference mismatch since training only ever sees pure noise. MeanVoiceFlow instead synthesizes a diffused source input during training as well, using the model’s own current average-velocity estimate (with a stop-gradient) to approximate it from a shuffled-speaker version of the target utterance, since nonparallel VC has no true paired source-target data. Half of each training batch uses this diffused-source input and half uses pure noise, to stabilize training while eliminating the train-inference mismatch. The underlying network reuses the exact U-Net architecture (12 convolution layers, 512 hidden channels, two downsampling/upsampling stages, gated linear units, weight normalization) from the FastVoiceGrad baseline, so novelty is concentrated in the training objective and inference procedure rather than the network structure itself.
Key Results
On VCTK any-to-any (zero-shot) VC with one-step inference, MeanVoiceFlow reaches nMOS 3.87 and SECS 0.886, outperforming one-step models trained from scratch without distillation (VoiceGrad-DM-1: nMOS 2.77; VoiceGrad-FM-1: nMOS 3.14) and reaching performance comparable to 30-step multistep baselines (VoiceGrad-DM-30: nMOS 3.79; VoiceGrad-FM-30: nMOS 3.79) and to one-step models that require distillation plus adversarial training (FastVoiceGrad: nMOS 3.73; FastVoiceGrad+: nMOS 3.81) (§4.3, Table 2). sMOS for MeanVoiceFlow is 2.92, close to but not statistically distinguishable from the strongest distillation-based baselines in the paper’s own significance testing. A component analysis on the zero-input constraint shows the full proposed configuration (SSIM loss + margin + zero-input-only application) reaches pMOS_s 3.90 / pMOS_n 3.77 / pMOS_v 4.08, versus 3.72 / 3.73 / 4.00 for the mean-flow-only baseline and worse pMOS_n/pMOS_v (over-smoothing) for element-wise L1/L2 variants (§4.2, Table 1). On a second dataset (LibriTTS train-clean, held-out speakers/sentences), the same relative ordering holds: MeanVoiceFlow again outperforms one-step VoiceGrad variants and approaches the 30-step versions (§4.4, Table 3).
Novelty Assessment
The contribution is a training-objective and inference-procedure innovation rather than a new network architecture: the paper explicitly reuses the baseline’s U-Net to isolate the effect of the training method. Applying mean flows (originally proposed for image generation) to nonparallel VC, and specifically to speech mel-spectrogram generation, is the paper’s main architectural-novelty claim. The structural margin reconstruction loss and conditional diffused-input training are targeted engineering solutions to two specific instabilities that arise when this transfer is made (JVP-based target instability, and training-inference input mismatch); both are ablated individually. The comparisons are internally consistent (same network backbone across all training-method baselines), which strengthens the claim that gains are attributable to the training procedure rather than confounded by architecture changes.
Field Significance
moderate — This paper demonstrates that one-step nonparallel VC trained purely from scratch, without a distillation teacher or adversarial training, can match distillation-and-adversarial-training one-step baselines on standard objective and subjective metrics. It extends mean flows, a recently proposed one-step generative training technique, into the voice conversion setting and identifies specific training instabilities that require targeted fixes (a zero-input reconstruction constraint and diffused-input training alignment) to make that transfer work in practice.
Claims
- supports: Average-velocity (mean flow) training can replace instantaneous-velocity flow matching to enable stable one-step nonparallel voice conversion trained entirely from scratch, without a distillation teacher or adversarial loss.
Evidence: MeanVoiceFlow (nMOS 3.87, SECS 0.886) matches or exceeds distillation- and adversarial-training-based one-step baselines FastVoiceGrad (nMOS 3.73) and FastVoiceGrad+ (nMOS 3.81) while requiring neither a pretrained teacher nor adversarial training. (§4.3, Table 2)
- complicates: Regularizing average-velocity training with a direct element-wise reconstruction loss trades off one quality dimension for another rather than improving quality uniformly.
Evidence: Adding an L1 or L2 zero-input reconstruction loss to the base mean-flow objective raised pMOS_s (3.72 to 3.81 / 3.80) but degraded pMOS_n (3.73 to 3.65 / 3.66) and pMOS_v (4.00 to 3.96 / 3.98) due to over-smoothing, whereas the proposed SSIM-based margin loss improved all three metrics together. (§4.2, Table 1)
- supports: Matching the input distribution seen during training to the distribution encountered at inference improves one-step conditional generation, beyond simply using a good input distribution at inference alone.
Evidence: Conditional diffused-input training, which exposes the model to diffused source data during training (not only at inference as in prior work), improved robustness to the inference-time mixing ratio and peak pMOS_s/SECS scores relative to a model trained only on pure noise. (§4.2, Figure 3)
- refines: The apparent quality gap between one-step and multi-step generative VC narrows substantially once the one-step model’s training procedure, not only its architecture, is specifically designed for one-step inference.
Evidence: One-step VoiceGrad-DM/FM variants trained with standard flow matching or diffusion objectives lag their 30-step counterparts by roughly 0.6-1.0 nMOS points, while MeanVoiceFlow trained with the mean-flow objective for one-step inference closes most of this gap (nMOS 3.87 vs. 3.79 for the 30-step versions) on the same VCTK evaluation protocol. (§4.3, Table 2)
Limitations and Open Questions
The paper reuses a fixed pretrained speaker encoder and bottleneck content extractor rather than training these jointly with the mean-flow model, so results are conditioned on the quality of those off-the-shelf components; the paper does not report sensitivity to swapping them out. Model size in parameters is not reported. Evaluation is limited to English-language corpora (VCTK, LibriTTS); no cross-lingual or low-resource evaluation is presented. The margin hyperparameter (0.3) and loss weight (1) for the zero-input constraint are set empirically without a reported sensitivity sweep beyond the ablation in Table 1. The authors themselves note that applying mean flows to speech more broadly remains underexplored and identify extension to other speech tasks as future work (§5).
Wiki Connections
- Voice Conversion — introduces a one-step nonparallel any-to-any VC training method, evaluated with standard VC-specific objective and subjective metrics (SECS, CER, nMOS, sMOS).
- Flow Matching — replaces the instantaneous-velocity formulation of conventional flow matching with the average-velocity formulation of mean flows to enable single-step sampling.
- Zero-Shot TTS — targets any-to-any (zero-shot) speaker conversion using a fixed speaker embedding extracted from a reference utterance, without speaker-specific fine-tuning.
- GAN Vocoder — relies on a pretrained HiFi-GAN vocoder to synthesize waveforms from the log-mel spectrograms the model generates.
- Speaker Adaptation — conditions generation on a speaker embedding of an unseen target speaker to adapt output identity without retraining.
- Subjective Evaluation — reports naturalness (nMOS) and speaker similarity (sMOS) listening tests with statistical significance testing against baselines.
- FasterVoiceGrad — same authors’ related prior work on faster one-step diffusion-based VC via distillation and adversarial training, part of the same research lineage this paper’s training-from-scratch approach departs from.
- LatentVoiceGrad — same authors’ related nonparallel VC work exploring latent diffusion and flow-matching formulations for the same VC setting.
- Vocoder-Projected Feature Discriminator — provides the enhanced discriminator used by the FastVoiceGrad+ baseline this paper compares against.
- HiFi-GAN — supplies the pretrained vocoder used to synthesize waveforms from the model’s log-mel spectrogram outputs.
- UTMOS — provides the predicted-MOS metric (pMOS_s) used as one of the paper’s primary objective evaluation measures.
- StableVC — a related flow-matching-based zero-shot VC system representative of the multi-step flow-matching VC lineage this paper’s one-step approach targets.