arXiv · 2026 · Preprint
Yongjoon Lee et al. (KAIST) · → Paper · Demo: ? · Code: ✓
Introduces Relativistic Adversarial Feedback (RAF), a GAN vocoder training objective that pairs self-supervised-learning-guided quality estimation with relativistic discriminator feedback to improve both in-domain fidelity and generalization to unseen speakers, languages, and recording conditions.
Problem
GAN-based neural vocoders synthesize waveforms efficiently in a single forward pass, but their standard adversarial objectives (LSGAN, HingeGAN) map real and fake samples to fixed decision boundaries rather than modeling the training distribution comprehensively. As zero-shot TTS and VC systems increasingly rely on vocoders to generalize across unseen speakers, languages, and recording conditions, this training limitation shows up as degraded out-of-distribution quality. Prior work addressed generalization mainly by scaling generator capacity (BigVGAN) or through slower diffusion/flow-matching alternatives, leaving the training objective itself underexplored as a lever for improving both fidelity and generalization without sacrificing GAN inference speed.
Method
RAF is a training framework, not a new generator or discriminator architecture; it modifies the adversarial objective used to train existing GAN vocoder backbones (BigVGAN-base, HiFi-GAN v1, Vocos). It has two components. The quality gap quantifies perceptual distance between a real and a generated waveform using frozen WavLM-large and HuBERT-large representations (the last convolutional layer of WavLM and the 22nd layer of HuBERT) plus a Multi-resolution STFT distance, concatenated into a three-dimensional target after per-component scaling. The discriminator gap is the softplus-transformed difference between the discriminator’s score for the fake sample and its score for the paired real sample, following relativistic pairing GAN (RpGAN): rather than assigning a single global decision boundary to all real/fake samples, the discriminator forms an individual decision boundary for each real/fake pair. The discriminator’s output layer is extended to three nodes so it can approximate each of the three quality-gap components. The adversarial objective for the discriminator minimizes the mean-squared error between the discriminator gap and the quality gap; the generator objective minimizes the discriminator gap directly, so iterative training drives the quality gap toward zero. A zero-centered gradient penalty (applied every 7 steps) stabilizes convergence, and mel-spectrogram reconstruction and feature-matching losses are added for training stability, following standard GAN vocoder practice.
A key training detail is segment size: because quality-gap estimation from short waveform segments is unreliable, RAF trains on 24,576-sample segments (longer than the original backbones’ default segment sizes), chosen based on a measured drop in quality-estimation error as segment length increases. All backbones were trained on LibriTTS (24 kHz, train-clean-100/360 + train-other-500) for 1M steps with batch size 16 and the AdamW optimizer, using the original discriminator combinations for each backbone (MRD+MPD for BigVGAN/Vocos, MSD+MPD for HiFi-GAN).
Key Results
On the LibriTTS-dev objective benchmark, RAF improves signal fidelity (M-STFT, PESQ, periodicity, V/UV F1) and perceptual quality (UTMOS, SCOREQ) over each backbone’s default adversarial objective: BigVGAN-base+RAF reaches PESQ 3.767 and UTMOS 3.651 versus 3.452/3.450 for BigVGAN-base+LSGAN, and BigVGAN-base+RAF at 0.5M steps already outperforms BigVGAN-base+LSGAN at 1M steps. Notably, BigVGAN-base (14M params) trained with RAF surpasses the much larger BigVGAN (112M params, LSGAN-trained) on UTMOS and SCOREQ, using about 12% of the parameters. Gains also hold for HiFi-GAN (v1) and Vocos. Across four unseen-domain test sets (LJSpeech, Deeply Korean, the multilingual UR dataset, and MUSDB18-HQ vocals), RAF improves most metrics for all three backbones, with the largest gains for HiFi-GAN (v1). Subjective SMOS evaluation on LibriTTS-test (30 English-native raters) and Deeply Korean (20 Korean-native raters) confirms the objective trends, with a larger RAF-over-LSGAN improvement margin on the out-of-domain Korean set (4.324 vs. 3.824) than on LibriTTS-test (4.592 vs. 4.526); a Wilcoxon signed-rank test found both differences statistically significant (p < 0.05). Ablations isolate the contribution of each RAF component (softplus transform, SSL-derived quality-gap terms, scaling hyperparameters), and a 13-configuration comparison against LSGAN, HingeGAN, RpGAN-GP, LSGAN+Q-recon, and two MetricGAN-RAF variants shows RAF achieves the best UTMOS/SCOREQ and second-best PESQ among quality-gap-based objectives when combined with long segments and gradient penalty. RAF also outperforms the other quality-gap objectives at generalizing to LJSpeech, Deeply Korean, and UR in a follow-up UTMOS comparison. Training cost increases: BigVGAN-base with RAF takes 9.4 GPU-days versus 5.9 for LSGAN under matched hardware.
Novelty Assessment
RAF’s novelty is in the training objective, not in generator or discriminator architecture: it combines SSL-derived perceptual quality estimation (previously used mainly in speech enhancement and MetricGAN-style optimization) with relativistic pairing (previously explored for vocoders only in a single prior study using a simple pointwise formulation) into a single adversarial feedback loop with an explicit, per-source quality-gap target. The paper is careful to distinguish RAF from the closest prior approach, MetricGAN, through two intermediate baselines (MetricGAN-RAF-v1/v2) and a toy two-mode experiment, showing that the generalization benefit comes specifically from the relativistic pairing formulation rather than merely from adding a quality-gap reconstruction term. The contribution is a training-recipe innovation validated across three existing backbones and thirteen training-objective configurations rather than a new model architecture; the underlying generators and discriminators are unmodified except for an expanded discriminator output layer.
Field Significance
moderate — RAF demonstrates that generalization gains attributed to scaling generator capacity (as in BigVGAN) can instead be obtained by changing the adversarial training objective, at a fraction of the parameter count. This reframes training-objective design as a lever for GAN vocoder generalization that is complementary to architecture scaling, and it directly targets a segment-size confound in quality-gap estimation that could affect other quality-metric-guided GAN training methods.
Claims
- supports: Incorporating self-supervised-learning-derived perceptual quality estimates into a discriminator’s training signal can improve both in-domain fidelity and out-of-distribution generalization of GAN vocoders.
Evidence: RAF, using WavLM-large and HuBERT-large features as quality-gap components, improves UTMOS and SCOREQ over the default adversarial objective for BigVGAN-base, HiFi-GAN (v1), and Vocos on LibriTTS-dev, and improves most objective metrics across four unseen datasets (LJSpeech, Deeply Korean, UR, MUSDB18-HQ). (§5.1.1, §5.1.2, Table 1, Table 2)
- supports: Reformulating the GAN adversarial objective to pair real and fake samples relatively, rather than mapping each to a fixed decision boundary, can substitute for increased generator capacity in reaching competitive perceptual quality.
Evidence: BigVGAN-base (14M params) trained with RAF outperforms BigVGAN (112M params) trained with LSGAN on UTMOS and SCOREQ, using roughly 12% of the parameters. (§5.1.1, Table 1)
- complicates: Gains from SSL-guided, relativistically-paired adversarial training come with a nontrivial increase in training cost relative to standard GAN objectives.
Evidence: Training BigVGAN-base with RAF took 9.4 GPU-days versus 5.9 for LSGAN under matched hardware, attributed to longer training segments, added SSL forward passes, and gradient-penalty regularization. (§5.1.1)
- refines: For quality-metric-guided GAN training, the generalization benefit stems specifically from relativistic pairing of real and fake samples, not merely from adding a perceptual quality-gap term to the loss.
Evidence: Across 13 training-objective configurations, RAF outperforms LSGAN+Q-recon and MetricGAN-RAF-v2 (both using the same SSL-derived quality gap without relativistic pairing) on UTMOS across LibriTTS, LJSpeech, Deeply Korean, and UR; a toy two-mode experiment shows RAF recovers both data modes fastest among the compared objectives. (§5.4.2, Table 6, Figure 3)
- complicates: Quality-gap-based adversarial training objectives require sufficiently long waveform segments to produce a reliable training signal.
Evidence: Measuring SCOREQ-based quality-estimation error across segment sizes on 10K LibriTTS samples showed a steep error decline between 8,192 and 16,384 samples, motivating a 24,576-sample training segment (longer than the backbones’ original defaults) for RAF. (§3.7, Figure 2)
Limitations and Open Questions
Warning
RAF materially increases training cost: the paper reports no lightweight alternative to the heavy frozen SSL models (WavLM-large, HuBERT-large), the extended segment length, and the gradient-penalty regularization it relies on, all three of which the authors identify as compute overheads relative to standard LSGAN training.
The authors also note the absence of a rigorous theoretical explanation for RAF’s convergence behavior, leaving the mechanism by which relativistic pairing improves mode coverage empirically demonstrated (via the toy experiment and ablations) but not formally proven. They flag a dual-use ethical risk: RAF-trained vocoders could improve the realism of voice-spoofing deepfakes, and suggest audio watermarking or compatibility with deepfake-detection frameworks as mitigations, while also noting RAF could help generate adversarial examples for training detection systems.
Wiki Connections
- GAN Vocoder — proposes a new adversarial training objective (relativistic, SSL-guided quality-gap minimization) applicable to existing GAN vocoder backbones, rather than a new generator or discriminator architecture.
- Self-Supervised Speech — uses frozen WavLM-large and HuBERT-large representations as a core component of the discriminator’s quality-gap estimation, not merely as an evaluation baseline.
- Evaluation Metrics — evaluates generalization with seven objective metrics (M-STFT, PESQ, periodicity, V/UV F1, UTMOS, SCOREQ full-reference and no-reference) across five source and unseen datasets.
- Subjective Evaluation — validates objective gains with SMOS listening tests conducted with human raters on both English (LibriTTS-test) and Korean (Deeply) speech, including a statistical significance test.