ASRU · 2025 · Conference
Nan Xu et al. (Alibaba Digital Media & Entertainment Group) · → Paper · Demo: ? · Code: ?
Proposes a learnable harmonic discriminator for GAN-based vocoders that replaces the fixed-resolution STFT spectrogram with a triangular filter bank aligned to harmonic structure, improving synthesis quality on both speech and, especially, singing voice.
Problem
Time-frequency discriminators are a standard component of GAN-based vocoders, but the dominant design takes a plain STFT spectrogram as input, which applies the same frequency resolution at every frequency bin. This uniform resolution is a poor match for harmonic structure: fundamental frequency and low harmonics need fine resolution, while fast-changing high harmonics benefit from coarser frequency but finer time resolution. The Constant-Q Transform (CQT) offers dynamic resolution and has been used in prior discriminator designs (MS-SB-CQT), but its fixed-Q construction only aligns even-power harmonics (h = 2^d), making odd harmonics hard to model, and its variable window length per frequency bin creates temporal misalignment across bins. These limitations are most damaging for singing voice, where harmonics are dense, fast-moving, and perceptually salient.
Method
The paper proposes UnivHD (Universal Harmonic Discriminator), which keeps a standard fixed-window STFT as the front end (avoiding CQT’s temporal asynchrony) but replaces the flat frequency binning with a learnable harmonic filter bank. Each filter is a triangular band-pass filter centered at the h-th multiple of a base center frequency, with bandwidth initialized from the equivalent rectangular bandwidth (ERB) formula and scaled by a single learnable parameter γ (constrained to γ ≥ 1 to respect the Nyquist limit). Because the h-th harmonic’s center frequency is defined as an exact multiple of the base frequency rather than a power-of-two multiple (as in CQT), the filter bank can represent odd harmonics directly. Stacking H harmonic filters across F base frequency bins over T time frames produces a 3-D harmonic tensor [H, F, T]. A half-harmonic filter (h = 0.5) is added below the fundamental to capture residual low-frequency energy that would otherwise be discarded.
The harmonic tensor feeds a hybrid convolution block (HCB) that treats the harmonic axis as a channel dimension: a depthwise separable convolution plus pointwise convolution models intra-harmonic structure, while a parallel normal 2-D convolution models inter-harmonic relationships; the two outputs are summed. The resulting latent representation passes through three multi-scale dilated convolution (MDC) blocks (dilation rates 1, 2, 4) and a final convolution layer, with MDC block outputs used for the feature-matching loss and the final layer output used for the adversarial discriminator loss. The discriminator has 0.31M parameters and is trained as a drop-in addition to existing vocoder generators (HiFi-GAN, iSTFTNet) without modifying the generator itself, using their original adversarial training recipes at 24kHz.
Key Results
Across both HiFi-GAN and iSTFTNet backbones, adding any time-frequency discriminator (MS-STFT, MS-SB-CQT, or UnivHD) improves over the vanilla vocoder, with the largest gains on singing voice (§IV-C1, Table I). UnivHD gives the best results among the three discriminators on nearly every metric and test condition: for HiFi-GAN on out-of-domain singing, MOS rises from 3.48 (baseline) to 3.86 (UnivHD), compared to 3.66 for MS-STFT and 3.75 for MS-SB-CQT (§IV-C1, Table I). On in-domain speech, HiFi-GAN + UnivHD reaches MOS 4.05 versus 3.85 for the baseline and 3.98 for MS-SB-CQT. Combining UnivHD with MS-STFT (HiFi-GAN +S+H) gives a further small objective improvement over either alone and also beats the MS-STFT+MS-SB-CQT combination (Table II), indicating the two discriminator families carry complementary information. Spectrogram and pitch-contour visualizations on an out-of-domain singing sample show UnivHD recovering harmonic structure and correcting a pitch-contour error that the plain HiFi-GAN and MS-SB-CQT variants exhibit (§IV-C1, Figs. 3-4). Ablations show the discriminator is robust to the number of harmonics H over a broad range (8-15) but degrades sharply outside it: too few harmonics under-resolves the spectrum, and H > 20 at 24kHz can push the first-harmonic search range past the fundamental frequency of high-pitched samples, degrading pitch modeling (§IV-C2). Removing DSConv, the normal convolution, or the half-harmonic representation each individually hurts objective metrics on singing, with DSConv removal causing the largest MCD and F0-RMSE degradation (Table III).
Novelty Assessment
The contribution is a genuinely new discriminator architecture, not a new vocoder generator: HiFi-GAN and iSTFTNet are used unmodified as backbones. Within the established line of time-frequency discriminators (TFGAN, Harmonic WaveGAN, UnivNet, MS-STFT/EnCodec, MS-SB-CQT), the specific idea of scaling filter center frequencies by integer harmonic order (rather than by powers of two, as CQT does) to guarantee odd-harmonic coverage while keeping a fixed-window STFT front end for temporal alignment is a targeted and well-motivated design choice, and the learnable bandwidth parameter is a small but clean generalization of the fixed ERB formula. The half-harmonic addition and the HCB’s split between depthwise-separable (intra-harmonic) and normal (inter-harmonic) convolutions are incremental architectural refinements validated by ablation. The evaluation is reasonably thorough (two vocoder backbones, in- and out-of-domain speech and singing, objective and subjective metrics, combination and ablation studies), though it compares against only two prior discriminators and does not report results against Avocodo, UnivNet, or Harmonic WaveGAN despite discussing them in related work.
Field Significance
Moderate — this paper offers a concrete, ablated improvement to a well-studied vocoder component (the time-frequency discriminator) and demonstrates a real mechanism (harmonic-order filter scaling with a fixed STFT front end) for addressing the specific odd-harmonic and temporal-alignment weaknesses of the prior CQT-based discriminator. Its scope is narrow: it modifies one module within existing GAN vocoder pipelines rather than proposing a new synthesis paradigm, and the gains, while consistent, are incremental rather than transformative.
Claims
- supports: Dynamic frequency resolution in a GAN vocoder’s time-frequency discriminator improves synthesis quality over fixed-resolution STFT discriminators, with the largest gains on signals with dense, fast-moving harmonic content such as singing voice.
Evidence: On out-of-domain singing, MS-SB-CQT and the proposed UnivHD (both dynamic-resolution) outperform the fixed-resolution MS-STFT discriminator on all four objective/subjective metrics for both HiFi-GAN and iSTFTNet backbones. (§IV-C1, Table I)
- refines: Aligning a discriminator’s frequency filters to exact integer multiples of a base frequency, rather than to power-of-two-spaced bins, extends dynamic-resolution time-frequency discriminators to cover odd harmonics that constant-Q-based designs systematically miss.
Evidence: Because CQT center frequencies are defined as f_c^k = f_min · 2^(k/B), the h-th harmonic h·f_c^k is only representable when h = 2^d; the proposed harmonic filter instead scales center frequencies by integer harmonic order h, so the k-th bin’s h-th harmonic is exactly h times the k-th bin’s first harmonic. (§III-A)
- supports: Combining a harmonic/dynamic-resolution discriminator with a fixed-resolution multi-scale STFT discriminator yields complementary gains beyond either alone.
Evidence: HiFi-GAN with both MS-STFT and UnivHD (+S+H) improves PESQ and MCD on in- and out-of-domain speech over HiFi-GAN with either discriminator alone, and also outperforms the MS-STFT + MS-SB-CQT combination on the same test sets. (§IV-C1, Table II)
- complicates: The number of harmonics modeled by a harmonic-filter discriminator is not a free hyperparameter to maximize; too few harmonics under-resolves the spectrum and too many can push the modeled range past the sample’s fundamental frequency, degrading pitch fidelity.
Evidence: Ablation over H ∈ {2, 8, 10, 12, 15, 20} shows near-flat performance in a mid-range but clear degradation at both extremes; at 24kHz sampling with H > 20 the discriminator cannot represent fundamental frequencies above roughly 600Hz within the first harmonic. (§IV-C2)
Limitations and Open Questions
The evaluation compares UnivHD against only two prior time-frequency discriminators (MS-STFT and MS-SB-CQT), despite the related-work discussion covering several other harmonic- and multi-resolution-based designs (Harmonic WaveGAN, UnivNet, Avocodo) that are not included as baselines. The MOS listening tests use a small pool (15 utterances per test condition, 10 raters), and confidence intervals are reported without a stated significance test. The method is only demonstrated as an add-on discriminator for two specific generator architectures (HiFi-GAN, iSTFTNet); its interaction with other generator families (e.g., diffusion- or flow-matching-based vocoders) is untested.
Wiki Connections
- GAN Vocoder — proposes a new discriminator module for the adversarial training loop used by GAN-based vocoders, evaluated as a drop-in addition to two established generators.
- Evaluation Metrics — reports a multi-metric comparison (PESQ, MCD, F0-RMSE, MOS) across in-domain and out-of-domain speech and singing test conditions for several discriminator variants.
- Subjective Evaluation — runs a 5-point MOS listening test with human raters across all discriminator and vocoder combinations, on both speech and singing test sets.
- HiFi-GAN — used unmodified as one of two generator backbones onto which the proposed discriminator is added.
- Vocos — cited as a prior multi-resolution spectrogram discriminator design operating on multiple frequency bands, discussed as related work.
- EnCodec — its Multi-Scale STFT (MS-STFT) discriminator is used as one of the two baseline discriminators compared against UnivHD, and later combined with it.
- BigVGAN — cited among the direct-generation GAN vocoder methods that time-frequency discriminators like the one proposed here are meant to improve.
- PeriodWave — cited as a recent flow-matching-based alternative to GAN-based waveform generation in the related-work discussion of direct generation methods.
- Singing Voice Synthesis and Conversion — trains and evaluates specifically on singing datasets (OpenSinger, M4Singer, Opencpop), with the discriminator’s largest gains concentrated on singing voice’s dense, fast-moving harmonic content.