arXiv · 2023 · Preprint
Hubert Siuzdak · → Paper · Demo: ✓ · Code: ✓
Vocos is a GAN-based vocoder that generates Fourier spectral coefficients directly rather than operating in the time-domain, eliminating transposed convolution upsampling and achieving over an order-of-magnitude speed improvement while matching state-of-the-art audio quality.
Problem
Time-domain GAN vocoders such as HiFi-GAN and BigVGAN have dominated neural vocoding, but they rely on transposed convolutions to upsample feature sequences by several hundred times, which is computationally expensive and prone to aliasing artefacts. The time-frequency domain offers a more perceptually aligned representation of audio and benefits from highly efficient FFT algorithms, yet direct reconstruction of complex-valued spectrograms from neural networks had remained elusive, primarily because the phase spectrum is difficult to model: it is periodic, wraps around the range (-pi, pi], and its perceptual relevance is not fully understood.
Method
Vocos is a GAN vocoder with an isotropic architecture: the temporal resolution of the feature sequence remains constant throughout the network, matching the hop-size resolution of the STFT. There are no transposed convolutions; the only upsampling step is the computationally efficient Inverse Short-Time Fourier Transform (ISTFT) applied at the output.
The generator backbone adapts ConvNeXt (Liu et al., 2022) blocks: an input projection layer maps mel-spectrogram features into a hidden dimensionality, followed by a stack of 1D depthwise convolutional blocks with inverted bottlenecks, GELU activations, and Layer Normalization. The output head projects hidden activations into n_fft+2 channels and splits them into magnitude and phase components. Magnitude is represented by exponentiating the raw output (M = exp(m)), ensuring non-negativity. Phase is handled by a unit-circle activation: the raw phase output p is mapped via cosine and sine to obtain a complex coefficient STFT = M * (cos(p) + j*sin(p)), implicitly enforcing phase wrapping into (-pi, pi] without requiring a clipped nonlinearity.
The discriminator combines the multi-period discriminator (MPD) from HiFi-GAN and the multi-resolution discriminator (MRD) from UniVNet. Training uses hinge loss formulation for the adversarial objective, together with mel-spectrogram reconstruction loss (L1) and feature matching loss. The mel-spectrogram variant is trained on LibriTTS (train-clean + train-other) at 24 kHz for 2 million iterations with a batch size of 16 and AdamW optimiser.
A second variant, scaled down to 7.9M parameters to match EnCodec, is trained on DNS Challenge clean speech and conditioned on EnCodec codec tokens rather than mel-spectrograms. This positions Vocos as a drop-in decoder for any system producing EnCodec tokens, including the Bark TTS system.
Key Results
On the mel-spectrogram reconstruction task (LibriTTS test set), Vocos achieves MOS 3.62 and SMOS 4.55, statistically indistinguishable from BigVGAN (MOS 3.64, SMOS 4.54) and significantly ahead of HiFi-GAN (MOS 3.54) and iSTFTNet (MOS 3.57) (Table 2, Wilcoxon signed-rank test, p > 0.05). In objective metrics, Vocos leads all baselines on VISQOL (4.66) and PESQ (3.70), and achieves the best periodicity score (0.101), indicating fewer harmonic artefacts than time-domain GANs (Table 1).
On out-of-distribution music (MUSDB18), Vocos achieves average VISQOL 4.55, slightly above BigVGAN’s 4.54 and well above HiFi-GAN’s 4.39 (Table 3).
The speed advantage is decisive: on GPU, Vocos runs at 6696x real-time versus BigVGAN’s 98x and HiFi-GAN’s 495x. On CPU, Vocos is 169x real-time versus BigVGAN’s 0.4x and HiFi-GAN’s 5.8x (Table 6). The gain comes from eliminating learnable upsampling layers; the ISTFT is a fixed transform with O(N log N) complexity.
For the EnCodec decoder variant, Vocos substantially outperforms the original EnCodec decoder in perceptual quality: at 12 kbps, Vocos scores MOS 4.00 vs EnCodec’s 3.08; at 1.5 kbps the gap is even wider (2.73 vs 1.09) (Table 5). UTMOS scores corroborate this at all bandwidths (Table 4).
Ablations show that the unit-circle phase activation is essential (removing it reduces PESQ from 3.70 to 3.57), ConvNeXt blocks outperform standard ResBlocks (PESQ 3.70 vs 3.53), and Snake activations (used in BigVGAN) do not help in the Fourier-domain setting (Table 1).
Novelty Assessment
The core innovation is the combination of a fixed-resolution (isotropic) architecture with a principled unit-circle phase activation that handles phase wrapping implicitly. Neither component is entirely new in isolation: iSTFTNet already explored ISTFT-based vocoders, and GAN training for spectral reconstruction existed in earlier work. What Vocos contributes is the complete elimination of transposed convolutions (unlike iSTFTNet, which retains most upsampling layers) and the specific phase parameterisation that makes GAN training of complex spectrogram generators stable and high-quality. The result is a clean design with a well-motivated inductive bias and a genuine speed-quality Pareto improvement over the existing field.
The ConvNeXt backbone choice is borrowed directly from computer vision; its application here is engineering integration, but the isotropic architecture insight and the ISTFT-only upsampling are genuinely architectural in the vocoder context.
The EnCodec decoder application extends utility to the codec-token TTS pipeline without requiring new architectural ideas, but it is practically significant as it enables Vocos to serve as a higher-quality replacement decoder for any EnCodec-based system.
Field Significance
Tip
High — Vocos demonstrates that Fourier-domain GAN vocoders can match time-domain vocoders in perceptual quality while achieving dramatically better computational efficiency. The unit-circle phase activation resolves the longstanding training-instability problem that had blocked spectral-domain GAN vocoders. The paper also establishes that Vocos can serve as a higher-quality drop-in replacement for the EnCodec decoder, providing a practical upgrade path for codec-based TTS pipelines built on VALL-E-style autoregressive generation.
Claims
- Maintaining constant temporal resolution throughout a GAN vocoder, with ISTFT as the sole upsampling step, eliminates aliasing artefacts and dramatically reduces inference cost without sacrificing perceptual quality. (§3.1, §4.3, Table 6)
- Implicit phase wrapping via a unit-circle activation is essential for stable GAN training of complex-valued spectrogram generators; alternatives that clamp or clip phase angles substantially degrade output quality. (§3.2, §4.1.1, Table 1)
- Fourier-domain vocoders reduce periodicity errors more effectively than time-domain GANs, suggesting that modelling harmonics in the frequency domain provides a stronger inductive bias for voiced speech. (§4.1.1, Table 1)
- ConvNeXt blocks with isotropic architecture outperform dilated ResBlocks in the Fourier-domain vocoder setting, even though dilated convolutions were motivated by the need to expand receptive fields in time-domain models. (§4.1.1, Table 1)
- A Fourier-domain GAN vocoder trained as a neural codec decoder can substantially improve perceptual quality over the original codec decoder across all bitrates without architectural changes to the upstream codec. (§4.2, Table 5)
Limitations and Open Questions
Warning
Vocos’s mel-spectrogram MOS scores are reported on LibriTTS using crowd-sourced listeners; the ground-truth MOS (3.81) is noticeably below what might be expected for studio speech, suggesting the evaluation pool or headphone compliance filtering may limit the discriminative power of the subjective test. Statistical equivalence with BigVGAN is shown, but the test may be underpowered for detecting small differences.
The speed benchmarks are conducted without hardware-specific optimisations and use batch size 16, which is unlikely to reflect latency-critical single-sample streaming scenarios. The CPU advantage (169x real-time) is particularly striking but is not validated in a streaming or low-latency deployment setting.
The MDCT variant explored in Appendix A performs worse than the ISTFT variant across all metrics, suggesting the overcomplete STFT representation provides a beneficial inductive bias. Whether MDCT-based generation could become competitive with improved training strategies remains open.
The model is trained exclusively on 24 kHz speech and music; performance at higher sample rates (44.1 kHz or 48 kHz), which matter for high-fidelity TTS applications, is not reported.
Wiki Connections
- gan-vocoder — Vocos extends this concept by replacing transposed-convolution upsampling with ISTFT, enabling Fourier-domain GAN vocoding.
- neural-codec — The EnCodec decoder variant makes Vocos directly applicable as a higher-quality replacement in codec-based TTS pipelines; see also 2210.13438 (EnCodec).
- evaluation-metrics — Uses UTMOS (2204.02152), VISQOL, PESQ, and crowd-sourced MOS/SMOS with careful listener selection criteria.
- subjective-evaluation — The paper’s listening test methodology (headphone requirement, music-hobby filter, 1560 ratings from 39 participants) illustrates practice in controlled crowd-sourced MOS collection.
- autoregressive-codec-tts — Vocos serves as a drop-in decoder for Bark (GPT-style TTS using EnCodec tokens), connecting to the broader VALL-E-style autoregressive pipeline; see also 2209.03143 (AudioLM).
- 2206.04658 — BigVGAN is the primary baseline; Vocos achieves statistical parity in MOS while running 70x faster on GPU.