arXiv · 2026 · Preprint
Shih-Heng Wang et al. (University of Southern California / Carnegie Mellon University / Meta) · → Paper · Demo: ✗ · Code: ✓
A controlled study that trains neural audio codecs from scratch under matched configurations to isolate whether they generalize to unseen languages, whether speech-only pretraining hurts non-speech performance, and whether adding non-speech data during pretraining helps.
Problem
Neural audio codecs (NACs) are widely used both as compression tools and as the token vocabulary for speech/audio language models, but three basic generalization questions about them remain empirically unresolved: whether NACs generalize to languages unseen during pretraining, whether NACs pretrained only on speech transfer to non-speech domains (environmental sounds, music, animal vocalizations), and whether adding non-speech data to pretraining helps both domains or introduces trade-offs. Prior work addressing related questions compares off-the-shelf NACs, which differ in architecture, training objective, and pretraining data simultaneously, making it impossible to attribute any observed generalization gap to a single factor.
Method
The authors train neural audio codecs from scratch using the ESPnet-Codec toolkit, adopting the SoundStream architecture (SEANet convolutional encoder/decoder, 32-level residual vector quantizer, using only the first 8 codebooks for experiments) with a standard GAN training objective combining reconstruction loss (time-domain plus multi-scale mel-spectrogram), adversarial loss (STFT-based and waveform-based discriminators, generator/discriminator/feature-matching terms), and quantization commitment loss. Holding architecture and training procedure fixed, they vary only the pretraining data coverage across three configurations: English (English-only subset of the MLS corpus), Multilingual (all 8 MLS languages), and Multilingual+Audio (MLS plus AudioSet non-speech audio).
Each resulting NAC is evaluated on two axes. Signal reconstruction quality is measured on speech (LJSpeech for English, CSS10 German/Spanish subsets) and non-speech (AudioSet test set, excluded from pretraining) using 11 metrics from the VERSA toolkit: six intrusive/reference-based metrics (MCD, F0-CORR, PESQ, S-BERT, CI-SDR, VISQOL), three non-intrusive metrics (DNSMOS, UTMOS, SHEET-SSQA), and two perceptual metrics (WER via Whisper-Large, speaker similarity via ESPnet-SPK). Downstream application quality is measured via text-to-speech: a FastSpeech2 model is modified to predict the NAC’s 8 codec streams directly (rather than mel-spectrograms) instead of speaker embeddings, isolating codec effects, and its output is decoded through each NAC’s own decoder. HiFi-GAN (trained per-language) and a continuous mel-spectrogram FastSpeech2+HiFi-GAN pipeline serve as non-discretized upper-bound references throughout.
Key Results
Language generalization (Q1): the English-only-pretrained NAC outperforms the Multilingual-pretrained NAC on both Spanish and German reconstruction (15/0 vs. 2/16 aggregate best/worst counts across Table 1’s per-language metric sets) and remains competitive on downstream TTS for both unseen languages, contrasting with self-supervised speech models, which the authors note typically degrade on languages unseen during pretraining.
Non-speech generalization (Q2): on AudioSet non-speech reconstruction, the Multilingual+Audio-pretrained NAC outperforms speech-only-pretrained variants across all three non-speech metrics (MCD, CI-SDR, VISQOL: 3/0 best/worst vs. 0/2 for English-only), confirming that speech-only pretraining does not transfer to non-speech domains.
Combined pretraining (Q3): incorporating non-speech data during pretraining does not measurably hurt speech-task performance, contrary to the authors’ initial hypothesis of a trade-off: the Multilingual+Audio NAC scores 6/6 best/worst on speech reconstruction (Table 1) and 2/3 on TTS (Table 3), comparable to the speech-only configurations, while gaining the non-speech generalization above.
Novelty Assessment
The paper’s contribution is methodological rigor applied to a previously confounded question: by pretraining NACs from scratch under strictly matched architecture, loss formulation, and training procedure, and varying only pretraining data coverage, it isolates data coverage as the causal variable, something prior off-the-shelf NAC comparisons could not do. None of the individual components (SoundStream architecture, GAN codec training, VERSA metrics, FastSpeech2-on-codec-tokens evaluation) are new; the contribution is the controlled experimental design and the resulting empirical answers to three previously untested generalization questions, plus the counterintuitive finding that non-speech pretraining data does not cost speech performance.
Field Significance
Field Significance
high — This paper answers three basic, previously untested generalization questions about neural audio codecs under experimental controls that prior off-the-shelf comparisons lacked, directly informing pretraining-data curation decisions for any downstream codec-based speech or audio generation system, and produces a counterintuitive result (non-speech data does not trade off against speech performance) that runs against the paper’s own initial hypothesis.
Claims
- supports: Neural audio codecs pretrained on a single language can generalize to reconstruction and downstream synthesis of unseen languages without requiring multilingual pretraining coverage, unlike self-supervised speech representation models, which typically degrade on unseen languages.
Evidence: The English-pretrained NAC achieves more best-metric counts than the Multilingual-pretrained NAC on Spanish and German reconstruction (15/0 vs. 2/16 aggregate best/worst) and remains competitive on downstream TTS for both unseen languages (§4.2 Q1, Table 1, Table 3).
- supports: Neural audio codecs pretrained exclusively on speech data generalize poorly to non-speech signal reconstruction relative to codecs pretrained with non-speech data included.
Evidence: On AudioSet non-speech reconstruction, the Multilingual+Audio-pretrained NAC outperforms speech-only-pretrained NACs on all three non-speech metrics (MCD, CI-SDR, VISQOL), which score 0/2 best/worst (§4.2 Q2, Table 2).
- complicates: Including non-speech audio in a neural codec’s pretraining data does not necessarily trade off reconstruction or downstream-task quality on speech, contrary to the expectation that broadening pretraining domain coverage degrades in-domain performance.
Evidence: The Multilingual+Audio-pretrained NAC improves non-speech reconstruction (3/0 best/worst) while maintaining comparable speech-task performance to speech-only pretrained variants (6/6 best/worst in speech reconstruction, 2/3 in TTS) (§4.2 Q3, Table 1, Table 2, Table 3).
- complicates: Comparing off-the-shelf neural audio codecs across studies conflates architecture, training-objective, and pretraining-data differences simultaneously, making it difficult to attribute any observed generalization result to a single factor.
Evidence: The paper motivates training all NACs from scratch under strictly matched architecture and loss configuration specifically because prior studies relying on off-the-shelf NACs “yield limited insights due to variations in NAC’s implementation” (§1, §2.1).
Limitations and Open Questions
All findings are demonstrated on a single NAC architecture (SoundStream via ESPnet-Codec, default configuration); the authors explicitly note this and plan to extend to EnCodec, DAC, and HiFi-Codec to test whether the language- and domain-generalization conclusions hold across architectures.
Downstream application evaluation covers only text-to-speech, an inherently speech-only task; despite the paper’s non-speech reconstruction findings, no non-speech downstream task (e.g., music generation, environmental sound classification) is evaluated to confirm the reconstruction-level findings carry over to non-speech generation applications. The TTS evaluation uses only a non-autoregressive FastSpeech2 backbone; the authors note that more powerful autoregressive architectures (e.g., VALL-E-style codec language models) might interact differently with codec choice and are left for future work. Reported UTMOS scores for Spanish and German TTS are notably low, which the authors attribute to small per-language training data and UTMOS’s own limited cross-lingual generalization rather than to the codecs themselves, but this leaves some uncertainty in the downstream cross-lingual TTS quality comparisons.
Wiki Connections
- Neural Audio Codecs — isolates pretraining data coverage as a controlled variable to test neural codec generalization to unseen languages and non-speech domains, independent of architecture or training-objective differences.
- Evaluation Metrics — applies an 11-metric battery (six intrusive, three non-intrusive, two perceptual) via the VERSA toolkit to comprehensively characterize codec generalization across languages and domains.
- Multilingual Text-to-Speech — trains and evaluates separate FastSpeech2-on-codec-tokens TTS models across English, German, and Spanish to test whether codec language generalization carries through to downstream synthesis quality.
- ESPnet-Codec — supplies the toolkit and architecture used to pretrain all NACs in this study, and its reported TTS results are used as an external reference point in Table 3.
- VERSA — supplies the evaluation toolkit providing all 11 signal-reconstruction and perceptual metrics used to assess NAC generalization.
- HiFi-GAN — trained per-language as the vocoder baseline (mel-spectrogram to waveform, no discretization), serving as an upper-bound reference against the discretized NAC reconstructions.
- FastSpeech 2 — modified to predict codec tokens directly instead of mel-spectrograms, forming the downstream TTS evaluation harness used to test each NAC’s synthesis quality.
- EnCodec — named as an alternative NAC architecture the authors plan to test in future work to validate whether the generalization findings hold beyond SoundStream.