arXiv · 2025 · Preprint
Ioannides et al. · → Paper · Demo: ? · Code: ✓
Proposes a two-stage speech tokenizer that pretrains a Joint-Embedding Predictive Architecture (JEPA) encoder with density-adaptive gating purely via self-supervised masked prediction, then fine-tunes it with finite scalar quantization and a HiFi-GAN decoder to produce a low-frame-rate, reversible discrete token stream.
Problem
Conventional neural speech codecs couple representation learning with waveform reconstruction from the start, forcing the encoder to prioritize low-level acoustic detail needed for high-fidelity resynthesis over higher-level semantic structure. This conflates two different goals: learning representations useful for downstream language-model-style speech generation, and preserving perceptual quality for reconstruction. The paper asks whether separating these concerns, by pretraining an encoder purely to predict masked latent representations (as in image-domain JEPA) before ever training a decoder, produces a speech tokenizer that is both semantically useful and highly compressed.
Method
The framework has two stages. Stage 1 trains a JEPA-style encoder using block-based temporal masking: an online (context) encoder and an EMA-updated target encoder both process the full waveform, and a predictor network must reconstruct the target encoder’s latent features for masked spans using only the visible context, with loss computed exclusively over masked positions. The encoder integrates a Density Adaptive Attention Mechanism (DAAM), an existing statistical-salience gating mechanism the same authors previously introduced for parameter-efficient fine-tuning, adapted here to gate temporal features based on learned Gaussian-mixture density estimates of each channel’s activation statistics, rather than pairwise self-attention similarity. The encoder itself is a convolutional downsampling stack (total stride 9600 at 24kHz, yielding a 2.5Hz latent frame rate) followed by 8 Conformer blocks, totaling 121.7M trainable parameters.
Stage 2 fine-tunes this pretrained encoder jointly with a quantizer and decoder. Quantization uses Finite Scalar Quantization (FSQ, an existing fixed, codebook-free scalar quantization scheme) over 128 dimensions, followed by a new mixed-radix packing scheme that groups G=7 FSQ dimensions per output token via modular arithmetic, reducing the raw 320 tokens/sec (one token per FSQ dimension) down to 47.5 tokens/sec while remaining exactly reversible. Reconstruction uses a HiFi-GAN-style decoder (69.2M parameters) with the same DAAM gating applied in its residual blocks, trained with L1 reconstruction loss, multi-resolution STFT loss, and adversarial loss from multi-period and multi-scale discriminators. Training uses roughly 9,000 hours of LibriLight audio at 24kHz across both stages, on 2x A100 GPUs.
Key Results
The paper reports no standard reconstruction-quality or downstream-task metrics (no MOS, WER, or similar); it explicitly frames its findings as qualitative and preliminary given a limited training budget. The two concrete quantitative results reported are: (1) a training-loss comparison showing that adding DAAM gating to the JEPA encoder lowers the converged masked-prediction MSE loss to roughly 0.09, versus roughly 0.17 for an otherwise identical JEPA encoder without DAAM, under matched training steps; and (2) a token-rate comparison in which the proposed mixed-radix-packed tokenizer operates at 2.5Hz / 47.5 tokens/sec, lower than the frame rates the paper lists for several existing codecs (Mimi at 12.5Hz, DualCodec at 12.5-25Hz, SoundStream and EnCodec at 75Hz, DAC at 86Hz) and than U-Codec’s 5Hz. No side-by-side reconstruction or intelligibility comparison against any of these baselines is reported.
Novelty Assessment
The individual building blocks (JEPA, DAAM, FSQ, HiFi-GAN) are all drawn from prior work, including the authors’ own earlier DAAM paper. The genuine new contributions are narrower: applying JEPA-style masked latent prediction to speech tokenization (rather than images), integrating DAAM gating into that encoder and into the HiFi-GAN decoder’s residual blocks, and the mixed-radix token-packing scheme that combines multiple FSQ dimensions into a single reversible integer token. The packing scheme itself is a real, self-contained algorithmic contribution independent of the JEPA framing. Overall this reads as an architectural combination paper with one genuinely new component (the packing scheme), evaluated only preliminarily.
Field Significance
low — This is a workshop paper (NeurIPS 2025 UniReps) presenting an early-stage combination of existing self-supervised, quantization, and vocoding techniques for speech tokenization, with the authors themselves characterizing the results as qualitative and preliminary due to limited compute. The low token rate is a notable engineering data point for the low-frame-rate codec design space, but the absence of any reconstruction-quality or downstream evaluation against the baselines it lists means the paper’s practical competitiveness is not yet established. Public code release supports independent follow-up.
Claims
- supports: Speech tokenizers can be built by first training a self-supervised encoder purely on masked latent-space prediction, with no waveform reconstruction objective, and only later attaching a quantizer and decoder for reconstruction.
Evidence: Stage 1 optimizes the JEPA encoder and predictor using only MSE loss on masked-region latent predictions, with no reconstruction, spectral, or adversarial loss present until Stage 2 fine-tuning is introduced separately. (§2.5, §3.4)
- supports: Statistical-salience gating over per-channel temporal statistics can improve self-supervised masked-prediction training efficiency for speech relative to an architecturally identical encoder without such gating.
Evidence: Under matched training steps, the JEPA encoder with DAAM gating converges to a lower masked-prediction MSE (~0.09) than the same encoder without DAAM (~0.17). (§3.1, Figure 4)
- supports: Combinatorial packing of multiple scalar-quantized code dimensions into a single token can substantially cut a neural tokenizer’s token rate without any codebook learning and while remaining exactly reversible.
Evidence: Packing G=7 of 128 FSQ dimensions per token via modular (mixed-radix) arithmetic reduces the raw token rate from 320 tokens/sec to 47.5 tokens/sec with perfect reversibility, placing the resulting frame rate below several existing codec baselines the paper tabulates. (§3.2, Tables 3-4)
- complicates: A neural tokenizer’s low frame rate or token rate is not sufficient on its own to establish competitiveness with existing speech codecs, absent standard reconstruction-quality or downstream evaluation.
Evidence: The paper explicitly states its evaluation is qualitative and preliminary due to limited computational budget, reporting a frame-rate comparison table and a training-loss curve but no WER, MOS, or reconstruction-fidelity comparison against any of the codec baselines it lists. (§6, §8)
Limitations and Open Questions
The paper reports no quantitative reconstruction-quality, intelligibility, or downstream-task evaluation against the codec and SSL baselines it compares against on frame rate alone; all reported findings are explicitly characterized by the authors as qualitative and preliminary.
The authors list several additional limitations: the block-masking strategy uses fixed span-length distributions that may not adapt to varying speech rate or linguistic structure; experiments are monolingual (English, LibriLight only), leaving generalization to tonal or morphologically rich languages untested; pretraining data scale (~9,000 hours) is modest relative to large-scale SSL speech systems; and the framework has not been extended to cross-modal (audio-visual or audio-text) settings.
Wiki Connections
- Neural Audio Codec — proposes a low-frame-rate, codebook-free (FSQ-based) speech tokenizer with a novel mixed-radix token-packing scheme, positioned against existing neural codec frame rates.
- Self-Supervised Speech — the encoder is pretrained entirely through JEPA-style self-supervised masked latent prediction before any reconstruction objective is introduced.
- GAN Vocoder — the Stage 2 decoder is a HiFi-GAN-style vocoder trained with multi-period and multi-scale adversarial discriminators to reconstruct waveforms from quantized tokens.
- Llama-Mimi — cited as a comparison point in the frame-rate table (Mimi’s semantic-distillation tokenizer operates at 12.5Hz versus this paper’s 2.5Hz).
- U-Codec — cited as a comparison point in the frame-rate table (U-Codec’s ultra-low-frame-rate codec operates at 5Hz versus this paper’s 2.5Hz).