arXiv · 2025 · Preprint
Della Libera et al. (Concordia University / Mila-Quebec AI Institute / Université Laval) · → Paper · Demo: ? · Code: ✓
FocalCodec-Stream extends the FocalCodec single-codebook codec to streaming inference via multi-stage causal distillation of WavLM, achieving 0.55-0.80 kbps at 80 ms theoretical latency while outperforming existing streamable codecs on naturalness, intelligibility, and downstream task performance.
Problem
Most neural audio codecs depend on full future context during encoding or decoding, making them unsuitable for real-time applications such as speech assistants and interactive dialogue systems. Existing streamable codecs either operate at higher bitrates (1.5-1.6 kbps), rely on multiple codebooks, neglect semantic content, or sacrifice downstream task performance to achieve low-latency operation. The original FocalCodec addressed the low-bitrate, single-codebook, and semantic-preservation goals but remained non-streamable. The challenge is enabling streaming inference without sacrificing the hybrid acoustic-semantic representations that make single-codebook codecs useful for speech language modeling.
Method
FocalCodec-Stream adapts the original FocalCodec architecture for streaming via a four-stage causal distillation process, targeting an 80 ms latency budget. The encoder consists of the first six layers of WavLM-large, made causal by replacing standard convolutions with causal convolutions and substituting full-context gated relative attention with sliding window chunked attention over 80 ms (4 feature frames). The compressor and decompressor, built on focal modulation networks, also use causal convolutions and replace global pooling with a large-kernel causal convolution acting as a learnable sliding window moving average. The binary spherical quantizer is naturally streamable. The Vocos decoder replaces ConvNeXt blocks with causal convolutions and substitutes the inverse STFT with a linear projection followed by flattening.
A lightweight refiner module is inserted after the decompressor to bridge the distribution gap between features from the causal encoder and the full-context WavLM features the decoder was trained on. The refiner operates chunk-wise with a residual feedforward layer, using the full 80 ms budget to improve alignment with WavLM representations before decoding.
The four distillation stages proceed progressively. Stage 1 distills only the positional embedding (an 8.4M-parameter convolution with 2.56 s receptive field) from the full-context teacher using an L2 loss. Stage 2 converts and distills the remaining attention and convolutional components, applying pairwise L2 losses at each of the six encoder layers with a reversed linear schedule that weights the deeper layers more heavily. Stage 3 trains the causal compressor-quantizer-decompressor on top of the distilled encoder. Stage 4 jointly fine-tunes the full system with the refiner, using the full Libri-Light corpus (60k hours). Training scales progressively from LibriTTS (585 hours) in Stage 1 to Libri-Light-medium (5k hours) in Stages 2-3. Three variants are trained with codebook sizes of 2,048, 4,096, and 65,536 entries at 50 Hz token rate.
Key Results
On English speech resynthesis (LibriSpeech test-clean), FocalCodec-S@50-65k achieves UTMOS 3.85 and dWER 3.68% at 0.80 kbps, compared to the non-streaming FocalCodec@50 (UTMOS 4.05, dWER 2.18% at 0.65 kbps). Against streaming competitors, it outperforms Mimi6 (UTMOS 3.44, dWER 4.77% at 0.83 kbps) and HILCodec (UTMOS 2.86, dWER 6.65% at 1.50 kbps) on both naturalness and intelligibility. On multilingual MLS evaluation, all three FocalCodec-Stream variants remain competitive with hybrid baselines (Mimi5/6) while substantially outperforming purely acoustic codecs, and surpass PAST (which collapses to 49.35% dWER on multilingual data due to English-only supervised fine-tuning).
For one-shot voice conversion on VCTK (Table 2), FocalCodec-S@50-65k achieves UTMOS 3.10, dWER 22.71%, and speaker similarity 92.5%, the only streaming codec to simultaneously achieve high intelligibility and high speaker fidelity. Mimi6 reaches comparable speaker similarity (91.3%) but at 110% dWER, while PAST achieves lower dWER (18.28%) but very poor similarity (68.5%).
Downstream task evaluation (Table 3) shows FocalCodec-Stream variants rank among the top streaming codecs on ASR (WER ~17%), speaker identification, keyword spotting, and intent classification, while significantly outperforming acoustic codecs on speech enhancement and separation. The performance gap with the non-streaming FocalCodec@50 narrows substantially on generative tasks (SS DNSMOS 3.68-3.69 vs. 3.71). Ablations confirm that both the refiner and Stage 4 fine-tuning contribute meaningfully, with Stage 4 removal causing the largest degradation (UTMOS 3.87 to 3.78, dWER 4.39% to 5.05%).
Novelty Assessment
The core novelty lies in the multi-stage causal distillation framework for adapting WavLM to streaming inference while preserving hybrid acoustic-semantic representations. The insight of distilling progressively from positional embeddings (Stage 1) to attention (Stage 2) to the full codec system (Stages 3-4), combined with the refiner module to close the distribution gap between causal and full-context features, is a genuine contribution. The architecture itself largely reuses components from FocalCodec and Vocos; streaming adaptation of each module (causal convolutions, sliding window attention) follows established practices in streaming ASR and vocoder work.
As an extension of the original FocalCodec, this paper primarily demonstrates that single-codebook low-bitrate coding and streamability are jointly achievable. The evaluation is thorough, covering both reconstruction and downstream task quality across speech resynthesis, voice conversion, and seven discriminative and generative tasks, using comparable bitrate configurations across all baselines.
Field Significance
Moderate — FocalCodec-Stream demonstrates that causal distillation of self-supervised encoders can enable streaming at bitrates (0.55-0.80 kbps) previously only accessible to non-streamable codecs, closing a practical gap for real-time speech generation applications. The paper’s primary value is as an engineering and evaluation contribution: it extends an existing codec design to a new deployment setting and provides a systematic multi-task comparison across reconstruction, voice conversion, and downstream tasks, which is useful for benchmarking purposes.
Claims
-
supports: Single-codebook binary quantization at sub-1 kbps bitrates can match or exceed multi-codebook streaming codecs on naturalness and intelligibility in speech resynthesis.
Evidence: FocalCodec-S@50-65k achieves UTMOS 3.85 and dWER 3.68% at 0.80 kbps with a single codebook of 65,536 entries, outperforming Mimi6 (0.83 kbps, 6 codebooks, UTMOS 3.44, dWER 4.77%) on both metrics. (§4.1, Table 2)
-
supports: Multi-stage causal distillation of self-supervised speech encoders preserves hybrid acoustic-semantic representations for downstream tasks under streaming constraints.
Evidence: FocalCodec-Stream variants trained via four-stage WavLM distillation outperform acoustic streaming codecs (EnCodec, AudioDec, HILCodec) on ASR, keyword spotting, and intent classification despite operating at lower bitrates; the 65k variant matches or surpasses PAST on all discriminative and generative tasks except ASR. (§4.2, Table 3)
-
complicates: Supervised domain-specific fine-tuning of hybrid codecs achieves strong in-domain intelligibility at the cost of multilingual generalization.
Evidence: PAST, fine-tuned on English data, achieves the lowest English dWER among streaming codecs (4.04%) but degrades severely on multilingual MLS (49.35% dWER), whereas FocalCodec-Stream (trained on English-only Libri-Light but without supervised task fine-tuning) retains competitive multilingual performance (19.88% dWER). (§4.1)
-
supports: Voice conversion quality in streaming codecs requires joint optimization of intelligibility and speaker fidelity; gains on one metric alone are insufficient for practical use.
Evidence: Mimi6 achieves competitive speaker similarity (91.3%) in one-shot VC on VCTK but at 110% dWER, while PAST achieves lower dWER (18.28%) at only 68.5% speaker similarity; FocalCodec-S@50-65k is the only streaming codec to simultaneously achieve high values on both (dWER 22.71%, Sim 92.5%). (§4.1, Table 2)
-
supports: A lightweight refiner module bridging causal and full-context feature distributions substantially improves perceptual quality in distilled streaming codecs.
Evidence: Ablation on FocalCodec-S@50-4k shows that removing the refiner degrades UTMOS from 3.87 to 3.84 and dWER from 4.39% to 4.65%; omitting Stage 4 fine-tuning (which jointly trains the refiner) has a larger effect, raising dWER to 5.05% and reducing speaker similarity from 96.3% to 95.8%. (§4.3, Table 4)
Limitations and Open Questions
Training data is limited to English (LibriTTS, Libri-Light), so the multilingual robustness observed on MLS reflects generalization rather than explicit multilingual training. A performance gap with the non-streaming FocalCodec@50 remains across most metrics, particularly in ASR WER (17% vs. 15.33%) and SI error rate (2.18% vs. 0.35%), reflecting the inherent cost of the 80 ms latency budget. Downstream generative tasks (TTS, speech language modeling) are left for future work, so performance of the codec’s discrete representations in autoregressive modeling pipelines is not yet demonstrated.
The paper does not provide listening tests or crowd-sourced MOS, relying instead on the automatic UTMOS predictor for naturalness assessment.
Wiki Connections
- Neural Audio Codec — FocalCodec-Stream proposes a streaming single-codebook codec operating at 0.55-0.80 kbps via causal distillation, extending the neural codec design space to real-time inference.
- Streaming TTS — the paper’s central contribution is enabling streaming speech coding at 80 ms latency without sacrificing the hybrid representations needed for downstream speech generation tasks.
- Self-Supervised Speech — WavLM-large forms the encoder backbone and serves as the distillation teacher throughout all four training stages, making self-supervised speech pre-training central to the codec design.
- Voice Conversion — the paper evaluates one-shot VC on VCTK as a proxy for the codec’s content-speaker disentanglement, reporting UTMOS, dWER, and speaker similarity results across streaming codecs.
- Evaluation Metrics — the paper evaluates across a comprehensive multi-task suite (resynthesis, VC, ASR, speaker ID, SER, KS, IC, SE, SS) aligned with the DASB benchmark, reporting UTMOS, dWER, SPK-SIM, WER, and DNSMOS.
- VALL-E (VALL-E) — cited in introduction as an application of neural codec tokens to zero-shot TTS; motivates the need for low-bitrate streamable codecs capable of supporting autoregressive speech language modeling.