arXiv · 2025 · Preprint
Qiang et al. · → Paper · Demo: ✓ · Code: ✓
SecoustiCodec disentangles semantic from paralinguistic information in a single-codebook streaming codec by combining frame-level cross-modal contrastive learning with a VAE+FSQ quantization scheme, achieving competitive reconstruction quality at 0.27 kbps with only 20 tokens/s.
Problem
Speech codecs designed as tokenisers for language models face a core tension: they must produce discrete tokens rich in semantic content while stripping speaker-dependent paralinguistic information (timbre, emotion), yet multi-codebook systems complicate downstream LM training and most single-codebook alternatives sacrifice either reconstruction quality, semantic purity, or streaming capability. Distillation-based disentanglement approaches (SpeechTokenizer, MimiCodec) rely on HuBERT or WavLM representations that themselves carry residual paralinguistic content, leaving incomplete separation. Meanwhile, existing single-codebook models (WavTokenizer, BigCodec, TAAE) use non-causal architectures that preclude real-time streaming in voice-dialogue applications.
Method
SecoustiCodec models three independent representations from speech: a single-codebook discrete semantic encoding S, a continuous acoustic embedding A, and a global-level paralinguistic embedding G, designed to satisfy the approximate relationship S + G ≈ A. The architecture is built around three coordinated sub-systems: acoustic modeling, semantic modeling, and paralinguistic modeling.
The speech encoder, semantic projection, acoustic projection, semantic connector, and speech decoder are all designed with causal operations to support streaming encoding and decoding. The speech encoder consists of causal SeaNet encoder blocks. The acoustic projection is a causally masked transformer with 8 layers and 8-head attention at 512 dimensions, using RoPE positional encoding. The semantic projection shares this transformer architecture and appends a VAE head that outputs mean and variance for latent sampling. Quantisation then applies FSQ (Finite Scalar Quantization) on the sampled latent, mapping it into a bounded low-rank codebook space. This VAE+FSQ combination addresses the long-tail token distribution problem that standard VQ-VAE suffers, achieving 98.06% codebook utilisation compared to substantially lower rates for VQ-VAE and SimVQ.
The paralinguistic encoder is a VAE with 6 convolutional layers and SE-ResNet blocks, extracting a fixed-length global vector from a 3-second paralinguistic reference window. A semantic connector (sharing parameters with the acoustic projection) then takes S and G as inputs to predict the acoustic embedding A, enforcing semantic completeness through an MSE loss against the ground-truth A.
Semantic disentanglement is achieved through frame-level cross-modal contrastive learning. Phoneme embeddings (from a convolutional phoneme encoder with 4 transformer layers) and speech frame embeddings are aligned in a joint multimodal space at frame resolution. A symmetric cross-entropy loss over the resulting (B×T_s) × (B×T_s) similarity matrix trains the speech encoder to maximise similarity between each speech frame and its corresponding phoneme while minimising similarity to all others. This approach outperforms an ASR-classification loss baseline in speaker similarity (0.71 vs 0.64 SPK-SIM) and emotion consistency on the voice conversion task.
Training follows a two-stage acoustic-constrained multi-stage optimisation strategy. Stage 1 trains only the acoustic modules using mel-spectrogram reconstruction loss. Stage 2 freezes those modules and trains semantic and paralinguistic modules, progressively introducing KL terms for both the semantic VAE and the paralinguistic VAE using linear warm-up schedules. This staged approach is critical: ablations without staging show significant degradation across all metrics. The vocoder is HiFi-GAN.
Key Results
On the primary reconstruction evaluation (Table I, evaluated on the combined LibriTTS + AISHELL-3 test set), SecoustiCodec at 0.27 kbps (20 tokens/s, single codebook, causal) achieves PESQ 1.77, SPK-SIM 0.92, Emo-Sim 0.93, and WER 11.58. At 1 kbps (80 tokens/s) it reaches PESQ 2.58, SPK-SIM 0.95, and WER 3.99.
In the single-codebook comparison, BigCodec (1 kbps, non-causal) is the only single-codebook baseline that outperforms SecoustiCodec on PESQ (2.55 vs 1.77 at comparable bitrates), but BigCodec does not support causal streaming. Compared to causal multi-codebook models — Encodec (1.5 kbps), MimiCodec (0.55 kbps) — SecoustiCodec(0.27kbps) exceeds or matches them on PESQ, SPK-SIM, Emo-Sim, LSD, and WER despite operating at lower bitrate and token rate.
Latency on a single V800 GPU thread: initial latency 12.08 ms (lower than Encodec’s 13.39 ms and MimiCodec’s 84.4 ms). The encoder RTF is 0.002 (same as Encodec); the full system RTF including HiFi-GAN vocoder is 0.04.
Voice conversion ablation (Table IV): replacing the paralinguistic embedding at the decoder stage, SecoustiCodec achieves SPK-SIM 0.71 and Emo-Sim 0.86, versus SecoustiCodec-ASR-Loss (SPK-SIM 0.64, Emo-Sim 0.69), confirming that frame-level contrastive learning produces cleaner semantic disentanglement than phoneme classification loss.
Novelty Assessment
The core novelty is the combination of three independently motivated design choices into a cohesive single-codebook streaming system: (1) explicit paralinguistic modeling as a bridge between semantic and acoustic encodings rather than treating residual as noise; (2) VAE+FSQ for quantisation, which demonstrably improves codebook utilisation and reduces long-tail token frequency compared to VQ-VAE and SimVQ; and (3) frame-level cross-modal contrastive learning rather than SSL distillation for semantic disentanglement.
Each component individually has antecedents — FSQ from prior work, contrastive text-audio alignment from CLIP-based models, and VAE paralinguistic encoders from style transfer research. The integration is the contribution. The claim of superiority over BigCodec, the most competitive single-codebook baseline, is qualified: BigCodec (1 kbps, non-causal) achieves PESQ 2.55 vs SecoustiCodec(1kbps) 2.58, a margin that is minimal. The streaming constraint genuinely differentiates SecoustiCodec, but comparisons against causal baselines (Encodec, MimiCodec) involve different bitrate operating points, making direct quality comparisons approximate.
The dependence on phoneme-level text labels during training is a notable constraint: the contrastive learning objective requires text-speech paired data with duration-aligned phoneme sequences, limiting applicability to low-resource or purely unsupervised settings.
Field Significance
Moderate — SecoustiCodec addresses a real bottleneck in codec design for speech LMs: the difficulty of achieving semantic disentanglement, high codebook utilisation, and causal streaming in a single-codebook framework simultaneously. The VAE+FSQ quantisation finding and the superiority of frame-level contrastive learning over HuBERT/WavLM distillation for paralinguistic removal are the most transferable results, offering concrete alternatives for codec designers targeting streaming voice-dialogue applications.
Claims
- Distillation from self-supervised models such as HuBERT or WavLM does not achieve true semantic disentanglement in speech codecs, as these representations inherently retain paralinguistic content. (§II.A, §V.B)
- Frame-level cross-modal contrastive learning between phoneme and speech representations produces cleaner semantic-paralinguistic separation than phoneme classification loss in neural codecs. (§V.D, Table IV)
- VAE-augmented finite scalar quantization (VAE+FSQ) achieves substantially higher codebook utilisation and reduced long-tail token distribution compared to VQ-VAE in single-codebook speech codecs. (§V.C, Figure 4, Table II)
- Explicit modeling of paralinguistic information as a reconstruction bridge between semantic and acoustic encodings improves both semantic completeness and reconstruction fidelity in low-bitrate streaming codecs. (§III.B, §V.B)
- Staged training that freezes acoustic modules before introducing semantic and KL losses is necessary for stable convergence in multi-objective codec training. (§III.E, §V.C, Table II)
Limitations and Open Questions
Warning
The contrastive learning objective requires duration-aligned phoneme-level text labels during training. The authors acknowledge this as a key limitation and flag unsupervised disentanglement as the primary future direction.
Evaluation is conducted only on English (LibriTTS) and Mandarin (AISHELL-3) data; generalisation to other languages, especially low-resource or tonal languages with different phoneme structures, is unverified. The model uses HiFi-GAN as its vocoder, which introduces separate dependencies, vocoder artefacts, and the bulk of the decoding RTF — excluding vocoder time, the decoder RTF drops to 0.001. Codebook utilisation of 98.06% is reported for the VAE-FSQ variant but without comparison to the final SecoustiCodec configuration in the ablation table directly; the connection between codebook saturation and downstream LM training quality is asserted but not demonstrated empirically in TTS or dialogue tasks.
Wiki Connections
This paper is most directly relevant to neural-codec, where it proposes a new design point for single-codebook streaming codecs with explicit semantic disentanglement. The frame-level contrastive alignment technique connects to disentanglement. The motivation for streaming support links to streaming-tts and spoken-language-model, as the codec is explicitly designed for voice-dialogue downstream tasks.
In-corpus papers cited: VALL-E (the foundational codec LM for TTS that motivates the semantic token design) and CosyVoice (a supervised semantic token TTS system that also uses contrastive-aligned semantic tokens).