arXiv · 2025 · Preprint
Collette et al. (Systems & Technology Research) · → Paper · Demo: ? · Code: ?
Applies the Vevo zero-shot voice transfer pipeline as a semantic codec for ultra-low bandwidth voice communication, transmitting only content-style tokens at 650 bps and sending compressed timbre samples per speaker, achieving downstream task performance comparable to Encodec at up to 4x lower bitrate.
Problem
Conventional audio codecs (Opus, EnCodec) encode all features of speech at a fixed quality level, including acoustic properties that are redundant across a conversation. For downstream tasks such as transcription, sentiment analysis, and speaker verification, many encoded bits carry information that is irrelevant to the task, wasting bandwidth. At the same time, low-level speaker timbre properties remain stable across an entire recording, yet most codecs re-encode them in every transmitted packet. The challenge is to craft a codec that allocates bits to semantically meaningful features and reuses persistent features such as speaker timbre to reduce long-term temporal redundancy.
Method
The system splits the Vevo generative pipeline across a sender-receiver link. Vevo consists of an autoregressive transformer that produces content-style tokens and a flow-matching transformer that reconstructs waveforms from those tokens combined with a timbre Mel spectrogram. At the sender, the audio is tokenised into content-style tokens using Vevo’s autoregressive encoder; these tokens use a codebook of size 2^13 at 50 Hz, requiring only 650 bps for real-time streaming. The receiver reconstructs audio by feeding the content-style tokens and a timbre Mel spectrogram into Vevo’s flow-matching decoder.
Speaker timbre is transmitted separately and only once per new speaker, detected via continuous diarisation. Two timbre encoding strategies are evaluated. In the codec-switching scheme, the system temporarily switches to EnCodec (at 1.5 or 3 kbps) for a few seconds to transmit a complete timbre sample; the receiver extracts a Mel spectrogram from the decoded audio. In the speaker-embedding scheme, a 128-dimensional speaker embedding from the Zonos TTS model is quantised to 8 bits and transmitted; the receiver uses Zonos to synthesise a timbre reference waveform from the embedding plus an arbitrary text prompt. Once a speaker’s timbre is registered, subsequent utterances from that speaker are decoded entirely from the 650 bps content-style stream, and the long-run average bitrate trends toward that figure as the number of registered speakers grows.
The timbre-embedding approach trades latency (d_sample + d_transmit seconds per new speaker) for lower total bitrate, since no audio codec stream is required for timbre. The codec-switching approach introduces a brief period at the codec’s nominal bitrate but avoids the latency of on-demand embedding generation.
Key Results
Evaluation uses 1,000 clips from VoxCeleb1 (5 to 20 seconds, English, single-speaker), balanced across four emotion classes. Baselines are Opus at 5 kbps and EnCodec at 1.5 and 3 kbps.
On the content-only path at 650 bps (no timbre), the system achieves WER 0.15 compared to Encodec’s 0.20 at 1.5 kbps, and sentiment classification accuracy of 59% vs 60% (within 1% at 2x lower bitrate). Speaker verification (SpkRec) also matches Encodec at 1.5 kbps.
When a 4-second timbre sample is added via Encodec@3kbps, WER improves to 0.13 and speaker verification reaches 0.78, a 4-percentage-point gain over Encodec@3kbps (0.74) at equivalent long-run bitrate.
Perceptual metrics show consistent advantage: all proposed configurations outperform Encodec on UTMOS and NISQA (e.g., Vevo content-only UTMOS 1.88 vs Encodec@1.5kbps UTMOS 1.26; NISQA 4.31 vs 2.04). However, low-level fidelity metrics (PESQ, STOI) are worse for all proposed configurations, reflecting that the generative reconstruction does not preserve signal-level characteristics.
Noise resilience testing (bit-flip rates 0.1% to 10%) shows no measurable degradation at 0.1% BER. At 1% BER, sentiment accuracy and speaker verification still exceed Opus at 5 kbps, though WER degrades to 0.32. Performance collapses at 10% BER across all metrics.
Novelty Assessment
The contribution is engineering integration: Vevo’s architecture and EnCodec already exist; the paper applies Vevo’s sender/receiver split as a novel framing for semantic communications. The idea of separating content tokens from timbre and transmitting them over different channels at different rates is creative and well-executed, but relies entirely on pre-trained models without any new training, fine-tuning, or architectural modification. The evaluation is thorough given the paper’s scope (three downstream tasks, two timbre schemes, noise resilience), but the test set is modest (1,000 clips from a single corpus) and restricted to English. The perceptual quality improvement over EnCodec on UTMOS and NISQA is noteworthy, though the full-reference metric degradation (PESQ, STOI) reveals the trade-off clearly.
Field Significance
Moderate — This paper shows that factorising speech into content-style tokens and per-speaker timbre representations enables a practical ultra-low bandwidth voice communication scheme, achieving roughly 2 to 4 times lower bitrate than EnCodec with comparable semantic task performance. The approach demonstrates that generative TTS and voice transfer models can be repurposed as semantic codecs without retraining, suggesting a broader principle for task-oriented speech compression. The evaluation is limited to English and a single corpus, and the latency constraints from timbre transmission remain unresolved for strict real-time applications.
Claims
-
supports: Factorising speech into semantic components (content, style, timbre) enables task-relevant transmission at lower bitrates than general-purpose codecs that encode all features uniformly.
Evidence: Content-style tokens at 650 bps achieve WER 0.15 and sentiment accuracy 59%, within 1% of EnCodec at 1.5 kbps (WER 0.20, accuracy 60%), while using approximately half the bitrate. (§3.3, Table 1)
-
supports: Reusing a compressed speaker reference transmitted once per speaker can maintain speaker similarity at lower average bitrate than encoding full audio continuously.
Evidence: Vevo with Zonos speaker embedding (1 sec timbre) achieves SpkrSim 0.54, matching EnCodec at 1.5 kbps (0.54), while long-run average bitrate approaches 650 bps as registered speakers accumulate. (§3.4, Table 1)
-
complicates: Generative reconstruction from semantic tokens improves perceptual quality scores but degrades low-level signal fidelity metrics relative to waveform-level codecs.
Evidence: Vevo configurations outperform EnCodec on UTMOS and NISQA across all bitrates, but score lower on PESQ and STOI because the flow-matching decoder was not trained to preserve signal-level characteristics. (§3.4, Table 1)
-
complicates: Semantic codec approaches face a latency and error-propagation trade-off that general-purpose codecs avoid.
Evidence: Timbre transmission introduces per-speaker latency L = d_sample + d_transmit; errors in that one-time transmission cause permanent voice reconstruction inaccuracies until a correction is transmitted. (§2.2, §4)
-
supports: Neural semantic token representations are moderately robust to channel bit errors, maintaining acceptable downstream task performance at realistic noise levels.
Evidence: At 0.1% bit-flip rate the system shows no measurable degradation; at 1% BER sentiment classification (0.63) and speaker verification (0.77) still exceed Opus at 5 kbps. Performance collapses only above 10% BER. (§3.5, Table 2)
Limitations and Open Questions
Warning
Timbre transmission errors are permanent: a corrupted one-time speaker embedding results in incorrect voice reconstruction for all subsequent utterances from that speaker until a retransmission is triggered. The paper does not propose an error-recovery mechanism.
The system does not handle overlapping speakers; the paper acknowledges this would require sender-side speaker separation at additional bitrate cost. Evaluation is restricted to English and does not address non-English languages, which are explicitly noted as out-of-distribution for several downstream models. The test set of 1,000 clips from VoxCeleb1 is relatively small for claims about general-purpose voice communication performance. The real-time latency introduced by speaker embedding generation and transmission is uncharacterised across different network conditions.
Wiki Connections
- Neural Audio Codec — the paper positions its approach relative to EnCodec and Opus, demonstrating that semantic token representations can substitute for traditional codecs on downstream tasks at lower bitrate.
- Voice Conversion — the paper leverages Vevo, a zero-shot voice imitation model, as its core reconstruction engine; the receiver-side decoding is functionally VC conditioned on a transmitted timbre reference.
- Zero-Shot TTS — Vevo’s zero-shot capability allows arbitrary speaker timbre to be applied from a short reference, which is central to the per-speaker timbre reuse scheme.
- Evaluation Metrics — the paper employs an ensemble of perceptual and task-specific metrics (UTMOS, NISQA, PESQ, STOI, WER, SpkrSim) and identifies a divergence between no-reference perceptual scores and full-reference signal fidelity measures.
- Speaker Adaptation — the timbre registration scheme, in which a per-speaker reference is transmitted once and reused for all subsequent utterances, is conceptually analogous to speaker adaptation in TTS.