arXiv · 2025 · Preprint
Siyu Wang et al. · → Paper · Demo: ✓ · Code: ✓
STCTS reconstructs natural-sounding voice communication at roughly 80 bits per second by explicitly splitting speech into text, sparse prosody keyframes, and an amortized speaker embedding, then resynthesizing with a conditioned TTS model at the receiver.
Problem
Voice communication in bandwidth-constrained settings (maritime satellite links, tactical radio, congested IoT networks) is expensive: traditional waveform codecs such as Opus degrade sharply below roughly 10 kbps, and even modern neural codecs such as EnCodec bottom out around 1 kbps. Prior attempts to push lower fall into two camps, each with a structural gap. Token-based semantic codecs (e.g., Vevo, A Novel Semantic Compression Approach for Ultra-low Bandwidth Voice Communication) reach roughly 650 bps by transmitting discrete acoustic tokens, but those tokens are opaque and cannot be independently inspected, upgraded, or selectively encrypted. Speech-to-text/text-to-speech (STT-TTS) pipelines for IoT and tactical links reach comparably low bitrates by transmitting only a transcript, but they discard prosody and speaker identity entirely, producing a generic, impersonal voice. Speech disentanglement methods factor speech into content, prosody, and timbre, but they transmit continuous frame-level latents at 50-100 Hz, which requires hundreds to thousands of bits per second and defeats the purpose of ultra-low-bitrate transmission. STCTS targets the gap between these three lines of work: expressive, speaker-preserving voice communication at a bitrate close to the pure-text baseline.
Method
STCTS is a three-stage pipeline built almost entirely from off-the-shelf pretrained components, rather than a jointly trained end-to-end model. At the sender, audio passes through Silero voice activity detection, FasterWhisper speech-to-text (STT) for linguistic content, classical signal-processing feature extraction (YIN pitch, RMS energy, syllable-nucleus speaking rate) for prosody, and an ECAPA-TDNN embedding (from SpeechBrain) for speaker timbre. Each stream is compressed with a strategy tailored to its temporal dynamics: text via Brotli with a conversation-specific adaptive dictionary (~70 bps); prosody via sparse keyframe sampling (0.1-1 Hz) with delta encoding, dead-zone non-uniform quantization, and Huffman coding (<14 bps); and the 192-dimensional speaker embedding transmitted once at call start (or on detected speaker change via cosine-similarity thresholding) and cached at the receiver thereafter, amortizing its cost toward zero over a call. Streams are sent over a WebRTC data channel with differentiated reliability: text and cache-miss timbre packets get high-priority retransmission, prosody keyframes get medium-priority retransmission, and prosody deltas are sent best-effort and simply interpolated across if lost.
At the receiver, text is decompressed, prosody is reconstructed at 100 Hz via cubic spline interpolation between the sparse received keyframes, and Coqui XTTS-v2 (a zero-shot voice-cloning TTS model with HiFi-GAN vocoding) synthesizes speech conditioned jointly on the transcript, the interpolated pitch/energy/rate contours, and the cached or newly received speaker embedding. The system defines three preset operating points (minimal, balanced, high-quality) that trade prosody update rate and STT/quantization precision for bitrate, with the prosody rate for each preset chosen from a separate empirical sweep (see Key Results). No component is trained or fine-tuned by the authors; the contribution is the decomposition, compression, and transport design, together with the empirical characterization of how sparsely prosody can be sampled without perceptible loss.
Key Results
On LibriSpeech test-clean, STCTS sustains 71.6-79.6 bps across its three quality modes (excluding the one-time speaker embedding), a 40-50x reduction versus Opus at 6 kbps and roughly 8x versus EnCodec at 1 kbps, while achieving NISQA-predicted MOS of 4.255-4.280 (Table I, Table II, §IV-C1). This is comparable to the reference Vevo numbers reported in the source paper (~650 bps, NISQA MOS 4.21), evaluated on a different test set and reported for context only. Opus and EnCodec score far higher on PESQ (2.284 and 1.334 vs. STCTS’s 1.138-1.324) and dramatically higher on STOI (0.906 and 0.805 vs. STCTS’s 0.150-0.162), but far lower on NISQA MOS (2.455 and 2.083). Word error rate for STCTS sits around 0.213-0.264 depending on mode and noise, well above Opus (0.032) and EnCodec (0.110), since STCTS’s WER reflects STT transcription accuracy rather than waveform-level distortion.
A separate parameter sweep varying the prosody update rate from 0.05 Hz to 20 Hz (§IV-B, Figure 5) found a bimodal relationship between sampling rate and NISQA quality: quality peaks both at sparse rates (0.05-1 Hz, MOS ~4.30-4.36 at 132-154 bps total) and at dense rates (>6 Hz, MOS 4.317 at 410 bps), with a “dead zone” in between (1-5 Hz) where quality is worse than either flank despite intermediate bitrate. This finding directly set the prosody rates used in the three quality presets (0.1, 0.5, and 1.0 Hz).
Under simulated channel noise (0.1-10% bit error rate) in high-quality mode, NISQA MOS degrades only slightly, from 4.298 (no noise) to a floor around 4.232 (10% BER), and speaker similarity stays within 0.658-0.669 across all noise levels, attributed to the high-priority retransmission given to text and speaker-embedding packets (§IV-C2, Table II, Table III).
Novelty Assessment
STCTS’s contribution is a communication system and compression protocol rather than a new generative model or architecture. Every core neural component (FasterWhisper, Silero VAD, ECAPA-TDNN, XTTS-v2) is used off-the-shelf and untrained by the authors, a design choice the paper makes explicit when contrasting itself with speech disentanglement work: it favors modularity and interpretability over learning a disentangled representation from scratch. The genuinely new elements are (1) the specific per-stream compression design (dead-zone delta quantization plus cubic-spline interpolation for prosody, amortized-and-cached transmission for timbre) that pushes an STT-TTS pipeline from a text-only baseline down toward a fully expressive one without a large bitrate penalty, and (2) the empirical discovery of a bimodal quality-vs-prosody-sampling-rate curve, which is a genuine, non-obvious systems-level finding rather than an incremental restatement of prior work. The overall framing (text as the semantic channel, rather than learned discrete audio tokens as in Vevo) is a deliberate engineering trade-off for interpretability and modularity, not a claim of higher fidelity or lower bitrate than token-based semantic codecs.
Field Significance
moderate — This paper contributes a carefully engineered, empirically validated system for ultra-low-bitrate voice communication built from existing components, plus a specific and reusable empirical finding (the bimodal prosody-sampling-rate/quality relationship) that could inform future sparse-conditioning designs in TTS-based communication or streaming synthesis. Its evaluation is limited to read audiobook speech from a single corpus and a single receiver TTS backbone, so the generality of the sampling-rate finding across TTS architectures and more spontaneous speech is not established by this paper alone.
Claims
- supports: Explicit text-prosody-timbre decomposition combined with heavily asymmetric per-stream compression can push a speech communication system’s sustained bitrate close to a pure-transcript baseline while retaining perceived naturalness and speaker identity.
Evidence: STCTS sustains 71.6-79.6 bps across three quality modes on LibriSpeech test-clean, a 40-50x reduction versus Opus (6 kbps) and ~8x versus EnCodec (1 kbps), with NISQA-predicted MOS of 4.255-4.280 and speaker similarity of 0.667-0.673. (§IV-C1, Table I, Table II)
- complicates: The relationship between prosody update rate and perceived speech quality in TTS-based reconstruction is not monotonic with bitrate; intermediate sampling rates can perform worse than both sparser and denser sampling.
Evidence: A sweep from 0.05 Hz to 20 Hz shows NISQA MOS peaks at 0.05-1 Hz (~4.30-4.36, 132-154 bps) and at >6 Hz (4.317, 410 bps), with a “mid-frequency valley” at 1-5 Hz that the authors attribute to perceptually salient interpolation discontinuities. (§IV-B, Figure 5)
- complicates: Frame-level intrusive intelligibility metrics such as STOI are unreliable indicators of quality for generative, TTS-based speech reconstruction because they assume waveform-level temporal alignment that such systems do not preserve.
Evidence: STCTS STOI scores (0.150-0.162) fall far below Opus (0.906) and EnCodec (0.805) despite high NISQA MOS (4.2+) and moderate WER (~0.235), which the authors trace to independent STT/TTS timing rather than genuine intelligibility loss. (§IV-C1, “STOI Analysis”)
- supports: Speaker identity can be preserved under significant channel degradation by transmitting a compact speaker embedding infrequently with high delivery priority, rather than streaming continuous acoustic features.
Evidence: Speaker similarity stays within 0.658-0.669 across simulated bit error rates of 0-10%, attributed to the amortized, cached, high-priority-retransmitted transmission strategy for the 384-byte ECAPA-TDNN embedding. (§IV-C2, Table II)
- complicates: Cascading an independent speech-to-text stage with a text-to-speech reconstruction stage imposes a transcription-accuracy floor that the surrounding compression system cannot fully compensate for.
Evidence: STCTS word error rate remains 0.213-0.264 across quality modes and noise conditions, substantially higher than Opus (0.032) or EnCodec (0.110), since errors originate in STT decoding rather than in the compression or transport layers. (§IV-C1, §IV-C2, Table II)
Limitations and Open Questions
Evaluated only on read audiobook speech from a single corpus and receiver TTS backbone
All reported results use LibriSpeech test-clean, which the authors themselves note does not capture spontaneous conversational phenomena (turn-taking, interruptions, overlapping speech, disfluencies) that a real deployment would encounter, and every reconstruction result depends on a single TTS backbone (XTTS-v2), so the bimodal prosody-sampling finding is not shown to generalize to other synthesis architectures.
The system also carries a theoretical end-to-end latency of roughly 1.5-2.0 seconds (excluding network propagation), driven by STT chunk buffering and TTS context requirements, which the authors say is acceptable for half-duplex push-to-talk use but not for rapid full-duplex interruption. Non-speech acoustic events (laughter, crying, background music) are filtered out by design and lost at the receiver, and overall reconstruction quality is bounded by upstream STT/TTS performance and language/model availability, since no component is retrained by the authors. The paper reports Vevo’s numbers from a different test set as reference only, so the direct comparison to Vevo should be read as illustrative rather than a controlled head-to-head result.
Wiki Connections
- Zero-Shot TTS — the receiver relies entirely on XTTS-v2’s zero-shot voice cloning, conditioning reconstruction on a transmitted speaker embedding rather than any speaker-specific training.
- Prosody Control — the system’s core contribution is an explicit, independently transmitted and quantized pitch/energy/speaking-rate signal that conditions TTS reconstruction separately from content and speaker identity.
- Speaker Adaptation — speaker identity is preserved across a call through a one-time, cached, high-priority speaker-embedding transmission rather than per-utterance re-estimation.
- Streaming TTS — the pipeline is explicitly designed for real-time operation, with streaming STT windows, streaming TTS synthesis, and a measured end-to-end latency budget.
- Evaluation Metrics — the paper documents a specific failure mode of intrusive intelligibility metrics (STOI) when applied to generative, resynthesis-based speech reconstruction rather than waveform-preserving codecs.
- A Novel Semantic Compression Approach for Ultra-low Bandwidth Voice Communication — used as the primary semantic-compression baseline (Vevo, ~650 bps token-based reconstruction); STCTS explicitly contrasts its text-based, interpretable representation against Vevo’s opaque discrete acoustic tokens while reporting comparable NISQA quality at a much lower bitrate.