arXiv · 2026 · Preprint
Kuzmin et al. (NTU / A*STAR / PolyU (HK)) · → Paper · Demo: ✓ · Code: ✓
Stream-Voice-Anon adapts a causal neural-audio-codec language model, originally built for streaming voice conversion, into a real-time speaker anonymization system by adding anonymization-specific conditioning and delay-sampling techniques.
Problem
Speaker anonymization must run in real time for applications like call centers, voice assistants, and live legal or medical recordings, yet streaming anonymization remains underexplored relative to offline approaches. Existing streaming anonymization pipelines mostly adapt conventional offline designs (continuous-feature voice conversion or DSP methods) rather than modern neural-codec language-model (NAC-LM) architectures, so they inherit speaker-information re-entanglement during vocoding and rely on privacy-enhancing steps such as k-means clustering that degrade utility. Meanwhile, causal LM-based streaming voice conversion has progressed rapidly and inherits the superior content/speaker disentanglement of offline NAC methods, but existing NAC-LM streaming systems are designed to preserve identity for voice conversion, not to conceal it, and therefore lack anonymization-specific mechanisms.
Method
Stream-Voice-Anon builds on a causal, NAC-LM-based streaming voice conversion architecture and adds anonymization-specific components on top. A causal content encoder converts 44.1 kHz audio to log-Mel features, downsamples through a ConvNeXt stack followed by an 8-layer causal decoder-only Transformer (RoPE, SwiGLU), and discretizes the output with a VQ layer (8192-entry codebook) to obtain speaker-invariant content tokens. Unlike its StreamVoice precursor, the content encoder is trained with an auxiliary ASR objective from scratch in parallel (using Whisper-tokenized transcripts) alongside distillation from HuBERT representations, both intended to strengthen the bottleneck’s separation of linguistic content from speaker identity. A separate pre-trained, frozen acoustic encoder extracts multi-codebook acoustic tokens aligned to the content encoder’s frame rate, decoded at inference by a Firefly-GAN vocoder (from FishSpeech) with 8 codebooks at ~21.5 Hz.
The core generative model is a two-stage autoregressive voice conversion (ARVC) module. Following an interleaved AR factorization (content token, then the acoustic codes for the same frame, repeated frame-by-frame), a Slow-AR transformer (12 layers) produces a per-frame latent state conditioned on the content token and a global speaker embedding (extracted with CAM++ or the SparkTTS global tokenizer and placed at the start of the sequence), and a lightweight Fast-AR transformer (4 layers) autoregressively decodes that frame’s 8 acoustic codebooks conditioned on the Slow-AR latent. Because emitting acoustic codes immediately after a compressed content token is difficult without look-ahead, the model is trained with a per-utterance random emission delay sampled as d ~ U{1,…,8} frames, generalizing the fixed-delay scheme of prior work (StreamVoice+) into a single model that can trade latency for accuracy at inference time without retraining.
Anonymization is introduced entirely at the conditioning stage, independent of the source speaker’s own voice. A prompt pool built from VCTK, ESD, VoxCeleb1, and CREMA-D supplies K reference utterances per anonymization request; several selection strategies are compared (single fixed speaker, single random utterance, four random VCTK utterances, one utterance per dataset, or four emotion-varied CREMA-D utterances). Selected prompts are shuffled, and their content and acoustic tokens are concatenated to form the diverse conditioning context fed to the ARVC decoder. In parallel, the K prompts’ speaker embeddings are averaged and linearly combined with a speaker embedding sampled from a Gaussian distribution (weight alpha = 0.9 toward the prompt average) to produce the anonymized target embedding used as the global conditioning signal. Because all of this conditioning can be precomputed independent of the source utterance, it can run ahead of time and does not add to streaming latency. The content encoder and ARVC are trained on LibriHeavy and CommonVoice for 400,000 steps on 8 H100 GPUs (AdamW, initial learning rate 1e-4, effective batch size 128).
Key Results
Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon is compared against the prior streaming SOTA, DarkStream, at matched latency budgets (~180-210 ms) and against an offline baseline, EASY. At 180 ms latency, the cross-ds-4rnd configuration reduces WER by up to 46% relative versus DarkStream’s best comparable configuration (4.71% vs. 9.52%), and the cremad-emo-4rnd configuration improves emotion-recognition UAR by up to 28% relative (44.59 vs. 34.73/34.49). Privacy against the lazy-informed attacker is comparable to slightly better than DarkStream (down to 46.53-47.72% EER, near the 50% random-guess ceiling), but privacy against the harder semi-informed attacker (an attacker whose ASV model has been fine-tuned on anonymized data) degrades by roughly 13-15% relative (18.63-18.98% vs. 21.83% EER for DarkStream) (§4.1, Table 1). The offline baseline EASY still outperforms both streaming systems substantially on every axis (2.70% WER, 63.81 UAR, 45.89% semi-informed EER), underscoring a persistent online/offline anonymization gap.
A prompt-diversity ablation shows that increasing conditioning diversity from a single fixed VCTK speaker to four cross-dataset utterances raises semi-informed EER from 15.92% to 18.98%, supporting the hypothesis that diverse prompts hinder an adapting attacker, while lazy-informed EER changes only marginally (44.96-47.72%) since that attacker does not adapt to the anonymization scheme (§4.2, Table 2). A separate experiment varying latency from 130-440 ms shows privacy is essentially invariant to latency, while intelligibility improves up to 180 ms and then flattens; the dynamic-delay model slightly trails a fixed-delay model (d=4, 240 ms) on WER but preserves stronger privacy and allows latency to be chosen post-hoc without retraining (§4.3, Figure 2). On latency and throughput, the system reaches 180 ms end-to-end latency on a laptop RTX 3060 GPU (RTF 0.93) and as low as 151 ms on an H200 server GPU, with larger audio chunks trading responsiveness for real-time margin (§4.4, Table 3).
Novelty Assessment
The contribution is primarily engineering integration: the interleaved AR factorization, two-stage Slow/Fast AR decoding, and neural-codec vocoder are adopted from prior streaming voice conversion work (StreamVoice, StreamVoice+, FishSpeech), and the pseudo-speaker Gaussian-mixing idea is explicitly inspired by prior offline codec-LM anonymization systems. The paper’s own contribution is repurposing this architecture for a different task (concealing identity rather than preserving it) via a set of anonymization-specific conditioning techniques: speaker-embedding mixing, diverse prompt selection strategies, and the dynamic per-utterance delay-sampling scheme. Of these, dynamic delay sampling is the one genuinely new training-procedure element, generalizing a fixed-delay baseline into a single model with inference-time latency flexibility; it is a modest but real methodological addition rather than a new architecture. The prompt-diversity ablation is a useful, well-isolated empirical contribution that identifies which conditioning choices matter most for robustness against adaptive attackers.
Field Significance
Moderate — the paper demonstrates that NAC-LM streaming voice conversion architectures can be adapted to speaker anonymization with substantial gains in intelligibility and emotion preservation over the prior streaming baseline at matched latency, while showing that concealment of identity against adaptive (semi-informed) attackers remains an unresolved trade-off distinct from the identity-preserving VC setting these architectures were originally designed for. It establishes per-utterance dynamic delay sampling as a reusable technique for latency-flexible streaming speech generation, and its prompt-diversity findings offer a concrete, ablated design lever for other streaming anonymization systems.
Claims
- supports: Neural-codec-based causal language-model architectures developed for streaming voice conversion can be repurposed for streaming speaker anonymization, yielding substantially better intelligibility and expressiveness than prior streaming anonymization pipelines built on conventional causal encoders at comparable latency.
Evidence: At matched ~180-200 ms latency, the proposed system reduces WER by up to 46% relative and improves emotion-recognition UAR by up to 28% relative versus the prior streaming SOTA, DarkStream. (§4.1, Table 1)
- supports: Sampling the emission delay per utterance during training, rather than fixing it, lets a single streaming autoregressive voice conversion model trade latency for accuracy at inference time without retraining.
Evidence: Training with d ~ U{1,…,8} enables selecting the delay at inference; compared with a fixed-delay model at the same 240 ms latency budget, the dynamic-delay model trails slightly on WER but preserves privacy better and removes the need to retrain for different latency targets. (§2.5, §4.3, Figure 2)
- complicates: Diversifying speaker-prompt conditioning in streaming anonymization mainly strengthens privacy against adaptive attackers, with limited additional benefit against naive attackers that already sit near chance performance.
Evidence: Increasing prompt diversity from a single fixed speaker to four cross-dataset utterances raises semi-informed EER from 15.92% to 18.98%, while lazy-informed EER stays within a narrow 44.96-47.72% band across all strategies. (§4.2, Table 2)
- complicates: Real-time neural-codec-LM anonymization systems that close the intelligibility gap with offline pipelines can still trail on anonymization strength once the attacker is allowed to adapt to the anonymization scheme, leaving a residual online/offline privacy gap.
Evidence: The best configuration’s semi-informed EER (18.98%) is both below the prior streaming baseline’s 21.83% and far below the offline baseline EASY’s 45.89%, showing degradation relative to DarkStream under the harder threat model despite gains elsewhere. (§4.1, Table 1)
Limitations and Open Questions
Warning
The system requires GPU acceleration and cannot operate in real time on CPU-only hardware, restricting deployment in resource-constrained or purely CPU-based settings, by the authors’ own admission.
Privacy against the semi-informed (adaptive) attacker degrades relative to the prior streaming baseline even as intelligibility and emotion preservation improve, and the paper does not resolve this trade-off beyond noting it as future work. No human-listening evaluation (MOS or similar) is reported anywhere in the paper; all quality, privacy, and utility claims rest on automatic metrics (WER, UAR, EER), so perceptual naturalness of the anonymized speech is untested. The content encoder’s auxiliary ASR objective requires transcript-paired training data (LibriHeavy), adding a data requirement beyond what a purely self-supervised content encoder would need. Comparisons are limited to one prior streaming baseline (DarkStream) and one offline baseline (EASY); broader comparison against other NAC-LM or DSP-based anonymization systems is not reported.
Wiki Connections
- Voice Conversion — repurposes a two-stage autoregressive voice-conversion architecture as the anonymization backbone, replacing identity-preserving speaker conditioning with anonymized pseudo-speaker embeddings.
- Streaming TTS — extends causal, low-latency streaming synthesis techniques (interleaved AR factorization, dynamic delay sampling) to the speaker anonymization task, reaching 180ms end-to-end latency on consumer GPU hardware.
- Neural Audio Codec — relies on a frozen multi-codebook neural audio codec (FishSpeech’s Firefly-GAN, 8 codebooks) as the discrete acoustic token space the causal LM predicts autoregressively.
- Self-Supervised Speech — distills the causal content encoder from HuBERT representations as part of obtaining speaker-invariant linguistic tokens.
- Speaker Adaptation — conditions the AR decoder on a mixed pseudo-speaker embedding formed by combining averaged prompt-derived speaker vectors with a Gaussian-sampled synthetic embedding.
- DarkStream — the direct prior-SOTA streaming speaker anonymization baseline this paper benchmarks against throughout, at matched latency budgets under the VoicePrivacy 2024 Challenge protocol.