arXiv · 2026 · Preprint
Ye Lu et al. (Harbin Institute of Technology) · → Paper · Demo: ? · Code: ?
Shows that the speech tokens and frontend features that end-to-end speech LMs consume from user audio retain enough speaker identity for an attacker to recover a usable speaker embedding from three seconds of intercepted frontend output.
Problem
End-to-end speech LMs replace the text bottleneck of cascaded ASR-LLM-TTS pipelines with speech tokens that keep prosody, emotion, and other acoustic cues. A proposed privacy measure is split inference: the tokenizer runs on the user’s device and only its output reaches the cloud model. It is unclear whether these compact, task-oriented tokens suppress speaker identity or only re-encode it. Prior audio privacy attacks targeted speaker-recognition systems, anonymised x-vectors, membership inference, or the text outputs of audio LMs. None measured voiceprint leakage from the tokens passed between a speech LM’s frontend and its backbone.
This paper is a privacy attack on the input side of speech LMs. It generates no speech and reconstructs no waveforms.
Method
The threat model is an eavesdropper on the frontend-to-backbone link, or a curious service provider. The adversary can query the target frontend as a black box and see its outputs, the number of codebooks, and each codebook’s vocabulary size. It does not know the codebook embedding values. It also holds a public auxiliary speech corpus and picks a frozen, pretrained speaker encoder F. F defines the speaker space the attack must recover (§III-A). The attack (SIA, speaker inversion attack) trains a recovery model that maps a random contiguous slice of frontend output to an embedding close to F(x) for the victim’s waveform x.
The recovery model is Audio BERT (AuB) plus a projection head (§III-C). AuB keeps its own trainable lookup table for each codebook and sums the per-codebook embeddings at each frame. For frontends that also emit continuous features, a one-layer MLP projects them and adds them to the token embedding. A learnable CLS token is prepended, and a BERT backbone (BERT; hidden size 384) encodes the sequence. The CLS output gives a 512-dimensional vector, and a two-layer MLP head maps it to the target speaker space. All AuB weights start from random initialisation. Parameter count is not reported.
The training recipe, SpInv, has two stages (§III-D). Each utterance gives two local views (student), two global views (EMA teacher), and one teacher view for the frozen speaker encoder. Stage A (distillation pretraining, 60 epochs) combines five losses:
- cosine alignment to the frozen-encoder target embedding;
- relational matching of within-batch pairwise cosine geometry;
- DINO self-distillation from the EMA teacher;
- VICReg variance and covariance regularisers against collapse.
Stage B (40 epochs) adds an ArcMargin speaker-classification loss on the CLS representation and keeps the Stage A losses at lower weights. At attack time, only AuB and the projection head run on the intercepted slice.
The four attacked frontends cover three interface types (§IV-A):
- residual codebooks: Moshi’s Mimi codec, with one semantic and seven acoustic streams at 12.5 frames/s;
- residual codebooks: the Higgs3 eight-codebook tokenizer at 25 frames/s, also used by OmniVoice;
- hybrid: Kimi-Audio’s 12.5 Hz semantic tokens plus time-aligned Whisper features;
- continuous: Qwen3-Omni’s continuous 12.5 Hz Audio Transformer features.
The default target space is ECAPA-TDNN (192-d). ERes2Net (192-d) and CAM++ (192-d and 512-d) are alternative target spaces. All data comes from VoxCeleb under speaker-disjoint splits.
Key Results
The paper reports EER and minDCF as fractions, not percentages. Cosine similarity between recovered and target embeddings (authors’ term CosSim) is recorded here as SPK-SIM.
Leakage validation (Table I). AuB is trained with speaker labels on VoxCeleb1-dev and tested on the speaker-disjoint VoxCeleb1 verification split. It reaches low EER on every frontend:
| Frontend | EER | minDCF |
|---|---|---|
| Kimi-Audio | 0.0034 | 0.0171 |
| Higgs3 | 0.0066 | 0.0132 |
| Moshi | 0.0160 | 0.0512 |
| Qwen3-Omni | 0.0236 | 0.0836 |
Kimi-Audio has the lowest EER, and Higgs3 has the lowest minDCF.
Embedding recovery (Tables II, III). SpInv is trained on VoxCeleb2-dev with 3 s of frontend output and the ECAPA-TDNN target space.
On VoxCeleb2-test (Table II):
- SPK-SIM is 0.7536 for Kimi-Audio, 0.7527 for Higgs3, 0.7136 for Moshi, and 0.6683 for Qwen3-Omni.
- Higgs3 has the lowest EER (0.0174 vs 0.0178 for Kimi-Audio) and minDCF (0.0627), and the highest accuracy (0.9828 vs 0.9825).
- Kimi-Audio has the highest SPK-SIM by 0.0009. Higgs3 and Kimi-Audio are effectively tied.
On VoxCeleb1 (Table III):
- SPK-SIM is 0.7359 for Kimi-Audio, 0.7060 for Higgs3, 0.7037 for Moshi, and 0.6594 for Qwen3-Omni.
- Kimi-Audio has the highest SPK-SIM, the lowest KL, and the lowest minDCF.
- Higgs3 has the lowest EER (0.0489) and the highest accuracy (0.9513).
Qwen3-Omni, the only purely continuous interface, is weakest on SPK-SIM, accuracy, EER, and minDCF in both tables. On KL, Moshi is weakest in both tables. All four frontends score lower on VoxCeleb1 than on VoxCeleb2-test. The three token-based frontends stay above 0.70 SPK-SIM on both sets.
Input duration (Table IV, Moshi). SPK-SIM rises steadily with the length of the intercepted slice: 0.4887 at 1 s, 0.6677 at 2 s, 0.7136 at 3 s, 0.7642 at 5 s, 0.7949 at 8 s, and 0.8065 at 10 s. The verification metrics are not monotonic beyond 8 s. EER is 0.0110 at 8 s and 0.0124 at 10 s, and accuracy is 0.9892 at 8 s and 0.9878 at 10 s. minDCF falls sharply between 2 s (0.4050) and 3 s (0.0885).
Target speaker space (Table V, Moshi). Recovery works in all four target spaces, with SPK-SIM from 0.7136 (ECAPA-TDNN) to 0.7658 (CAM++ 192-d). CAM++ 512-d has the lowest EER (0.0135) and the highest accuracy (0.9867). ERes2Net has the lowest minDCF (0.0882, vs 0.0885 for ECAPA-TDNN).
The paper has no comparison baselines. It compares no other attacks, no simpler probe on the same tokens, and no waveform-level upper bound. It tests no defences, and it reports results without variance.
Novelty Assessment
The main contribution is empirical: a speaker-disjoint measurement of how much speaker identity survives in the input interfaces of four current speech LMs. The authors describe it as the first systematic study of voiceprint leakage from speech tokens. The attack model is an engineering integration of established parts: BERT encoding of summed codebook embeddings, embedding distillation, DINO self-distillation, VICReg regularisation, and ArcFace-style margin loss. The paper reports no ablation of the two stages or of the individual losses, so the contribution of each part is unknown. Supporting discrete, hybrid, and continuous interfaces in one model is convenient but simple: per-codebook lookup tables plus an additive MLP projection.
Field Significance
Moderate. The paper gives direct evidence that semantic-plus-acoustic codec tokens, and even an LLM-oriented continuous audio encoder, carry recoverable speaker identity into the speech LM backbone. Running the tokenizer on the device therefore does not, by itself, protect voiceprints. The evidence comes from automatic speaker-verification metrics on VoxCeleb only. Its practical value is as a cautionary data point for speech LM deployment and for work on privacy-preserving speech tokenizers.
Claims
- complicates: Discrete speech tokens used as speech LM inputs do not remove speaker identity; a learned probe on the token stream can verify unseen speakers with low error.
Evidence: An AuB probe trained on VoxCeleb1-dev reaches EER 0.0160 on Mimi tokens (Moshi), 0.0066 on Higgs3 tokens, and 0.0034 on Kimi-Audio frontend output on the speaker-disjoint VoxCeleb1 verification split. (§IV-B, Table I)
- supports: Speaker embeddings in a chosen pretrained speaker space can be recovered from a few seconds of speech LM frontend output, without access to the waveform or the codebook embedding values.
Evidence: SpInv trained on VoxCeleb2-dev reaches 0.7136 to 0.7536 cosine similarity to ECAPA-TDNN target embeddings on VoxCeleb2-test from 3 s of Moshi, Higgs3, or Kimi-Audio output, with EER 0.0174 to 0.0305. The three stay above 0.70 on VoxCeleb1. (§IV-C, Tables II, III)
- refines: Continuous LLM-oriented audio-encoder features leak somewhat less recoverable speaker identity than residual-codebook or hybrid token interfaces, but the leakage remains substantial.
Evidence: Qwen3-Omni’s continuous frontend gives the lowest recovery on both test sets (SPK-SIM 0.6683 and 0.6594; EER 0.0378 and 0.0783), yet its leakage-probe EER is still 0.0236. (§IV-B, §IV-C, Tables I-III)
- refines: Voiceprint recovery from speech tokens grows with the length of the exposed segment and saturates after several seconds.
Evidence: On Moshi tokens, SPK-SIM rises from 0.4887 at 1 s to 0.7136 at 3 s and 0.8065 at 10 s. EER and accuracy are best at 8 s and slightly worse at 10 s. (§IV-D, Table IV)
- supports: Speaker leakage from speech tokens does not depend on one speaker-encoder space; recovery transfers across target encoders and embedding sizes.
Evidence: With Moshi tokens, SpInv reaches SPK-SIM 0.7136 (ECAPA-TDNN), 0.7562 (ERes2Net), 0.7658 (CAM++ 192-d), and 0.7167 (CAM++ 512-d), with EER 0.0135 to 0.0443. (§IV-D, Table V)
Limitations and Open Questions
Warning
The paper has no baseline attacks, no simpler probes, no defences, and no ablation of SpInv’s two stages or five losses. The study shows that leakage exists, but it cannot show how much of the recovery is due to the specific SpInv recipe.
- The attack recovers speaker embeddings, not waveforms. The authors list waveform-level voice reconstruction as future work, so the actual impersonation risk is not measured (§V).
- All experiments use VoxCeleb celebrity speech from YouTube. Broader in-the-wild data is not tested (§V).
- The threat model assumes query access to the target frontend for building training pairs. The paper does not discuss deployments where the frontend is not separately exposed.
- Higgs3 is an autoregressive conversational TTS model, so its tokenizer is evaluated as a speech frontend rather than as a dialogue model’s input interface.
- Model size, layer count of the BERT backbone, code, and variance across runs are not reported.
- The paper proposes privacy-preserving tokenizers as future work but does not test any.
Wiki Connections
- Spoken Language Model — attacks the speech-input interfaces of four existing speech LMs with real VoxCeleb audio and measures how much speaker identity they pass to the backbone.
- Neural Audio Codec — treats speech tokenizers (Mimi residual codebooks, the Higgs3 tokenizer) as the object of study and shows that their tokens preserve speaker-discriminative information.
- Moshi — attack target; its Mimi codec tokens are the default interface for all ablations.
- Kimi-Audio — attack target with a hybrid semantic-token plus Whisper-feature interface; has the lowest leakage-probe EER (Higgs3 has the lowest minDCF).
- Qwen3-Omni — attack target representing purely continuous frontend features; the hardest interface to invert but still leaky.
- OmniVoice — named as another system that uses the attacked Higgs3 tokenizer as its audio frontend; not evaluated directly.
- BERT — the backbone of the AuB recovery model.
- SpeechGPT — cited as an earlier modular speech-LLM design that the token-based end-to-end systems under attack move away from.