arXiv · 2025 · Preprint
Soh Hui Hui et al. · → Paper · Demo: ? · Code: ?
SecureSpeech assembles an ASR + NER + LLM + TTS pipeline for dual anonymization of speaker identity and spoken content, using natural language speaker descriptions to generate privacy-preserving pseudo-speakers via Parler-TTS without requiring source speaker audio.
Problem
Speaker anonymization faces two distinct threats: voice identity theft (where voice cloning reconstructs a speaker’s characteristics from short samples) and content-based re-identification (where named entities in speech transcripts reveal identity even after the voice is altered). Prior work addresses these threats in isolation, and combined systems have relied on embedding-space speaker manipulation that gives users little intuitive control over how the anonymized voice sounds. The ASR + NER + TTS pipeline approach existed before but was constrained by the limited voice diversity of earlier TTS systems.
Method
The pipeline operates in four sequential stages. A wav2vec2-large-960h-lv60-self model, fine-tuned on LibriSpeech-960h with a CTC objective, transcribes the input utterance. A DeBERTa-L model with 24 transformer layers performs token classification to identify named entities (PERSON, PLACE, ORG, WHEN, NORP, QUANT, LAW) with timestamps in the transcription. Llama-3.2 then receives the tagged entities and replaces them with contextually coherent alternatives via few-shot prompt engineering.
The synthesis stage uses Parler-TTS Large v1, a decoder-only Transformer. The input is the cleaned transcript prepended to the sequence, and a natural language speaker description is encoded by a T5 encoder and injected into the decoder via cross-attention. The decoder produces discrete audio tokens decoded by the Descript Audio Codec (DAC) into a waveform. Parler-TTS was trained on approximately 45,000 hours of audio. Speaker descriptions specify gender, pitch, pitch modulation, speaking rate, accent, and recording conditions, giving users direct readable control over the pseudo-speaker profile without any reference audio.
Key Results
All three TTS-based systems compared (VALL-E, XTTS-v2, and the prompt-based approach) achieve FAR=0% on SLUE-VoxPopuli, indicating full speaker anonymization as measured by ECAPA-TDNN-based speaker verification (Table 1). The prompt-based system achieves WER 15.5% and PMOS 4.01 (Torchaudio-SQUIM), comparable to XTTS-v2 (WER 14.17%, PMOS 3.99) and substantially better in intelligibility than VALL-E (WER 33.69%). Anonymized speech achieves lower WER than original speech (23.67%) for most systems, because TTS synthesises from corrected text rather than disfluent natural speech.
Accent specification substantially affects intelligibility: WER ranges from 12.07% (Slovak) to 23.76% (Italian) across 38 accent descriptions while privacy (FAR=0%) holds across all (Table 2). Speaking rate choice also matters: normal rate achieves WER 12.39%, while very fast speech degrades to 18.56% (Table 3). The content protection components achieve ASR WER 19.00% on original speech and NER F1 71.80% on predicted transcriptions, with named entity replacement accuracy of 99.95% among detected entities.
Novelty Assessment
The contribution is engineering integration. All components are existing systems assembled into a dual-anonymization pipeline: wav2vec2 for ASR, DeBERTa-L for NER, Llama-3.2 for entity replacement, and Parler-TTS for synthesis. The specific novelty is substituting embedding-space speaker manipulation with readable natural language speaker descriptions, which provides transparent and flexible user control. The paper is the first to apply prompt-based TTS to speaker anonymization in this combined content-and-identity setting.
The empirical evaluation is limited by two factors. First, FAR=0% is achieved by all TTS-based approaches because TTS training speakers are disjoint from the evaluation set, making the metric uninformative for comparing methods. Second, the NER recall of 71.80% means a non-trivial portion of named entities are missed regardless of replacement accuracy, leaving real gaps in content privacy. Evaluation uses a single dataset (SLUE-VoxPopuli) with 161 speakers.
Field Significance
Low - SecureSpeech connects the speaker anonymization literature to prompt-based instruction-conditioned TTS, demonstrating a transparent alternative to embedding-space speaker manipulation. The components are fully off-the-shelf and the empirical findings are difficult to interpret given the evaluation design, but the pipeline structure can serve as a reference for future privacy-oriented TTS applications.
Claims
-
supports: Instruction-conditioned TTS enables speaker anonymization without relying on source speaker audio or speaker embeddings.
Evidence: Parler-TTS with natural language speaker descriptions achieves FAR=0% on SLUE-VoxPopuli, matching zero-shot TTS baselines (VALL-E, XTTS-v2) that require reference audio, while offering explicit control over gender, pitch, accent, speaking rate, and channel conditions. (§3.3, §3.4.1, Table 1)
-
complicates: False acceptance rate is an unreliable discriminator of speaker anonymization quality when evaluation design guarantees disjoint training and test speakers.
Evidence: FAR=0% is achieved by every TTS-based system regardless of architecture or conditioning, because TTS models are trained on speakers absent from the evaluation set; the metric cannot distinguish the proposed method from any TTS-based pipeline in this setting. (§3.4.1)
-
complicates: ASR + NER pipeline-based content privacy is limited by recognition and detection errors that allow a meaningful share of sensitive content to pass through.
Evidence: The ASR component achieves 19.00% WER on original speech and DeBERTa-L NER achieves only 71.80% F1 on predicted transcriptions, meaning many named entities are missed at the detection stage despite 99.95% replacement accuracy among detected entities. (§3.4.1, §4)
-
complicates: Generative variability in prompt-based TTS limits consistent voice identity reproduction across utterances sharing the same speaker description.
Evidence: Parler-TTS outputs may sound like different persons across generations even with an identical speaker description, undermining multi-utterance anonymization coherence and making deterministic identity control difficult. (§4)
-
supports: Speaker attribute choice in instruction-conditioned TTS introduces measurable variation in downstream speech intelligibility that practitioners must account for.
Evidence: WER varies from 12.07% (Slovak accent) to 23.76% (Italian accent) across 38 accent descriptions, and from 12.39% (normal rate) to 18.56% (very fast) across speaking rate settings, while speaker privacy (FAR=0%) remains constant throughout. (§3.4.2, Tables 2 and 3)
Limitations and Open Questions
Warning
FAR=0% is observed for all TTS-based systems including both baselines, because the evaluation speakers are disjoint from TTS training speakers. The privacy metric cannot distinguish the proposed approach from any TTS pipeline in this experimental design, which materially limits the privacy claims.
The NER component (71.80% F1) misses a substantial share of named entities, so content privacy is incomplete even when replacement accuracy is near-perfect among detected entities. WER-based content privacy evaluation does not capture semantic re-identification risks from context, paraphrase, or implicit references.
The pipeline is irreversible by design, making it unsuitable for forensic or audit contexts where original content must be recoverable. Prosody cannot be preserved without risking speaker re-identification, since prosodic patterns carry speaker-discriminative information. The study covers six speaker attribute dimensions but uses a single dataset, limiting generalisability.
Wiki Connections
- Instruction-Conditioned TTS - SecureSpeech uses Parler-TTS’s natural language speaker descriptions as the primary mechanism for generating pseudo-speakers without requiring any reference audio.
- Zero-Shot TTS - Parler-TTS is the zero-shot TTS synthesis backbone; VALL-E and XTTS-v2 serve as zero-shot TTS baselines in the speaker anonymization evaluation.
- Evaluation Metrics - the paper applies a privacy-utility evaluation framework combining FAR (speaker privacy), WER (content intelligibility), and PMOS (Torchaudio-SQUIM audio quality) to speaker anonymization pipelines.
- Neural Audio Codec - Parler-TTS relies on the Descript Audio Codec (DAC) for discrete audio token generation, making codec quality a dependency of the anonymized speech output.
- VALL-E (VALL-E) - used as a speaker anonymization baseline representing zero-shot TTS conditioned on source speaker audio; achieves FAR=0% but substantially degraded intelligibility (WER 33.69%) compared to the prompt-based approach.