arXiv · 2025 · Preprint
Chen et al. · → Paper · Demo: ? · Code: ?
Introduces HARMGEN, a suite of five attacks that coerce Large Audio-Language Model (LALM) based TTS systems into vocalizing harmful text content, and evaluates the effectiveness of both reactive and proactive defenses against it.
Problem
Prior misuse research on speech synthesis has concentrated almost entirely on speaker impersonation: making a TTS system sound like a specific target voice to deceive listeners or biometric systems. This paper identifies a distinct, largely unexamined threat surface: misuse of the linguistic content a TTS system is willing to speak, independent of whose voice it uses. Large Audio-Language Models (LALMs) used as TTS engines (e.g., GPT-4o-mini-audio, Gemini-2.5-live, Qwen-omni-turbo) undergo safety alignment and are typically deployed behind input/output content filters, so directly requesting harmful speech is usually refused. Standard LLM jailbreak techniques transfer poorly to this setting because a TTS-oriented LALM will vocalize whatever artifacts a jailbreak prompt introduces (role-play framing, special characters, paraphrases), corrupting the intended harmful utterance and, for longer content, exceeding context limits. The paper asks whether a TTS-specific attack methodology can compel LALM-based TTS systems to speak explicitly harmful content while evading both model-level safety alignment and pipeline-level content moderation.
Method
The authors first establish two baseline attacks: B1 feeds harmful text directly to a “read this aloud” prompt, and B2 embeds the harmful text as audio input with a “transcribe and speak” prompt. Testing these against two conventional TTS models and three LALM-based TTS models shows that conventional TTS models comply unconditionally (0% refusal), while LALM-based models refuse the majority of harmful requests via built-in safety alignment (§4.3).
Building on this, the paper introduces HARMGEN, five advanced attacks organized into two families. The text-modality family (harmful semantic concealment) keeps all input in text but disguises its toxicity: Concat uses a toxic span detector to split a harmful sentence into segments at toxic-word boundaries, synthesizes each segment separately (each individually appearing benign to the LALM), and concatenates the resulting audio with brief silences; Shuffle randomly permutes word order until the LALM stops refusing, then uses a forced aligner (Montreal Forced Aligner) to obtain word-level timestamps in the synthesized audio and reassembles the words into the correct order. The multi-modal family (audio modality exploitation) keeps the text prompt largely benign but smuggles only the toxic words in through the audio channel: Read supplies an audio clip of someone speaking the toxic word and asks the LALM to recognize and re-speak it within a templated sentence; Spell supplies letter-by-letter audio spelling of the word; Phoneme supplies IPA phoneme audio for the word (using the CMU Pronouncing Dictionary and grapheme-to-phoneme conversion for out-of-dictionary words), constrained to a candidate word list to aid recognition. Toxic span detection throughout uses Mudes. All attacks are evaluated as black-box, prompt-only interactions with five commercial LALM-based TTS systems (GPT-4o-mini-audio, GPT-4o-mini-tts, GPT-5-nano paired with CosyVoice 2.0 as an external vocoder, Gemini-2.5-live, Qwen-omni-turbo) across three datasets: Ethos (English hate speech), Mul-ZH (Chinese, derived from TOXICN), and a custom single-harmful-word dataset (Self). Effectiveness is measured with trial-level refusal rate (R1), sentence-level refusal rate (R2, the fraction of sentences refused in all 10 repeated trials), and a toxicity score (TS) computed on the ASR-obtained transcript of the output audio via Detoxify (English) or COLD (Chinese).
For countermeasures, the paper evaluates reactive defenses deployable by downstream platforms after audio has been generated (deepfake-audio detection with AASIST2, and transcribe-then-moderate text detection using Whisper plus OpenAI’s moderation API) and a proactive defense deployable by the TTS provider itself before release (moderating the model-emitted text that many LALMs return alongside the audio, buffered across segmented Concat queries and re-ordering-checked for Shuffle queries).
Key Results
Under at least one HARMGEN attack, all five commercial LALM-based TTS models that initially refused a majority of hate-speech prompts were driven to synthesize 100% of them at least once, with substantially elevated toxicity scores (§5.4.1). The text-modality attacks (Concat, Shuffle) generally outperform the multi-modal attacks (Read, Spell, Phoneme) on the primary R2/R1/TS ranking, since the latter can fail when the LALM misrecognizes the injected audio word, though the multi-modal attacks were judged (by the authors, informally) to yield more natural prosody since they synthesize the full sentence rather than disjoint segments. Combining attacks compounds effectiveness: Shuffle+Spell reduces R1 to 13.6% on GPT-4o-mini-audio/Ethos, versus 75.5% for Shuffle alone and 35.2% for Spell alone (§5.4.5). Attacks generalize across four tested output voice styles, across harmful categories (harassment, hate, illicit, violence), and to sentences whose harmful semantics carry no explicit harmful word.
On the defense side, the state-of-the-art deepfake detector AASIST2 achieves no more than 25% (GPT-4o-mini-audio) or 70% (GPT-5-nano) accuracy at a 0.5 threshold, and over 75% and 55% Equal Error Rate respectively, indicating it substantially underperforms on the high-fidelity outputs these attacks produce (§6.1.1). Reactive transcribe-then-moderate text detection catches at least 66% of harmful audio in the unperturbed case (Table 2), but its detection ratio drops from 82.93% to 57.07% (Concat) and from 78.19% to 43.29% (Shuffle) once the transcription model (Whisper) is targeted with imperceptible adversarial perturbations. In contrast, proactive moderation of the LALM’s own emitted text (before release) detects 57-93% of attack instances across all five HARMGEN attacks and cannot be bypassed post hoc since the audio has not yet left the provider’s pipeline (Table 3, §6.2).
Novelty Assessment
The contribution is primarily a novel evaluation/attack methodology rather than a new speech-generation architecture: HARMGEN combines existing components (toxic span detection, forced alignment, letter/phoneme audio synthesis) into new attack procedures targeting a threat model the authors argue is previously unexamined for TTS specifically (content-centric misuse, as distinct from the extensively studied speaker-impersonation deepfake threat). The audio-modality smuggling attacks (Read/Spell/Phoneme) exploit a genuinely TTS/LALM-specific attack surface that has no analogue in pure-text LLM jailbreaks, since it relies on the model’s willingness to treat audio input as content to transcribe and re-speak. The countermeasure analysis, comparing reactive audio/text-based detection against proactive provider-side text moderation, is a useful empirical contribution but does not itself introduce a new detection architecture.
Field Significance
Tip
high — this paper opens a content-centric misuse dimension for TTS/LALM safety evaluation that had previously been overshadowed by speaker-impersonation deepfake research, and demonstrates concretely that current commercial safety alignment and reactive content-moderation pipelines are systematically bypassable for this threat.
The paper provides the first systematic red-teaming study organized around what a TTS system says rather than whose voice it uses, spanning five commercial LALM-based systems and two languages. Its concrete finding that provider-side proactive text moderation substantially outperforms downstream reactive audio/text detection gives TTS providers and platform operators an actionable, evaluated mitigation rather than only a demonstrated vulnerability.
Claims
- supports: Large Audio-Language Model based TTS systems can be induced to bypass built-in safety alignment by smuggling disallowed content through the audio input channel rather than the text channel, even when the corresponding text-only jailbreak fails.
Evidence: The Read, Spell, and Phoneme attacks encode only the toxic words as audio (read, letter-spelled, or phoneme-pronounced) while keeping the surrounding text prompt benign; across five commercial LALM-based TTS systems this reduced refusal rates far below the text-only B1 baseline, e.g. driving models that refused a majority of Ethos prompts under B1 to synthesize 100% of them under at least one attack. (§5.4.1, Figure 3)
- complicates: Concealing the surface-level toxicity of input text (e.g., by fragmenting or reordering harmful sentences before synthesis) can simultaneously evade a system’s safety-alignment refusal and its input/output content moderation, without requiring any audio-modality access.
Evidence: The Concat and Shuffle attacks, which operate purely in the text modality, achieve 0% trial- and sentence-level refusal on GPT-4o-mini-tts and Gemini-2.5-live for the Self dataset (versus 100% for the baseline attack), and remain effective on sentences whose harmful semantics carry no explicit harmful word. (§5.4.1, §5.4.4, Figure 3, Figure 6)
- complicates: Deepfake-audio detectors calibrated on existing spoofed-speech benchmarks can substantially underperform when applied to high-fidelity audio produced by modern LALM-based TTS systems, weakening their reliability as a reactive content-safety countermeasure.
Evidence: The leaderboard-topping AASIST2 detector achieves no more than 25% accuracy and over 75% Equal Error Rate on attack-generated audio from GPT-4o-mini-audio, and detection is markedly better on GPT-5-nano’s output only because that pipeline relies on an external, lower-fidelity conventional TTS vocoder (CosyVoice 2.0) rather than end-to-end LALM synthesis. (§6.1.1, Figure 8)
- refines: Reactive, transcript-based content moderation applied after audio synthesis is materially less robust than proactive moderation of a model’s own emitted text applied before audio is released, because the former can be defeated by adversarial perturbations targeting the transcription step while the latter has no post-hoc bypass opportunity.
Evidence: Adding imperceptible adversarial perturbations to the synthesized audio (targeting the Whisper transcription model) reduces the reactive detection ratio from 82.93% to 57.07% for the Concat attack and from 78.19% to 43.29% for the Shuffle attack, whereas proactive moderation of model-emitted text (unavailable for post-hoc evasion) detects 57-93% of instances across all five attacks. (§6.1.2, §6.2, Table 2, Table 3)
Limitations and Open Questions
Warning
The paper’s countermeasure evaluation only tests text-modality harmfulness detection (transcript-then-moderate, or direct model-emitted text moderation) because, as the authors state, no available detector operates directly on the audio modality for harmful content; true audio-native content moderation therefore remains unassessed by this work.
Additional limitations acknowledged in the paper’s own discussion (§7): the Concat and Shuffle attacks degrade prosodic naturalness because they synthesize disjoint text segments or scrambled word orders and then reassemble audio, and Shuffle’s reconstruction quality further depends on forced-aligner accuracy. The Read, Spell, and Phoneme attacks depend on toxic span detection accuracy; missed or spurious spans reduce completeness, and sentences with many toxic words create a harder placeholder-disambiguation problem for the LALM. The Spell attack does not generalize to languages without letter-wise spelling (e.g., Chinese), and Phoneme requires reliable grapheme-to-phoneme resources that may be scarce for low-resource languages. All evaluated systems are commercial and closed-source; open-source LALM-based TTS models, where safety mechanisms could plausibly be removed via fine-tuning rather than prompt-level attack, are explicitly out of scope.
Wiki Connections
- Evaluation Metrics — introduces trial-level and sentence-level refusal-rate metrics (R1, R2) alongside a transcript-based toxicity score, extending TTS evaluation methodology from audio-quality metrics into safety/misuse-resistance measurement.
- Spoken Language Model — targets LALM-based TTS systems specifically, exploiting their ability to consume external audio input (word readings, spellings, phoneme sequences) and re-render it as speech, a capability with no analogue in text-only or conventional TTS pipelines.
- CosyVoice 2 — used as the external conventional-TTS vocoder to give the text-only GPT-5-nano model audio output capability for evaluation, and its lower-fidelity output is shown to be substantially easier for deepfake detectors to flag than end-to-end LALM-generated speech.