arXiv · 2025 · Preprint
Zhao et al. (Shanghai Jiao Tong University) · → Paper · Demo: ? · Code: ✗
A watermark-free TTS traceability framework that jointly trains a TTS generator and a discriminator with aligned (non-adversarial) objectives, enabling attribution of synthesized speech to its source model without embedding explicit watermark information in the audio.
Problem
Prior approaches to tracing synthetic speech fall into two families, and both have fundamental weaknesses. Deepfake detection methods perform binary classification (real vs. fake) but cannot identify which specific model generated an utterance, and they degrade rapidly on out-of-domain TTS systems. Explicit watermarking methods embed n-bit information into audio or the vocoder stage, but this introduces audible quality degradation, creates security exposure (forged watermarks), and is architecturally constrained to codec-based TTS pipelines. A third gap affects both families: none of the existing approaches can trace synthesized speech back to a specific acoustic model when the vocoder is replaced, because the vocoder’s own fingerprints tend to dominate.
Method
The framework pairs a TTS generator with a dedicated discriminator and trains both jointly using aligned optimization objectives, inverting the standard GAN dynamic. In a GAN, the generator minimizes detectability while the discriminator maximizes it. Here, both components are trained to maximize the discriminator’s ability to recognize audio from its paired generator, producing output that carries subtle, human-imperceptible signatures unique to that generator’s learned distribution.
The generator is F5-TTS (a flow-matching, Diffusion Transformer-based system), selected for its end-to-end differentiability, which is required so that the discriminator’s BCE loss can backpropagate through the entire synthesis pipeline. The discriminator uses wav2vec 2.0 as a feature extractor and a 9-layer LCNN as the binary classifier. Training proceeds in 10 iterative loops, each comprising three stages: (1) fine-tune F5-TTS with a combined loss of the standard F5-TTS objective and the discriminator’s BCE signal; (2) use the fine-tuned model to synthesize a new discriminator training set with speaker and text effects held constant via voice cloning; (3) train the discriminator on this refreshed dataset. To prevent the discriminator from merely learning the difference between fine-tuned and original F5-TTS outputs, audio from the original open-source checkpoint is included as a negative class during training loops, then removed in a final 10-epoch cleanup pass. The vocoder (Vocos) is frozen throughout and only the acoustic model (Mel spectrogram stage) is trained, making traceability vocoder-robust by design.
Key Results
In-domain classification (F5-TTS outputs vs. real audio) is near-trivially easy for both the baseline independent discriminator and the jointly trained system, with both achieving AUC above 0.999 and EER below 0.3% (Table 2). The meaningful comparison is out-of-domain generalization, where the discriminator must determine whether audio was generated by the jointly trained F5-TTS variant when it also sees real audio and audio from three other TTS systems (CosyVoice, CosyVoice 2, E2-TTS). Joint training improves AUC from 0.8823 to 0.9421, EER from 18.99% to 11.50%, and accuracy from 85.60% to 89.38% (Table 3). Speech quality is preserved and marginally improved: WER decreases from 2.202% to 2.033%, speaker similarity increases from 0.659 to 0.661, and UTMOS rises from 3.926 to 3.958 on the LibriSpeech-PC test-clean set (Table 5). Robustness testing across nine audio manipulation scenarios shows strong stability against resampling, time-stretching, reverb, volume scaling, and MP3 compression, but meaningful degradation under additive noise (MUSAN, AUC drop of 0.1233 out-of-domain) and pitch shifting (AUC drop of 0.0901 out-of-domain) (Table 4). The comparisons within each configuration are internally consistent, though the baseline and proposed system use different F5-TTS checkpoints (original vs. fine-tuned), which is correctly accounted for.
Novelty Assessment
The inverted GAN objective, applying aligned rather than adversarial losses to generator and discriminator, is a straightforward but genuinely new idea applied to TTS attribution. Prior watermarking work for TTS (TraceableSpeech, WMCodec, P2Mark) all embed explicit bit sequences; this paper is the first to demonstrate model-specific traceability without any such embedding. That said, the constituent components are entirely pre-existing: F5-TTS is the unchanged base model, wav2vec 2.0 + LCNN is a well-established anti-spoofing discriminator architecture, and the iterative fine-tuning loop draws on standard practice. The evaluation covers one TTS backbone (F5-TTS) and one discriminator configuration, which limits how far the generalization results transfer. The claim that the method is vocoder-robust is mechanically well-reasoned but tested only implicitly (audio is generated with Vocos throughout). The restriction to end-to-end differentiable TTS models explicitly excludes VALL-E-type discrete token architectures, which represent a substantial portion of the current frontier.
Field Significance
Moderate — This paper introduces watermark-free TTS attribution as a distinct problem framing, separating model traceability from explicit watermark injection. The joint training mechanism is conceptually clean and the empirical gains over independent discriminator training are real, though the evaluation is single-model and the practical deployment scenario (needing access to the fine-tuned model weights to produce traceable audio) constrains applicability. The contribution is most useful as a proof of concept that implicit, training-time traceability is achievable, providing a direction for follow-on work covering discrete token-based TTS and stronger adversarial attacks.
Claims
-
supports: Joint training of a TTS generator and a speech discriminator with aligned optimization objectives improves out-of-domain generalization of model-specific attribution compared to training the discriminator independently.
Evidence: Jointly training F5-TTS and a wav2vec 2.0 + LCNN discriminator improves out-of-domain AUC from 0.8823 to 0.9421 and EER from 18.99% to 11.50%, while independent discriminator training on the same data shows substantially weaker generalization. (§5.4, Table 3)
-
supports: Watermark-free TTS traceability through joint training can preserve or slightly improve synthesis quality relative to the base model.
Evidence: The jointly trained F5-TTS achieves WER 2.033% (vs. 2.202% baseline), UTMOS 3.958 (vs. 3.926), and speaker similarity 0.661 (vs. 0.659) on LibriSpeech-PC test-clean, indicating no quality penalty from the added traceability objective. (§5.6, Table 5)
-
complicates: TTS attribution discriminators based on audio feature patterns are vulnerable to additive noise and pitch manipulation, but robust to common audio processing operations.
Evidence: Out-of-domain AUC drops by 0.1233 under additive noise (MUSAN) and 0.0901 under pitch shifting, while remaining stable (within 0.025 AUC) under resampling, time-stretching, reverb, volume scaling, and MP3 compression. (§5.5, Table 4)
-
complicates: Differentiability of the full synthesis pipeline from text input to output waveform is a necessary precondition for watermark-free traceability via joint generator-discriminator training, excluding discrete token-based TTS architectures.
Evidence: The paper explicitly notes that VALL-E-type models using non-differentiable discrete token generation are incompatible with this framework because the discriminator’s loss cannot backpropagate through the discrete generation step. (§6)
Limitations and Open Questions
Warning
The method is evaluated on a single TTS backbone (F5-TTS) with a single discriminator architecture, and the out-of-domain test uses only three other TTS systems. Generalization of the approach to a broader range of TTS models and adversarial conditions is untested.
The framework is architecturally restricted to TTS models that support end-to-end differentiation from text input to waveform, ruling out discrete token-based systems (VALL-E and its descendants). Additive noise and pitch shifting substantially degrade discriminator performance, suggesting that an adversary with access to these transformations could evade attribution. The paper does not evaluate whether the discriminator’s traceability signal remains effective after the fine-tuned model is further fine-tuned by a third party or after the vocoder is swapped at inference time (though vocoder-robustness is claimed by design). Code release is conditional on paper acceptance and not yet available.
Wiki Connections
- Flow Matching — this paper uses F5-TTS, a flow-matching TTS system, as the generator backbone and introduces a novel training objective on top of it for traceability.
- Zero-Shot TTS — the system inherits F5-TTS’s zero-shot voice cloning capability, using reference audio and text to control speaker identity during synthesis.
- Self-Supervised Speech — wav2vec 2.0, a self-supervised pre-training model, serves as the core feature extractor in the traceability discriminator.
- Evaluation Metrics — the paper uses EER and AUC as primary traceability metrics, alongside WER and UTMOS for synthesis quality, establishing a multi-dimensional evaluation protocol for the attribution task.
- F5-TTS — the TTS generator in this paper’s framework; the proposed method fine-tunes F5-TTS and uses its flow-matching architecture to enable end-to-end differentiable traceability training.
- CosyVoice (CosyVoice) — used as an out-of-domain negative class in generalization testing to assess whether the discriminator correctly identifies audio not generated by the target model.
- CosyVoice2 (CosyVoice 2) — also used as an out-of-domain test system alongside CosyVoice to evaluate discriminator generalization.
- VALL-E (VALL-E) — cited as an example of discrete token-based TTS that is explicitly incompatible with the proposed joint training framework due to non-differentiable generation.