arXiv · 2026 · Preprint
Liao et al. · → Paper · Demo: ✗ · Code: ✓
Introduces RVCBench, a benchmark of 225 speakers and 14,370 utterances that systematically stress-tests 11 modern voice cloning models across input shifts, generation challenges, output post-processing, and adversarial perturbations, revealing substantial robustness gaps not captured by clean-setting quality benchmarks.
Problem
Voice cloning (VC) systems can synthesize speech that closely matches a target speaker from only seconds of reference audio, but practical deployments inevitably introduce noisy reference audio, imperfect or adversarial text prompts, and downstream processing (compression, transmission) that existing benchmarks do not stress-test. Prior VC benchmarks (e.g., CloneEval, EmergentTTS-Eval) are largely quality-centric under clean settings and cover limited robustness failure modes, while general audio robustness benchmarks (e.g., Aha-Bench, AudioWatermarkBench) target hallucination or watermark resilience broadly but are not tailored to VC-specific stressors such as reference-audio domain shifts, cross-lingual and long-context identity drift, output post-processing, or proactive anti-cloning perturbations. No prior work evaluates VC robustness as a full-pipeline problem.
Method
RVCBench formalizes a VC pipeline as a generation model producing cloned speech from a (potentially perturbed) reference audio and a text prompt, and evaluates robustness along four dimensions covering the full pipeline: input robustness, generation robustness, output robustness, and audio-perturbation robustness. The benchmark is built from eight public sources (including VCTK, LibriTTS, AISHELL-1, EMIME, and VoiceBank+DEMAND) reprocessed into 18 robustness evaluations spanning 10 tasks, 225 speakers, and 14,370 utterances.
Input robustness is evaluated with RVC-AudioShift (reference audio spanning 12 accents, both genders, and multiple age groups from VCTK) and RVC-TextShift (LLM-generated prompts with realistic formatting irregularities, plus scam-content prompts drawn from a public robocall dataset). Generation robustness covers RVC-Multilingual (single-language English and Chinese cloning plus cross-lingual English-Mandarin cloning via EMIME), RVC-LongContext (extended text prompts and long reference audio), and RVC-Expression (VCTK speakers paired with robocall-style prompts augmented with explicit emotion cues, scored via audio-text emotion alignment using an LLM-as-judge). Output robustness covers RVC-Compression (six compression conditions: MP3/AAC/Opus at 64/32/24/16 kbps plus a telephone narrowband simulation) and RVC-Detectability (a zero-shot SpeechLLM-as-judge deepfake detector distinguishing ground-truth, benign cloned, and scam cloned speech). Audio-perturbation robustness covers RVC-PassiveNoise (VCTK mixed with 10 environmental noise types and multi-speaker interference at varying decibel levels), RVC-AdvNoise (attack effectiveness against proactive anti-clone defenses such as SafeSpeech, its SPEC variant, POP, and Enkidu), and RVC-AntiProtect (whether denoising with DEMUCS can recover cloneability from adversarially protected reference audio).
The 11 evaluated VC models are grouped by generative architecture: autoregressive codec-token LMs (FishSpeech, SparkTTS, MOSS-TTSD, Higgs Audio v2), diffusion/flow acoustic models (StyleTTS-2, PlayDiffusion, OZSpeech), and hybrid LM-plus-flow-generator systems (CosyVoice 2, GLM-TTS, VibeVoice, and the omni-modal MGM-Omni). Generation quality is measured with Speaker Similarity (SIM), automatic MOS (DNSMOS), Mel-cepstral distortion (MCD), real-time factor (RTF), and word error rate (WER); deepfake detectability is measured with EER, minDCF, and detector accuracy.
Key Results
Across clean-setting English and Chinese cloning (Tables 1-2), hybrid models (CosyVoice 2, GLM-TTS, MGM-Omni) achieve the strongest overall performance, autoregressive LM-based models competitively maintain spectral consistency and speaker identity, and diffusion-based models improve content accuracy but struggle with multilingual support; even the best model (CosyVoice 2) reaches only 0.60 SIM on English-VC-LibriTTS, well below theoretical upper bounds. Under RVC-AudioShift, WER and MCD both vary substantially by accent, with Indian-accented references consistently the hardest to preserve and Canadian/Australian accents the easiest. Under RVC-TextShift, nearly all models show increased WER on hallucination-inducing and scam-style prompts relative to clean VCTK prompts, with roughly half the models degrading by close to 50% on hallucination content.
Language shifts cause the sharpest instability: moving from English (MOS ≈4, WER often <0.1) to Chinese-VC degrades perceptual quality across all models and dramatically increases WER, and cross-lingual cloning further compounds the loss, primarily through content-preservation failure. Long-form generation degrades similarly: WER rises and MCD increases with longer text prompts across all backends, with CosyVoice 2 and MGM-Omni the most stable but none immune. Under RVC-Compression, all models show increased MCD and reduced STOI, with narrowband telephone simulation the most damaging condition. Under RVC-Detectability, most models remain readily detectable by the SQ-LLM judge (SparkTTS is the notable exception, at 62.5% detector accuracy), and detectability does not correlate with generation quality: FishSpeech and CosyVoice 2 both achieve strong generation quality and high (>85%) detection accuracy. Under RVC-PassiveNoise, roughly half the evaluated models are relatively robust to noisy references while the rest degrade substantially, and all models degrade under multi-speaker interference. Under RVC-AdvNoise, SafeSpeech and its SPEC variant are the most effective anti-clone perturbations against most models, though MOSS-TTSD and SparkTTS lack resilience under both noisy and adversarial conditions; DEMUCS denoising only partially restores cloneability after SPEC protection, and MOSS-TTSD is the one model that can partially exploit denoising artifacts to exceed clean-reference performance.
Novelty Assessment
RVCBench proposes no new VC architecture; its contribution is a comprehensive, standardized robustness testbed spanning the full VC deployment pipeline (input, generation, output, perturbation), which prior work addressed only piecemeal (VC benchmarks under clean settings only, or general audio robustness benchmarks not tailored to VC-specific stressors). The scale (225 speakers, 14,370 utterances, 10 tasks, 11 models across three architecture families) and the inclusion of adversarial anti-cloning defenses and their counteraction as a first-class robustness axis are the paper’s distinguishing contributions relative to prior VC benchmarks such as CloneEval.
Field Significance
Tip
High — RVCBench is the first benchmark to systematically evaluate VC robustness as a full-pipeline problem rather than isolated failure modes, and its empirical findings (universal degradation under compression, persistent deepfake detectability, differential vulnerability to anti-cloning perturbations) surface concrete, reproducible gaps between reported clean-setting VC quality and deployment-realistic robustness.
This paper provides a reusable, open-source testbed and a structured taxonomy of VC robustness dimensions (input, generation, output, perturbation) that future VC systems can be benchmarked against, and its cross-architecture comparison (autoregressive-LM, diffusion/flow, hybrid) demonstrates that no single architecture family dominates across all robustness dimensions.
Claims
- supports: Voice cloning systems that perform well under clean, in-domain reference audio and prompts do not necessarily generalize their speaker-identity and content fidelity to realistic deployment shifts.
Evidence: Across 11 models spanning three architecture families, the best clean-setting SIM score (CosyVoice 2, 0.60 on English-VC-LibriTTS) remains well below theoretical upper bounds, and performance further deteriorates under accent shifts, text-prompt irregularities, and language shifts even for the strongest models. (§4.1-4.3, Tables 1-2)
- complicates: Cross-lingual and non-English voice cloning remains substantially less robust than English cloning, even for models that report strong multilingual capability.
Evidence: Moving from English-VC (WER often below 0.1, MOS ≈4) to Chinese-VC and cross-lingual English-Mandarin cloning produces a marked WER increase and perceptual quality drop across all 8 models evaluated on these settings. (§4.3, Tables 1-2)
- complicates: High voice-cloning fidelity does not imply resistance to automated deepfake detection, and the two properties can be independent design axes.
Evidence: FishSpeech and CosyVoice 2 achieve strong generation quality on Tables 1-2 yet are also among the most reliably detected by the SQ-LLM judge (85.17% and 85.22% accuracy respectively), while SparkTTS is the hardest to detect (62.50% accuracy) despite weaker generation quality. (§4.4, Table 3)
- complicates: Proactive anti-cloning perturbations that protect reference audio from unauthorized cloning are not reliably neutralized by generic denoising, but their effectiveness and recoverability vary substantially by target VC model.
Evidence: Denoising SPEC-protected reference audio with DEMUCS only partially restores cloning quality for most models, with SIM and MOS remaining below clean-reference baselines in most cases, while MOSS-TTSD is the one evaluated model that can exploit denoising artifacts to match or exceed its clean-reference performance. (§4.5, Figure 11)
Limitations and Open Questions
The benchmark evaluates 11 specific open and commercial VC models current as of early 2026; as the field moves quickly, newer models are not covered and would need to be added to keep RVCBench current. Emotion-alignment scoring in RVC-Expression relies on an LLM-as-judge (Gemini) rather than human raters, and deepfake detectability is assessed with a single automated SQ-LLM judge rather than a panel of detectors, so both findings are bounded by the reliability of these automated judges rather than validated against human perceptual studies. The proactive-perturbation evaluation (RVC-AdvNoise, RVC-AntiProtect) reproduces anti-cloning defenses (SafeSpeech, SPEC, POP, Enkidu) using a fixed surrogate TTS model (BERT-VITS2), which may not represent worst-case transferability to all 11 target VC architectures.
Wiki Connections
- Voice Conversion — provides the first full-pipeline robustness benchmark for zero-shot voice cloning, complementing existing quality-centric VC evaluation with systematic stress tests across input, generation, output, and perturbation conditions.
- Evaluation Metrics — introduces a 10-task, 18-evaluation robustness testbed with a standardized metric suite (SIM, MOS, MCD, RTF, WER, EER, minDCF) applied consistently across model families.
- Spoken Language Models — evaluates several autoregressive codec-token and hybrid LM-based VC systems (FishSpeech, SparkTTS, CosyVoice 2, GLM-TTS) alongside diffusion-based systems within a unified robustness framework.
- ClonEval — cited as a prior VC benchmark that RVCBench extends by covering robustness failure modes beyond clean-setting quality comparison.
- Degrading Voice — the cited survey of VC robustness whose gap analysis directly motivates RVCBench’s systematic, full-pipeline benchmark design.
- CosyVoice 2 — one of 11 evaluated VC models; achieves the strongest overall clean-setting performance among evaluated systems.
- Fish-Speech — one of 11 evaluated VC models, representing the autoregressive codec-token LM family.
- Spark-TTS — one of 11 evaluated VC models; shows the lowest deepfake-detector accuracy among evaluated systems despite weaker generation quality.
- VibeVoice — one of 11 evaluated VC models, representing the hybrid LM-plus-flow-generator family.
- GLM-TTS — one of 11 evaluated VC models, representing the hybrid LM-plus-flow-generator family.
- SpeechLLM-as-Judges — its SQ-LLM model is used as the zero-shot deepfake detector for RVCBench’s output-robustness evaluation.