arXiv · 2025 · Preprint
Christop et al. (Adam Mickiewicz University) · → Paper · Demo: ✓ · Code: ✓
ClonEval introduces a reproducible, architecture-agnostic evaluation framework for voice cloning TTS, combining a WavLM-based speaker-similarity protocol, an open-source evaluation library, and a public leaderboard with per-emotion performance breakdowns.
Problem
Evaluating voice cloning models reproducibly across research groups has been hampered by inconsistent methodology. Prior landmark systems (VALL-E, Voicebox) each employed slightly different metric implementations and data choices, and relied on human MOS scores that external parties cannot replicate without repeating the full listening test. There is no standardized, open benchmark that accepts arbitrary voice cloning models as black boxes and produces comparable results across labs or across time.
Method
ClonEval imposes three requirements on a voice cloning model: it must accept a reference audio sample and a text prompt as inputs and produce a single generated audio sample as output. No other assumptions are made about model internals, architecture, or public availability of weights.
Evaluation proceeds in two automated stages. In the generation stage, the model produces speech samples for all reference/text pairs drawn from the benchmark datasets. In the scoring stage, a pretrained WavLM model extracts speaker embeddings from both the reference and the generated audio; the cosine similarity between each pair is computed and averaged across all samples in the dataset. The primary evaluation dataset is LibriSpeech test-clean. Four additional emotional speech datasets (CREMA-D, RAVDESS, SAVEE, TESS) probe how well each model transfers expressive characteristics alongside speaker identity. Acoustic features from Librosa (pitch, mel-spectrogram, spectral centroid, flatness, roll-off, RMS, tempogram, chromagram) are also extracted to provide a supplementary signal.
The software library operates on two directories of matched audio files, identified by filename, and allows emotion labels to be encoded in filenames for per-category aggregation. Outputs include a per-pair metric file and a dataset-level summary. Results are published to an open leaderboard where vendors can submit their own systems for community comparison.
Key Results
Five open-source voice cloning models were evaluated on ClonEval (WavLM cosine similarity, averaged across all five datasets, Table 1):
- XTTS-v2: 0.8356 (best overall)
- SpeechT5: 0.8298
- VALL-E X: 0.7862
- WhisperSpeech: 0.7837
- OuteTTS-0.2-500M: 0.7499
All models achieved their highest speaker similarity on LibriSpeech test-clean (0.88–0.91 range), with substantially lower scores on emotional datasets. SAVEE proved the most challenging condition, where OuteTTS reached only 0.54. In the per-emotion breakdown (Table 3), fear consistently produced the lowest scores across models (e.g., XTTS-v2: 0.7929), while neutral achieved the highest (XTTS-v2: 0.848). Disgust and anger also ranked near the bottom for most models. The acoustic feature analysis (Table 2) showed high cosine similarity for mel-spectrogram and spectral roll-off, but lower similarity for pitch, spectral flatness, and tempogram, consistent with same-speaker but different-content generation.
Novelty Assessment
The contribution is evaluation infrastructure rather than methodological novelty. WavLM cosine similarity for speaker identity measurement was established by VALL-E and Voicebox; ClonEval does not introduce a new metric. What is new is the packaging: a fully deterministic protocol with fixed data splits, an open-source evaluation library, and a public leaderboard capable of accepting arbitrary black-box models from any vendor. The inclusion of emotional speech datasets to probe emotion transfer fidelity adds diagnostic value absent from prior practice. The paper is a short conference submission (ICASSP under review) with appropriately narrow scope; it does not claim algorithmic novelty.
Field Significance
Moderate - ClonEval addresses a genuine reproducibility gap in voice cloning evaluation by standardizing data, metrics, and output format into a single deployable package. Its practical value lies in enabling fair cross-paper comparisons and in surfacing systematic weaknesses, particularly on emotional speech, that single-dataset evaluations miss. A shared benchmark is a prerequisite for meaningful progress tracking, and the public leaderboard lowers the barrier to entry for independent evaluation.
Claims
-
supports: Automated speaker-similarity metrics can serve as architecture-agnostic proxies for voice cloning quality, enabling reproducible evaluation without human raters.
Evidence: ClonEval evaluates five open-source models as black boxes using WavLM cosine similarity, requiring no human listeners or model internals access, while revealing systematic performance differences across systems and conditions. (§3, §4)
-
complicates: Benchmark evaluations using only neutral read-speech overestimate the practical voice cloning performance of current TTS systems.
Evidence: All five models achieved WavLM cosine similarity in the 0.88–0.91 range on LibriSpeech test-clean, but scores dropped substantially on emotional datasets; OuteTTS fell to 0.54 on SAVEE, revealing that neutral-data performance does not generalize to expressive conditions. (§7.1, Table 1)
-
complicates: Emotion transfer fidelity varies systematically by emotion category, with high-arousal negative emotions consistently harder to clone than neutral or positive-valence speech.
Evidence: Across all five evaluated models in Table 3, fear produced the lowest or near-lowest per-emotion scores (e.g., XTTS-v2: 0.7929 for fear vs. 0.848 for neutral), and disgust and anger also ranked consistently below neutral for most models. (§7.3, Table 3)
Limitations and Open Questions
The benchmark is English-only; generalization of the protocol and leaderboard to multilingual voice cloning is deferred to future work. Speaker similarity via WavLM cosine similarity does not capture naturalness, intelligibility, or audio artefacts, so a system with high speaker similarity scores may still produce perceptually poor speech. The paper acknowledges the absence of human evaluation and plans to add it. Because the leaderboard relies on vendors self-reporting generated samples, it cannot rule out benchmark-specific optimizations. WavLM representations may not generalize equally across all speaker demographics and acoustic environments, which could introduce bias in rankings for out-of-distribution reference voices.
Wiki Connections
- Evaluation Metrics — ClonEval contributes a standardized evaluation protocol and open-source tooling for measuring voice cloning quality via speaker-similarity metrics, filling a reproducibility gap in the field.
- Zero-Shot TTS — the voice cloning task evaluated by ClonEval (conditioning on a reference audio clip to match speaker identity at inference time) is the defining scenario for zero-shot TTS systems.
- Voice Conversion — voice cloning occupies the intersection of zero-shot TTS and VC; ClonEval’s speaker-similarity evaluation methodology is relevant to any system that transfers speaker identity from a reference sample.
- Emotion Synthesis — the inclusion of four emotional speech datasets (CREMA-D, RAVDESS, SAVEE, TESS) provides a diagnostic lens on how well voice cloning systems transfer expressive emotional characteristics alongside speaker identity.