IWSLT 2026 · 2026 · Workshop
Mo Ahtasam et al. (IIT Patna / Aligarh Muslim University) · → Paper · Demo: ✗ · Code: ✓
Runs four pretrained zero-shot TTS systems on English-prompted cross-lingual voice cloning into six target languages from ACL-60/60 for the IWSLT 2026 Cross-Lingual Voice Cloning track, scoring intelligibility and speaker similarity with automatic metrics only.
Problem
Cross-lingual voice cloning (CLVC) asks a system to speak target-language text in the voice of a speaker who has no recordings in that language. The task is the subject of a dedicated track at IWSLT 2026. The paper frames three difficulties: phonemes that have no English counterpart (Arabic emphatics, Mandarin tones), speaker traits such as pitch and rhythm that are entangled with language-specific prosody, and the need to render the target text without omissions or hallucinations. It sets out to compare recent multilingual zero-shot TTS systems under one pipeline and to characterize how they trade target-language intelligibility against speaker identity.
Method
The paper builds no model. It wraps four existing pretrained systems in a common inference and scoring pipeline. English reference audio from the ACL-60/60 corpus supplies the voice prompts (the speakers are unseen by all systems, following the IWSLT 2026 protocol), and the target-language text comes from the corpus’s parallel fields. Six target languages are used: Arabic, German, French, Japanese, Russian and Chinese, for roughly 3,144 synthesized utterances (§3.1). Reference audio is converted to mono and stored as 16-bit PCM WAV with no silence trimming or loudness normalization; target text is Unicode-normalized and passed to each system’s own tokenizer or grapheme-to-phoneme front end rather than a shared phoneme space (§3.1, §3.2).
The four systems, with the parameter counts the paper gives (§3.4), are MOSS-TTS (8B), Qwen3-TTS (1.7B), VoxCPM2 (2B) and CosyVoice 3 (0.5B). The paper groups them as autoregressive (MOSS-TTS, Qwen3-TTS), “diffusion” (VoxCPM2) and “flow-matching” (CosyVoice3, described as directly modelling waveform manifolds). Those labels are the paper’s own. VoxCPM2 is a tokenizer-free autoregressive TTS model whose local diffusion-transformer decoder is trained with a flow-matching objective (the paper itself also calls it a “diffusion autoregressive” model), and CosyVoice 3 pairs an autoregressive language model over speech tokens with a flow-matching acoustic stage (the paper likewise calls it “large language model-based”). All four systems therefore include autoregressive generation, and the paper’s autoregressive-versus-diffusion comparison rests on its own labels rather than on a clean architectural split. The only per-system inference settings reported are classifier-free guidance of 2.0 with 10 denoising steps for VoxCPM2, and CosyVoice3 run “with or without” a reference transcript, without saying which setting produced the results (§3.3).
Each system runs in its own conda environment under a subprocess orchestrator that exchanges JSON over stdin and stdout, on three consumer RTX 2080 GPUs, with batch inference that falls back to single samples under memory pressure (§1, §3.5). Outputs are peak-normalized. Content consistency is scored with Whisper large-v3 using language-directed decoding: WER for Arabic, German, French, Japanese and Russian, and CER for Chinese, after removing Arabic diacritics and lowercasing and stripping punctuation for alphabetic scripts (§3.2, §3.5). Speaker similarity is the cosine similarity between ECAPA-TDNN embeddings of the reference and cloned audio, extracted with SpeechBrain at 16 kHz and clipped to a valid range (§3.3). No human listening test and no speech-quality metric is reported, although the introduction names speech quality as one of the three official IWSLT criteria.
Key Results
Table 1 is the only place the paper gives per-system, per-language values. Rebuilt from the PDF text layer and checked against the table image, it reads as follows (speaker similarity / error rate; “not supported” marks languages the system does not officially support, §4.1):
| System | ar (WER) | de (WER) | fr (WER) | ja (WER) | ru (WER) | zh (CER) |
|---|---|---|---|---|---|---|
| CosyVoice3 | not supported | 0.68 / 0.18 | 0.67 / 0.15 | 0.70 / 0.15 | 0.70 / 0.20 | 0.69 / 0.17 |
| MOSS-TTS | 0.54 / 0.31 | 0.53 / 0.35 | 0.51 / 0.30 | 0.52 / 0.30 | 0.53 / 0.38 | 0.49 / 0.43 |
| Qwen3-TTS | not supported | 0.50 / 0.22 | 0.50 / 0.09 | 0.48 / 0.20 | 0.50 / 0.25 | 0.53 / 0.10 |
| VoxCPM2 | 0.72 / 0.25 | 0.65 / 0.30 | 0.74 / 0.26 | 0.70 / 0.28 | 0.68 / 0.35 | 0.72 / 0.30 |
Per language in Table 1, the lowest error rate belongs to CosyVoice3 for German (0.18), Japanese (0.15) and Russian (0.20), and to Qwen3-TTS for French (0.09 WER) and Chinese (0.10 CER). The highest speaker similarity belongs to VoxCPM2 for French (0.74) and Chinese (0.72) and to CosyVoice3 for German (0.68 against VoxCPM2’s 0.65) and Russian (0.70 against 0.68); Japanese is a tie between CosyVoice3 and VoxCPM2 at 0.70. The text’s statement that VoxCPM2 has the highest speaker similarity across all languages (§4.2) therefore holds in Table 1 only for Arabic, French and Chinese. On Arabic only two systems are scored, so VoxCPM2’s lead there (0.25 WER and 0.72 similarity against MOSS-TTS’s 0.31 and 0.54; a gap of 0.06 WER and 0.18 similarity in VoxCPM2’s favour) is a two-way comparison, not a ranking over four systems.
The paper’s stated system means (§4.1, presented as the content of Figure 2) are 0.16 error and 0.50 similarity for Qwen3-TTS, 0.28 and 0.70 for VoxCPM2, 0.18 and 0.69 for CosyVoice3, and 0.35 and 0.52 for MOSS-TTS. The Qwen3-TTS and CosyVoice3 means cover five languages and exclude Arabic, so they are not directly comparable with the six-language means of the other two systems. MOSS-TTS, the largest system, has the highest error rate of the four in every language in Table 1 and speaker similarity of 0.49 to 0.54, below every CosyVoice3 and VoxCPM2 cell.
The paper reads its results as an inverse relationship between content accuracy and speaker similarity (§4.1, §4.3). Table 1 supports that reading only for the Qwen3-TTS versus VoxCPM2 pair: CosyVoice3 combines the lowest error rate with the highest similarity in German and Russian, and MOSS-TTS is weak on both axes. The paper also calls Arabic the most challenging target with “2-3x higher error rates” (§4.4), but in Table 1 Arabic is VoxCPM2’s lowest-error language and sits below MOSS-TTS’s German, Russian and Chinese error rates; Arabic’s difficulty in this study shows up as missing system support rather than as higher scores.
Novelty Assessment
The contribution is a small shared-track evaluation. The systems, the evaluation corpus and the scorers (Whisper large-v3 and ECAPA-TDNN similarity) are all existing components, and the pipeline engineering (per-model environments, consumer-GPU deployment) is practical rather than methodological. The language set adds Arabic, Japanese and Russian alongside the more commonly reported Chinese, French and German, which gives a small amount of cross-system evidence on lower-coverage targets. The architectural conclusions (that diffusion suits morphologically complex low-resource languages) go beyond what four systems with mixed and loosely labelled architectures, and one two-system Arabic comparison, can establish.
Field Significance
Low — The paper adds a single automatic-metric data point on how current pretrained multilingual zero-shot TTS systems handle English-prompted cloning into six languages, with no human evaluation and no system of its own. Its most usable observations are that speaker similarity in this setting is mostly a property of the system rather than of the target language, and that official language coverage, not just model quality, decides which systems can be compared on lower-resource targets.
Claims
- complicates: Target-language intelligibility and speaker similarity in zero-shot cross-lingual voice cloning do not trade off uniformly across systems; one system can lead on both.
Evidence: The paper reports an inverse relationship between error rate and similarity, but in its own per-language table CosyVoice3 has both the lowest WER and the highest speaker similarity for German (0.18, 0.68) and Russian (0.20, 0.70), while the largest system, MOSS-TTS, is weak on both axes. (§4.1, §4.3, Table 1)
- supports: Parameter count alone does not predict cross-lingual voice cloning quality.
Evidence: The 8B MOSS-TTS records the highest error rate of the four systems in all six target languages and speaker similarity of 0.49 to 0.54, below every value of the 0.5B CosyVoice3 (0.67 to 0.70) and the 2B VoxCPM2 (0.65 to 0.74). (§4.1, §4.4, Table 1)
- supports: When the source language is fixed, speaker similarity in cross-lingual cloning depends more on the system than on the target language.
Evidence: Within each system, ECAPA-TDNN similarity varies over a narrow band across languages (CosyVoice3 0.67 to 0.70, MOSS-TTS 0.49 to 0.54, Qwen3-TTS 0.48 to 0.53, VoxCPM2 0.65 to 0.74), while every CosyVoice3 and VoxCPM2 value (0.65 or higher) exceeds every MOSS-TTS and Qwen3-TTS value (0.54 or lower). (§4.2, Table 1)
- complicates: Uneven official language coverage in pretrained multilingual TTS systems limits cross-system comparison on lower-resource target languages.
Evidence: Only MOSS-TTS and VoxCPM2 officially support Arabic, so the Arabic comparison reduces to two systems and the five-language means of CosyVoice3 and Qwen3-TTS cannot be compared directly with the six-language means of the others. (§4.1, §6)
Limitations and Open Questions
Reported numbers disagree across the paper
The per-language bars in Figure 3 do not reproduce Table 1 (for example, the MOSS-TTS Japanese error bar rises above the 0.4 gridline against 0.30 in Table 1, and the Qwen3-TTS German error bar sits well below the 0.2 gridline against 0.22), and the Qwen3-TTS mean error bar in Figure 2 sits near the 0.1 gridline against the 0.16 stated in §4.1. Table 1’s bold marks also do not follow its caption (MOSS-TTS’s 0.43 Chinese CER, the highest in its column, is bold). This page uses Table 1 throughout; which source reflects the final runs is not stated.
- Evaluation is automatic only (Whisper large-v3 WER/CER and ECAPA-TDNN cosine similarity), and the paper itself notes that these metrics may miss perceived naturalness and fine acoustic detail (§6). Speech quality, which the paper lists as an official track criterion, is not measured.
- Japanese is scored with WER (§3.5, Figure 3 caption), but the paper does not say how Japanese text is segmented into words, which makes the Japanese error rates hard to interpret or compare.
- The autoregressive-versus-diffusion analysis rests on the paper’s own architecture labels. VoxCPM2 and CosyVoice 3 both include autoregressive generation, and with four systems that differ in size, training data and language coverage at once, the conclusion that diffusion suits morphologically complex languages is not isolated from those other factors.
- Only English source speakers from a single corpus are used, with no statistical tests beyond the error bars in the figures, and emotion or fine-grained prosody transfer is not evaluated (§6).
- Some inference settings are under-specified: whether CosyVoice3 received the reference transcript is not stated (§3.3). The citations given for VoxCPM2 and CosyVoice3 are the earlier VoxCPM paper (VoxCPM) and an ICASSP 2025 CosyVoice demonstration paper rather than the reports for the evaluated versions.
Wiki Connections
- Zero-Shot TTS — Compares four pretrained zero-shot cloning systems on English-prompted cross-lingual cloning into six languages, with automatic intelligibility and speaker-similarity scores only; the paper builds no system of its own.
- MOSS-TTS — One of the four evaluated systems; the largest, and one of only two with official Arabic support, it records the highest error rate in every target language.
- Qwen3-TTS — Evaluated system with the lowest French and Chinese error rates in Table 1 but speaker similarity of only 0.48 to 0.53, and no Arabic support.
- VoxCPM2 — Evaluated system that the paper labels “diffusion”; it leads speaker similarity on Arabic, French and Chinese and is the stronger of the two Arabic-capable systems.
- CosyVoice 3 — Evaluated system that the paper labels “flow-matching”; it pairs the lowest German, Japanese and Russian error rates with high speaker similarity, which cuts against the paper’s trade-off reading.
- VoxCPM — The paper cites this earlier VoxCPM report when describing VoxCPM2.
- Whisper — Whisper large-v3 with language-directed decoding is the ASR model behind every WER and CER figure.
- ECAPA-TDNN — Its speaker embeddings supply the cosine speaker-similarity metric.
- SpeechBrain — Toolkit used to extract the ECAPA-TDNN embeddings.