IWSLT 2026 · 2026 · Workshop
Abebe et al. (Shaggar Institute of Technology / Trinity College Dublin) · → Paper · → Preprint · Demo: ? · Code: ✓
An IWSLT 2026 cross-lingual voice cloning submission that resynthesizes the ACL 60/60 development set with three zero-shot cloning models, keeps the best candidate per utterance by an automatic CER plus speaker-similarity score, and uses that synthetic data to train one LoRA adapter per language (Arabic, French, Chinese) on top of OmniVoice.
Problem
The IWSLT 2026 Cross-Lingual Voice Cloning shared task asks systems to speak scientific text in Arabic, French, or Chinese in the voice of an English-speaking presenter (§1, §5). Scientific speech brings technical terminology and code-switching, and the authors argue that the main obstacle to adapting large multilingual TTS models to this domain is the lack of paired, in-domain, cross-lingual training speech (§1). Foundation models such as OmniVoice cover many languages but, in the authors’ framing, need domain-specific fine-tuning to perform well on academic text (§1).
Method
Best-of-N ensemble distillation (§3). The source data is the ACL 60/60 development split: 884 utterances per language for Arabic, French, and Chinese (2,652 in total), each pairing target-language text with a reference clip of the original English speaker from an ACL 2022 talk (§3.1). Three teacher models synthesize a candidate for every utterance: OmniVoice, VoxCPM, and Chatterbox (Resemble AI) (§3.2). Each candidate is scored as S_comb = 0.5 × (1 − CER) + 0.5 × SIM, where CER comes from Whisper large-v3 transcription and SIM is the cosine similarity of ECAPA-TDNN speaker embeddings against the reference; the highest-scoring candidate is kept (§3.2, Eq. 1). OmniVoice wins most utterances in every language (73.4% AR, 76.24% FR, 68% ZH); Chatterbox contributes only French samples (23.76%) and VoxCPM only Arabic and Chinese samples (26.6% and 32%) (Table 1). The selected set is further filtered by a minimum quality threshold (threshold value not reported) and split into training and development partitions (§4.1).
Per-language LoRA fine-tuning (§4.1 to §4.3). The fine-tuned model is OmniVoice, described as a 0.6B-parameter model with a Qwen3-0.6B backbone (§3.2, §4.2). Audio is tokenized with a HIGGS-based acoustic tokenizer (§4.1). Instead of one multilingual adapter, the authors train a separate Rank-Stabilized LoRA (RSLoRA) adapter for each language, applied to self-attention, feed-forward, and audio projection layers (§4.2). The authors describe the objective as an autoregressive cross-entropy loss over generated audio tokens (§4.3), although the OmniVoice reference they cite is titled as a diffusion language model; the paper does not reconcile the two descriptions. Training runs for 400 steps per language on one NVIDIA A40 each, with mixed precision and a cosine learning-rate schedule; LoRA rank and other hyperparameters are deferred to the code repository (§4.3).
Inference (§4.4). A 20-second segment of the reference audio is extracted with energy-based voice activity detection, the input text is split into chunks of up to 200 characters at sentence boundaries, each chunk is synthesized with temperature 0.8 and top-p 0.9, and the chunks are concatenated (§4.4).
Key Results
All evaluation is automatic: WER and CER for intelligibility and ECAPA-TDNN cosine similarity (SIM) for speaker fidelity (§5). WER is not reported for Chinese, where the authors treat CER as the meaningful metric (§5).
Pretrained system comparison on blindset-4 (Table 2). On a 4-speaker subset of the blind test set, base OmniVoice has the highest SIM in all three languages: 0.703 in Arabic (Chatterbox 0.680, VoxCPM2 0.607, XTTS-V2 0.501), 0.753 in French (Chatterbox 0.619, VoxCPM2 0.575, Qwen3-TTS 0.533, XTTS-V2 0.445), and 0.702 in Chinese (Chatterbox 0.653, VoxCPM2 0.569, Qwen3-TTS 0.522, XTTS-V2 0.511). It does not lead on intelligibility anywhere. In Arabic, VoxCPM2 has the lowest WER (0.209) and CER (0.072) against OmniVoice’s 0.238 and 0.076. In French, Qwen3-TTS has the lowest WER (0.050) and CER (0.011) against OmniVoice’s 0.079 and 0.020. In Chinese, OmniVoice has the highest CER of the five systems (0.219), with Qwen3-TTS lowest (0.090). Qwen3-TTS was not run on Arabic because it does not support the language (§5). XTTS-V2 and Chatterbox are the other baselines.
Effect of LoRA fine-tuning on blindset-full (Table 3). Fine-tuned minus baseline OmniVoice (negative is better for WER and CER, positive is better for SIM):
| Lang | WER | CER | SIM |
|---|---|---|---|
| AR | 0.244 → 0.228 (−0.016) | 0.077 → 0.060 (−0.017) | 0.734 → 0.726 (−0.008) |
| FR | 0.079 → 0.082 (+0.003) | 0.025 → 0.030 (+0.005) | 0.753 → 0.760 (+0.007) |
| ZH | not reported | 0.200 → 0.205 (+0.005) | 0.719 → 0.732 (+0.013) |
So fine-tuning improves intelligibility only in Arabic, at a small SIM cost, while in French and Chinese it slightly raises SIM and slightly worsens error rates. The authors read this as a trade-off between the two objectives (§5.1). No variance, confidence intervals, or significance tests are reported, and the differences are small. Fine-tuned systems were evaluated only on blindset-full, so they are never compared against the non-OmniVoice baselines on the same test set.
The prose in §5.1 quotes numbers that do not match the peer-reviewed Table 3: it gives the fine-tuned Arabic model a CER of 0.071 and SIM of 0.723 (Table 3: 0.060 and 0.726), and credits the fine-tuned Chinese model with SIM 0.719 (Table 3 gives 0.719 for the baseline and 0.732 for the fine-tuned model). This was checked against the PDF text. The quoted figures equal the Table 3 values of the arXiv preprint, so the text appears not to have been updated with the new table. This page uses the Table 3 values.
Novelty Assessment
The system combines existing parts: pretrained zero-shot cloning models as teachers, best-of-N selection with off-the-shelf ASR and speaker-verification scorers, and RSLoRA adaptation of a released TTS model. The specific contributions are applying multi-teacher selection to build in-domain cross-lingual cloning data from ACL 60/60, and choosing per-language adapters over one shared adapter. The paper does not test that choice against a shared multilingual adapter, so the claim that a unified adapter “dilutes” language-specific nuances (§4.2) is a premise, not a finding. Since OmniVoice supplies 68% to 76% of the selected samples (Table 1), the fine-tuning data is largely self-distilled, and the paper does not separate the effect of the other teachers’ samples from that of in-domain text exposure.
Field Significance
low — A shared-task system paper with automatic metrics only. It gives a small data point that distilling from a teacher ensemble and adding per-language LoRA to a strong multilingual cloning model changes intelligibility and speaker similarity by small amounts in opposite directions, depending on the language. The blindset-4 comparison adds a useful observation: among the five pretrained systems, the one with the highest speaker similarity never had the lowest error rate in any language.
Claims
- complicates: Fine-tuning a multilingual zero-shot cloning model on best-of-N distilled in-domain data does not reliably improve both intelligibility and speaker similarity; the direction of the gain can depend on the target language.
Evidence: Per-language RSLoRA on OmniVoice lowers Arabic WER (0.244 to 0.228) and CER (0.077 to 0.060) while SIM drops (0.734 to 0.726); in French and Chinese SIM rises (0.753 to 0.760, 0.719 to 0.732) while CER worsens (0.025 to 0.030, 0.200 to 0.205). (§5.1, Table 3)
- supports: In cross-lingual voice cloning, the system that best preserves speaker identity is not necessarily the most intelligible, so the two objectives have to be reported separately.
Evidence: On blindset-4, OmniVoice has the highest SIM in Arabic (0.703), French (0.753), and Chinese (0.702), but VoxCPM2 has the lowest Arabic WER/CER (0.209/0.072), Qwen3-TTS the lowest French WER/CER (0.050/0.011) and Chinese CER (0.090), and OmniVoice the highest Chinese CER (0.219). (§5.1, Table 2)
- refines: When automatic scorers pick synthetic training data from a teacher ensemble, the selection depends on language, and no single teacher dominates across languages.
Evidence: With selection by 0.5 × (1 − Whisper CER) + 0.5 × ECAPA-TDNN SIM over 884 utterances per language, Chatterbox wins 23.76% of French samples but none in Arabic or Chinese, while VoxCPM wins 26.6% (AR) and 32% (ZH) but none in French; OmniVoice wins the rest. (§3.2, Table 1)
Limitations and Open Questions
Evaluation rests on automatic scorers that overlap with the selection criterion
Distilled samples were chosen by Whisper large-v3 CER and ECAPA-TDNN similarity (§3.2), and the evaluation uses ECAPA-TDNN similarity and ASR error rates from the same kind of tools (the Limitations section names Whisper/ECAPA as the evaluation proxies). This makes the fine-tuning gains optimistic. There are no human listening tests, no significance testing, and the Table 3 differences are at most 0.017 absolute. The §5.1 prose also quotes Arabic and Chinese numbers that disagree with Table 3 (see Key Results).
- The baseline comparison (Table 2) uses a 4-speaker subset, and the fine-tuning ablation (Table 3) uses the full blind set. The fine-tuned system is therefore never compared directly against Chatterbox, XTTS-V2, VoxCPM2, or Qwen3-TTS.
- Per-language adapters are motivated by a claim that a shared adapter dilutes language-specific detail (§4.2), but no shared-adapter condition is reported. The authors also note that the per-language design multiplies the number of adapters (§Limitations).
- The distilled set is small (884 utterances per language before an unreported quality filter), and training lasts 400 steps; LoRA rank, learning rate, and filter threshold are deferred to the code (§4.1, §4.3).
- Fine-tuning speakers come from ACL 2022 talks (ACL 60/60) and evaluation voices from ACL 2023 talks (§3.1, §5). The paper does not state whether the two speaker sets are disjoint, and it does not discuss OmniVoice’s pretraining speakers, so it cannot be established that the evaluated voices are unseen.
- The paper describes OmniVoice training as autoregressive cross-entropy over audio tokens (§4.3), while the cited OmniVoice reference is a diffusion language model; how the LoRA objective relates to the base model’s training objective is not explained.
Wiki Connections
- Diffusion TTS — The fine-tuned base model is OmniVoice, a diffusion-language-model TTS system, which the paper adapts with per-language RSLoRA on distilled scientific-domain data.
- OmniVoice — The base model fine-tuned here, also the main teacher in the distillation ensemble (68% to 76% of selected samples) and the strongest pretrained system on speaker similarity in Table 2.
- VoxCPM — One of the three distillation teachers; it supplies 26.6% of Arabic and 32% of Chinese selected samples and none of the French ones.
- VoxCPM2 — Evaluated as a pretrained baseline on blindset-4, where it has the lowest Arabic WER and CER.
- Qwen3-TTS — Pretrained baseline with the lowest French and Chinese error rates on blindset-4 but lower speaker similarity than OmniVoice; not run on Arabic.
- Whisper — Whisper large-v3 provides the CER used in the best-of-N selection score, and Whisper also serves as the intelligibility proxy in evaluation.
- ECAPA-TDNN — Provides the speaker embeddings used both in the selection score and in the SIM evaluation metric.
- HW-TSC IWSLT 2026 cloning submission — Another system submitted to the same IWSLT 2026 Cross-Lingual Voice Cloning shared task.
- Balancing Linguistic Intelligibility and Speaker Identity — A same-track IWSLT 2026 paper benchmarking four pretrained cross-lingual cloning systems with automatic intelligibility and similarity metrics, the same trade-off this paper reports.