IWSLT 2026 · 2026 · Workshop

Seymanur Akti et al. (Karlsruhe Institute of Technology / Carnegie Mellon University) · → Paper · → Preprint · Demo: ✗ · Code: ✗

Adapts a pretrained multilingual autoregressive TTS model to English-to-Arabic/French/Chinese voice cloning by adding native-script language tags to every text prompt and fine-tuning with GRPO on an ASR-plus-speaker-similarity reward, with no target-language reference speech.

Problem

The IWSLT 2026 Cross-Lingual Voice Cloning track asks systems to speak French, Arabic, and Chinese text in the voice of an English reference speaker. The data are ACL conference talks, so references carry varied speaker accents and the target text is full of technical terms, acronyms, and named entities. In-context cloning models such as VALL-E, F5-TTS, Qwen3-TTS, and CosyVoice 3 preserve speaker identity well, but the authors identify two failure modes in this cross-lingual setting. The first is accent leakage, where English phonetics carry into the target language. The second is mispronunciation of domain-specific words. The introduction also notes that cascaded TTS-plus-VC pipelines add their own errors, such as speaker leakage (Akti et al., 2025).

Method

The system starts from FishAudio-S2-Pro, the model described in the Fish Audio S2 Technical Report (the paper cites that report and links the fishaudio/fish-speech repository). It is an autoregressive multilingual TTS model with in-context cloning from a reference speech-text pair. The paper does not report model size, codec, or any change to the decoder side. The authors add three components on top of it.

  • Language-tag prompting (§3.1). The base model infers language implicitly from the text. The authors prefix every text input, including the reference transcript, with a free-form language tag. They compare English-word tags ([arabic], [french], [chinese]) with native-script tags (Arabic script for Arabic, [français], [普通话]). They report that native-script tags gave better cross-lingual pronunciation, but give no numbers for that comparison.
  • GRPO fine-tuning (§3.2). Target-language speech from the reference speakers is not available, so the model is adapted with GRPO (DeepSeekMath) instead of supervised training. The reward is the mean of (1 − CER) and speaker similarity, both scaled to [0, 1]. CER comes from MMS-1B-all ASR and speaker similarity from a WavLM-base-plus speaker-verification model. LoRA (rank 64, α = 16) is applied to the attention, MLP, and output layers. The authors note that the original S2 recipe updated only the MLP layers. Other settings are group size 8, AdamW at learning rate 1e-5, and KL coefficient β = 0.1. Fine-tuning uses the ACL 60/60 development set with language-tagged prompts.
  • Reference retrieval by lexical matching (§2, §3.3). For the blind test set, where the reference is a long recording, VibeVoice-ASR segments and transcribes it into chunks of about 2 to 10 seconds. At inference time the system picks reference chunks that share words with the target text, so the model hears the speaker say the domain terms.

For evaluation the authors use different scorers from the reward models to limit reward hacking. CER comes from Whisper-large-v3 (Whisper), speaker similarity (SSIM) from a SpeechBrain ECAPA verifier (SpeechBrain), and predicted MOS from UTMOS. A SpeechBrain VoxLingua107 language-ID model measures target-language confidence (§4.1).

Key Results

All quantitative results are on the ACL 60/60 evaluation set with gold segmentation, using automatic metrics only (Table 1). Blind-test scores for the submitted system are not reported.

SystemCER ar / fr / zh (%) ↓SSIM ar / fr / zh (%) ↑UTMOS ar / fr / zh ↑
Baseline, no tags6.57 / 3.10 / 11.3764.05 / 60.75 / 62.392.94 / 2.86 / 2.90
Baseline + language tags6.39 / 2.90 / 12.0563.77 / 60.21 / 62.492.94 / 2.85 / 2.88
GRPO + language tags (submitted)6.38 / 2.78 / 10.9964.15 / 60.83 / 62.522.93 / 2.88 / 2.89
  • Language tags alone help two of three languages on CER. CER falls for Arabic (6.57 to 6.39) and French (3.10 to 2.90) but rises for Chinese (11.37 to 12.05). SSIM drops slightly for Arabic and French and rises slightly for Chinese. UTMOS ties for Arabic and drops by 0.01 to 0.02 for French and Chinese.
  • GRPO fine-tuning gives the lowest CER and highest SSIM in all three languages. All differences are small: CER improves over the untagged baseline by 0.19 (ar), 0.32 (fr), and 0.38 (zh) points, and SSIM by 0.08 to 0.13 points. UTMOS does not follow. The submitted system is the best on UTMOS only for French (2.88). It is 0.01 below both baselines for Arabic (2.93 vs 2.94) and 0.01 below the untagged baseline for Chinese (2.89 vs 2.90).
  • Tags raise target-language ID confidence (Table 2). Language-ID probability rises in every language: Arabic 89.87 to 93.42, French 88.68 to 90.23, Chinese 90.99 to 92.13. The reported averages are 89.85 and 91.64. The table compares prompts with and without tags and does not name the model variant. The reported 91.64 average differs from the mean of its three per-language values (91.93) as printed in the PDF.
  • Lexical matching is shown only qualitatively (Table 3). Six entities are transcribed phonetically under matched and non-matched references. With a matched reference, terms such as VALSE and SVAMP are spoken as words. With a non-matched reference they are spelled out letter by letter. The matched condition uses the reference utterance whose translation is the target text, not the retrieval pipeline from §3.3, and no CER or other score is given for matched versus non-matched subsets.

The comparisons are fair in that all three systems share the base model and test data. However, the gains are a few tenths of a point, and the paper reports no significance tests or variance.

Novelty Assessment

The contribution is engineering on a strong pretrained model. Language tags use free-form prompting that the base model already supports. GRPO with ASR and speaker-similarity rewards follows the recipe the base model was already post-trained with, here with LoRA on more layers and task-specific data. Retrieving reference segments that share words with the target text is a simple inference-time idea. It addresses a real problem for technical-domain cloning, but the paper does not measure its effect. The most transferable observation is that explicit target-language tags measurably raise language-ID confidence of the output (Table 2), which offers one way to reduce accent leakage without retraining.

Field Significance

low — A single shared-task system paper with automatic metrics only and small Table 1 differences. It gives a data point that explicit language tags and RL fine-tuning with ASR and speaker-similarity rewards help cross-lingual cloning only slightly on top of a strong multilingual autoregressive TTS model. It also shows that the gains vary by language and metric.

Claims

  • supports: Explicit target-language conditioning in the text prompt reduces source-language accent leakage in cross-lingual voice cloning.

    Evidence: Adding language tags to every prompt raises language-ID probability of the generated speech in all three target languages (Arabic 89.87 to 93.42, French 88.68 to 90.23, Chinese 90.99 to 92.13). The authors report that speaker accent traits are retained. (§4.1, Table 2)

  • complicates: Prompt-level language tags on a model not trained with them do not lower error rates uniformly across target languages.

    Evidence: Without fine-tuning, tags lower CER for Arabic and French but raise it for Chinese (11.37 to 12.05), and SSIM falls slightly for Arabic and French. (§4, Table 1)

  • supports: Reinforcement learning with automatic intelligibility and speaker-similarity rewards can adapt a TTS model to a new conditioning format without target-language reference speech.

    Evidence: GRPO with a reward of the mean of (1 − CER) and speaker similarity, evaluated with different ASR and verification models from the reward, gives the lowest CER (6.38 / 2.78 / 10.99) and highest SSIM in all three languages. The gains over the untagged baseline are at most 0.38 CER points. (§3.2, §4, Table 1)

  • complicates: RL fine-tuning on intelligibility and speaker-similarity rewards does not reliably improve predicted naturalness.

    Evidence: After GRPO, UTMOS is 0.01 below both baselines for Arabic and 0.01 below the untagged baseline for Chinese, and higher only for French. (§4, Table 1)

  • supports: In-context cloning models mispronounce acronyms and technical terms unless the reference audio contains them.

    Evidence: When the reference contains the same terms (VALSE, SVAMP, ViLBERT), they are spoken as words. With a non-matching reference from the same speaker they are spelled out or split into parts, shown qualitatively for six entities. (§4.2, Table 3)

Limitations and Open Questions

Warning

The submitted system’s blind-test results are not reported, and the reference-retrieval component is never evaluated. Table 3 compares an oracle matched reference with a random one on six entities, without any metric. The abstract’s claim that lexical matching “yields consistent improvements on matched subsets” therefore has no quantitative support in the paper.

  • Only automatic metrics are used: CER, SSIM, UTMOS (a predicted MOS), and language-ID probability. No human listening test checks accent, naturalness, or speaker similarity.
  • Differences in Table 1 are mostly a few tenths of a point, with no variance or significance testing. Per-language effects differ in direction.
  • The abstract states that language prompting gives the largest gains. In Table 1, tags give the larger CER reduction for Arabic and French, but GRPO gives the larger one for Chinese, where tags alone made CER worse.
  • The native-script versus English-word tag comparison and the KL-coefficient sweep are described only in prose, with no numbers.
  • The paper does not say whether development-set and evaluation-set speakers are disjoint, and the base model’s training data are not described. Whether evaluated speakers are unseen is therefore unknown.
  • Only English-to-Arabic, French, and Chinese are tested, all in the ACL talk domain.

Wiki Connections

  • Autoregressive Codec TTS — The system fine-tunes the autoregressive language model of FishAudio-S2-Pro with LoRA and steers it with prompt-level language tags.
  • Multilingual TTS — One GRPO-adapted model handles Arabic, French, and Chinese targets from English references, and the paper measures accent leakage with a language-ID model.
  • RLHF Speech — GRPO fine-tuning uses only automated feedback, rewarding low MMS-ASR CER and high WavLM speaker similarity, and is evaluated with separate scorers.
  • Fish Audio S2 Technical Report — FishAudio-S2-Pro is the base model; the paper extends its GRPO post-training recipe to attention and output layers for cross-lingual cloning.
  • DeepSeekMath — Cited as the source of GRPO, the RL algorithm used for task adaptation.
  • Whisper — Whisper-large-v3 computes evaluation CER, kept separate from the MMS ASR used in the reward.
  • SpeechBrain — Provides the ECAPA speaker-verification model for evaluation SSIM and the VoxLingua107 language-ID model for the accent-leakage analysis.
  • UTMOS — Supplies the predicted-MOS naturalness score in Table 1.
  • HW-TSC IWSLT 2026 Cross-Lingual Voice Cloning — Another submission to the same IWSLT 2026 cross-lingual voice cloning track.
  • Balancing Linguistic Intelligibility and Speaker Identity — A same-track IWSLT 2026 paper that compares multiple cloning systems on the intelligibility and speaker-identity trade-off.
  • One Voice, Many Tongues — A same-track IWSLT 2026 submission that adapts a different pretrained TTS model with LoRA.