arXiv · 2026 · Preprint

Huaxuan Wang et al. (NetEase Youdao) · → Paper · Demo: ✓ · Code: ✓

Confucius4-TTS clones an unseen speaker in 14 languages without a transcript of the reference audio. It does this by replacing the paired text-audio prompt of a CosyVoice-style two-stage system with a speaker embedding pooled from self-supervised speech features by a speaker encoder trained jointly with the LLM.

Problem

Many zero-shot TTS systems condition on a paired text-audio prompt in the VALL-E style. They therefore need an accurate transcript of the reference clip at inference time. In-the-wild reference audio is often untranscribed, especially for low-resource languages and dialects. This makes cross-lingual voice cloning hard to deploy. Prior transcript-free approaches rely on forced alignment (Cross-Lingual F5-TTS) or on fine-tuning with synthetic prompt pairs (X-Voice). The authors want transcript-free cloning that needs neither, while keeping a transcript-based continuation mode available in the same model.

Method

The system has three stages: an autoregressive text-to-semantic (T2S) model, a semantic-to-acoustic (S2A) conditional flow-matching decoder, and a pre-trained BigVGAN vocoder.

T2S. A 24-layer decoder-only Transformer (hidden size 1280; parameter count not reported) predicts discrete semantic tokens from a frozen pre-trained MaskGCT semantic tokenizer. Its input sequence is a speaker embedding, then the target-text embeddings, then a boundary token, then the semantic tokens. Text uses the BPE tokenizer of a pre-trained LLM (Mistral 7B) with no grapheme-to-phoneme step. The LLM embedding table is frozen, and a small learnable MLP projects it to the T2S width. The target language is set through a short natural-language instruction prepended to the text, following the CosyVoice series format. Training uses the autoregressive cross-entropy loss on pairs of two different utterances from the same speaker (one reference, one target).

Learnable speaker encoder. An ECAPA-TDNN with attentive statistics pooling runs on frame-level w2v-BERT 2.0 features of the reference audio. It produces one vector at the T2S hidden dimension, which is prepended to the sequence. The speaker encoder is trained jointly with the T2S backbone rather than taken from a speaker-verification model. The paper argues that self-supervised features carry both content and speaker information, which lets the encoder extract timbre without the transcript. The paper does not state whether w2v-BERT 2.0 itself is updated; its list of jointly optimized parts names the speaker encoder, text projection, semantic-token embedding and Transformer only.

Dual-mode inference. T2S is trained only in the reference-cloning layout (speaker embedding plus target text). At inference the same model can also run continuation cloning: the reference transcript and the reference semantic tokens are prepended as a prefix (Table 1). Reference cloning is the default and needs no transcript.

S2A. A Diffusion Transformer trained with optimal-transport conditional flow matching (L1 vector-field loss) generates 80-dim mel-spectrograms. Its frame-level condition joins semantic-token embeddings, the continuous T2S hidden states at the semantic-token positions (to reduce the quantization bottleneck), and a frozen CAM++ speaker-verification embedding. A reference mel prompt is prepended along time. A learnable placeholder replaces the semantic condition over the prompt frames, so the prompt needs no transcript or tokens. Conditions are dropped jointly for classifier-free guidance (strength 0.7 at inference, 25 Euler steps).

Training. Training uses about 500k hours across Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay and Vietnamese. The data pass through separation, VAD segmentation, multi-ASR filtering (error rate below 2.5%) and speaker clustering, and include roughly 1,000 hours of synthetic speech per language. Training runs in two stages on 32 A40 GPUs: T2S plus speaker encoder first, then S2A with T2S frozen.

Key Results

Evaluation covers four public benchmarks plus a human ranking study. All objective results use WER/CER from Whisper large-v3 or Paraformer and SIM from the Seed-TTS-eval WavLM-large verification model. Most baselines are evaluated in transcript-conditioned configurations (marked †). The comparison therefore pits Confucius4-TTS’s transcript-free mode against systems given more information.

  • CV3-Eval cross-lingual (Table 2). Confucius4-TTS has the lowest error rate on four of six directions: en→zh 6.16, zh→en 3.19, ja→en 3.44, ko→en 3.42. VoxCPM2 is lower on ja→zh (4.20 vs. 4.87) and ko→zh (1.20 vs. 1.28). The six-direction average is 3.73. Averaging the same table gives 4.22 for VoxCPM2, 7.19 for CosyVoice 3-1.5B and 12.29 for OmniVoice. These baseline averages are computed here, not reported by the paper. Transcript-conditioned CosyVoice 2 (48.10) and OmniVoice (52.64) fail badly on ja→zh.
  • X-Voice, source→Chinese (Table 3). CER is lowest on de, fr, ko and vi sources (for example 2.70 vs. 3.01 for X-Voice on fr→zh). X-Voice is lower on en, ja and th, by 0.15, 0.11 and 0.03 points. OmniVoice degrades sharply on ja→zh (60.88).
  • Seed-TTS-eval, intra-lingual (Table 4). Reference cloning reaches WER 1.49 / SIM 0.700 on test-en and CER 0.94 / SIM 0.765 on test-zh. Qwen3-TTS has lower error on both (1.24, 0.77). Seed-TTS has the highest SIM on both (0.762, 0.796). Confucius4-TTS’s SIM is below Seed-TTS, Qwen3-TTS, OmniVoice and VoxCPM2 on English and Chinese. It is above the other transcript-free system, X-Voice, on all four columns.
  • MiniMax-MLS-Test, 11 languages (Table 5). Reference cloning has the lowest WER on German (0.47) and Thai (1.56) among the eight columns. SIM ranges from 0.723 (French) to 0.812 (Korean) and is never the highest in a row: OmniVoice or VoxCPM2 lead SIM in most rows. Malay, one of the 14 training languages, is not evaluated on any benchmark.
  • Reference vs. continuation cloning (Tables 4, 5). Adding the reference transcript and tokens raises SIM in all 11 MiniMax-MLS languages and on both Seed-TTS-eval subsets (for example French 0.723 to 0.755). It raises WER in 9 of 11 languages (Italian 1.30 to 3.26) and on both Seed-TTS-eval subsets. Japanese and Vietnamese are the two exceptions, with slightly lower error.
  • Human ranking (Table 6). Native or fluent raters rank six systems blind on four internal cross-lingual directions (number of raters and utterances not reported). Confucius4-TTS has the best overall rank on en→zh (1.97), zh→ko (1.20) and zh→ja (1.50). VoxCPM2 is better on zh→en (1.95 vs. 2.28). Confucius4-TTS ranks first in timbre similarity on all four directions. ElevenLabs and VoxCPM2 lead on emotional expressiveness for zh→en, where Confucius4-TTS is third.

Novelty Assessment

The contribution is primarily engineering integration. Its parts come from prior systems:

  • The two-stage LLM T2S plus flow-matching S2A layout and the language instruction format follow the CosyVoice series.
  • The jointly trained ECAPA-TDNN speaker encoder follows MiniMax-Speech and Qwen3-TTS.
  • Speaker encoding from self-supervised features follows the IndexTTS line and transcript-free flow-matching work.
  • Feeding T2S hidden states to the acoustic decoder follows IndexTTS2.
  • The semantic tokenizer is MaskGCT’s.

The new part is how these are combined for transcript-free cross-lingual cloning, together with dual-mode inference from a single training layout. The paper does not ablate its central design choices. No experiment compares the SSL-based speaker encoder with a mel-based one or with a frozen verification encoder, isolates the T2S hidden-state conditioning, or measures the synthetic-data share. The results show that the integrated system works but do not show which component is responsible.

Field Significance

Moderate — The paper shows that a CosyVoice-style semantic-token system conditioned only on a pooled speaker embedding can match or beat transcript-conditioned systems on cross-lingual intelligibility across several public benchmarks. On the same benchmarks, it trails them on speaker similarity. With released checkpoints, it can serve as an open transcript-free baseline for cross-lingual zero-shot TTS.

Claims

  • supports: Conditioning an autoregressive semantic-token TTS model on a single pooled speaker embedding, rather than on a paired text-audio prompt, yields strong cross-lingual intelligibility without needing a reference transcript.

    Evidence: Transcript-free reference cloning gets the lowest error on four of six CV3-Eval cross-lingual directions against transcript-conditioned CosyVoice 2/3 and OmniVoice (for example zh→en 3.19 vs. 5.39 for CosyVoice 3-1.5B). It also gets the lowest CER on four of seven X-Voice source→Chinese directions. (§4.2, Tables 2, 3)

  • supports: In zero-shot TTS, a prefix-style reference prompt improves speaker similarity over a global speaker embedding but tends to reduce intelligibility.

    Evidence: Continuation cloning with the reference transcript and tokens raises SIM in all 11 MiniMax-MLS-Test languages and on both Seed-TTS-eval subsets. WER rises in 9 of 11 languages (Italian 1.30 to 3.26) and on Seed-TTS-eval test-en (1.49 to 1.68). (§4.3, §4.4, Tables 4, 5)

  • complicates: Transcript-free cloning via a global speaker embedding still lags prompt-continuation systems on objective speaker similarity, even when human raters prefer its timbre.

    Evidence: On Seed-TTS-eval, reference-cloning SIM (0.700 en, 0.765 zh) is below Seed-TTS, Qwen3-TTS, OmniVoice and VoxCPM2, and is never the row maximum on MiniMax-MLS-Test. Human raters nonetheless rank its timbre similarity first on all four internal cross-lingual directions. (§4.3, §4.4, §4.5, Tables 4, 5, 6)

  • complicates: Transcript-conditioned zero-shot TTS can fail on some cross-lingual reference directions, so a low average error rate can hide direction-specific collapse.

    Evidence: On CV3-Eval ja→zh, transcript-conditioned CosyVoice 2 reaches 48.10 CER and OmniVoice 52.64, against 4.87 for the transcript-free system. OmniVoice also reaches 60.88 CER on X-Voice ja→zh against 2.85 to 8.16 on the other six directions. (§4.2, Tables 2, 3)

Limitations and Open Questions

No ablations

The paper does not ablate its key design choices: the SSL input to the speaker encoder, joint vs. pre-trained speaker encoders, T2S hidden-state conditioning in S2A, or the synthetic training data. The claim that SSL features are what enables transcript-free extraction is therefore argued, not tested.

  • The training corpus (about 500k hours, partly synthetic) is internal and not described per language. Language-resampling weights are not given.
  • Most baselines run in transcript-conditioned modes. MiniMax-Speech and ElevenLabs numbers on MiniMax-MLS-Test are copied from the MiniMax-Speech paper rather than re-run.
  • The human evaluation uses an internal test set. It reports neither the number of raters nor the number of utterances, and covers only four directions, all involving Chinese.
  • Malay is a training language but is not evaluated. The X-Voice evaluation covers only Chinese targets, and CV3-Eval covers only Chinese or English targets.
  • Model size, latency and real-time factor are not reported. Streaming is listed as future work.

Wiki Connections

  • Autoregressive Codec TTS — The T2S stage is a decoder-only LLM that predicts discrete MaskGCT semantic tokens, following the CosyVoice-style supervised-semantic-token pattern.
  • Flow Matching — The S2A stage is an optimal-transport conditional flow-matching DiT that converts semantic tokens and T2S hidden states to mel-spectrograms with classifier-free guidance.
  • Zero-Shot TTS — The paper evaluates prompt-based cloning of unseen speakers in transcript-free reference mode and transcript-based continuation mode within one model.
  • Multilingual TTS — The model is trained on 14 languages and evaluated on 11 languages intra-lingually (MiniMax-MLS-Test) plus Chinese/English targets cross-lingually.
  • Self-Supervised Speech — The learnable speaker encoder pools w2v-BERT 2.0 representations of the reference audio, which the paper credits for transcript-free speaker extraction.
  • Subjective Evaluation — Blind human ranking by native or fluent raters compares six systems on timbre, naturalness, pronunciation, emotion and overall quality across four cross-lingual directions.
  • X-Voice — A transcript-free cross-lingual baseline and benchmark source. Confucius4-TTS reaches transcript-free cloning without X-Voice’s synthetic-prompt fine-tuning and beats it on Seed-TTS-eval.
  • VoxCPM2 — The strongest overall baseline, leading on two CV3-Eval directions, most MiniMax-MLS SIM rows and the zh→en human overall rank.
  • OmniVoice — An evaluated baseline whose transcript-conditioned configuration collapses on ja→zh in both CV3-Eval and X-Voice.
  • CosyVoice 3 — Source of the CV3-Eval benchmark and of the instruction text format; its 0.5B and 1.5B models are cross-lingual baselines.
  • CosyVoice 2 — A cross-lingual baseline on CV3-Eval with much higher error, especially for Japanese references.
  • MiniMax-Speech — Source of the learnable-speaker-encoder design and the MiniMax-MLS-Test benchmark, and a multilingual baseline whose MiniMax-MLS-Test numbers are copied from that paper (the human study instead uses the commercial MiniMax-Speech-2.8-HD).
  • Qwen3-TTS — Source of the ECAPA-TDNN speaker-encoder recipe and text-projection MLP, and the strongest Seed-TTS-eval intelligibility baseline.
  • MaskGCT — Provides the frozen pre-trained semantic tokenizer whose tokens the T2S model predicts.