arXiv · 2026 · Preprint

Antonis Asonitis et al. (AGIGO / ETH Zurich / Sapienza University of Rome / University of Zurich) · → Paper · Demo: ✓ · Code: ✓

GRAFT lets a codec language model TTS take a short spoken example of a single word, from any speaker, in place of that word’s text, so the word follows the example’s pronunciation while the sentence stays in the cloned target voice.

Problem

Codec language model TTS systems built on pretrained text LLMs inherit a strong text prior. This helps on common words but often mispronounces rare proper nouns, brand names, loanwords and technical terms, because spelling underdetermines pronunciation. Existing per-word fixes need a written phonetic specification: G2P lexicons assume one canonical pronunciation and generalise poorly to unseen words, lexicon-reading models need an entry, and weight-editing methods such as SonoEdit need a target transcription. Transcribing a spoken example with a neural phone recogniser gives a phoneme string, but the paper argues that these phonemizers discard diacritics and so lose tone and stress detail. No prior system lets a non-expert fix a word’s pronunciation simply by saying the word, while keeping the output in a different target voice.

Method

GRAFT keeps the architecture of Qwen3-TTS (Qwen3-TTS-12Hz-0.6B-Base, 16 codebooks at 12.5 Hz) unchanged and adds no parameters. The input is a target text, a target-voice reference clip, and one or more (word, isolated recording) pairs. The prompt contains a style slot for the reference clip and a body in which ungrafted words keep their text tokens. Each grafted word is replaced by a graft slot holding its codec tokens, never added alongside its text. Codec embeddings are summed across all 16 codebooks and written into the placeholder positions; the talker transformer then generates the output codec frames autoregressively (§III-A, §III-B). At inference the word clip is trimmed with a voice activity detector and encoded with the model’s own codec, so no forced alignment is needed (§III-E).

The training data teaches the model to copy pronunciation while ignoring the hint speaker. Each training utterance is voice-converted to a random other speaker with Seed-VC (20 steps), force-aligned with the Qwen3-TTS aligner, and cut into per-word clips that are re-encoded with the codec. The training target is always the codec of the original, unconverted utterance, so voice conversion artefacts never enter the target (§III-C). Eligible words (at least 0.04 s and four characters) are grafted with probability 0.5. With probability 0.3 the graft comes from the same word in a different utterance, which breaks the coarticulation of an in-context cut. With probability 0.5 it is augmented by decoding, perturbing (noise, loudness, pitch and speed, telephone band, compression, reverberation, variable silence) and re-encoding (§III-C, Eq. 1).

Figure 2: Training-data construction. From a Speaker A utterance we make a voice-converted copy in a different voice, force-align it to find word boundaries, and re-encode the target word ('Nguyen') with the codec as a per-word graft. The graft replaces that word's text in the prompt, and the model is trained to predict the codec of the original Speaker A utterance. Since the graft is in a different voice from the target, the model learns to copy pronunciation while ignoring the hint speaker.

The model is fine-tuned with the standard next-token cross-entropy on talker codec tokens plus an auxiliary sub-talker loss weighted 0.3. Training uses about 7.3k hours (3.2M utterances, 28.4k speakers) from VCTK, LibriTTS, MLS and Common Voice in English, German, French, Spanish and Italian, and takes 48 hours on a single 96 GB GPU (§III-D, Table I).

Key Results

Evaluation uses a new benchmark of 500 rare words per language (wordfreq Zipf 1.3 to 3.3) from Wikimedia Lingua Libre isolated recordings, each placed in a carrier phrase, with the human recording as the hint (§IV-A). The main metric, D-PER, is the phoneme error rate between the hint and the target word cut from the output, both transcribed with the ZIPA phone recogniser. Whisper similarity (WS) compares Whisper-encoder embeddings of the two word clips (§IV-C). Phoneme-input baselines receive the hint transcribed to espeak phonemes; text-input baselines get only the spelling (§IV-B).

  • Versus the identical text-only base (Table III, D-PER): English 0.37 to 0.29, German 0.49 to 0.30, French 0.41 to 0.28, Spanish 0.38 to 0.29, Italian 0.40 to 0.25, a 22 to 39% reduction, significant in every language (paired Wilcoxon, p < 10^-7). The English per-word spread also narrows (±0.16 vs ±0.42) (§V-A).
  • Versus open-source systems: GRAFT has the lowest D-PER, the lowest hardest-10% D-PER, and the highest WS in all five languages. In English the closest D-PER competitors are F5-TTS (0.38) and CosyVoice 2 (0.39); phoneme-input systems score 0.43 (Matcha-TTS) to 0.65 (StyleTTS 2) even when given the hint’s phonemes (Table III).
  • Costs: sentence WER rises over the base in every language (English 0.09 vs 0.02; German 0.11 vs 0.07; French 0.07 vs 0.05; Spanish 0.09 vs 0.02; Italian 0.11 vs 0.06). Removing the target word drops English WER from 0.09 to 0.03 (base 0.02 to 0.01), so most of the gap sits on the grafted word, which Whisper also often fails to transcribe from the clean human recording (§V-A). English SSIM is 0.76, above the base (0.73) but below MaskGCT (0.80) and F5-TTS (0.79). UTMOS is above the base in four languages and below it in German (3.99 vs 4.03) (Table III).
  • Human study (English, 30 words, 20 paid listeners, 600 pairwise comparisons, Bradley-Terry): GRAFT ranks first, 150 points ahead of the next system, with its 95% CI lower bound (2107) above every baseline’s point estimate (§VI, Figure 3).
  • Oracle phonemes: on 225 CMUdict-covered English words, phoneme baselines given human-curated CMUdict pronunciations score D-PER 0.45 to 0.69, against 0.30 for GRAFT with audio (Table V). Note that D-PER is scored against the hint, not the dictionary form.
  • Ablations (100 ACL 60/60 words, Table II): without voice conversion, D-PER is lowest (0.16 vs 0.19) but SSIM collapses (0.20 vs 0.68) and UTMOS and WER worsen (3.89 vs 4.43; 0.18 vs 0.05). ECAPA scores on the cut target word show the no-VC model places the word in the hint voice (0.67 hint vs 0.10 target), while GRAFT renders it in the target voice (0.47 target vs 0.12 hint) (§V-A). Without augmentation, D-PER rises to 0.25.
  • Multiple hints (100 carriers, Table IV): moving from one to five grafted words raises D-PER from 0.282 to 0.376 and WER from 0.049 to 0.206 (peak 0.245 at four), while UTMOS stays between 4.440 and 4.494.

The comparisons are fair in that all systems get the same carrier and speaker reference, but GRAFT alone receives the pronunciation as audio, and the benchmark scores faithfulness to the supplied hint rather than canonical correctness, as the paper itself states (§V-A).

Novelty Assessment

The closest prior mechanism is WESCON, which splices an acoustic prompt at a chosen word to control emotion. GRAFT applies the same per-word splicing to pronunciation and adds the key ingredient: voice-converted per-word hints with cross-utterance swaps and augmentation, so the hint speaker can differ from the target. The ablation shows this data construction is what separates pronunciation transfer from copying the hint waveform. The method is a new conditioning interface and training-data recipe on an unchanged backbone, not a new network design. The multilingual difficult-word benchmark with openly licensed isolated recordings, and the D-PER and WS target-word metrics, are a secondary evaluation contribution. All results come from one 0.6B backbone.

Field Significance

Moderate — The paper shows that a codec language model TTS can take an acoustic example of a single word as a pronunciation control, with no extra parameters, and that voice-converted training hints are needed to stop the model copying the hint voice. It also releases a five-language benchmark for target-word pronunciation fidelity. Its evidence is limited to one backbone and non-tonal languages.

Claims

  • supports: Splicing a spoken example of a word into a codec language model’s prompt, in place of its text tokens, can control that word’s pronunciation more faithfully than text input or phoneme input derived from the same example.

    Evidence: On 500 rare words per language, GRAFT lowers target-word D-PER over the identical text-only backbone by 22 to 39% in five languages and beats phoneme-input baselines given the hint’s espeak phonemes; in a 600-comparison English listening test it ranks first by 150 Bradley-Terry points. (§V-A, §VI, Table III, Figure 3)

  • supports: Training on voice-converted conditioning examples paired with original-voice targets lets a TTS model transfer content from a reference in one voice while keeping a different target voice.

    Evidence: Without voice conversion, grafted words sit on the hint speaker (ECAPA 0.67 hint vs 0.10 target) and utterance SSIM collapses to 0.20; with it, grafted words carry the target identity (0.47 target vs 0.12 hint) and SSIM is 0.68. (§V-A, Table II)

  • complicates: Per-word acoustic pronunciation control trades sentence-level intelligibility for target-word fidelity, and the cost grows with the number of controlled words.

    Evidence: GRAFT’s WER exceeds the base in all five languages (English 0.09 vs 0.02), and with five hints in one carrier WER rises from 0.049 to 0.206 while D-PER rises from 0.282 to 0.376. (§V-A, §VI, Table III, Table IV)

  • complicates: Sentence-level ASR WER is a weak measure of rare-word pronunciation, since one wrong word barely changes it and ASR often fails on correctly pronounced rare words.

    Evidence: Whisper-large-v3 returns the exact word from the clean isolated human recording only 75% of the time in English (66% overall), and on over half of GRAFT’s ASR misses the rendered pronunciation still matches the hint (D-PER at most 0.34). (§V-A)

  • refines: Supplying a human-curated dictionary pronunciation to phoneme-conditioned TTS does not close the gap to direct acoustic conditioning when fidelity is scored against a recorded reference.

    Evidence: On 225 CMUdict-covered English words, phoneme baselines reach D-PER 0.45 to 0.69 with oracle CMUdict phonemes and 0.43 to 0.65 with hint-derived phonemes, against 0.30 for GRAFT with audio. (§VI, Table V)

Limitations and Open Questions

  • GRAFT is shown on one 0.6B backbone; scaling to larger models is untested (§VI).
  • The interface works on single words; multi-word spans are future work (§VI).
  • The benchmark languages are non-tonal, so the motivating advantage over phonemizers that drop tone marks is not measured; tonal-language transfer is open (§VI).
  • Mistrimmed or low-quality hints can produce a wrong pronunciation; reliability depends on matching the trimmed training distribution (§VI).
  • D-PER and the human study both score faithfulness to the supplied hint, not canonical correctness, and GRAFT is the only system that receives the hint as audio. The paper argues that WS and the human study rule out a channel-match artefact, but the benchmark does not test whether the output is correct when the hint itself is wrong.
  • The source of the target-speaker reference clips in the benchmark, and whether those speakers are absent from training, is not reported, so voice cloning of unseen speakers is not separately established.
  • The human study covers English only (30 words); other languages rely on objective metrics.

Wiki Connections

  • Autoregressive Codec TTS — GRAFT fine-tunes an autoregressive codec language model and adds a per-word codec-token conditioning slot without new parameters.
  • Multilingual TTS — The model is fine-tuned on five languages and evaluated on a new difficult-word benchmark in each of them.
  • Subjective Evaluation — A blind pairwise listening study with Bradley-Terry scoring measures pronunciation closeness to a reference recording.
  • Qwen3-TTS — GRAFT fine-tunes the Qwen3-TTS-12Hz-0.6B-Base model and uses it as the text-only baseline.
  • WESCON — The closest prior per-word splicing mechanism, used for emotion; GRAFT adapts it to pronunciation and adds voice-converted training.
  • Seed-VC — Used only in data preparation to voice-convert training utterances so per-word hints come from a different speaker.
  • SonoEdit — A per-word pronunciation fix by weight editing that needs a written phonetic specification, which GRAFT avoids.
  • F5-TTS — Text-input baseline with the closest English D-PER to GRAFT and higher speaker similarity.
  • CosyVoice 2 — Text-input baseline that matches the base model on English UTMOS and trails GRAFT on target-word D-PER.
  • MaskGCT — Phoneme-input baseline given hint phonemes; highest SSIM but higher D-PER and WER than GRAFT.
  • XTTS — Multilingual text-input baseline evaluated in all five languages.