arXiv · 2026 · Preprint

Changi Hong et al. (Gwangju Institute of Science and Technology / AunionAI) · → Paper · Demo: ? · Code: ?

Improves lip-sync in Korean-English automated dubbing without touching the video, by having an LLM paraphrase the translated text and picking the candidate whose predicted duration and vowel sequence best match the source speech, scored by DTW over vowel embeddings taken from a cross-lingual TTS text encoder.

Problem

Automated dubbing has to satisfy isochrony (matching pauses and speaking duration) and lip-sync (matching visible mouth movements). Prior work handled isochrony by length-controlled machine translation or prosodic alignment in the TTS stage, and handled lip-sync either by conditioning TTS on video embeddings or by deepfake re-rendering of the speaker’s lips. The authors argue these approaches either modify the original video or work mainly between languages with similar word order, and do not address structurally distant pairs such as Korean and English. The paper was accepted to ICPR 2026; this page covers the arXiv record.

Method

The work sits inside a cascaded dubbing pipeline (source separation, BGM-robust ASR with forced alignment, speaker clustering, NMT, then TTS) that the authors describe as background (§2.1). Their contribution is a text-side synchronization stage inserted between NMT and TTS, plus a modified cross-lingual TTS.

TTS backbone. A YourTTS model (VITS-style: posterior encoder, prior encoder with transformer text encoder and duration predictor, HiFi-GAN generator trained adversarially) is trained on LibriTTS-360 (904 speakers) and KMSSS (550 Korean speakers) for Korean-to-English (K2E) and English-to-Korean (E2K) cross-lingual synthesis, with language embeddings. The integer-duration predictor of vanilla YourTTS is replaced with a differentiable duration predictor (following Parallel Tacotron 2), so gradients flow through durations and speaking rate can be controlled more precisely. Training uses AdamW for 200 epochs on four A100s; model size is not reported.

Isochrony (ISO). Source and target texts are phonemized (g2pk for Korean, phonemizer for English) and blank-interleaved. Pauses in the source audio are detected either by an RMS energy threshold (frames below 0.01 energy for at least 250 ms, then boundary refinement) or by CTC forced alignment of word boundaries. Pause frames are removed, a source speaking rate is computed as the ratio of the pause-free source frame length to the TTS-predicted duration of the source text, and this rate is applied to the TTS-predicted duration of each target candidate. A candidate is accepted if its predicted length is within about 30 frames (roughly 350 ms) of the source and its LaBSE cross-lingual cosine similarity to the source is at least 0.75. Otherwise GPT-4o generates paraphrases, for up to 60 iterations; if nothing passes, the most semantically similar candidate is kept.

Phonetic synchronization (PS). The authors build a Korean-English vowel distance table from the trained TTS: per-sentence mean vectors for each vowel are taken from a linear projection of the text encoder embeddings over 6,000 Korean and 5,000 English training sentences, clustered into five K-means centroids per vowel, and averaged; the Korean-English vowel distance is the Euclidean distance between averaged centroids (Fig. 5). GPT-4o then produces 60 paraphrase candidates of the isochronous target sentence. Each candidate and the source are reduced to vowel indices (non-vowels set to a null index), expanded by predicted per-phoneme durations, and compared by DTW with a Sakoe-Chiba band using the vowel distance as local cost. PS picks the candidate with the lowest DTW distance.

PS-Comet. Because PS ignores meaning, PS-Comet selects the candidate maximizing a weighted sum of the normalized, inverted DTW score (weight 1.6) and a COMET score (weight 0.4). The weighting is motivated by a FLORES analysis showing DTW and COMET are only weakly (negatively) correlated (Pearson r = -0.327).

Key Results

Lip-sync is measured with SyncNet-based LSE-D (lower is better) and LSE-C (higher is better); naturalness only with UTMOS. All evaluation sets are small: 20 clips each from a Korean lip-reading corpus and TCD-TIMIT, and 15 film clips for the voice-actor comparison.

  • Lip-reading datasets (Table 2, averaged over Korean and English). Baseline TTS scores LSE-D 12.671 / LSE-C 1.128 / UTMOS 2.453. ISO alone gives 12.640 / 1.065 / 2.432, slightly worse than baseline on LSE-C and UTMOS. ISO + PS gives 12.378 / 1.175 / 2.562, and ISO + PS-Comet 12.175 / 1.404 / 2.614, the best TTS row on all three metrics. Ground truth video audio is 9.980 / 3.537 / 2.749, so a large lip-sync gap to real speech remains. The §4.3 prose gives PS a UTMOS of 2.362, but the Table 2 image (table-2.png) shows 2.562.
  • Voice-actor set (Table 1). For K2E, PS-Comet reaches LSE-D 10.561 / LSE-C 1.457 versus professional voice actors at 11.118 / 1.260 and the baseline TTS (“Origin”) at 10.898 / 1.191. For E2K, PS-Comet reaches 10.182 / 0.744 versus voice actors at 10.852 / 0.668 and baseline at 10.906 / 0.703; plain PS ties PS-Comet on E2K LSE-D (10.182) and is marginally higher on LSE-C (0.750). The paper frames the baseline TTS as worse than voice actors, but the table is mixed: baseline beats voice actors on K2E LSE-D and E2K LSE-C and loses on the other two cells. Swapping RMS for CTC pause detection worsens PS-Comet on every cell (K2E 11.057 / 1.169, E2K 10.377 / 0.713), although the authors report CTC sounds more natural in the videos, which they take as a limitation of LSE metrics.
  • Against deepfake lip editing (Table 3, voice-actor average). Commercial deepfake post-processing on top of PS-Comet gives far better lip-sync (LSE-D 8.965, LSE-C 2.528) than PS-Comet TTS alone (10.372, 1.101), but lowers video quality (VMAF 86.208 versus 98.229) and takes 19 min 29 s versus 1 min 34 s per 10 s video. The Table 3 baseline LSE-D (11.092) does not equal the mean of the Table 1 “Origin” cells (10.898 and 10.906), while the other baseline and PS-Comet cells do match the Table 1 averages (checked against table-1.png and table-3.png).
  • Semantic cost (Table 4). Paraphrasing for lip-sync lowers source-target LaBSE similarity from 76.5 (initial translation) to 67.9 (PS) and 68.2 (PS-Comet) for K2E, and from 80.0 to 73.2 and 76.7 for E2K. SBERT and LASER show the same ordering.

No human listening or viewing results appear in the main text; subjective tests and the French-language experiments are deferred to supplementary material that is not part of the parsed paper.

Novelty Assessment

The contribution is primarily engineering integration. Length-matched paraphrasing for isochrony is established in dubbing research; the new element is the PS selection criterion, a DTW alignment over duration-expanded vowel sequences whose local cost is derived from the trained TTS text encoder’s vowel embeddings, combined with COMET for meaning. The TTS itself is YourTTS with a differentiable duration predictor substituted in, and candidate generation is delegated to GPT-4o. The lip-sync gains over the no-PS baseline are consistent but modest (LSE-D 12.671 to 12.175 on lip-reading data), measured only with SyncNet metrics on 15 to 40 clips, with no main-text human evaluation and no comparison against other text-side dubbing methods such as isometric or isochrony-aware MT. The vowel-centroid procedure (Eq. 8) is garbled in the parsed text, so its exact form is not fully recoverable from the main paper.

Field Significance

Moderate. The paper shows that lip-sync in automated dubbing can be improved purely on the text side, by choosing among LLM paraphrases using a vowel-level phonetic alignment score derived from the TTS model, and quantifies the resulting trade-off against semantic fidelity and against video-editing approaches. It is a useful data point for structurally distant language pairs, but the small evaluation and SyncNet-only measurement keep it closer to a proof of concept than a benchmarked advance.

Claims

  • supports: Selecting among paraphrases of a translated sentence by phonetic similarity to the source utterance can improve lip-sync of dubbed speech without modifying the video.

    Evidence: ISO + PS-Comet lowers averaged LSE-D from 12.671 to 12.175 and raises LSE-C from 1.128 to 1.404 over the same TTS without synchronization on Korean and English lip-reading clips; on film clips it also beats professional voice-actor dubbing on both SyncNet metrics in both directions. (§4.3, Table 2; §4.2, Table 1)

  • complicates: Optimizing dubbing text for phonetic or lip-sync alignment trades away semantic fidelity to the source, which must be reintroduced explicitly.

    Evidence: LaBSE source-target similarity drops from 76.5 to 67.9 (K2E) and 80.0 to 73.2 (E2K) under pure PS; adding a COMET term recovers only part of it (68.2 and 76.7). (§4.5, Table 4; §3.3)

  • complicates: Duration matching alone is not sufficient for lip-sync and can slightly degrade synchronization and naturalness metrics in automated dubbing.

    Evidence: ISO without PS changes averaged LSE-D only from 12.671 to 12.640 while LSE-C falls from 1.128 to 1.065 and UTMOS from 2.453 to 2.432; on the E2K voice-actor set ISO also lowers LSE-C (0.703 to 0.666). (§4.3, Table 2; §4.2, Table 1)

  • complicates: SyncNet-based lip-sync metrics (LSE-D/LSE-C) may not capture the naturalness of pause placement and timing in dubbed speech.

    Evidence: CTC-based pause detection scores worse than RMS on every LSE cell (K2E LSE-D 11.057 vs 10.561) yet the authors judge its video results perceptually better. (§4.2, Table 1)

  • refines: Video-side lip editing still achieves much stronger lip-sync than text-side synchronization, at a large cost in video fidelity and compute.

    Evidence: Deepfake post-processing reaches LSE-D 8.965 / LSE-C 2.528 versus 10.372 / 1.101 for PS-Comet TTS, but VMAF falls from 98.229 to 86.208 and processing time rises from 1 min 34 s to 19 min 29 s per 10 s video. (§4.4, Table 3)

Limitations and Open Questions

Warning

All results come from SyncNet LSE metrics and UTMOS on very small sets (20 clips per lip-reading corpus, 15 film clips); the main text reports no human ratings, no translation-quality comparison against other dubbing MT methods, and no significance testing, so the size and perceptual relevance of the gains are uncertain.

  • The authors themselves note the LSE metrics disagree with their perceptual judgment on pause detection, which weakens conclusions drawn from them.
  • Candidate generation depends on GPT-4o (up to 60 paraphrases per sentence), a closed model, which limits reproducibility and adds latency.
  • The approach is cascaded and uses no visual information; the authors point to unified audio-visual dubbing models as an alternative direction.
  • The upstream ASR, NMT, separation and speaker-clustering modules are described but not evaluated, and it is not stated whether the experiments use ASR output or reference transcripts.
  • Speaker identity preservation is cited as the reason for choosing YourTTS but is not measured.
  • Two reporting discrepancies: the §4.3 prose UTMOS for PS (2.362) differs from Table 2 (2.562), and the Table 3 baseline LSE-D (11.092) is not the average of the Table 1 baseline cells.
  • French and other language-pair results are only in supplementary material.

Wiki Connections

  • Multilingual TTS — Trains a cross-lingual YourTTS on Korean and English data with language embeddings and derives a cross-lingual vowel distance table from its text encoder.
  • Prosody Control — Controls dubbed-speech timing by estimating a pause-free source speaking rate and selecting target text whose TTS-predicted duration matches, with a differentiable duration predictor; evaluated through ISO ablations.
  • GAN Vocoder — The trained TTS uses a VITS-style posterior/prior encoder with an adversarially trained HiFi-GAN generator.
  • HiFi-GAN — Supplies the generator and discriminator design inside the trained YourTTS backbone.
  • LibriTTS — LibriTTS-360 provides the English half of the cross-lingual TTS training data.
  • UTMOS — Used as the only naturalness metric for the dubbed speech.
  • XTTS — Cited as support for YourTTS-family speaker similarity in dubbing; not used as a baseline.