arXiv · 2026 · Preprint
Zhiyuan Zhao et al. (International Digital Economy Academy (IDEA)) · → Paper · Demo: ✓ · Code: ✓
Releases LEMAS-Dataset, a 150,000-hour, 10-language open speech corpus with word-level timestamps and confidence scores, and validates it with two benchmark models: LEMAS-TTS (a multilingually stabilized F5-TTS extension) and LEMAS-Edit (a multilingual extension of VoiceCraft for speech editing).
Problem
Generative speech models perform strongly in high-resource languages like English and Chinese but degrade in multilingual settings, and the paper argues this is primarily a data problem rather than an architecture problem. Existing large-scale corpora are either monolingual (Libri-Light, GigaSpeech, WenetSpeech), scale- or domain-limited when multilingual (TEDx, MLS), or scale to hundreds of thousands of hours only by relying on minimally filtered web-crawled data with unreliable annotations (YODAS). Datasets with rigorous word-level timestamps, which are needed for both robust TTS training and precise speech editing, exist for single languages (WenetSpeech4TTS) but not at multilingual scale.
Method
LEMAS-Dataset aggregates audio-text pairs from GigaSpeech, GigaSpeech2, WenetSpeech4TTS, Emilia, MLS, multilingual TEDx, Alcaim, Golos, and YODAS, normalizes them into a unified schema, and re-aligns all of it with the Multilingual MMS Forced Aligner (a wav2vec-based CTC alignment model trained on 1,100+ languages), after first romanizing transcripts with Uroman to avoid language-specific pronunciation dictionaries. Word-level alignments are retained only above per-dataset confidence thresholds, alongside duration/pause constraints, language-specific speech-rate normalization, and character/script validation. The result is 150,144 hours across 10 languages (Chinese, English, Russian, Spanish, Portuguese, German, French, Italian, Indonesian, Vietnamese) with word-level timestamps and per-word confidence scores, split into a training partition and a curated, per-language-balanced 500-utterance evaluation partition (with roughly 20% of eval samples manually verified as natural speech).
LEMAS-TTS extends the F5-TTS non-autoregressive flow-matching / DiT architecture for multilingual stability. It replaces mixed character/word tokenization with a unified phonetic front-end (tonal Pinyin for Chinese, IPA via eSpeak-NG elsewhere, with explicit language-identifier tokens and pause tags derived from the dataset’s word timestamps), and adds two auxiliary training objectives: a CTC alignment loss on decoder outputs to enforce monotonic acoustic-linguistic correspondence, and an accent-adversarial objective (a gradient-reversal-layer accent classifier supervised with pseudo-labels from an off-the-shelf language-ID model) to suppress cross-lingual accent leakage, inspired by IndexTTS2’s disentanglement approach. A Prosody Encoder (adapted from Seamless Expressive, using an ECAPA-TDNN backbone over filterbank features) produces a prosody embedding injected into the DiT for explicit prosody transfer from a reference clip. At inference, the paper also redesigns F5-TTS’s Sway Sampling and classifier-free-guidance schedules as tunable one-parameter families (a quadratically decaying CFG schedule and a power-law time-warping reparameterization) to improve stability on long multilingual utterances, and adds a reference-free mode that samples speaker characteristics internally instead of conditioning on a reference clip.
LEMAS-Edit extends VoiceCraft, a decoder-only autoregressive architecture that treats speech editing as masked token infilling, from its original English-only training to a 7-language multilingual setting via warm-start initialization from VoiceCraft’s 330M-parameter checkpoint (architecture unchanged). It adds a history-aware repetition penalty that scales with generation length to suppress token-looping artifacts, and an adaptive re-generation mechanism that estimates a target speaking rate from the reference audio, imposes a corresponding max-rate constraint during decoding, and triggers automatic re-generation with relaxed constraints (wider mask boundaries, higher repetition penalty) when output length or an internal anomaly flag indicates a failed edit. The editing front end uses multilingual Whisper for transcription and the MMS aligner for precise word-level timestamps to construct editing masks, with a choice of two denoising backends and automatic chunking with cross-fade stitching for long recordings.
Key Results
On multilingual TTS (10 languages, 10,000 samples), LEMAS-TTS outperforms the contemporaneous open-source multilingual baseline OpenAudio-S1-mini on both WER and speaker similarity in every language. The prosody-aware LEMAS-TTS variant reaches an average WER of 6.39% and SIM of 0.539, versus 8.06%/0.547 for the non-prosody variant and 12.27% (excluding an anomalous Vietnamese result) for OpenAudio-S1-mini. The prosody encoder consistently lowers WER (improved pronunciation stability) while slightly lowering speaker similarity, a trade-off the authors attribute to reduced expressive richness. For speech editing, LEMAS-Edit is evaluated via a human A/B preference test (6 listeners, 20 utterances across 7 languages, edits generated by randomly replacing words/phrases via ChatGPT); results show overall balanced listener judgments across languages with only minor language-specific preferences, which the authors present as evidence that LEMAS-Dataset supports robust, architecture-agnostic multilingual editing evaluation rather than as a head-to-head win over a specific baseline.
Novelty Assessment
The dataset construction pipeline (MMS-based multilingual forced alignment via romanization, confidence-scored filtering, balanced per-language evaluation curation) is the paper’s most substantial and independently verifiable contribution: it is a genuine scale and annotation-quality advance over comparable multilingual corpora (Table 1 in the paper shows LEMAS-Dataset is the only compared corpus combining multilingual coverage, 150k+ hour scale, and word-level timestamps). LEMAS-TTS and LEMAS-Edit are both explicitly framed as validation vehicles for the dataset rather than architectural contributions in their own right; F5-TTS and VoiceCraft are extended, not redesigned, and most of the individual mechanisms (CTC alignment loss, GRL-based adversarial disentanglement, prosody encoders, repetition penalties for autoregressive decoding) are adaptations of established techniques rather than new inventions. The TTS comparison uses a single baseline (OpenAudio-S1-mini); the paper does not compare against other strong multilingual systems mentioned in its own related work (CosyVoice 3, XTTS-v2), which would have given a fuller picture of where LEMAS-TTS sits in the current multilingual TTS landscape.
Field Significance
Tip
High, at 150,000+ hours across 10 languages with word-level timestamps, confidence scores, open code, dataset, and demo release, LEMAS-Dataset is a substantial open infrastructure contribution to multilingual generative speech research, filling a specific gap (large-scale, high-quality, temporally annotated, and genuinely multilingual) that the paper documents concretely against prior corpora. Its practical value depends on adoption by others training or fine-tuning multilingual TTS and editing systems, which is not yet demonstrated within the paper itself.
Claims
- supports: Unifying diverse scripts into a shared phonetic representation, augmented with explicit language identifiers, stabilizes large-scale multilingual flow-matching TTS training and improves cross-lingual intelligibility.
Evidence: LEMAS-TTS, extending F5-TTS with a unified phonetic front-end, CTC alignment loss, and accent-adversarial disentanglement, achieves lower WER than the OpenAudio-S1-mini baseline in every one of 10 evaluated languages (average WER 6.39% vs. 12.27%). (§6.1, Table 4)
- supports: An explicit accent-adversarial disentanglement objective, applied via a gradient-reversal accent classifier, is a mechanism for reducing cross-lingual accent leakage in multilingual zero-shot TTS.
Evidence: An accent classifier attached to the conditioning pathway via a Gradient Reversal Layer, supervised with pseudo-labels from an off-the-shelf language-identification model, is introduced specifically because flow-matching models without such regularization were observed to leak higher-resource-language prosody into lower-resource-language synthesis. (§4.1)
- complicates: Adding explicit prosody conditioning to a multilingual flow-matching TTS model improves pronunciation stability at some cost to speaker similarity.
Evidence: The prosody-aware LEMAS-TTS variant achieves lower average WER than the variant without a prosody encoder (6.39% vs. 8.06%) but slightly lower average speaker similarity (0.539 vs. 0.547), which the authors attribute to prosodic conditioning trading expressive richness for articulation stability. (§6.1, Table 4)
- supports: Fine-grained, confidence-scored word-level alignment at large multilingual scale is a practical enabler for extending an autoregressive speech-editing model beyond the single language it was originally trained on.
Evidence: LEMAS-Edit, warm-started from a 330M-parameter English-only VoiceCraft checkpoint and fine-tuned on word-level-aligned multilingual subsets covering 7 languages, extends VoiceCraft’s masked-infilling editing to multilingual audio without architectural changes, and achieves balanced A/B naturalness preference across languages in human evaluation. (§5.1, §6.2, Figure 4)
Limitations and Open Questions
- Subjective evaluation of LEMAS-TTS was not conducted for languages other than Chinese and English due to a lack of native evaluators, so the paper’s TTS quality claims for the other 8 languages rest on objective metrics (WER, SIM) alone.
- The Vietnamese WER result for the OpenAudio-S1-mini baseline was excluded from the reported average as “unusually high,” a judgment call made by the authors rather than an independently audited exclusion criterion.
- TTS comparison uses a single contemporaneous baseline (OpenAudio-S1-mini); other strong multilingual systems the paper’s own related work discusses (CosyVoice 3, XTTS-v2) are not included in the head-to-head evaluation.
- The paper does not apply speech enhancement or noise removal during dataset construction by design, preserving in-the-wild acoustic variability; this is a deliberate trade-off rather than a flaw, but it means downstream models trained on LEMAS-Dataset inherit whatever acoustic noise the source corpora contain.
Wiki Connections
- Flow Matching — LEMAS-TTS extends F5-TTS’s non-autoregressive flow-matching and DiT backbone with multilingual stabilization objectives rather than proposing a new flow-matching formulation.
- Multilingual TTS — trains and validates both benchmark models across 10 (TTS) and 7 (editing) languages using a shared phonetic representation and language-identifier conditioning, the paper’s central contribution.
- Zero-Shot TTS — LEMAS-TTS targets robust zero-shot multilingual synthesis from a reference clip, building on F5-TTS’s zero-shot cloning paradigm at larger multilingual scale.
- Disentanglement — trains an accent classifier via a Gradient Reversal Layer on the conditioning pathway to explicitly suppress accent information and force accent-invariant representations, evaluated through the resulting reduction in cross-lingual accent leakage.
- Prosody Control — adds a dedicated Prosody Encoder that extracts a prosody embedding from reference audio and injects it into the DiT for explicit, controllable prosody transfer across languages.
- Subjective Evaluation — evaluates LEMAS-Edit’s speech-editing naturalness via a 6-listener A/B preference test across 7 languages.
- F5-TTS — LEMAS-TTS is built directly on F5-TTS’s flow-matching DiT architecture, adding multilingual stabilization objectives and a redesigned CFG/sampling schedule.
- VoiceCraft — LEMAS-Edit is a multilingual extension of VoiceCraft, warm-started from its 330M-parameter English checkpoint without architectural changes.
- Seed-TTS — the paper’s speaker-similarity (SIM) evaluation protocol follows the one established in this paper.
- IndexTTS2 — the accent-adversarial disentanglement mechanism in LEMAS-TTS is explicitly inspired by this paper’s disentanglement approach.
- CosyVoice 2 — discussed in related work as a hybrid discrete-plus-flow-matching multilingual TTS approach, part of the landscape LEMAS-TTS is positioned against.
- CosyVoice — discussed in related work as an early hybrid semantic-token-plus-flow-matching multilingual zero-shot TTS system.