ACL · 2025 · Conference
Dua et al. (Oracle AI) · → Paper · Demo: ? · Code: ?
SpeechWeave is an end-to-end pipeline for generating diverse, normalized multilingual text and speaker-standardized speech audio to serve as training data for TTS systems, achieving 10–48% greater linguistic and phonetic diversity than direct LLM prompting and 97% normalization accuracy in English.
Problem
Training high-quality TTS systems for commercial, domain-specific applications requires large quantities of text and speech data covering target business domains. Sourcing such data is difficult: web scraping introduces licensing concerns and domain gaps, and publicly available corpora (LJSpeech, LibriSpeech) are predominantly audiobook-style and lack coverage of professional domains such as healthcare, finance, or automotive. LLMs are a natural candidate for text synthesis, but direct prompting with a fixed domain label produces low-diversity output — experiments with Mistral-7b-Instruct show that even high temperature and top-p settings yield near-identical sentences when the prompt is unchanged. Post-hoc text normalization tools (e.g., NeMo’s WFST normalizer) also fail to generalize across semiotic class variations (addresses, dates, phone numbers, email addresses), achieving only 67% normalization accuracy in English. And for speaker standardization, using human voice artists at scale is cost-prohibitive.
Method
SpeechWeave consists of four components arranged in an end-to-end pipeline: a keyphrase sampler, an entity sampler with at-source normalization, a text script generator, and an audio generation module with cross-lingual voice cloning.
The keyphrase sampler addresses prompt repetitiveness via multi-step prompting. Rather than asking an LLM to generate domain sentences directly, the pipeline first generates business subdomains, then randomly selects one, prompts for a creative paragraph about it, and extracts keyphrases from that paragraph. Keyphrases are stored with fuzzy deduplication (token sort ratio + Levenshtein distance) to prevent near-duplicates from accumulating. The final text-generation prompt then includes specific keyphrases alongside semiotic class entities, forcing the LLM to produce lexically varied sentences.
The entity sampler handles text normalization at the point of generation rather than as a post-processing step. It encodes generation recipes for nine semiotic class types (addresses, phone numbers, email addresses, URLs, dates, times, percentages, person names with salutations, and percentages) and generates both the surface form and its normalized equivalent simultaneously. Because normalization rules are embedded in the generator, accuracy is deterministic for supported entity types; a lightweight post-processor handles residual LLM-introduced entities (acronym expansion, numeral conversion, hyphen/bracket removal).
The audio generation module takes normalized text and generates base speech audio using MeloTTS (Zhao et al., 2023), then applies OpenVoice V2’s tone color converter (Qin et al., 2024) with reference recordings from proprietary voice artists to standardize speaker identity. The tone color converter is language-agnostic, allowing English reference voices to be used for Spanish audio, which enables consistent speaker standardization across languages without requiring separate voice artist recordings per language. The pipeline is evaluated on a generated dataset of 3,000 samples spanning 16 business domains, 5 sentence types, 9 semiotic classes, and 2 reference speakers (male and female) in English and Spanish.
Key Results
For text diversity, SpeechWeave achieves grouped mean pairwise similarity scores 45.8% lower than direct LLM prompting in English and 44.4% lower in Spanish (Table 3). Type-Token Ratio (TTR) and Moving Average TTR (MATTR) also exceed both direct prompting and the LibriSpeech public datasets. Diphone coverage exceeds the baseline by 17.4% in English and 9.7% in Spanish, though LibriSpeech’s longer sentences give it an edge of 5.7% (English) and 15.2% (Spanish) over SpeechWeave.
For normalization accuracy, at-source generation achieves 0.97 (English) and 0.94 (Spanish), compared to NeMo’s 0.67 and 0.54 on the same 500-sentence evaluation set per language (Table 4).
For generated audio quality, the pipeline achieves NISQA-estimated MOS of 4.95 (English) and 4.87 (Spanish), SNR of 59.82 dB and 53.01 dB, and WER of 9.32% and 15.21% against the source normalized text (Table 5). Note that MOS here is predicted by the NISQA automated model, not human raters; these values should not be directly compared to human MOS results.
For downstream TTS training, fine-tuning StyleTTS 2 from a LibriTTS checkpoint on SpeechWeave-generated data reduces WER by 40% relative in English (15.37% to 9.36%) and 27% in Spanish (85.05% to 48.44%) on LibriSpeech test sets (Table 6). The Spanish baseline WER is high because StyleTTS 2 has no Spanish-trained checkpoint, so the SpeechWeave data is performing language adaptation, not just domain adaptation.
Novelty Assessment
The contribution is engineering integration. Each core component relies on existing, well-understood technology: multi-step keyphrase prompting adapts techniques from prompt diversity literature (TinyStories, step-recall prompting); at-source normalization is a deterministic rule-based system rather than a learned model; MeloTTS provides base audio generation; and OpenVoice V2 provides the voice cloning component. None of these individually is novel.
What SpeechWeave does well is assemble these parts into a coherent data generation pipeline and evaluate it end-to-end, including a downstream TTS fine-tuning experiment that demonstrates practical utility. The at-source normalization finding is the strongest individual result: the accuracy gap over NeMo (97% vs. 67% in English) is substantial and the mechanism is principled. The keyphrase diversity gains are real but less surprising given the prompting diversity literature that motivates them. The downstream WER reductions are encouraging but the evaluation is limited to two languages and one downstream model.
Field Significance
Moderate — SpeechWeave provides a practical solution to a recurring pain point in commercial TTS development: generating domain-specific, properly normalized training data at scale without manual recording. Its at-source normalization approach demonstrates a meaningful accuracy advantage over post-hoc normalizers for training data generation, which is a result applicable to any text-conditional synthesis pipeline. The pipeline’s design is clear and reproducible in principle, though the dependence on Mistral-7b-Instruct and OpenVoice V2 means results will vary with model choice.
Claims
-
supports: Structured keyphrase infusion via multi-step LLM prompting substantially reduces semantic repetition in generated TTS training corpora compared to single-prompt domain generation.
Evidence: SpeechWeave’s keyphrase pipeline achieves 45.8%/44.4% lower grouped mean pairwise similarity in English/Spanish and 17.4%/9.7% higher diphone coverage relative to direct domain prompting with Mistral-7b-Instruct, even at high temperature (Table 3). (§3.1, §4.1)
-
supports: Generating semiotic class entities with embedded normalization rules at source achieves higher normalization accuracy for TTS training text than applying a post-hoc normalizer to LLM-generated output.
Evidence: At-source entity normalization reaches 0.97 accuracy in English and 0.94 in Spanish, compared to 0.67 and 0.54 for NeMo’s WFST text normalizer on the same 500-sentence evaluation sets per language (Table 4). (§3.2, §4.2.1)
-
supports: Fine-tuning a pre-trained TTS model on synthetic pipeline-generated data with controlled diversity and normalization can meaningfully improve downstream speech intelligibility.
Evidence: StyleTTS 2 fine-tuned on SpeechWeave data shows 40% relative WER reduction in English (15.37% to 9.36%) and 27% in Spanish (85.05% to 48.44%) on LibriSpeech test sets; the Spanish result also demonstrates cross-lingual adaptation from a monolingual English checkpoint (Table 6). (§4.2.3)
-
complicates: Automated speech quality metrics from predictive models should not be treated as equivalent to human MOS scores when evaluating synthetic TTS training data.
Evidence: SpeechWeave reports MOS of 4.95 (English) and 4.87 (Spanish) estimated by the NISQA neural quality predictor rather than human raters; the paper notes this explicitly. Automated MOS predictors are known to diverge from human judgments, particularly for TTS systems that differ from the predictor’s training distribution. (§4.2.2, Table 5)
Limitations and Open Questions
Warning
The evaluation covers only English and Spanish, and improvements in normalization accuracy and diversity may not hold for morphologically rich or lower-resource languages, which the paper acknowledges as future work.
The entity sampler supports nine semiotic classes; any entity type not included in the recipes will not be normalized at source, and the post-processor may handle them incorrectly. The extent of this gap depends on how often such entities appear in target domains.
The pipeline depends on Mistral-7b-Instruct for text generation and OpenVoice V2 for voice cloning, and performance is expected to vary with model substitution. No ablation is provided over LLM or TTS model choice, so the contribution of each component to the overall result is not fully isolated.
The downstream evaluation uses WER from an ASR model to assess TTS quality, which measures intelligibility rather than naturalness or speaker similarity. Prosody, expressiveness, and voice consistency are not evaluated beyond the automated MOS and SNR scores.
Wiki Connections
- Multilingual TTS — SpeechWeave targets multilingual TTS data generation, evaluating its pipeline on English and Spanish with cross-lingual speaker standardization via voice cloning.
- Evaluation Metrics — the paper applies TTR, MATTR, diphone coverage, pairwise semantic similarity, normalization accuracy, NISQA-estimated MOS, SNR, and WER to evaluate training data quality across text and audio dimensions.
- Speaker Adaptation — speaker standardization via cross-lingual voice cloning (OpenVoice V2) is a core component, enabling consistent speaker identity across languages in generated training data.
- Voice Conversion — OpenVoice V2’s tone color converter is used as a cross-lingual VC component to standardize speaker characteristics, applied after base TTS audio generation.