concept: multilingual-tts last_updated: '2026-07-29' paper_count: 75 papers: - id: '2206.04658' published_date: "2022-06-09" entry_date: '2026-07-29' year: 2022 venue: arXiv task: - TTS architecture: - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - gan_based_multilingual_synthesis claims: - claim_id: periodic_activation_functions_with_learnable_frequency_parameters_improve_gan_vocoder role: supports claim: Periodic activation functions with learnable frequency parameters improve GAN vocoder robustness to out-of-distribution audio, including unseen recording environments, languages, and non-speech signals. source: §3.2, §4.5, §4.6 evidence: Periodic activation functions with learnable frequency parameters improve GAN vocoder robustness to out-of-distribution audio, including unseen recording environments, languages, and non-speech signals. confidence: high relevance: medium - claim_id: anti_aliasing_via_low_pass_filtering_around_nonlinear_activations_reduces role: supports claim: Anti-aliasing via low-pass filtering around nonlinear activations reduces high-frequency artifacts in neural vocoders and is complementary to periodic inductive bias. source: §3.3, §4.7, Table 4 evidence: Anti-aliasing via low-pass filtering around nonlinear activations reduces high-frequency artifacts in neural vocoders and is complementary to periodic inductive bias. confidence: high relevance: medium - claim_id: gan_vocoders_benefit_significantly_from_scaling_model_capacity_when_paired role: supports claim: GAN vocoders benefit significantly from scaling model capacity when paired with appropriate architectural improvements and training stabilisation techniques, whereas naive scaling without those changes leads to early collapse. source: §3.4, §4.7 evidence: GAN vocoders benefit significantly from scaling model capacity when paired with appropriate architectural improvements and training stabilisation techniques, whereas naive scaling without those changes leads to early collapse. confidence: high relevance: medium - claim_id: similarity_mos_smos_which_requires_direct_comparison_with_ground_truth role: supports claim: Similarity MOS (SMOS), which requires direct comparison with ground truth, is a more sensitive evaluation metric than standard MOS for distinguishing speaker fidelity under diverse and out-of-distribution conditions. source: §4.3 evidence: Similarity MOS (SMOS), which requires direct comparison with ground truth, is a more sensitive evaluation metric than standard MOS for distinguishing speaker fidelity under diverse and out-of-distribution conditions. confidence: high relevance: low - claim_id: training_on_diverse_multi_condition_speech_data_including_noisy_recordings role: supports claim: Training on diverse, multi-condition speech data (including noisy recordings) is a necessary complement to architectural improvements for achieving universal vocoding performance. source: §4.7, Table 5 evidence: Training on diverse, multi-condition speech data (including noisy recordings) is a necessary complement to architectural improvements for achieving universal vocoding performance. confidence: high relevance: medium limitations: - BigVGAN is trained and evaluated on 24 kHz audio only. Generalisation to 44.1 kHz or 48 kHz audio (common in music and broadcast) is not tested and the model architecture is not designed for it. - The 112M model synthesises at 44.72x real-time, which is slower than HiFi-GAN (93.75x) and BigVGAN-base (70.18x). For latency-sensitive applications, the architectural overhead of filtered Snake activations is a concrete trade-off. The paper acknowledges that spectral normalisation, anti-aliased upsampling layers, and larger discriminators all degraded quality; the practical lessons are reported informally but without a principled explanation of why regularisation harms audio GANs when it helps image GANs. MOS evaluations are conducted with native English-speaking Mechanical Turk workers even for non-English and OOD conditions, which may introduce evaluator bias. The model is trained exclusively on speech despite being evaluated on music and instruments, and the extrapolation limits of this approach for broader audio generation are not characterised. caveats: [] - id: '2212.04356' published_date: "2022-12-06" entry_date: '2026-07-29' year: 2022 venue: arXiv task: - evaluation architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: weakly_supervised_asr_training_at_sufficient_scale_produces_models_that role: supports claim: Weakly supervised ASR training at sufficient scale produces models that generalise robustly across recording conditions, speakers, and languages without dataset-specific fine-tuning. source: §3.3, Table 2 evidence: Weakly supervised ASR training at sufficient scale produces models that generalise robustly across recording conditions, speakers, and languages without dataset-specific fine-tuning. confidence: high relevance: medium - claim_id: models_evaluated_solely_on_in_distribution_benchmarks_can_appear_superhuman role: supports claim: Models evaluated solely on in-distribution benchmarks can appear superhuman while remaining significantly below human performance out-of-distribution, indicating that benchmark-focused evaluation overstates robustness. source: §3.3, Figure 2 evidence: Models evaluated solely on in-distribution benchmarks can appear superhuman while remaining significantly below human performance out-of-distribution, indicating that benchmark-focused evaluation overstates robustness. confidence: high relevance: low - claim_id: multitask_and_multilingual_joint_training_introduces_negative_transfer_in_small role: supports claim: Multitask and multilingual joint training introduces negative transfer in small models but confers benefits at large scale. source: §4.3, Figure 9 evidence: Multitask and multilingual joint training introduces negative transfer in small models but confers benefits at large scale. confidence: high relevance: high - claim_id: per_language_zero_shot_asr_performance_scales_predictably_with_per role: supports claim: Per-language zero-shot ASR performance scales predictably with per-language training data, with WER roughly halving for every 16-fold increase in supervision hours. source: §3.4, Figure 3 evidence: Per-language zero-shot ASR performance scales predictably with per-language training data, with WER roughly halving for every 16-fold increase in supervision hours. confidence: high relevance: medium limitations: - Long-form transcription remains unreliable without explicit decoding heuristics (beam search, temperature fallback, voice activity detection, initial timestamp constraints) to prevent repetition loops, truncation of edge segments, and hallucination. These are workarounds rather than principled solutions (§4.5). Performance on languages with non-Indo-European scripts (Hebrew, Telugu, Chinese, Korean) is worse than the training-data trend predicts, likely reflecting tokenisation mismatch and linguistic distance. Language identification accuracy on Fleurs is penalised by having no training data for 20 of 102 Fleurs languages, upper-bounding accuracy at 80.4% regardless of model capacity (§3.6). - The text normalisation procedure carries a risk of being overfitted to Whisper's transcription style, which could bias WER comparisons in Whisper's favour (§4.4). Fine-tuning properties are not studied in this paper, leaving open how the model's robustness advantage transfers to supervised fine-tuning settings where prior work performs. caveats: [] - id: '2303.03926' published_date: "2023-03-07" entry_date: '2026-07-29' year: 2023 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models claims: - claim_id: large_scale_multilingual_codec_language_models_can_transfer_speaker_identity role: supports claim: Large-scale multilingual codec language models can transfer speaker identity, emotion, and acoustic environment across languages from a single source utterance without paired bilingual data. source: §3, §5.3, Table 3 evidence: Large-scale multilingual codec language models can transfer speaker identity, emotion, and acoustic environment across languages from a single source utterance without paired bilingual data. confidence: high relevance: high - claim_id: language_id_conditioning_is_essential_for_native_sounding_accent_in role: supports claim: 'Language ID conditioning is essential for native-sounding accent in cross-lingual codec TTS: removing it significantly degrades accent scores even while modestly improving speaker similarity.' source: §5.5, Table 6 evidence: 'Language ID conditioning is essential for native-sounding accent in cross-lingual codec TTS: removing it significantly degrades accent scores even while modestly improving speaker similarity.' confidence: high relevance: high - claim_id: in_context_learning_with_acoustic_token_prompts_provides_stronger_cross role: supports claim: In-context learning with acoustic token prompts provides stronger cross-lingual voice preservation than speaker embedding approaches across both TTS and speech-to-speech translation tasks. source: §5.3, §5.4, Tables 3, 5 evidence: In-context learning with acoustic token prompts provides stronger cross-lingual voice preservation than speaker embedding approaches across both TTS and speech-to-speech translation tasks. confidence: high relevance: high - claim_id: the_ar_nar_two_stage_codec_language_model_architecture_extends role: supports claim: The AR/NAR two-stage codec language model architecture extends naturally to cross-lingual generation by treating bilingual phoneme sequences as concatenated prompts. source: §3.2, §3.4 evidence: The AR/NAR two-stage codec language model architecture extends naturally to cross-lingual generation by treating bilingual phoneme sequences as concatenated prompts. confidence: high relevance: high limitations: - Evaluations cover only English and Chinese, and test sets are small (40 speakers, 1373 samples for TTS; 14 speakers, 350 utterances for S2ST on EMIME). Generalisation to typologically distant language pairs, more than two languages, or lower-resource settings is untested. - The EMIME dataset is bilingual by construction (same speakers recorded in both languages), making the ASV upper bound (tgt vs. src) unusually informative but also artificially favourable for a cross-lingual system. The ASV gap between the model and the upper bound for English-to-Chinese (0.48 vs. 0.58) suggests substantial room for improvement in voice transferability in that direction. - 'The language ID ablation reveals an inherent tension: stronger accent control comes at the cost of voice similarity. The right operating point depends on application, and the paper does not explore soft or learnable trade-off mechanisms. No model size figures are reported, making cost comparison difficult. S2ST quality is bottlenecked by translation quality; the paper does not disentangle acoustic and linguistic error sources beyond the oracle-text upper bound analysis.' caveats: [] - id: '2306.12925' published_date: "2023-06-22" entry_date: '2026-07-29' year: 2023 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models claims: - claim_id: initializing_a_speech_text_llm_from_a_pretrained_text_only role: supports claim: Initializing a speech-text LLM from a pretrained text-only checkpoint substantially outperforms training from scratch at equivalent model scale. source: §5.4.2, Table 6 evidence: Initializing a speech-text LLM from a pretrained text-only checkpoint substantially outperforms training from scratch at equivalent model scale. confidence: high relevance: medium - claim_id: audio_tokenizer_quality_is_a_primary_bottleneck_in_llm_based role: supports claim: 'Audio tokenizer quality is a primary bottleneck in LLM-based speech generation and understanding: stronger semantic tokenizers yield large downstream gains independent of LM scale.' source: §5.4.3, Table 7 evidence: 'Audio tokenizer quality is a primary bottleneck in LLM-based speech generation and understanding: stronger semantic tokenizers yield large downstream gains independent of LM scale.' confidence: high relevance: medium - claim_id: a_unified_multimodal_vocabulary_that_interleaves_text_and_audio_tokens role: supports claim: A unified multimodal vocabulary that interleaves text and audio tokens enables zero-shot speech translation to language pairs not seen during speech training, by inheriting translation capability from text pretraining. source: §5.2, Table 3 evidence: A unified multimodal vocabulary that interleaves text and audio tokens enables zero-shot speech translation to language pairs not seen during speech training, by inheriting translation capability from text pretraining. confidence: high relevance: medium - claim_id: training_on_combined_tasks_that_decompose_complex_speech_operations_into role: supports claim: Training on combined tasks that decompose complex speech operations into intermediate text steps improves performance over direct end-to-end decoding. source: §5.4.4, Table 8 evidence: Training on combined tasks that decompose complex speech operations into intermediate text steps improves performance over direct end-to-end decoding. confidence: high relevance: medium - claim_id: voice_identity_preservation_in_cross_lingual_speech_synthesis_can_exceed role: supports claim: Voice identity preservation in cross-lingual speech synthesis can exceed high-quality TTS-based references when an audio LM is conditioned on a short spoken prompt. source: §5.3, Table 4 evidence: Voice identity preservation in cross-lingual speech synthesis can exceed high-quality TTS-based references when an audio LM is conditioned on a short spoken prompt. confidence: high relevance: high limitations: - The entire system depends on the quality of the audio tokenizer, which is not released and requires access to Google-internal USM models. The best-performing configuration (AudioPaLM-2 with USM-v2 tokens) is not reproducible externally; the published ablations use the multilingual w2v-BERT tokenizer as the weakest condition, suggesting that reported performance at USM-v2 quality cannot be independently verified. - Adding S2ST tasks modestly degrades ASR and AST performance, suggesting that sharing model capacity across output modalities introduces trade-offs that are not fully resolved by the training mixture design (§5.4.5, Table 9). The paper evaluates primarily on speech translation tasks; generative speech quality at naturalness is only assessed in the S2ST with voice transfer setting, not for open-ended TTS. The subjective evaluations were conducted on an earlier version of AudioPaLM using AudioLM decoding rather than SoundStorm, meaning the best-performing decoder was not evaluated subjectively. Evaluation benchmarks for generative audio tasks more generally remain underdeveloped relative to text, a limitation the paper explicitly notes. caveats: [] - id: '2312.01479' published_date: "2023-12-03" entry_date: '2026-07-29' year: 2023 venue: arXiv task: - TTS architecture: - GAN - VAE relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - gan_based_multilingual_synthesis - vae_multilingual_latent_models claims: - claim_id: decoupling_tone_color_transfer_from_style_and_language_generation_enables role: supports claim: Decoupling tone color transfer from style and language generation enables flexible post-cloning control over emotion, accent, and rhythm without requiring paired multi-attribute training data. source: §2.1, §2.2 evidence: Decoupling tone color transfer from style and language generation enables flexible post-cloning control over emotion, accent, and rhythm without requiring paired multi-attribute training data. confidence: high relevance: low - claim_id: using_ipa_as_a_universal_phoneme_representation_in_a_normalising role: supports claim: Using IPA as a universal phoneme representation in a normalising flow allows voice cloning to generalise to unseen languages without retraining on massive multilingual speaker data. source: §2.3, §3 "Importance of IPA" evidence: Using IPA as a universal phoneme representation in a normalising flow allows voice cloning to generalise to unseen languages without retraining on massive multilingual speaker data. confidence: high relevance: high - claim_id: feed_forward_voice_cloning_pipelines_encoder_flow_decoder_without_autoregressive role: supports claim: Feed-forward voice cloning pipelines (encoder-flow-decoder without autoregressive or diffusion components) achieve real-time or faster inference, making large-scale commercial deployment feasible. source: §3 "Fast Inference with Low Cost" evidence: Feed-forward voice cloning pipelines (encoder-flow-decoder without autoregressive or diffusion components) achieve real-time or faster inference, making large-scale commercial deployment feasible. confidence: high relevance: medium - claim_id: cross_study_quantitative_comparison_of_voice_cloning_systems_is_systematically role: supports claim: Cross-study quantitative comparison of voice cloning systems is systematically confounded by differences in test set composition, training data scale, and evaluation protocol. source: §3 evidence: Cross-study quantitative comparison of voice cloning systems is systematically confounded by differences in test set composition, training data scale, and evaluation protocol. confidence: high relevance: low limitations: - The paper reports no quantitative metrics anywhere. All evaluation is qualitative, with audio samples hosted on a demo website. This makes it impossible to independently verify the claimed advantages over VALL-E, XTTS, YourTTS, or Voicebox on any standard benchmark. - The base TTS model is trained on only four speakers across English, Chinese, and Japanese. Adding support for additional languages requires collecting at least one native single-speaker dataset, which while much cheaper than a full MSML system, still represents a non-trivial data acquisition effort. Style preservation through the tone color converter is imperfect for highly expressive inputs, with emotion occasionally being neutralised. The proprietary training data for both the base model and the MSML converter set cannot be replicated by the research community despite the open-source code and weights release. caveats: [] - id: '2406.04904' published_date: "2024-06-07" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - gan_based_multilingual_synthesis claims: - claim_id: multilingual_zero_shot_tts_training_degrades_speaker_similarity_compared_to role: complicates claim: Multilingual zero-shot TTS training degrades speaker similarity compared to monolingual training on the same data, reflecting a fundamental trade-off in cross-lingual speaker conditioning. source: §4.1, Table 2, Table 3 evidence: Multilingual zero-shot TTS training degrades speaker similarity compared to monolingual training on the same data, reflecting a fundamental trade-off in cross-lingual speaker conditioning. confidence: high relevance: high - claim_id: a_perceiver_resampler_based_speaker_conditioning_encoder_producing_multiple_fixed role: supports claim: A Perceiver Resampler-based speaker conditioning encoder, producing multiple fixed-length embeddings from variable-length reference audio, improves voice cloning robustness in massively multilingual autoregressive TTS over single-embedding approaches. source: §2 evidence: A Perceiver Resampler-based speaker conditioning encoder, producing multiple fixed-length embeddings from variable-length reference audio, improves voice cloning robustness in massively multilingual autoregressive TTS over single-embedding approaches. confidence: high relevance: high - claim_id: evaluating_multilingual_tts_models_against_monolingual_baselines_on_the_same role: supports claim: Evaluating multilingual TTS models against monolingual baselines on the same language produces misleading comparisons, because the multilingual model's per-language training data is substantially reduced. source: §3.2, §4.1 evidence: Evaluating multilingual TTS models against monolingual baselines on the same language produces misleading comparisons, because the multilingual model's per-language training data is substantially reduced. confidence: high relevance: high - claim_id: a_small_amount_of_target_speaker_fine_tuning_data_approximately role: supports claim: A small amount of target-speaker fine-tuning data (approximately 10 minutes) substantially improves speaker similarity in cross-lingual zero-shot synthesis, including extreme prosody styles such as whispering. source: §5 evidence: A small amount of target-speaker fine-tuning data (approximately 10 minutes) substantially improves speaker similarity in cross-lingual zero-shot synthesis, including extreme prosody styles such as whispering. confidence: high relevance: high - claim_id: low_frequency_codec_codebook_entries_can_be_pruned_without_quality role: supports claim: Low-frequency codec codebook entries can be pruned without quality loss and improve expressiveness in multilingual discrete-token TTS. source: §2 evidence: Low-frequency codec codebook entries can be pruned without quality loss and improve expressiveness in multilingual discrete-token TTS. confidence: high relevance: high limitations: - Speaker similarity lags behind monolingual specialists in English, and the multilingual evaluation uses cross-lingual prompting (English speaker references for non-English languages), which may understate true within-language similarity. No human listening test was conducted for multilingual outputs beyond subjective English comparisons. - The paper lacks ablations isolating the contribution of the Perceiver Resampler versus the larger reference representation alone. The VQ-VAE compression (21.53 Hz, 1 codebook) is highly compact compared to EnCodec at 75 Hz with 8 codebooks; the quality ceiling this imposes is not characterised against higher-fidelity codecs. Arabic and CJK language results remain weakest by CER (Table 4), and the causes, whether limited training data, romanisation quality, or tokeniser coverage, are not analysed. The authors acknowledge future intent to disentangle speaker and prosody for cross-speaker prosody transfer, which the current architecture does not support. caveats: [] - id: '2407.05361' published_date: "2024-07-07" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - infrastructure current_role: influential method_family: [] claims: - claim_id: training_speech_generation_models_on_large_scale_in_the_wild role: supports claim: Training speech generation models on large-scale in-the-wild data, after automated quality filtering, can match audiobook-derived training data on formal-domain benchmarks while improving performance on spontaneous speech generation. source: §V.A, Table V evidence: AR+SoundStorm and VoiceBox trained on Emilia-En (46k hours) match MLS-trained counterparts on LibriSpeech-Test WER/S-SIM/FSD, but improve WER, S-SIM, and FSD on Emilia-Test (spontaneous speech) for both architectures. confidence: high relevance: medium - claim_id: increasing_in_the_wild_training_set_size_continues_to_improve role: supports claim: Increasing in-the-wild training-set size continues to improve zero-shot TTS quality at fixed model capacity, with the largest gains at smaller scales and diminishing but still-positive returns as scale grows. source: §V.B, Figure 5 evidence: Scaling training data from 5k to 134k hours steadily reduces AR+SoundStorm WER (5.2% to 4.2% on LibriSpeech-Test; 5.7% to 4.9% on Emilia-Test), with gains concentrated below 46k hours and convergence emerging around 100k hours. confidence: high relevance: medium - claim_id: training_a_single_model_on_pooled_multilingual_in_the_wild role: complicates claim: Training a single model on pooled multilingual in-the-wild data enables crosslingual generation but trades off same-language quality relative to a monolingual model trained on comparable per-language data volume. source: §V.C evidence: The Emilia-Large-trained (216k hours, 6 languages) multilingual AR+SoundStorm model scores WER 4.9%, S-SIM 0.636, and FSD 15.31 on English, all slightly worse than an English-only 134k-hour model (WER 4.5%, S-SIM 0.65, FSD 14.8). confidence: high relevance: high - claim_id: automated_quality_filtering_pipelines_for_in_the_wild_speech_data role: complicates claim: Automated quality-filtering pipelines for in-the-wild speech data necessarily discard a large majority of raw source audio to reach a training-quality bar. source: §III, Table 2 evidence: On a 666.94-hour benchmark sample, Emilia-Pipe's six stages retain only 38.75% of the input (258.44 hours, 96,117 clips) after DNSMOS, language-confidence, and duration-outlier filtering. confidence: high relevance: medium - claim_id: crosslingual_voice_transfer_in_zero_shot_tts_remains_substantially_harder role: complicates claim: Crosslingual voice transfer in zero-shot TTS remains substantially harder than same-language transfer even when the training corpus itself is multilingual and balanced across the target languages. source: §V.C, Table VI evidence: Crosslingual WER and speaker similarity degrade relative to matched-language pairs across all six languages (e.g., VoiceBox S-SIM falls from 0.588 En-En to 0.386 En-Zh; AR+SoundStorm WER rises from 6.3% Ko-Ko to 8.3% En-Ko). confidence: high relevance: high limitations: - Subjective evaluation in the audiobook-vs-in-the-wild comparison (§V.A) uses only sixteen samples (eight per evaluation set) and twelve English-speaking evaluators, limiting the statistical reliability of the reported CMOS and SMOS differences. No confidence intervals or significance tests are reported for any subjective metric. - The authors themselves note three further limitations. The speaker diarisation model is imperfect and can leave a small proportion of segments containing overlapping speakers, which may degrade downstream generation quality. Emilia-Pipe constrains segments to 3-30 seconds, and generating speech outside that range is observed to produce unexpected outcomes, meaning pipeline hyperparameters may need adjustment for other use cases. Emilia-Large covers only six languages due to resource constraints, and the authors explicitly invite community contributions to extend language coverage. Additionally, all multilingual and crosslingual evaluation (§V.C) relies entirely on objective metrics (WER, S-SIM, FSD) without subjective naturalness assessment, and Emilia-Test is drawn from the same distribution as Emilia's training data, which may favour Emilia-trained models' apparent generalisation relative to MLS-trained ones. caveats: [] - id: '2407.05407' published_date: "2024-07-07" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: inserting_a_vector_quantizer_into_a_supervised_asr_encoder_yields role: supports claim: Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. source: §2.1, §5.1, Table 7 evidence: Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. confidence: high relevance: medium - claim_id: in_autoregressive_codec_lm_tts_both_the_text_tokenizer_and role: supports claim: In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. source: §5.2, Table 7 evidence: In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. confidence: high relevance: low - claim_id: asr_re_ranking_is_an_effective_post_hoc_method_for role: complicates claim: ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. source: §5.3, Tables 8, 9 evidence: ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. confidence: high relevance: medium - claim_id: instruction_fine_tuning_on_a_modest_amount_of_labelled_data role: supports claim: Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. source: §2.4, §5.4, Table 10 evidence: Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. confidence: high relevance: medium - claim_id: high_quality_tts_synthesized_speech_can_serve_as_effective_training role: supports claim: High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. source: §5.5, Table 11 evidence: High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. confidence: high relevance: medium limitations: - '- Only a single VQ codebook (4096 codes) is used; multi-level RVQ and its effect on quality vs. compression is left for future work. - The choice of VQ insertion layer (after layer 6 of 12) is not ablated — optimal placement is unresolved. - Cross-lingual cloning omits prompt prosody to prevent leakage, which may reduce naturalness in target language. - Instruction fine-tuning data amounts (556h) are modest; broader paralinguistic coverage remains open. - No subjective (MOS) evaluation in the main paper; relies entirely on objective WER/CER/SS metrics.' caveats: [] - id: '2409.00750' published_date: "2024-09-01" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS - VC architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models claims: - claim_id: non_autoregressive_masked_generative_transformers_can_achieve_human_level_speaker role: supports claim: Non-autoregressive masked generative transformers can achieve human-level speaker similarity in zero-shot TTS without requiring explicit text-speech alignment or phone-level duration supervision. source: §4.2.1, Table 2 evidence: Non-autoregressive masked generative transformers can achieve human-level speaker similarity in zero-shot TTS without requiring explicit text-speech alignment or phone-level duration supervision. confidence: high relevance: low - claim_id: replacing_k_means_quantisation_of_ssl_features_with_vq_vae role: supports claim: Replacing k-means quantisation of SSL features with VQ-VAE vector quantisation reduces information loss in tonal languages and improves downstream acoustic token prediction. source: §3.2.1 evidence: Replacing k-means quantisation of SSL features with VQ-VAE vector quantisation reduces information loss in tonal languages and improves downstream acoustic token prediction. confidence: high relevance: medium - claim_id: masked_generative_tts_substantially_outperforms_autoregressive_tts_on_hard_text role: supports claim: Masked generative TTS substantially outperforms autoregressive TTS on hard-text robustness (tongue twisters, repeating phrases) while maintaining competitive naturalness on standard benchmarks. source: §4.2.2, Appendix J, Table 13 evidence: Masked generative TTS substantially outperforms autoregressive TTS on hard-text robustness (tongue twisters, repeating phrases) while maintaining competitive naturalness on standard benchmarks. confidence: high relevance: medium - claim_id: parallel_iterative_decoding_in_masked_generative_models_yields_constant_inference role: supports claim: Parallel iterative decoding in masked generative models yields constant inference cost regardless of output length, in contrast to autoregressive decoding whose cost scales linearly with utterance duration. source: §4.2.2 evidence: Parallel iterative decoding in masked generative models yields constant inference cost regardless of output length, in contrast to autoregressive decoding whose cost scales linearly with utterance duration. confidence: high relevance: medium - claim_id: zero_shot_style_cloning_via_in_context_learning_extends_to role: supports claim: Zero-shot style cloning via in-context learning extends to accent and emotion transfer without task-specific architectural changes. source: §4.3, Tables 4–5 evidence: Zero-shot style cloning via in-context learning extends to accent and emotion transfer without task-specific architectural changes. confidence: high relevance: low limitations: - Speech content editing is acknowledged as "not very robust" by the authors, who attribute this to a training objective mismatch (mask-and-predict vs. fill-in-mask). The editing capability is demonstrated qualitatively only, with no quantitative evaluation reported. - 'Training uses 100K hours of English and Chinese speech from Emilia, with multilingual extension at far smaller data budgets (2,500–8,200 hours per language). Multilingual performance is uneven: French and German show higher WER in cross-lingual dubbing, and the authors note limitations from insufficient retraining of all components on expanded data.' - Duration control requires either a ground-truth length or the flow-matching duration predictor; errors in predicted duration propagate to WER. The gap between predicted-length and ground-truth-length WER is measurable (e.g., 2.634 vs. 2.012 on LibriSpeech test-clean). - Inference steps of 25-50 for T2S plus the S2A step schedule add latency compared to single-pass systems, though the paper does not report real-time factor or wall-clock comparisons. - Emotion control requires post-training fine-tuning on labelled data; it is not available zero-shot from the base model alone. caveats: [] - id: '2411.01156' published_date: "2024-11-02" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - gan_based_multilingual_synthesis claims: - claim_id: eliminating_grapheme_to_phoneme_conversion_by_directly_feeding_raw_text role: supports claim: Eliminating grapheme-to-phoneme conversion by directly feeding raw text to an LLM backbone is viable for multilingual TTS and can improve handling of context-dependent polyphonic words. source: §1, §3 evidence: Eliminating grapheme-to-phoneme conversion by directly feeding raw text to an LLM backbone is viable for multilingual TTS and can improve handling of context-dependent polyphonic words. confidence: high relevance: high - claim_id: hierarchical_decomposition_of_autoregressive_token_generation_into_semantic_level_and role: supports claim: Hierarchical decomposition of autoregressive token generation into semantic-level and acoustic-level stages improves codebook stability in grouped scalar quantization. source: §3.1, §3.1.1 evidence: Hierarchical decomposition of autoregressive token generation into semantic-level and acoustic-level stages improves codebook stability in grouped scalar quantization. confidence: high relevance: medium - claim_id: grouped_finite_scalar_vector_quantization_achieves_higher_codebook_utilisation_than role: supports claim: Grouped Finite Scalar Vector Quantization achieves higher codebook utilisation than residual vector quantization alternatives, mitigating dead-code collapse. source: §3.2.2, §3.2.3 evidence: Grouped Finite Scalar Vector Quantization achieves higher codebook utilisation than residual vector quantization alternatives, mitigating dead-code collapse. confidence: high relevance: medium - claim_id: real_time_tts_inference_with_low_first_packet_latency_is role: supports claim: Real-time TTS inference with low first-packet latency is achievable on consumer GPU hardware through standard inference optimisations without architectural compromise. source: §4.2 evidence: Real-time TTS inference with low first-packet latency is achievable on consumer GPU hardware through standard inference optimisations without architectural compromise. confidence: high relevance: low limitations: - The entire experimental evaluation is conducted on a proprietary test set with undisclosed composition and size. No public benchmark is used, making it impossible to independently verify the claimed superiority over CosyVoice and F5-TTS or to compare against the broader literature. - The MOS evaluation uses "naive listeners" rather than trained raters or crowdsourced panels following standard listening test protocols (e.g. ITU-T P.800), which may inflate scores relative to conventional evaluations. The paper does not report model size, training compute, or inference memory requirements in full, limiting reproducibility. DPO training details are omitted from the main training description. The paper does not evaluate cross-lingual transfer or accent preservation, which are claimed motivations for the non-G2P design. It is also unclear how the system handles low-resource languages beyond the eight listed in the training data. caveats: [] - id: '2412.02612' published_date: "2024-12-03" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: speech_text_interleaved_pre_training_at_trillion_token_scale_enables role: supports claim: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. source: §4.1, Table 4 evidence: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. confidence: high relevance: medium - claim_id: single_codebook_supervised_speech_tokenizers_derived_from_asr_models_can role: supports claim: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. source: §3.1, Table 1 evidence: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. confidence: high relevance: medium - claim_id: streaming_interleaved_generation_templates_alternating_text_and_speech_token_output role: supports claim: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. source: §3.3 evidence: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. confidence: high relevance: low - claim_id: end_to_end_speech_language_models_that_include_dedicated_speech role: supports claim: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. source: §5.2, Table 6 evidence: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. confidence: high relevance: medium - claim_id: decoupling_the_text_and_speech_output_subtasks_during_fine_tuning role: supports claim: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. source: §4.2.2 evidence: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. confidence: high relevance: medium limitations: - The paper reports no subjective listening test (MOS/MUSHRA) on the chat model output; UTMOS is used as a proxy for speech naturalness, and the chat evaluation relies on GPT-4o scoring of ASR transcriptions, introducing cascaded error from both the vocoder quality and the Whisper transcription step. - 'The 175bps tokenizer trades acoustic fidelity for compactness: VisQOL at 12.5Hz (2.52) is lower than SpeechTokenizer variants and the 50Hz variant of the same system. This may limit voice cloning quality and the fidelity of paralinguistic feature reproduction (accent, fine-grained emotion), though the paper does not directly evaluate these.' - Instruction-following for speech style (emotion, dialect, rate) is described and demonstrated qualitatively but not evaluated quantitatively; it is unclear how reliably the model follows complex or combined style instructions. - 'The streaming thoughts ratio (13 text : 26 speech tokens) and block size (b=0.8s) are empirically chosen hyperparameters; their sensitivity and generalisability to other tokenizer frame rates or LLM sizes is not studied.' caveats: [] - id: '2501.06282' published_date: "2025-01-10" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: large_scale_multi_task_training_across_heterogeneous_speech_tasks_substantially role: supports claim: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. source: §3.4, §4.1–4.5 evidence: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. confidence: high relevance: medium - claim_id: aligned_multimodal_architectures_can_achieve_instruction_controlled_speech_style_emotion role: complicates claim: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. source: §3.2, §4.4, Table 18 evidence: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. confidence: high relevance: high - claim_id: full_duplex_spoken_dialogue_at_sub_second_latency_is_achievable role: supports claim: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. source: §3.1, §4.5, Table 21 evidence: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. confidence: high relevance: low - claim_id: an_autoregressive_streaming_voice_decoder_that_interleaves_text_hidden_states role: supports claim: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. source: §3.2, §4.4, Table 17 evidence: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. confidence: high relevance: low limitations: - The instruction-following voice generation evaluation uses a single in-house Chinese test set of 122 turns, making external validation of the 98.4% accuracy figure impossible. Code and model weights had not been released at time of publication. - LoRA-only updates to the text LLM during training limit the model's ability to follow diverse instructions; more comprehensive LLM updates with higher-quality text data remain unexplored. Long-tail pronunciation errors persist in end-to-end audio generation, partly due to special symbols that the decoder cannot reliably convert to speech. The full-duplex module still relies on external acoustic echo cancellation (AEC) and voice activity detection (VAD) modules, meaning a truly end-to-end duplex system has not been achieved. Performance on speech emotion recognition in low-resource languages shows mixed results (e.g., Polish at 55.9% F1), indicating that cross-lingual generalization is uneven despite the broad multilingual training. caveats: [] - id: 2025.coling-main.685 published_date: "2025-01-19" entry_date: '2026-07-29' year: 2025 venue: COLING task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: large_scale_naturally_spoken_multilingual_corpora_enable_cross_lingual_zero role: supports claim: Large-scale, naturally-spoken multilingual corpora enable cross-lingual zero-shot TTS capabilities in models that lack them when trained on single-language or audiobook-derived data. source: §4, Table 2 evidence: XTTS_v2 trained on the VoxpopuliTTS large subset (10,000h per language) achieves cross-lingual synthesis in EN, FR, and ES with MOS scores of 3.37, 3.52, and 3.31, respectively, demonstrating the dataset's utility for zero-shot multilingual TTS. confidence: high relevance: high - claim_id: naturalness_and_conversational_style_in_training_data_trades_off_against role: complicates claim: Naturalness and conversational style in training data trades off against intelligibility metrics during TTS model training. source: §4 evidence: The authors attribute relatively high WER values (8.25–9.20%) to the naturalness and conversational complexity of VoxpopuliTTS, suggesting that models trained on naturalistic in-the-wild data face greater challenges in transcript fidelity than those trained on clean studio recordings. confidence: high relevance: medium - claim_id: asr_confidence_scores_provide_an_effective_proxy_for_filtering_hallucinated role: supports claim: ASR confidence scores provide an effective proxy for filtering hallucinated transcripts in automatic speech processing pipelines for TTS data curation. source: §2.2 evidence: Per-word logit thresholding (confidence < 0.7 discarded) is used to remove FasterWhisper hallucinations, and sentence-final punctuation is used to identify semantically incomplete segments for merging, both validated as effective in the pipeline design. confidence: high relevance: medium - claim_id: quality_stratified_corpus_subsets_introduce_a_trade_off_between_data role: complicates claim: Quality-stratified corpus subsets introduce a trade-off between data scale and audio quality, requiring practitioners to select the appropriate tier for their task. source: §3, Table 1 evidence: VoxpopuliTTS is split into small (~1,000h, DNSMOS ~3.87), medium (~5,000h, DNSMOS ~3.71), and large (~10,000h, DNSMOS ~3.39) subsets per language, with quality decreasing as scale increases. confidence: high relevance: medium limitations: - The paper does not compare VoxpopuliTTS against Emilia or WenetSpeech4TTS in a controlled training experiment, making it impossible to assess whether the processing pipeline or the dataset composition offers measurable advantages over alternatives. Coverage is limited to three European languages (English, French, Spanish), so the approach's scalability to morphologically complex or lower-resource languages is untested. The evaluation relies on a single model family (XTTS_v2) and does not include ablations that isolate the contribution of the confidence filter or the segment merging step. Human MOS evaluations are reported with confidence intervals but the number of raters, the test items, and the evaluation protocol are not described. caveats: [] - id: '2502.05512' published_date: "2025-02-08" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - gan_based_multilingual_synthesis claims: - claim_id: speaker_conditioning_via_a_multi_reference_conformer_perceiver_improves_zero role: supports claim: Speaker conditioning via a multi-reference Conformer Perceiver improves zero-shot voice cloning stability and timbre consistency over single-vector speaker embeddings. source: §2.3, Table 4 evidence: Speaker conditioning via a multi-reference Conformer Perceiver improves zero-shot voice cloning stability and timbre consistency over single-vector speaker embeddings. confidence: high relevance: low - claim_id: direct_waveform_decoding_from_lm_hidden_states_via_a_gan role: supports claim: Direct waveform decoding from LM hidden states via a GAN vocoder achieves competitive audio quality with faster inference than diffusion-based intermediate representation decoding. source: §2.4, Table 5 evidence: Direct waveform decoding from LM hidden states via a GAN vocoder achieves competitive audio quality with faster inference than diffusion-based intermediate representation decoding. confidence: high relevance: medium - claim_id: fsq_reaches_near_100_codebook_utilisation_with_less_training_data role: supports claim: FSQ reaches near-100% codebook utilisation with less training data than VQ, though VQ converges to similar utilisation with sufficient data scale. source: §3.3.2, Figure 2 evidence: FSQ reaches near-100% codebook utilisation with less training data than VQ, though VQ converges to similar utilisation with sufficient data scale. confidence: high relevance: medium - claim_id: hybrid_character_pinyin_tokenisation_enables_reliable_correction_of_chinese_polyphonic role: supports claim: Hybrid character-pinyin tokenisation enables reliable correction of Chinese polyphonic character mispronunciations at inference time without requiring a separate grapheme-to-phoneme module. source: §3.3.1, Table 2 evidence: Hybrid character-pinyin tokenisation enables reliable correction of Chinese polyphonic character mispronunciations at inference time without requiring a separate grapheme-to-phoneme module. confidence: high relevance: high limitations: - Model size is not reported, and the training data pipeline uses proprietary internet-sourced audio with pseudo-labels from commercial ASR; neither the data nor the code is released, limiting reproducibility. - The system is limited to Chinese and English, with acknowledged weak emotional expression replication. Instruction-based voice generation is explicitly unsupported. The MOS evaluation relies on 100 samples from an unspecified test set distribution, and the SPK-SIM metric uses ERes2Net rather than a standardised model, making direct comparison with published baselines difficult. The paper does not report streaming latency or real-time factor, despite positioning the hybrid architecture as streaming-capable. caveats: [] - id: '2503.14345' published_date: "2025-03-18" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: spontaneous_scripting_from_an_llm_is_roughly_as_important_as role: supports claim: Spontaneous scripting from an LLM is roughly as important as the acoustic modeling choice for perceived spontaneity in long-form dialogue TTS. source: §4.2.2, Table 3 evidence: Spontaneous scripting from an LLM is roughly as important as the acoustic modeling choice for perceived spontaneity in long-form dialogue TTS. confidence: high relevance: low - claim_id: full_sequence_interleaving_of_text_and_speech_codes_extended_to role: supports claim: Full-sequence interleaving of text and speech codes, extended to 40,000-token context windows with speaker-change tokens, enables coherent long-form zero-shot multi-speaker synthesis that turn-level concatenation cannot match. source: §3.2.1, Tables 1–2 evidence: Full-sequence interleaving of text and speech codes, extended to 40,000-token context windows with speaker-change tokens, enables coherent long-form zero-shot multi-speaker synthesis that turn-level concatenation cannot match. confidence: high relevance: low - claim_id: curriculum_learning_progressively_exposing_a_codec_lm_to_increasing_dialogue role: supports claim: Curriculum learning, progressively exposing a codec LM to increasing dialogue complexity, is an effective strategy for developing long-context and spontaneous generation capability without requiring matched long-context data from the outset. source: §3.2.1 evidence: Curriculum learning, progressively exposing a codec LM to increasing dialogue complexity, is an effective strategy for developing long-context and spontaneous generation capability without requiring matched long-context data from the outset. confidence: high relevance: low - claim_id: automatic_speaker_similarity_metrics_cosine_embedding_similarity_can_disagree_with role: supports claim: Automatic speaker similarity metrics (cosine embedding similarity) can disagree with subjective speaker similarity ratings in long-form generation settings, particularly when the acoustic model attends to prosodic rather than purely timbral features. source: §4.2.1 evidence: Automatic speaker similarity metrics (cosine embedding similarity) can disagree with subjective speaker similarity ratings in long-form generation settings, particularly when the acoustic model attends to prosodic rather than purely timbral features. confidence: high relevance: low - claim_id: chunk_wise_autoregressive_decoding_with_a_causal_chunk_mask_provides role: supports claim: Chunk-wise autoregressive decoding with a causal chunk mask provides a practical solution to the continuity and memory constraints of mel-spectrogram reconstruction from long semantic code sequences. source: §3.2.2 evidence: Chunk-wise autoregressive decoding with a causal chunk mask provides a practical solution to the continuity and memory constraints of mel-spectrogram reconstruction from long semantic code sequences. confidence: high relevance: medium limitations: - All evaluation is conducted on a small internal test set (4 knowledge sources for podcast, 7 podcasts for the script ablation), and training data is entirely proprietary (~515K hours). Results cannot be independently reproduced, and generalisability to other domains or languages beyond Chinese and English is unverified. - The notably lower SIM-O score for English (0.53 vs. 0.75 baseline) reveals unresolved speaker consistency challenges in the English long-context setting, which the authors attribute to an imbalanced training curriculum favouring audiobook over conversational English data. Hallucinations in speaker attribution (utterances assigned to the wrong speaker) emerge from the interplay of semantic token timbre leakage, diarization errors in training data, and ambiguous filler-word interpretations; no mitigation is proposed beyond discussion. The two-speaker restriction (host + guest) is a deliberate scope limitation; extension to three or more speakers is left as future work. Evaluation is subjective-only for multi-speaker interactions; no standardised zero-shot TTS benchmark (LibriSpeech, VCTK) is used, limiting direct comparison to single-speaker zero-shot systems. caveats: [] - id: iclr-2025-hQvX9MBowC published_date: "2025-04-24" entry_date: '2026-07-29' year: 2025 venue: ICLR task: - TTS architecture: - diffusion - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - diffusion_multilingual_tts - transformer_encoder_decoder_multilingual_tts claims: - claim_id: diffusion_transformer_backbones_are_better_suited_to_tts_than_u role: supports claim: Diffusion Transformer backbones are better suited to TTS than U-Net backbones once domain-specific conditioning factors (phonemes, durations) are removed. source: §5.2, Table 4 evidence: Under matched training conditions, replacing the DiT backbone with a U-Net (and a U-Net variant without down/up-sampling) increases WER from 2.93 to 3.7 and drops SIM-r from 0.588 to 0.389 on the English cross-sentence task. confidence: high relevance: medium - claim_id: predicting_total_target_length_and_generating_variable_length_sequences_outperforms role: supports claim: Predicting total target length and generating variable-length sequences outperforms fixed-length generation with padding in diffusion-based TTS. source: §5.2, Table 5 evidence: Fixed-length modeling with padding reaches WER 6.81-8.89, while a learned speech length predictor with variable-length generation reaches WER 5.36-5.58 under otherwise identical settings. confidence: high relevance: medium - claim_id: aligning_text_and_speech_latent_representations_improves_cross_attention_conditioned role: supports claim: Aligning text and speech latent representations improves cross-attention-conditioned generation quality, independent of model or training-data scale. source: §5.2, Tables 6-7 evidence: A speech codec fine-tuned with an auxiliary language-modeling loss against a frozen text encoder (Mel-VAE++) improves WER/SIM over the unaligned codec regardless of which text encoder (ByT5 or SpeechT5) is paired with it, and a jointly text-speech-trained text encoder (SpeechT5, 85M params) outperforms a larger text-only encoder (ByT5-base, 415M params) trained on more data. confidence: high relevance: medium - claim_id: removing_domain_specific_alignment_factors_from_ldm_based_tts_narrows role: complicates claim: Removing domain-specific alignment factors from LDM-based TTS narrows but does not eliminate the gap to phoneme-duration-based systems in speaker similarity. source: §5.1, Table 2 evidence: DiTTo-en-XL reaches SIM-r 0.6554 on the cross-sentence task, below Voicebox's reported 0.681 (a phoneme/duration-based non-autoregressive model), even though DiTTo-en-XL is faster and matches or exceeds Voicebox on WER. confidence: high relevance: low - claim_id: codec_compression_ratio_not_codec_reconstruction_quality_alone_determines_suitability role: complicates claim: Codec compression ratio, not codec-reconstruction quality alone, determines suitability as a diffusion target for variable-length TTS. source: §5.2, Table 7 evidence: DAC achieves higher PESQ and ViSQOL codec-reconstruction scores than Mel-VAE, but its 7-8x longer latent sequences make training and inference substantially less efficient and degrade end-to-end WER/SIM relative to the more compressed but lower-fidelity Mel-VAE. confidence: high relevance: low limitations: - Code and pretrained weights are not released at publication (only demo samples), and most baseline comparisons (Voicebox, VALL-E, NaturalSpeech 2/3) use numbers copied from the original papers' reported results rather than reproduced under DiTTo's own evaluation pipeline, limiting independent verification of head-to-head rankings. - The speaker-similarity gap to phoneme-duration-based systems like Voicebox persists even at the largest model scale tested, suggesting that removing domain-specific factors trades off some speaker-identity preservation for simplicity and speed. The paper also does not explore natural-language instruction following or fine-grained prosody control, both noted as future directions rather than addressed in this work. Multilingual results are reported on only 100 held-out examples per language, which is a narrow evaluation slice given nine languages with very different phonological properties. caveats: [] - id: 2025.naacl-short.69 published_date: "2025-04-29" entry_date: '2026-07-29' year: 2025 venue: NAACL task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: multilingual_training_on_typologically_related_low_resource_languages_improves_tts role: supports claim: Multilingual training on typologically related low-resource languages improves TTS objective quality over monolingual training for all participating languages. source: §5, Table 2 evidence: Multilingual models consistently outperform monolingual baselines across F0-RMSE, MCD, PESQ, and VUV F1 for all four speakers (Ojibwe JJ, NJ; Mi'kmaq MJ; Maliseet AT); MOS for Ojibwe JJ improves from 2.71 (monolingual) to 3.42 (multilingual FNet). confidence: high relevance: high - claim_id: attention_free_sequence_mixing_architectures_can_match_self_attention_performance role: supports claim: Attention-free sequence mixing architectures can match self-attention performance in flow-matching TTS while halving GPU memory requirements. source: §5, Tables 2–3 evidence: Hydra achieves objective metrics close to self-attention MatchaTTS across all four speakers, and attention-free models consume 2.3–2.5G vs. 4.6G GPU memory at batch size 400 (float32). confidence: high relevance: medium - claim_id: subquadratic_complexity_in_attention_free_architectures_does_not_translate_to role: complicates claim: Subquadratic complexity in attention-free architectures does not translate to inference latency gains in small TTS models. source: §5, Table 3 evidence: Self-attention has higher throughput than Mamba2, Hydra, and FNet in float32 on an A100 at batch size 400, because the model is small enough that architectural complexity advantages are not observable. confidence: high relevance: low - claim_id: standard_mos_based_subjective_evaluation_protocols_can_produce_unreliable_results role: complicates claim: Standard MOS-based subjective evaluation protocols can produce unreliable results when applied to Indigenous language TTS due to cultural norms around negative judgement. source: §5 evidence: One of two Ojibwe raters assigned the highest score (5/5) to all voices including synthetic ones; MOS surveys could not be completed for Mi'kmaq and Maliseet due to lack of available raters; the authors argue the task paradigm itself is culturally misaligned. confidence: high relevance: low limitations: - The custom datasets are not publicly released due to lack of consent from voice donors, preventing replication of the reported results. MOS evaluation is available only for one Ojibwe speaker (with two raters), making subjective quality assessment effectively impossible for three of the four speakers. - Recording in home environments introduces ambient noise that limits overall synthesis quality. The community-centered evaluation observation is preliminary and anecdotal rather than a systematic study of cultural evaluation norms. Model deployment for educational use, which the paper motivates, is planned but not yet demonstrated, leaving a gap between system performance and real-world utility. caveats: [] - id: 2025.americasnlp-1.1 published_date: "2025-05-04" entry_date: '2026-07-29' year: 2025 venue: AmericasNLP 2025 task: - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts - gan_based_multilingual_synthesis claims: - claim_id: transfer_learning_from_a_closely_related_language_can_enable_intelligible role: supports claim: Transfer learning from a closely related language can enable intelligible TTS for an indigenous language with fewer than 5 hours of training data. source: §4.1, §5.1 evidence: Transfer learning from a closely related language can enable intelligible TTS for an indigenous language with fewer than 5 hours of training data. confidence: high relevance: medium - claim_id: subjective_evaluation_by_native_speakers_is_necessary_when_objective_metrics role: supports claim: Subjective evaluation by native speakers is necessary when objective metrics such as PESQ and POLQA cannot be applied due to absent high-quality reference recordings. source: §5.3 evidence: Subjective evaluation by native speakers is necessary when objective metrics such as PESQ and POLQA cannot be applied due to absent high-quality reference recordings. confidence: high relevance: low - claim_id: pronunciation_drift_in_a_native_speaker_living_outside_the_language role: supports claim: Pronunciation drift in a native speaker living outside the language community can cause natural speech to score lower on intelligibility than synthetic speech on standardised evaluation. source: §5.4 evidence: Pronunciation drift in a native speaker living outside the language community can cause natural speech to score lower on intelligibility than synthetic speech on standardised evaluation. confidence: high relevance: low - claim_id: gan_based_vocoders_initialized_from_a_universal_pretrained_model_converge role: supports claim: GAN-based vocoders initialized from a universal pretrained model converge rapidly when the target speaker is similar to the pretraining distribution. source: §4.2, §5.2 evidence: GAN-based vocoders initialized from a universal pretrained model converge rapidly when the target speaker is similar to the pretraining distribution. confidence: high relevance: low limitations: - '- Corpus is only 4h 37m from a single female speaker, limiting prosodic and phonetic diversity and voice coverage. - Transfer learning from Spanish may introduce Spanish phonotactic biases that are not present in Shipibo-Konibo. - Evaluation relied entirely on subjective metrics (no PESQ/POLQA due to lack of reference data); subjectivity and evaluator bias are acknowledged. - Two writing conventions for Shipibo-Konibo exist; the 2015 standard was used, limiting applicability in communities using older orthographies. - The speaker''s extended residence in Lima introduced pronunciation drift that complicates the natural vs. synthetic comparison. - The model has not been tested on adaptation to other Panoan or Amazonian languages yet, though the framework is intended to generalize.' caveats: [] - id: '2505.07916' published_date: "2025-05-12" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - vae_multilingual_latent_models claims: - claim_id: jointly_training_a_speaker_encoder_with_an_autoregressive_tts_model role: supports claim: Jointly training a speaker encoder with an autoregressive TTS model yields better intelligibility and competitive speaker similarity compared to using a fixed speaker verification embedding. source: §3.6, Table 4 evidence: Jointly training a speaker encoder with an autoregressive TTS model yields better intelligibility and competitive speaker similarity compared to using a fixed speaker verification embedding. confidence: high relevance: low - claim_id: zero_shot_voice_cloning_conditioned_on_untranscribed_reference_audio_produces role: complicates claim: Zero-shot voice cloning conditioned on untranscribed reference audio produces lower word error rates than one-shot conditioning with a paired text-audio exemplar, at the cost of slightly reduced speaker similarity. source: §3.2, Table 1; §3.5, Table 3 evidence: Zero-shot voice cloning conditioned on untranscribed reference audio produces lower word error rates than one-shot conditioning with a paired text-audio exemplar, at the cost of slightly reduced speaker similarity. confidence: high relevance: low - claim_id: augmenting_a_vae_with_a_normalizing_flow_on_the_latent role: supports claim: Augmenting a VAE with a normalizing flow on the latent space (Flow-VAE) improves both waveform reconstruction quality and downstream TTS speaker similarity over a standard VAE with the same architecture. source: §3.7, Tables 5–6 evidence: Augmenting a VAE with a normalizing flow on the latent space (Flow-VAE) improves both waveform reconstruction quality and downstream TTS speaker similarity over a standard VAE with the same architecture. confidence: high relevance: low - claim_id: speaker_encoder_representations_trained_without_text_dependencies_support_cross_lingual role: supports claim: Speaker encoder representations trained without text dependencies support cross-lingual synthesis with high intelligibility, outperforming prompt-based systems that require transcribed reference audio in cross-lingual scenarios. source: §3.5, Table 3 evidence: Speaker encoder representations trained without text dependencies support cross-lingual synthesis with high intelligibility, outperforming prompt-based systems that require transcribed reference audio in cross-lingual scenarios. confidence: high relevance: high - claim_id: disentangled_speaker_embeddings_from_a_task_specific_encoder_enable_parameter role: supports claim: Disentangled speaker embeddings from a task-specific encoder enable parameter-efficient per-speaker adaptation by fine-tuning only the speaker embedding vector, preserving base model generalization. source: §4.3 evidence: Disentangled speaker embeddings from a task-specific encoder enable parameter-efficient per-speaker adaptation by fine-tuning only the speaker embedding vector, preserving base model generalization. confidence: high relevance: low limitations: - The training dataset is proprietary and the model size is not reported, which makes independent replication impossible. All results are from internal or public leaderboard evaluations only; no code release is indicated. - The leaderboard evaluation (Artificial Analysis Arena) uses preference judgements that may be influenced by sample selection; the Arena snapshot is from a single date (May 12, 2025) and rankings can shift as more models are added. The claim of "top position" is time-bound. - WER metrics for several tested languages (Cantonese, French, Hindi) remain substantially higher than for well-resourced languages, indicating multilingual coverage is uneven. The cross-lingual evaluation is limited to seven target languages using Chinese source speakers, leaving broader cross-lingual generalization untested. - Emotion control via LoRA requires separate training runs per emotion category and discrete emotion labels, which limits granularity. The text-to-voice extension relies on structured attribute tags alongside free-text descriptions, constraining the range of expressible timbres. No human evaluation of the LoRA emotion control or text-to-voice outputs is reported. caveats: [] - id: '2505.13000' published_date: "2025-05-19" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - codec - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models claims: - claim_id: directly_encoding_ssl_features_into_the_first_rvq_layer_preserves role: supports claim: Directly encoding SSL features into the first RVQ layer preserves significantly more semantic content than distilling SSL representations into codec tokens, particularly for tonal languages where pitch information is phonemically critical. source: §4.2, Table 2 evidence: Directly encoding SSL features into the first RVQ layer preserves significantly more semantic content than distilling SSL representations into codec tokens, particularly for tonal languages where pitch information is phonemically critical. confidence: high relevance: low - claim_id: neural_audio_codecs_operating_at_lower_frame_rates_with_more role: supports claim: Neural audio codecs operating at lower frame rates with more quantization layers achieve superior audio quality at equivalent bitrates compared to higher frame rate codecs with fewer quantization layers. source: §4.3, Table 3 evidence: Neural audio codecs operating at lower frame rates with more quantization layers achieve superior audio quality at equivalent bitrates compared to higher frame rate codecs with fewer quantization layers. confidence: high relevance: medium - claim_id: semantic_enhancement_of_the_first_rvq_layer_improves_downstream_tts role: supports claim: Semantic enhancement of the first RVQ layer improves downstream TTS speaker similarity as well as intelligibility, because higher semantic fidelity in RVQ-1 enables the waveform stream's quantizers to focus on acoustic detail rather than recovering content information. source: §4.4, Table 4 evidence: Semantic enhancement of the first RVQ layer improves downstream TTS speaker similarity as well as intelligibility, because higher semantic fidelity in RVQ-1 enables the waveform stream's quantizers to focus on acoustic detail rather than recovering content information. confidence: high relevance: low - claim_id: the_quality_gap_between_distillation_based_and_direct_encoding_semantic role: supports claim: The quality gap between distillation-based and direct-encoding semantic codecs is substantially larger in Mandarin than in English, revealing a systematic limitation of distillation approaches for tonal languages. source: §4.2, Table 2 evidence: The quality gap between distillation-based and direct-encoding semantic codecs is substantially larger in Mandarin than in English, revealing a systematic limitation of distillation approaches for tonal languages. confidence: high relevance: medium limitations: - The 12.5Hz DualCodec-based TTS systems consistently underperform their 25Hz counterparts on both WER and speaker similarity (Table 4, Table 6), indicating that the quality upper bound of the 12.5Hz variant is not yet competitive with the best open-source systems at 50Hz despite the frame rate reduction improving inference speed. - The paper evaluates TTS only on Seed-TTS-Eval; no subjective TTS listening tests are reported, so the MUSHRA gains in codec reconstruction may not fully translate to perceived TTS naturalness. Speaker similarity scores with DualCodec-VALLE remain below those of MaskGCT baselines that use separate semantic and acoustic tokenizers, suggesting the unified approach has not yet matched the best-performing two-stage pipeline design. The DualCodec encoder is substantially heavier than baselines (628M vs 38M for Mimi) due to the frozen w2v-BERT-2.0 model, increasing training-time compute, though the decoder remains lightweight for inference. caveats: [] - id: '2505.17589' published_date: "2025-05-23" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: differentiable_reward_optimization_over_discrete_speech_tokens_rather_than_synthesized role: supports claim: Differentiable reward optimization over discrete speech tokens, rather than synthesized audio, enables scalable post-training for LLM-based TTS without the computational overhead of full audio generation. source: §2.2 evidence: Differentiable reward optimization over discrete speech tokens, rather than synthesized audio, enables scalable post-training for LLM-based TTS without the computational overhead of full audio generation. confidence: high relevance: medium - claim_id: supervised_multi_task_training_of_a_speech_tokenizer_on_tasks role: supports claim: Supervised multi-task training of a speech tokenizer on tasks including emotion recognition, language identification, and speaker analysis captures richer paralinguistic content than ASR-only objectives, with measurable downstream gains in content consistency and speaker similarity. source: §2.1, §5.3 evidence: Supervised multi-task training of a speech tokenizer on tasks including emotion recognition, language identification, and speaker analysis captures richer paralinguistic content than ASR-only objectives, with measurable downstream gains in content consistency and speaker similarity. confidence: high relevance: low - claim_id: current_zero_shot_tts_systems_infer_emotional_tone_primarily_from role: supports claim: Current zero-shot TTS systems infer emotional tone primarily from text semantics; expression of text-unrelated emotions from a reference prompt remains a significant unsolved challenge. source: §5.2.3, Table 9 evidence: Current zero-shot TTS systems infer emotional tone primarily from text semantics; expression of text-unrelated emotions from a reference prompt remains a significant unsolved challenge. confidence: high relevance: medium - claim_id: scaling_training_data_and_model_size_in_two_stage_codec role: supports claim: Scaling training data and model size in two-stage codec TTS produces consistent gains up to a point of diminishing returns, with data scaling showing stronger marginal returns than model size scaling beyond 0.5B parameters on current benchmarks. source: §4.2, §5.1 evidence: Scaling training data and model size in two-stage codec TTS produces consistent gains up to a point of diminishing returns, with data scaling showing stronger marginal returns than model size scaling beyond 0.5B parameters on current benchmarks. confidence: high relevance: low - claim_id: existing_tts_benchmarks_based_on_studio_quality_audiobook_data_underestimate role: supports claim: Existing TTS benchmarks based on studio-quality audiobook data underestimate the difficulty of in-the-wild voice cloning, where noisy reference audio and domain diversity expose robustness gaps not visible in standard evaluations. source: §4.4 evidence: Existing TTS benchmarks based on studio-quality audiobook data underestimate the difficulty of in-the-wild voice cloning, where noisy reference audio and domain diversity expose robustness gaps not visible in standard evaluations. confidence: high relevance: medium limitations: - Speaker similarity lags Seed-TTS across all SEED-TTS-Eval conditions, and the authors attribute this to data volume differences rather than architectural factors. Since pretraining data is not released, the degree to which CosyVoice 3's improvements are reproducible without Alibaba's internal 1M-hour corpus is unclear. - 'The DiffRO reward mechanism exhibits a trade-off between intelligibility and speaker similarity: RL post-training slightly reduces speaker similarity while improving WER. Multi-reward DiffRO with emotion targets further degrades pronunciation, suggesting that reward balancing in multi-objective post-training for TTS is an open engineering problem. Timbre control via textual instructions is absent and noted as future work. Singing voice generation remains unsatisfactory due to insufficient training data coverage in the tokenizer and LM stages. Performance on Korean and Japanese lags other languages due to data volume limitations and language-specific encoding challenges (kana conversion, multiple pronunciations).' caveats: [] - id: '2507.15272' published_date: "2025-07-21" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - diffusion_multilingual_tts claims: - claim_id: conditioning_duration_prediction_on_reference_audio_from_the_same_speaker role: supports claim: Conditioning duration prediction on reference audio from the same speaker can improve speaker similarity in zero-shot diffusion TTS without modifying the training objective. source: §2.2, Table 2 evidence: Cross-attention over an unrelated reference mel-spectrogram (same speaker, different content) improves SIM-O over a speaker-embedding-conditioned Grad-TTS baseline across all seven Indian languages tested. confidence: high relevance: low - claim_id: classifier_free_guidance_with_a_dataset_mean_unconditional_condition_improves role: supports claim: Classifier-free guidance with a dataset-mean unconditional condition improves speaker consistency at inference time in diffusion-based TTS for unseen speakers. source: §2.3, Table 2 evidence: A2TTS applies CFG at inference, following UnitSpeech, to amplify speaker conditioning without changing the training procedure; speaker similarity gains are observed across all languages evaluated. confidence: high relevance: low - claim_id: diffusion_based_zero_shot_tts_can_achieve_reasonable_speaker_similarity role: complicates claim: Diffusion-based zero-shot TTS can achieve reasonable speaker similarity, but intelligibility on out-of-domain benchmarks degrades substantially relative to ground-truth speech. source: §Appendix A, Tables 3-9 evidence: A2TTS TTS WER on FLEURS, CommonVoice, and IndicTTS is 1.5-4x higher than the corresponding ASR baseline on the same test sets across Hindi, Marathi, Punjabi, Bengali, Gujarati, Tamil, and Malayalam. confidence: high relevance: low - claim_id: language_specific_diffusion_tts_models_can_be_trained_on_standard role: supports claim: Language-specific diffusion TTS models can be trained on standard benchmark speech corpora to achieve multi-speaker zero-shot synthesis in low-resource settings. source: §3.1, Table 2 evidence: A2TTS trains separate models for seven Indian languages using only the IndicSUPERB dataset (except Malayalam and Tamil which also use IndicTTS), achieving SIM-O above 0.70 for all languages on the IndicSUPERB evaluation set. confidence: high relevance: low limitations: - No naturalness evaluation (MOS or automated predictors) is reported. Speaker similarity (SIM-O) and CER are the only metrics, so claims about speech quality and naturalness cannot be verified from the paper's own experiments. - The model is only compared to a single baseline (Grad-TTS with speaker embeddings), without evaluating against UnitSpeech (from which the speaker encoder is borrowed), YourTTS, or other zero-shot multilingual systems. This makes it difficult to assess how much of the improvement comes from the cross-attention duration predictor versus the overall system design. - The paper trains language-specific models rather than a single multilingual model, which limits scalability to new languages. The authors acknowledge that out-of-domain speakers and languages not covered by IndicSUPERB would require fine-tuning. High training cost (2,000 epochs) is also noted as a practical constraint. - The elevated TTS WER relative to ASR baselines on CommonVoice and FLEURS suggests intelligibility issues on speakers and recording conditions not well represented in IndicSUPERB training data. caveats: [] - id: '2507.16875' published_date: "2025-07-22" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - flow_matching_multilingual_tts claims: - claim_id: duration_prediction_strategy_choice_in_zero_shot_tts_presents_language role: complicates claim: Duration prediction strategy choice in zero-shot TTS presents language-dependent trade-offs between intelligibility and speaker similarity that are not consistent across multilingual settings. source: §5, Table 3, Table 5 evidence: The speaker-prompted (PFlow-style) predictor improves Tamil intelligibility (WER 0.295 vs. 0.312) and speaker similarity (SMOS 4.56 vs. 4.41) but degrades Marathi intelligibility substantially (WER 0.365 vs. 0.224 for infill), while Bengali human evaluation favours infill on both dimensions. confidence: high relevance: high - claim_id: speaker_conditioned_audio_prompting_enables_duration_prediction_without_forced_alignment role: supports claim: Speaker-conditioned audio prompting enables duration prediction without forced alignment, yielding competitive or superior speaker similarity in low-resource multilingual TTS. source: §3.4.2, §5.1, Table 4, Table 5 evidence: The 84M-parameter speaker-prompted predictor, conditioning on a 3-second mel spectrogram via cross-attention, achieves higher Sim-o scores than the infill predictor for Tamil, Telugu, and Bengali, and substantially higher human SMOS for Hindi (4.14 vs. 3.57), without using explicit duration context at inference. confidence: high relevance: high - claim_id: pooled_low_resource_training_corpora_for_multilingual_tts_contain_significant role: complicates claim: Pooled low-resource training corpora for multilingual TTS contain significant quality variance that requires aggressive filtering before flow-matching models can be reliably trained. source: §4.2.2 evidence: WER-based filtering using IndicWhisper reduced the Hindi training set from 1.3M to 1M utterances; a partial recovery via CTC alignment scores added 30K back, indicating that roughly 23% of the raw data was too noisy or misaligned to be usable. confidence: high relevance: high - claim_id: the_relative_benefit_of_speaker_prompted_duration_conditioning_over_text role: refines claim: The relative benefit of speaker-prompted duration conditioning over text-and-alignment conditioning is not uniform and depends on the phonological and prosodic characteristics of the target language. source: §5.1, §5.2, Table 3, Table 4, Table 5 evidence: Tamil, with complex phoneme structures and high prosodic variability, benefits consistently from speaker prompting across both objective WER and human SMOS; Marathi shows a clear regression; Hindi and Bengali exhibit mixed patterns depending on test subset. confidence: high relevance: high limitations: - All experiments are conducted on speech infilling (50% sentence masking), not full-sentence TTS generation. Intelligibility and speaker similarity results may not transfer directly to the more common single-utterance TTS evaluation setting. - The cross-sentence completion task is described but no quantitative results are reported in the paper text. Comparisons are against ground-truth durations and the two proposed predictors only; no external TTS baselines from the Indian language literature are included. Training data for some languages is below 1,000 hours, and the five languages evaluated represent a small fraction of India's linguistic diversity. All models train in FP32, which is resource-intensive and may limit reproducibility for lower-compute settings. No code or model checkpoints are publicly released. caveats: [] - id: '2507.21138' published_date: "2025-07-22" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: rl_alignment_with_composite_perceptual_rewards_improves_multiple_speech_quality role: supports claim: RL alignment with composite perceptual rewards improves multiple speech quality dimensions simultaneously over SFT-only baselines in autoregressive codec TTS. source: §3.5, Table 8 evidence: GRPO with combined WER + speaker similarity + DNSMOS rewards achieves gains in all three metrics over the SFT baseline, while single-reward models improve only their target dimension; TTS-1 WER drops from 7.9% (SFT) to 6.3% (RL-aligned). confidence: high relevance: low - claim_id: audio_pre_training_on_large_scale_raw_speech_substantially_improves role: supports claim: Audio pre-training on large-scale raw speech substantially improves subsequent SFT quality in LLM-based TTS beyond what instruction-tuned LLM initialisation provides. source: §3.4, Figure 5 evidence: Initialising SFT from an audio pre-trained LLaMA-3.2-1B checkpoint yields lower SFT loss, approximately 15% lower WER, and approximately 3% higher speaker similarity than initialising from the base LLaMA-3.2-1B-Instruct checkpoint. confidence: high relevance: medium - claim_id: scaling_speechlm_parameter_count_in_autoregressive_codec_tts_consistently_improves role: supports claim: Scaling SpeechLM parameter count in autoregressive codec TTS consistently improves intelligibility and speaker fidelity across languages. source: §4, Figure 8, Table 8 evidence: TTS-1-Max (8.8B) achieves 5.1% overall WER and higher SIM across all 11 evaluated languages compared to TTS-1 (1.6B) at 6.3% WER, with performance correlating with pre-training loss differences. confidence: high relevance: low - claim_id: style_conditioning_via_discrete_text_tags_conflicts_with_speaker_identity role: complicates claim: Style conditioning via discrete text tags conflicts with speaker identity preservation in single-codebook codec TTS architectures. source: §3.6 evidence: Direct prepending of style markup tags during SFT produced no effect on synthesized speech because the single-codebook design entangles acoustic and semantic information; successful style control required constructing paired neutral/stylized utterances from the same speaker and applying LoRA fine-tuning. confidence: high relevance: low - claim_id: streaming_audio_delivery_in_autoregressive_tts_introduces_audible_artifacts_and role: complicates claim: Streaming audio delivery in autoregressive TTS introduces audible artifacts and volume inconsistencies at segment boundaries that require specific engineering mitigations independent of the generative model's quality. source: §5.1 evidence: Without concatenation restricted to non-voicing regions and context-aware decoder decoding with extended audio prompt context, segment boundaries introduce clicks and volume drops; these are engineering-layer problems independent of SpeechLM quality. confidence: high relevance: low limitations: - Model weights are not publicly released, making independent benchmarking and replication impossible. All evaluations use proprietary or internal test sets; the internal TTS arena covers only English and uses approximately 20 annotators with a modest vote count per pair. - The training data is drawn from a proprietary mixture of public and licensed sources whose exact composition is not disclosed, limiting reproducibility. The evaluation framework does not include standard public TTS benchmarks (e.g., LibriTTS or VCTK test sets), making direct numerical comparison with published systems that do report on these benchmarks difficult. - The paper notes that speaker similarity metrics fluctuate with emotionally expressive speech, suggesting that current automated evaluation protocols may not fully capture perceptual quality in dynamic scenarios. The audio markup system, while effective for English, shows "reduced fidelity" when generalizing style control to non-English languages. Prompt audio caching can cause prosodic bleed from the reference audio into generated speech, and longer sequences generated from short prompts may degrade in quality. caveats: [] - id: '2507.18119' published_date: "2025-07-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: modality_alignment_spoken_language_models_can_internalize_paralinguistic_cues_dialect role: supports claim: Modality-alignment spoken language models can internalize paralinguistic cues (dialect, age, emotion, non-speech vocalizations) through staged training without requiring explicit user instructions at inference time. source: §5.2, Table 7, Table 8 evidence: GOAT-SLM achieves 50.73% average dialect-following accuracy across five Chinese dialects without explicit prompting, versus 18.91% for the next-best model; it also reaches 40.91% on non-speech vocal response tasks where all other models score below 10%. confidence: high relevance: high - claim_id: initializing_a_speech_generation_head_from_a_pretrained_llm_text role: supports claim: Initializing a speech generation head from a pretrained LLM text head promotes parameter reuse and can preserve linguistic competence in the speech branch without full retraining. source: §3, §5.1, Table 6 evidence: The Speak module is initialized from Write module weights; the model retains multi-turn dialogue accuracy of 84% despite not training on multi-turn data, attributed to well-aligned input embeddings from this parameter sharing strategy. confidence: high relevance: medium - claim_id: incorporating_paralinguistic_awareness_into_a_spoken_language_model_introduces_a role: complicates claim: Incorporating paralinguistic awareness into a spoken language model introduces a trade-off with general semantic question-answering performance. source: §5.1, Table 4 evidence: GOAT-SLM's capability on general AQA (Table 4) declines compared to models without paralinguistic training, remaining at average level across eight QA datasets; the authors explicitly attribute this to the expanded training objectives. confidence: high relevance: medium - claim_id: dialectal_speech_generation_quality_is_uneven_across_dialects_with_closely role: complicates claim: Dialectal speech generation quality is uneven across dialects, with closely related varieties more reliably acquired than typologically distant ones. source: §5.2, Table 10 evidence: In subjective evaluation of dialectal spoken response (Table 10), GOAT-SLM exceeds 90% consistency for Cantonese, Henan, Shanghainese, and Sichuanese, but achieves only 57.2% for northeastern Mandarin, which the authors note is lexically similar to standard Mandarin and may suffer from training ambiguity. confidence: high relevance: medium - claim_id: confidence_based_gradient_masking_during_speech_token_training_can_improve role: supports claim: Confidence-based gradient masking during speech token training can improve pronunciation stability by selectively updating tokens where the model's predictions are reliable. source: §4.3 evidence: GOAT-SLM applies confidence-based automatic gradient masking in Stage 3, masking low-confidence tokens during backpropagation; the technique is credited with significantly enhanced pronunciation stability and speech fidelity. confidence: high relevance: medium limitations: - The evaluation is restricted to Chinese-language dialects and Mandarin-English bilingual content; generalization to non-Chinese languages, especially under dialectal conditions, is untested. The TELEVAL benchmark is developed by the same research group, introducing potential evaluation-training alignment that may not reflect performance on independently constructed benchmarks. - The paper relies on GOAT-TTS to generate synthetic speech training targets in Stage 3-2, creating a dependency on a proprietary tool whose capabilities bound the upper limit of the attribute-aware training data. The paper does not ablate the impact of LLM-initialized speech branch weights versus random initialization in a controlled experiment, leaving the primary architectural design claim without direct causal evidence. - Northeastern Mandarin generates only 57.2% dialectal consistency in subjective evaluation, revealing that dialects with high lexical overlap with the training language (standard Mandarin) are harder to acquire distinctly, likely because the model defaults to standard Mandarin patterns. caveats: [] - id: 2025.acl-industry.42 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: - GAN - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - gan_based_multilingual_synthesis - transformer_encoder_decoder_multilingual_tts claims: - claim_id: accurate_grapheme_to_phoneme_conversion_is_the_single_most_important role: supports claim: Accurate grapheme-to-phoneme conversion is the single most important preprocessing component for intelligibility in tonal-language TTS, contributing more to WER reduction than tokenisation or pause prediction combined. source: §6, Table 3 evidence: Accurate grapheme-to-phoneme conversion is the single most important preprocessing component for intelligibility in tonal-language TTS, contributing more to WER reduction than tokenisation or pause prediction combined. confidence: high relevance: high - claim_id: proprietary_general_purpose_tts_systems_degrade_substantially_on_domain_specific role: supports claim: Proprietary general-purpose TTS systems degrade substantially on domain-specific vocabulary, while systems trained with targeted vertical-domain data show much smaller performance drops in specialised settings. source: §6, Table 2 evidence: Proprietary general-purpose TTS systems degrade substantially on domain-specific vocabulary, while systems trained with targeted vertical-domain data show much smaller performance drops in specialised settings. confidence: high relevance: medium - claim_id: low_resource_tts_systems_can_achieve_competitive_speaker_similarity_in role: supports claim: Low-resource TTS systems can achieve competitive speaker similarity in zero-shot voice cloning relative to voice conversion models when trained with a style embedding module, even without speaker-specific fine-tuning. source: §6, Table 4 evidence: Low-resource TTS systems can achieve competitive speaker similarity in zero-shot voice cloning relative to voice conversion models when trained with a style embedding module, even without speaker-specific fine-tuning. confidence: high relevance: low - claim_id: contextual_prosody_modelling_via_a_tonal_phoneme_aware_bert_encoder role: supports claim: Contextual prosody modelling via a tonal phoneme-aware BERT encoder improves naturalness scores in tonal-language TTS beyond what standard duration and pitch predictors achieve. source: §5 evidence: Contextual prosody modelling via a tonal phoneme-aware BERT encoder improves naturalness scores in tonal-language TTS beyond what standard duration and pitch predictors achieve. confidence: high relevance: low limitations: - '- The approach has been validated only on Thai; generalization to other tonal or low-resource languages is claimed but not demonstrated. - Model size and full architecture details are not reported, limiting reproducibility. - The zero-shot comparison is against OpenVoice (a VC model) rather than other zero-shot TTS systems. - Data collection from TikTok and YouTube raises questions about license and long-term data availability.' caveats: [] - id: 2025.acl-long.313 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: influential method_family: - flow_matching_multilingual_tts claims: - claim_id: explicit_phoneme_level_alignment_is_not_required_for_robust_non role: supports claim: Explicit phoneme-level alignment is not required for robust non-autoregressive zero-shot TTS; modality-specific representation refinement before feature concatenation can resolve text-speech alignment failures in alignment-free architectures. source: §3.2, §5.1 evidence: Explicit phoneme-level alignment is not required for robust non-autoregressive zero-shot TTS; modality-specific representation refinement before feature concatenation can resolve text-speech alignment failures in alignment-free architectures. confidence: high relevance: medium - claim_id: deep_entanglement_of_semantic_and_acoustic_features_in_a_shared role: supports claim: Deep entanglement of semantic and acoustic features in a shared transformer produces systematic alignment failures in non-autoregressive TTS that persist across training scale and cannot be addressed by re-ranking. source: §3.2, §5.1, Table 4 evidence: Deep entanglement of semantic and acoustic features in a shared transformer produces systematic alignment failures in non-autoregressive TTS that persist across training scale and cannot be addressed by re-ranking. confidence: high relevance: medium - claim_id: inference_time_flow_step_scheduling_that_densifies_early_integration_steps role: supports claim: Inference-time flow step scheduling that densifies early integration steps improves faithfulness and speaker similarity in flow-matching TTS without any retraining. source: §3.2, §5.2, Table 5 evidence: Inference-time flow step scheduling that densifies early integration steps improves faithfulness and speaker similarity in flow-matching TTS without any retraining. confidence: high relevance: low - claim_id: non_autoregressive_flow_matching_tts_trained_at_100k_hour_scale role: supports claim: Non-autoregressive flow-matching TTS trained at 100K-hour scale can match autoregressive systems on intelligibility while achieving substantially lower real-time factor. source: §5, Table 1 evidence: Non-autoregressive flow-matching TTS trained at 100K-hour scale can match autoregressive systems on intelligibility while achieving substantially lower real-time factor. confidence: high relevance: medium - claim_id: absence_of_standardised_benchmark_subsets_with_released_prompt_lists_is role: supports claim: Absence of standardised benchmark subsets with released prompt lists is a structural barrier to fair cross-paper comparison in zero-shot TTS; community-released evaluation sets directly reduce this problem. source: §4 evidence: Absence of standardised benchmark subsets with released prompt lists is a structural barrier to fair cross-paper comparison in zero-shot TTS; community-released evaluation sets directly reduce this problem. confidence: high relevance: low limitations: - Speaker similarity (SPK-SIM 0.66–0.67 on English test sets) remains below ground truth (0.69–0.73) and well below Seed-TTS DiT (0.790 EN, 0.809 ZH), which operates at unknown but much larger training scale. Whether this gap is architectural or purely a function of training data volume is not resolved by this paper. - 'The mel spectrogram representation creates a sequence-length bottleneck: mel frames are significantly longer than the text token sequence, making computation and attention cost scale unfavourably with utterance length. The paper identifies more efficient continuous representations as the primary remaining efficiency direction.' - Duration estimation via character-count ratio introduces uncontrolled variance at inference, particularly for prosodically complex or phonetically irregular text. A lightweight learned duration predictor could improve rhythm and naturalness at moderate complexity cost. - Fine-grained paralinguistic control — emotion, speaking style, prosodic emphasis — is absent. The infilling framing naturally mirrors the reference prompt's style, but there is no mechanism to override or interpolate paralinguistic attributes. Broader multilingual coverage beyond English and Chinese is also unevaluated. caveats: [] - id: 2025.acl-long.598 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: preference_alignment_with_cross_model_preference_pairs_yields_larger_intelligibility role: supports claim: Preference alignment with cross-model preference pairs yields larger intelligibility improvements than single-model self-comparison pairs for TTS post-training. source: §5.2, Table 5 evidence: Preference alignment with cross-model preference pairs yields larger intelligibility improvements than single-model self-comparison pairs for TTS post-training. confidence: high relevance: medium - claim_id: dpo_objectives_can_be_derived_for_flow_matching_and_masked role: supports claim: DPO objectives can be derived for flow-matching and masked generative TTS architectures by operating in velocity space and masked token distributions, respectively, without requiring explicit reward models. source: §4.2, §4.3 evidence: DPO objectives can be derived for flow-matching and masked generative TTS architectures by operating in velocity space and masked token distributions, respectively, without requiring explicit reward models. confidence: high relevance: medium - claim_id: intelligibility_gains_from_preference_alignment_on_english_and_chinese_data role: supports claim: Intelligibility gains from preference alignment on English and Chinese data generalize to unseen languages (Japanese, Korean, German, French) without degrading speaker similarity. source: §5.1, Table 10 evidence: Intelligibility gains from preference alignment on English and Chinese data generalize to unseen languages (Japanese, Korean, German, French) without degrading speaker similarity. confidence: high relevance: low - claim_id: preference_datasets_constructed_from_weaker_tts_generators_effectively_improve_stronger role: supports claim: Preference datasets constructed from weaker TTS generators effectively improve stronger models outside the training distribution, demonstrating weak-to-strong generalization in speech alignment. source: §5.1, Table 4 evidence: Preference datasets constructed from weaker TTS generators effectively improve stronger models outside the training distribution, demonstrating weak-to-strong generalization in speech alignment. confidence: high relevance: medium - claim_id: iterative_preference_alignment_yields_diminishing_returns_across_rounds_suggesting_that role: supports claim: Iterative preference alignment yields diminishing returns across rounds, suggesting that base model capability is the primary ceiling on alignment effectiveness. source: §5.3, Table 6 evidence: Iterative preference alignment yields diminishing returns across rounds, suggesting that base model capability is the primary ceiling on alignment effectiveness. confidence: high relevance: medium limitations: - '- INTP covers only English and Chinese; while generalization to unseen languages is shown, coverage of low-resource languages remains weak. - The computational cost of alignment (8 H100 GPUs per run) limits accessibility for smaller labs. - Iterative alignment shows diminishing returns between rounds, suggesting the base model''s capability is the ceiling. Better foundation models will be needed. - The paper does not evaluate anti-spoofing robustness; improved intelligibility alignment may inadvertently affect spoofing resilience. - All evaluation uses the SeedTTS test sets, which are English and Chinese. No evaluation on truly out-of-distribution natural speech (e.g., radio or phone call recordings) is provided.' caveats: [] - id: 2025.acl-long.681 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: large_scale_instruction_datasets_that_span_diverse_acoustic_and_paralinguistic role: supports claim: Large-scale instruction datasets that span diverse acoustic and paralinguistic dimensions improve speech-text LLM generalization on instruction-following benchmarks beyond what smaller task-specific corpora achieve. source: §5.2, Table 3 evidence: SIFT-LLM outperforms O-ASQA-LLM (same backbone, fine-tuned on 2.7M-example OpenASQA) across all benchmarks, including 46.1% vs. 22.9% on EvalSIFT closed-ended and 57.4% vs. 45.9% on Dynamic-Superb. confidence: high relevance: medium - claim_id: instruction_fine_tuning_on_diverse_speech_tasks_reduces_specialized_performance role: complicates claim: Instruction fine-tuning on diverse speech tasks reduces specialized performance on foundational tasks relative to the pre-trained model checkpoint. source: §5.3, Table 4 evidence: SIFT-LLM WER on LibriSpeech test-clean rises from 2.5% (50K pre-training checkpoint) to 3.5% after instruction fine-tuning, consistent with behavior observed in Qwen2-Audio and its instruction fine-tuned variant. confidence: high relevance: medium - claim_id: codec_representations_that_jointly_encode_acoustic_and_semantic_content_enable role: supports claim: Codec representations that jointly encode acoustic and semantic content enable more accurate controllable speech generation than semantic-only representations. source: §5.4, Table 6 evidence: X-codec2 (fused semantic and acoustic codes) achieves QWK of 0.69 for pitch variation versus 0.15 for HuBERT codes (semantic-only) on SIFT-LLM GEN controllable generation evaluations. confidence: high relevance: low - claim_id: semantic_only_discrete_speech_representations_are_insufficient_for_reliable_speaker role: complicates claim: Semantic-only discrete speech representations are insufficient for reliable speaker-dependent controllable generation, as they primarily encode content rather than prosodic and acoustic identity. source: §5.4, §F.3 evidence: The HuBERT-code vocoder setup achieves approximately 50% gender accuracy against instruction-specified gender, compared to 95.8% for X-codec2; pitch variation QWK from original audio degrades substantially in HuBERT re-synthesis (0.11 QWK, Table 18 Appendix F.3). confidence: high relevance: low limitations: - SIFT-LLM does not achieve state-of-the-art results on foundational speech tasks; the instruction fine-tuning trades off specialized ASR and translation capability, limiting its use as a drop-in replacement for task-specialized models. - Controllable generation evaluation relies on automatically extracted acoustic features compared against instruction-specified targets, which indirectly assesses controllability but cannot measure perceptual quality or naturalness. The QWK and MAE metrics measure categorical agreement rather than human-perceived fidelity. - The LLM-as-a-judge evaluation methodology for open-ended and classification tasks introduces judge model dependency; results may shift with different judge LLMs, and the interaction between judge capability and task difficulty is not systematically studied. - Hallucination when queried about content unrelated to the input audio is noted as an open issue; attempts to mitigate it by adding off-topic instructions with higher weight degraded speech understanding performance, suggesting a robustness-capability tension. caveats: [] - id: 2025.acl-short.81 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS - evaluation architecture: - autoregressive-LM - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - transformer_encoder_decoder_multilingual_tts claims: - claim_id: long_form_training_audio_10_20_seconds_per_segment_with role: supports claim: Long-form training audio (10–20 seconds per segment) with explicit speaker identities improves zero-shot TTS quality for low-resource tonal languages compared to training on short-segment corpora. source: §3.1, §4 evidence: Long-form training audio (10–20 seconds per segment) with explicit speaker identities improves zero-shot TTS quality for low-resource tonal languages compared to training on short-segment corpora. confidence: high relevance: low - claim_id: multilingual_voice_cloning_models_such_as_xtts_v2_exhibit_architectural role: supports claim: Multilingual voice-cloning models such as XTTS-v2 exhibit architectural failure modes on short input sequences that are not corrected by data augmentation with short clips. source: §4 evidence: Multilingual voice-cloning models such as XTTS-v2 exhibit architectural failure modes on short input sequences that are not corrected by data augmentation with short clips. confidence: high relevance: high - claim_id: autoregressive_codec_language_models_vall_e_voicecraft_generalize_better_than role: supports claim: Autoregressive codec language models (VALL-E, VoiceCraft) generalize better than Tortoise-based models to short-sentence inputs in low-resource language fine-tuning. source: §4, Table 2 evidence: Autoregressive codec language models (VALL-E, VoiceCraft) generalize better than Tortoise-based models to short-sentence inputs in low-resource language fine-tuning. confidence: high relevance: high - claim_id: a_dataset_curation_pipeline_based_on_dual_asr_agreement_filtering role: supports claim: A dataset curation pipeline based on dual-ASR agreement filtering produces higher-quality transcriptions for audiobook audio than single-model transcription alone, enabling more reliable TTS training. source: §2.1 evidence: A dataset curation pipeline based on dual-ASR agreement filtering produces higher-quality transcriptions for audiobook audio than single-model transcription alone, enabling more reliable TTS training. confidence: high relevance: medium limitations: - The paper does not evaluate code-switching scenarios (mixed Vietnamese-English input), which is relevant in practice. The dataset is audiobook domain only, so speaking style coverage is narrower than general-purpose datasets. All models are fine-tuned rather than trained from scratch, which means performance is bounded by the pre-trained model's multilingual capacity. The architecture issue observed with XTTS-v2 on short sentences is identified but not resolved. The dataset is released for non-commercial use only, which limits industrial adoption. It is also unclear how the system handles tonal phonology beyond phonemizer outputs, and no ablation on augmented data proportion is presented. caveats: [] - id: 2025.findings-acl.687 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - singing - TTS architecture: - flow-matching - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_multilingual_tts - vae_multilingual_latent_models claims: - claim_id: boundary_masking_during_training_improves_zero_shot_singing_synthesis_naturalness role: supports claim: Boundary masking during training improves zero-shot singing synthesis naturalness by forcing the model to learn smooth phoneme and note transitions without relying on precise alignment annotations. source: §3.2, Table 4 evidence: BBC Encoder masks 8 tokens at each phoneme/note boundary; ablating this masking causes CMOS-Q to drop -0.36 in style transfer and -0.39 in style control, the largest single-component quality degradation. confidence: high relevance: medium - claim_id: contrastive_alignment_of_cross_modal_style_embeddings_singing_speech_natural role: supports claim: Contrastive alignment of cross-modal style embeddings (singing, speech, natural language) enables a single encoder to support style transfer, speech-to-singing, and instruction-conditioned synthesis without modality-specific architectures. source: §3.3, Table 4 evidence: Custom Audio Encoder trained with CLIP-style triplet contrastive loss achieves unified style space across modalities; removing it drops CMOS-S by -0.37 (style transfer) and CMOS-C by -0.41 (style control). confidence: high relevance: medium - claim_id: language_conditioned_mixture_of_experts_routing_in_a_flow_matching role: supports claim: Language-conditioned mixture-of-experts routing in a flow-matching transformer improves multilingual singing synthesis quality by directing language-family-specific token processing to specialised experts. source: §3.4, Table 4, Table 7 evidence: Lingual-MOE ablation (replaced with standard FFN) causes CMOS-Q to drop -0.29 in style transfer; Stylistic-MOE ablation causes CMOS-S -0.26 and CMOS-C -0.33; four experts per group is the optimal configuration before diminishing returns. confidence: high relevance: high - claim_id: f0_supervision_applied_to_intermediate_transformer_representations_improves_both_synthesis role: supports claim: F0 supervision applied to intermediate transformer representations improves both synthesis quality and style fidelity in zero-shot singing voice synthesis. source: §3.4, Table 4 evidence: Removing F0 supervision from the first block's output causes CMOS-Q -0.33 and CMOS-S -0.24 in style transfer, and CMOS-Q -0.31 and CMOS-C -0.27 in style control. confidence: high relevance: medium - claim_id: multi_level_style_control_via_natural_language_prompts_in_singing role: complicates claim: Multi-level style control via natural language prompts in singing synthesis requires costly manual annotation, limiting dataset scale and introducing labelling errors. source: §6, Appendix B evidence: Style labels (emotion, singing method, vocal range, word-level techniques) annotated by music experts at $300/hour, then converted to natural language via GPT-4o; the paper cites labelling cost and annotation errors as primary limitations constraining generalisation. confidence: high relevance: medium limitations: - 'The training and test sets share the same underlying singer pool: 30 "unseen" singers are held out from a pool drawn from the same data collection. This limits the assessment of generalisation to truly out-of-distribution singers or languages not present in training (all 9 test languages are also in training).' - Generation speed does not meet real-time industrial requirements at current inference settings, despite 25-step ODE inference; streaming inference is deferred to future work. - Multi-level style annotation requires music expert labour and is therefore not easily scalable; automatic labelling tools are noted as future work but not yet demonstrated. - The model's potential for misuse in unauthorised dubbing is acknowledged. Vocal watermarking is proposed as a mitigation but has not been implemented in this paper. caveats: [] - id: 2025.findings-acl.71 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: explicit_listening_comprehension_objectives_improve_cross_modal_understanding_in_multimodal role: supports claim: Explicit listening comprehension objectives improve cross-modal understanding in multimodal LLMs fine-tuned for spoken QA. source: §3.4, Table 3 evidence: Removing the story listening comprehension sub-task from DAMSEL causes the largest single-task performance degradation (Table 3, ablations on ASK-QA with Speech-Qwen), confirming that transcription supervision is the most critical auxiliary signal. confidence: high relevance: medium - claim_id: multi_task_auxiliary_task_design_can_substitute_for_large_scale role: supports claim: Multi-task auxiliary task design can substitute for large-scale data collection in adapting MLLMs to spoken question answering. source: §3.2, Figure 4, Table A7 evidence: DAMSEL with Speech-Qwen surpasses the prior state-of-the-art on Spoken-SQuAD using only 10% of training data (66.38 EM vs. DDNet 64.1 EM trained on 100% of data). confidence: high relevance: medium - claim_id: data_centric_multi_task_fine_tuning_yields_consistent_gains_even role: supports claim: Data-centric multi-task fine-tuning yields consistent gains even for frontier MLLMs with extensive prior multi-modal pre-training. source: §3.1 evidence: Gemini Pro fine-tuned with DAMSEL on 1% of ASK-QA improves 5.7% relative over single-task tuning; improvement persists with full data (1.6% relative), despite Gemini having seen large-scale multi-modal training data and ASK-QA being newly synthesised. confidence: high relevance: medium - claim_id: tts_synthesised_speech_datasets_have_quality_ceilings_that_limit_model role: complicates claim: TTS-synthesised speech datasets have quality ceilings that limit model generalisation to naturalistic paralinguistic variation. source: §Limitations, §B.1 evidence: The paper notes that WER-filtered ASK-QA transcriptions may not perfectly match synthesised speech, and that current TTS lacks perfect controllability for speaking rate and pitch. The dynamic evaluation pipeline also depends on TTS to regenerate spoken turns, compounding generation quality issues across multi-turn rollouts. confidence: high relevance: medium - claim_id: spoken_qa_performance_gains_from_multi_task_learning_are_largest role: refines claim: Spoken QA performance gains from multi-task learning are largest in the low-data regime and diminish as training data scales. source: §3.1, §3.3, Table 2 evidence: Gemini Pro's DAMSEL advantage over single-task tuning narrows from 16.13% relative EM improvement (1% data) to 1.7% (full data) on SD-QA, and from 5.7% to 1.6% relative on ASK-QA. confidence: high relevance: medium limitations: - Performance relies on ground-truth transcriptions as listening comprehension targets. In practice, transcriptions from Spoken-SQuAD and SD-QA are derived from ASR systems with known errors. The paper acknowledges it is unclear whether slight transcription noise improves robustness or degrades performance. - A key open question is whether the DAMSEL data generation process scales to MLLM post-training. The paper demonstrates gains with fine-tuning but notes that verifying post-training effects is computationally infeasible in this work. The dynamic multi-turn evaluation for ASK-QA also depends on an LLM-based action classifier and user simulator, introducing additional sources of evaluation variance that are not thoroughly characterised. - The model currently targets auditory semantic understanding. The paper notes that more nuanced SCA tasks (such as monitoring user frustration in task guidance) require sensitivity to different paralinguistic dimensions not addressed by the current auxiliary tasks. caveats: [] - id: 2025.findings-acl.75 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: discrete_speech_unit_sequences_can_be_compressed_into_text_like role: supports claim: Discrete speech unit sequences can be compressed into text-like representations via n-gram language modelling, providing effective text-free guidance for both cross-modal and cross-lingual learning in speech translation. source: §2.1, §3.2, Table 2 evidence: Unit language (2-gram merged mHubert units, K=3) achieves average ASR-BLEU of 21.5 on VoxPopuli across four language directions, matching recognised-text training and yielding +1.2 BLEU over the baseline that uses raw discrete units alone. confidence: high relevance: high - claim_id: combining_cross_modal_and_cross_lingual_auxiliary_objectives_in_multi role: complicates claim: Combining cross-modal and cross-lingual auxiliary objectives in multi-task sequence-to-sequence speech training can produce gradient interference that limits the gains of each individual loss. source: §2.3, §3.2, §4.3 evidence: Simultaneously applying source (L_CM) and target (L_CL) unit language losses without task prompts yields combined gains no greater than either loss alone, due to CM guidance at middle encoder layers disrupting CL representation learning in upper layers, as confirmed by attention localness analysis. confidence: high relevance: high - claim_id: learnable_task_specific_prompt_vectors_can_separate_competing_auxiliary_objectives role: supports claim: Learnable task-specific prompt vectors can separate competing auxiliary objectives within a shared Transformer encoder by enforcing distinct intermediate representations for each task. source: §2.4, §3.2, Table 2 evidence: Two prompt vectors (b_CM and b_CL) prepended at different encoder layer boundaries, with a negative MSE diversification loss, fully recover the +1.2 BLEU combined gain from unit language guidance that is otherwise lost to task interference. confidence: high relevance: medium - claim_id: sequence_length_compression_is_a_critical_enabler_of_cross_lingual role: refines claim: Sequence length compression is a critical enabler of cross-lingual alignment in textless speech translation, and n-gram-merged unit sequences provide substantially better compression than BPE pseudo-languages. source: §4.1, §4.6, Figure 4, Table 6 evidence: Unit language lengths fall consistently between raw units and character-level text across all four VoxPopuli language pairs; unit language CL training outperforms BPE pseudo-language by +0.5 BLEU average, attributed to n-gram contextual information absent in BPE merges. confidence: high relevance: high limitations: - 'The evaluation uses ASR-BLEU only: output speech is recognised by ASR and scored with BLEU against text references. Naturalness, fluency, and speaker consistency of the synthesised speech are not assessed. The paper''s own Limitations section acknowledges this gap.' - The method is validated on four European language directions (Spanish, French, English) from VoxPopuli. Generalisation to typologically distant or low-resource languages, where phonological unit distributions may differ substantially, is unexamined. Unit language construction with K=3 requires approximately 12 hours of preprocessing per 160M-unit corpus, which may limit applicability at scale. The task prompt mitigates but does not fully eliminate the CM/CL conflict; the remaining interaction between the two objectives is not resolved. caveats: [] - id: '2507.17527' published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: end_to_end_speech_to_speech_translation_trained_with_reinforcement role: supports claim: End-to-end speech-to-speech translation trained with reinforcement learning can achieve substantially lower latency than cascaded commercial systems while surpassing them in translation quality. source: §3.3, Table 1 evidence: On RealSI (zh-en), the system achieves VIP 79.5 and SVIP 67.8 at AL 5.18 seconds, versus the best commercial competitor at VIP 53.2 and SVIP 3.0 at AL 48.21. confidence: high relevance: low - claim_id: two_stage_rl_training_that_separates_fine_grained_process_rewards role: supports claim: Two-stage RL training that separates fine-grained process rewards from holistic sequence-level outcome rewards outperforms applying either strategy in isolation for streaming speech generation. source: §4.3, Table 6 evidence: The combined two-stage scheme achieves higher VIP and lower AL than both single-turn-only and multi-turn-only variants on internal benchmarks; multi-turn-only trades translation quality for latency, while single-turn-only cannot exploit global latency constraints. confidence: high relevance: low - claim_id: rl_reward_design_for_speech_generation_is_prone_to_exploitation role: complicates claim: RL reward design for speech generation is prone to exploitation when individual reward signals are applied without complementary constraints. source: §4.2, Table 5 evidence: Training with the Time Compliance Reward alone caused the model to reduce audio duration by ~35% and text token count by ~15%, yielding an 8-point BLEURT drop while satisfying the temporal constraint; adding an adversarial quality reward restored translation fidelity. confidence: high relevance: medium - claim_id: reinforcement_learning_applied_after_sft_improves_latency_more_substantially_than role: supports claim: Reinforcement learning applied after SFT improves latency more substantially than translation quality in simultaneous speech translation. source: §4.1, Tables 3, 4 evidence: RL reduces FLAL from 3.90 to 2.37 and AL from 2.82 to 2.58 on RealSI, while VIP improves more modestly from 75.1 to 79.5 compared to the SFT baseline. confidence: high relevance: low - claim_id: automated_mt_metrics_such_as_bleurt_and_comet_do_not role: complicates claim: Automated MT metrics such as BLEURT and COMET do not fully capture the quality of simultaneous speech interpretation in realistic long-form scenarios. source: §3.2, Appendix A evidence: The paper introduces SVIP as a composite human evaluation metric combining translation quality, delivery latency, speech rate, pronunciation, and fluency, arguing that automated metrics fail to assess the pragmatic requirements of live spoken interpretation. confidence: high relevance: medium limitations: - Training data, model weights, and all commercial baseline identities are proprietary. The headline results cannot be independently reproduced or verified. - Evaluation covers only Chinese-to-English and English-to-Chinese translation; generalisation to other language pairs is untested. The paper does not isolate the contribution of voice cloning quality from translation quality within the SVIP metric, making it difficult to assess how each component affects the composite score. Future work identified by the authors includes broader language coverage, improved voice personalization, and enhanced speech stability and expressiveness. caveats: [] - id: '2503.11026' published_date: "2025-07-30" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: conditioning_a_flow_matching_mel_spectrogram_generator_on_rich_multimodal role: supports claim: Conditioning a flow matching mel-spectrogram generator on rich multimodal speaker representations produces more consistent speaker identity in zero-shot cross-lingual speech synthesis than injecting a single speaker embedding at the vocoder stage. source: §5.4, Table 1 evidence: MAVFlow achieves an average 36% improvement in speaker similarity (SS) over AV2AV across four language pairs on MuAViC, using OT-CFM conditioned on x-vector speaker embeddings plus facial emotion embeddings, while AV2AV uses d-vector conditioning in the vocoder only. confidence: high relevance: high - claim_id: higher_quality_intermediate_mel_spectrogram_synthesis_propagates_benefits_to_downstream role: supports claim: Higher-quality intermediate mel-spectrogram synthesis propagates benefits to downstream talking-face generation even when the face decoder itself is unchanged. source: §5.5, Table 5 evidence: MAVFlow improves LSE-C by +0.87, LSE-D by -0.49, and FID by -0.61 relative to AV2AV on LRS3 visual evaluation, despite using the same Wav2Lip face decoder, suggesting that the mel quality bottleneck affects face sync accuracy. confidence: high relevance: medium - claim_id: visual_emotion_conditioning_is_insufficient_on_its_own_to_improve role: complicates claim: Visual emotion conditioning is insufficient on its own to improve emotion expression in synthesized speech and requires concurrent audio speaker conditioning to be effective. source: §5.6, Table 8 evidence: Adding only visual guidance to the CFM model marginally maintains speaker similarity (SS 0.056 vs 0.057 without guidance) but reduces emotion accuracy from 28.66% to 26.83% on CREMA-D; the combination of audio and visual guidance is needed to reach 36.46%. confidence: high relevance: low - claim_id: paralinguistic_and_linguistic_generation_objectives_are_compatible_in_zero_shot role: supports claim: Paralinguistic and linguistic generation objectives are compatible in zero-shot cross-lingual speech synthesis; improving speaker fidelity does not require sacrificing translation accuracy. source: §5.4, Tables 1 and 3 evidence: MAVFlow maintains competitive ASR-BLEU scores (26.97 Es-En vs 26.57 for AV2AV and 28.66–30.55 for cascaded systems) while substantially improving speaker similarity, using the same unit translation module as AV2AV. confidence: high relevance: high - claim_id: emotion_recognition_accuracy_in_synthesized_cross_lingual_speech_remains_far role: complicates claim: Emotion recognition accuracy in synthesized cross-lingual speech remains far below ground-truth levels even with multimodal conditioning, indicating that paralinguistic preservation is an unsolved challenge. source: §5.4, Table 2; §5.6, Table 7 evidence: MAVFlow achieves 36.46% audio emotion accuracy vs a GT upper bound of 81.95% on CREMA-D, even with dual audio-visual guidance; additional training on an emotion-rich dataset (CREMA-D) raises this to 51.46% but still far below GT. confidence: high relevance: high limitations: - The system relies on emotional cues from facial video alone; audio-side paralinguistics such as prosody and timbre variation are not used as emotion conditioning signals, which may limit emotion transfer when visual input is unavailable or low quality. The unit extractor and unit-to-unit translation modules are adopted unchanged from AV2AV, so improvements in semantic translation quality require addressing those upstream components separately. The Duration Length Regulator interpolates to match source duration, which may introduce length-related artifacts when source and translated speech have substantially different natural lengths. Evaluation is limited to five European languages with English as the target; generalization to typologically distant language pairs (e.g., tonal languages, right-to-left scripts) is untested. caveats: [] - id: 2025.unlp-1.11 published_date: "2025-07-31" entry_date: '2026-07-29' year: 2025 venue: UNLP task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: context_aware_sentence_level_neural_models_outperform_word_level_neural role: supports claim: Context-aware sentence-level neural models outperform word-level neural baselines for lexical stress prediction in morphologically complex languages. source: §3.2.4, Table 2 evidence: ByT5 fine-tuned on sentence-level synthetic data achieves 87.7% word-level accuracy, compared to 73.2% for the word-level Ukrainian Accentor, by incorporating broader sentence context during inference. confidence: high relevance: medium - claim_id: neural_models_do_not_surpass_dictionary_based_systems_on_contextual role: complicates claim: Neural models do not surpass dictionary-based systems on contextual stress disambiguation even when trained at the sentence level. source: §3.2.4, Table 2 evidence: ByT5 G2P achieves 58.1% ambiguous word accuracy vs. 64.3% for Ukrainian Word Stress (First); the dictionary system's POS-based disambiguation and guaranteed fallback remain advantageous for heteronym resolution. confidence: high relevance: medium - claim_id: hybrid_pipelines_combining_neural_and_dictionary_based_stress_prediction_outperform role: supports claim: Hybrid pipelines combining neural and dictionary-based stress prediction outperform either method alone for low-resource languages. source: §3.2.4, Table 2 evidence: Combining ByT5 G2P with Ukrainian Word Stress (OnAmbiguity.Skip) achieves 92.5% word-level and 52.0% sentence-level accuracy, exceeding both standalone neural (87.7%, 35.3%) and dictionary-only (88.7%, 41.5%) results. confidence: high relevance: medium - claim_id: carefully_designed_rule_based_g2p_systems_can_achieve_very_low role: supports claim: Carefully designed rule-based G2P systems can achieve very low word error rates for languages with relatively transparent orthographies, substantially outperforming naive mapping. source: §4.4, Table 4 evidence: The rule-based Ukrainian phonemizer reaches 1.23% WER on a constructed phonologically diverse dataset, vs. 48.75% WER for a naïve letter-to-phoneme baseline, with errors concentrated in abbreviations and complex consonant clusters. confidence: high relevance: high limitations: - The phonemizer does not handle abbreviations or numerical expressions, and the stress prediction model is trained entirely on automatically generated pseudo-labels with no manually verified training sentences. Both limitations constrain applicability to controlled text in standard Ukrainian orthography. - The stress model's contextual disambiguation is limited by sparse heteronym representation in the synthetic training corpus, which relies on an ASR-based annotation pipeline with inherent labelling errors. The evaluation benchmark contains only 1,026 sentences, making sentence-level accuracy estimates noisy. Neither system accounts for regional dialects or non-standard language varieties. The phonemizer operates at word level and cannot handle sentence-level phenomena such as cross-word assimilation. Full-text phonemization evaluation is deferred to future work. caveats: [] - id: 2025.ccl-1.80 published_date: "2025-08-01" entry_date: '2026-07-29' year: 2025 venue: workshop task: - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: projecting_phoneme_representations_from_two_languages_into_a_shared_latent role: supports claim: Projecting phoneme representations from two languages into a shared latent space reduces cross-lingual phoneme confusion and improves naturalness in code-switched speech synthesis. source: §3.2, Table 2 evidence: Projecting phoneme representations from two languages into a shared latent space reduces cross-lingual phoneme confusion and improves naturalness in code-switched speech synthesis. confidence: high relevance: high - claim_id: per_token_language_id_conditioning_helps_a_multilingual_codec_lm role: supports claim: Per-token language ID conditioning helps a multilingual codec LM distinguish phonetic characteristics across languages in code-switched synthesis, though its impact is smaller than that of shared phoneme representations. source: §3.3, Table 4 evidence: Per-token language ID conditioning helps a multilingual codec LM distinguish phonetic characteristics across languages in code-switched synthesis, though its impact is smaller than that of shared phoneme representations. confidence: high relevance: high - claim_id: codec_language_model_architectures_outperform_vae_based_seq2seq_systems_for role: supports claim: Codec language model architectures outperform VAE-based seq2seq systems for code-switched TTS when trained on monolingual data only. source: §4.3.2, Table 3 evidence: Codec language model architectures outperform VAE-based seq2seq systems for code-switched TTS when trained on monolingual data only. confidence: high relevance: low - claim_id: code_switched_tts_systems_can_be_trained_effectively_from_monolingual role: supports claim: Code-switched TTS systems can be trained effectively from monolingual corpora alone, without requiring real bilingual or code-switched training audio. source: §3.1, §4.1 evidence: Code-switched TTS systems can be trained effectively from monolingual corpora alone, without requiring real bilingual or code-switched training audio. confidence: high relevance: medium limitations: - '- Proprietary Lao dataset is not publicly available, limiting reproducibility. - Evaluation uses a small number of listeners (10 Lao, 10 English), raising statistical concerns. - RMSE is used as the sole objective metric; no WER, CER, or SPK-SIM reported. - The approach does not generalize beyond Lao-English without new proprietary data per language. - No streaming or real-time inference analysis. - How the model handles intra-word code-switching (vs. inter-sentence) is not evaluated.' caveats: [] - id: '2508.14049' published_date: "2025-08-05" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: large_multilingual_tts_systems_built_on_semantic_token_intermediaries_transfer role: supports claim: Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models. source: §2.1, §5.1 evidence: Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models. confidence: high relevance: high - claim_id: decoupling_the_text_to_semantic_and_semantic_to_acoustic_stages role: supports claim: Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining. source: §2, §4.1 evidence: Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining. confidence: high relevance: medium - claim_id: flow_matching_is_a_viable_replacement_for_diffusion_in_the role: supports claim: Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity. source: §2.3, §5.1 evidence: Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity. confidence: high relevance: medium - claim_id: intelligibility_in_low_resource_languages_with_limited_training_data_remains role: complicates claim: Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system. source: §5.1, Table 2 evidence: Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system. confidence: high relevance: high limitations: - The evaluation relies exclusively on WER over 10 sentences per language with no MOS, SMOS, or naturalness scores. This makes it impossible to assess audio quality, expressiveness, or speaker similarity relative to baselines — core dimensions for a TTS system. - 'English dominates the training set at 58%, which may explain strong English results but raises questions about whether true cross-lingual transfer or data dominance is responsible. The system lacks prosody and pace control conditioning in M1, which the authors flag as future work. Zero-shot speaker fidelity for M2 is acknowledged as limited compared to infilling-based approaches like Seamless. Fine-tuning introduces hallucination that requires careful intervention (freezing classification heads only), suggesting the LM component is sensitive to distribution shift. Languages with fewer training hours (Assamese: 48h, Dogri: 8h, Rajasthani: 20h) show substantially weaker results.' caveats: [] - id: '2508.04195' published_date: "2025-08-06" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: treating_paralinguistic_vocalizations_as_inline_vocabulary_tokens_in_asr_decoding role: supports claim: Treating paralinguistic vocalizations as inline vocabulary tokens in ASR decoding enables joint transcription of lexical and non-verbal content within a single unified sequence, without requiring a separate event detection module. source: §Paralinguistic Aware Speech Recognition evidence: Treating paralinguistic vocalizations as inline vocabulary tokens in ASR decoding enables joint transcription of lexical and non-verbal content within a single unified sequence, without requiring a separate event detection module. confidence: high relevance: medium - claim_id: automatically_annotated_data_at_scale_can_match_or_exceed_the role: supports claim: Automatically annotated data at scale can match or exceed the training effectiveness of smaller human-annotated sets for paralinguistic-enhanced TTS, measured by CER and perceptual quality. source: §Main Results, Table 5 evidence: Automatically annotated data at scale can match or exceed the training effectiveness of smaller human-annotated sets for paralinguistic-enhanced TTS, measured by CER and perceptual quality. confidence: high relevance: medium - claim_id: fine_tuning_zero_shot_tts_models_with_paralinguistic_vocabulary_extensions role: supports claim: Fine-tuning zero-shot TTS models with paralinguistic vocabulary extensions enables token-level control over non-verbal vocalization placement without material degradation of lexical quality or speaker similarity. source: §Paralinguistic-enhanced TTS Experiments, Table 5 evidence: Fine-tuning zero-shot TTS models with paralinguistic vocabulary extensions enables token-level control over non-verbal vocalization placement without material degradation of lexical quality or speaker similarity. confidence: high relevance: low - claim_id: listener_preference_for_para_enhanced_tts_is_strong_even_when role: supports claim: Listener preference for para-enhanced TTS is strong even when paralinguistic tag recall remains below 65%, suggesting that the mere presence of contextually appropriate non-verbal cues improves perceived naturalness. source: §Human Evaluation, Table 6 evidence: Listener preference for para-enhanced TTS is strong even when paralinguistic tag recall remains below 65%, suggesting that the mere presence of contextually appropriate non-verbal cues improves perceived naturalness. confidence: high relevance: medium limitations: - The NVSpeech corpus is restricted to Chinese (Mandarin) and draws primarily from game-style voice acting rather than spontaneous conversational speech, which may limit generalization to natural dialogue scenarios. The auto-labeled data quality depends entirely on the SenseVoice ASR model's precision, and label errors accumulate at scale without a second annotation pass. - Paralinguistic tag recall in TTS reaches only 61.9%, leaving substantial room for improvement in ensuring that specified non-verbal cues are faithfully rendered. The paper evaluates TTS on its own in-domain and open-domain splits rather than a shared benchmark, making direct comparison with other paralinguistic TTS systems (such as EmoCtrl-TTS or ELaTE) difficult. Model size and inference cost are not reported. English experiments (Appendix D) use a small consolidated dataset and a single speaker source, limiting cross-lingual generalization claims. Potential misuse risks from highly naturalistic TTS with non-verbal behaviors (deepfakes, audio manipulation) are acknowledged but not technically mitigated. caveats: [] - id: '2508.05385' published_date: "2025-08-07" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: automated_annotation_pipelines_for_non_verbal_vocalizations_can_match_or role: supports claim: Automated annotation pipelines for non-verbal vocalizations can match or exceed manually-annotated datasets in downstream NV generation and understanding tasks, while scaling at substantially lower cost. source: §5.1, Table 2; §5.2, Tables 4–5 evidence: Automated annotation pipelines for non-verbal vocalizations can match or exceed manually-annotated datasets in downstream NV generation and understanding tasks, while scaling at substantially lower cost. confidence: high relevance: medium - claim_id: positional_accuracy_of_non_verbal_tag_annotations_is_a_more role: supports claim: Positional accuracy of non-verbal tag annotations is a more important determinant of NV controllability in TTS than dataset size alone. source: §5.1, Table 2 evidence: Positional accuracy of non-verbal tag annotations is a more important determinant of NV controllability in TTS than dataset size alone. confidence: high relevance: medium - claim_id: frame_level_nv_detection_models_trained_exclusively_on_one_language role: supports claim: Frame-level NV detection models trained exclusively on one language can generalise to structurally different languages without retraining, suggesting that acoustic features of non-verbal sounds are largely language-agnostic. source: §2.2; §3 evidence: Frame-level NV detection models trained exclusively on one language can generalise to structurally different languages without retraining, suggesting that acoustic features of non-verbal sounds are largely language-agnostic. confidence: high relevance: medium - claim_id: rule_based_nv_data_augmentation_produces_measurably_worse_nv_controllability role: supports claim: Rule-based NV data augmentation produces measurably worse NV controllability in fine-tuned TTS systems compared to models trained on naturally-occurring vocalizations. source: §5.1, Table 2 evidence: Rule-based NV data augmentation produces measurably worse NV controllability in fine-tuned TTS systems compared to models trained on naturally-occurring vocalizations. confidence: high relevance: medium limitations: - The dataset is heavily skewed toward Chinese (over two-thirds of samples), not because the pipeline is language-dependent but because the crawled source material is predominantly Chinese. This imbalance means English NV performance, while competitive, may underperform on broader English benchmarks, and the paper does not provide results on out-of-domain English test sets. - The detection model is trained and evaluated on simulated test data (augmented speech + NV clips), not on naturally-occurring NV events in the wild. Whether the ~91% F1 reflects real-world performance is untested. The evaluation covers only six NV categories despite the dataset spanning ten. NVS is also relatively small (38K samples, ~131 hours) compared to comparable speech datasets used for general TTS; the observed SSIM and WER gaps relative to the base F5-TTS suggest this is a binding constraint. caveats: [] - id: '2508.07302' published_date: "2025-08-10" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts claims: - claim_id: language_agnostic_emotional_embeddings_from_pre_trained_models_can_serve role: supports claim: Language-agnostic emotional embeddings from pre-trained models can serve as a reliable cross-lingual retrieval signal for zero-shot emotion transfer in TTS. source: §III.C, §IV.C evidence: Language-agnostic emotional embeddings from pre-trained models can serve as a reliable cross-lingual retrieval signal for zero-shot emotion transfer in TTS. confidence: high relevance: high - claim_id: retrieval_augmented_prompting_reduces_foreign_accent_artefacts_in_cross_lingual role: supports claim: Retrieval-augmented prompting reduces foreign-accent artefacts in cross-lingual emotional speech synthesis more effectively than direct prosody transfer between typologically distant languages. source: §III.C, §IV.E evidence: Retrieval-augmented prompting reduces foreign-accent artefacts in cross-lingual emotional speech synthesis more effectively than direct prosody transfer between typologically distant languages. confidence: high relevance: high - claim_id: flow_matching_alignment_between_discrete_codec_tokens_and_mel_spectrograms role: supports claim: Flow-matching alignment between discrete codec tokens and mel-spectrograms improves speaker identity preservation as well as prosodic naturalness in multilingual synthesis. source: §III.B, §IV.E evidence: Flow-matching alignment between discrete codec tokens and mel-spectrograms improves speaker identity preservation as well as prosodic naturalness in multilingual synthesis. confidence: high relevance: high - claim_id: clustering_based_retrieval_strategies_over_large_emotional_speech_pools_maintain role: supports claim: Clustering-based retrieval strategies over large emotional speech pools maintain higher accuracy and lower latency than exhaustive cosine similarity search as pool size grows. source: §IV.D, Table II evidence: Clustering-based retrieval strategies over large emotional speech pools maintain higher accuracy and lower latency than exhaustive cosine similarity search as pool size grows. confidence: high relevance: low - claim_id: two_stage_fine_tuning_first_on_phonetics_then_on_expressiveness role: supports claim: Two-stage fine-tuning — first on phonetics, then on expressiveness — enables effective emotion adaptation in low-resource target languages from a strong multilingual foundation model. source: §III.D, §IV.A evidence: Two-stage fine-tuning — first on phonetics, then on expressiveness — enables effective emotion adaptation in low-resource target languages from a strong multilingual foundation model. confidence: high relevance: high limitations: - All evaluation is conducted on internal, non-public datasets with a narrow test configuration (one Chinese speaker, proprietary Thai data). There is no standard benchmark, no released code, and no cross-lab reproducibility path. Claims about emotion transfer quality are difficult to verify independently. - The evaluation covers only the Chinese-to-Thai direction. The paper claims the framework is language-agnostic, but this is stated as future work rather than demonstrated. The listener panel is small (15 raters) and the Thai subset is particularly small (5 raters), raising questions about statistical reliability. The EMOS metric used here is a custom MOS variant not directly comparable to published results elsewhere. Comparisons with Typhoon2-Audio are limited to CER only, leaving the emotional quality comparison against a strong 8B-parameter baseline unanswered. caveats: [] - id: '2508.07426' published_date: "2025-08-10" entry_date: '2026-07-29' year: 2025 venue: ASRU task: - TTS architecture: - transformer-enc-dec - GAN - VAE relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts - gan_based_multilingual_synthesis - vae_multilingual_latent_models claims: - claim_id: accent_label_quality_exerts_a_larger_effect_on_accent_synthesis role: supports claim: Accent label quality exerts a larger effect on accent synthesis fidelity than raw data volume, and filtering noisy self-reported labels consistently outperforms using them directly. source: §V-C, Table VI evidence: Accent label quality exerts a larger effect on accent synthesis fidelity than raw data volume, and filtering noisy self-reported labels consistently outperforms using them directly. confidence: high relevance: medium - claim_id: speaker_accent_disentanglement_in_low_resource_settings_benefits_from_voice role: supports claim: Speaker-accent disentanglement in low-resource settings benefits from voice conversion-based timbre augmentation, even when the augmented data introduces acoustic degradation. source: §V-D, Table VIII evidence: Speaker-accent disentanglement in low-resource settings benefits from voice conversion-based timbre augmentation, even when the augmented data introduces acoustic degradation. confidence: high relevance: low - claim_id: automatic_accent_identification_metrics_and_human_perceptual_evaluations_of_accent role: supports claim: Automatic accent identification metrics and human perceptual evaluations of accent plausibility frequently diverge, limiting the reliability of either metric alone as a benchmark. source: §V-B, §V-E evidence: Automatic accent identification metrics and human perceptual evaluations of accent plausibility frequently diverge, limiting the reliability of either metric alone as a benchmark. confidence: high relevance: medium - claim_id: large_zero_shot_tts_systems_can_produce_competitive_accented_speech role: supports claim: Large zero-shot TTS systems can produce competitive accented speech through style copying without explicit accent conditioning, often matching or exceeding dedicated accent-controllable models on naturalness. source: §V-B, Table VII evidence: Large zero-shot TTS systems can produce competitive accented speech through style copying without explicit accent conditioning, often matching or exceeding dedicated accent-controllable models on naturalness. confidence: high relevance: medium - claim_id: geolocation_based_accent_label_discovery_is_extensible_to_languages_and role: supports claim: Geolocation-based accent label discovery is extensible to languages and accent varieties without existing labeled speech, but precision varies substantially across accent regions depending on broadcast training data coverage. source: §III-A, Table III evidence: Geolocation-based accent label discovery is extensible to languages and accent varieties without existing labeled speech, but precision varies substantially across accent regions depending on broadcast training data coverage. confidence: high relevance: medium limitations: - The paper evaluates English accent synthesis only, despite XTTS-v2 supporting multiple languages. The claim that geolocation-based label discovery generalises to any accent in any language is theoretical; precision on Canada (13.3%), Scotland (9.4%), Germany (15.9%), Ireland (20.4%), and Malaysia (7.1%) is poor, limiting practical applicability to a subset of the 11 accents studied. - 'Human evaluation is expensive and was conducted for only 4 of the 11 or more accents included in training, leaving uncertain whether objective metric rankings carry over to human preference for the remaining accents. The kNN-VC augmentation results are inconsistent: human evaluators found no clear benefit for augmented versus non-augmented systems in Australian and Scottish evaluations, contradicting objective metric improvements. The WER gap between the proposed systems and CosyVoice2 (13.1% vs. 7.1%) is notable and unexplained in depth. The geolocation model was trained on radio broadcasts, which may introduce domain mismatch when applied to crowd-sourced microphone speech.' caveats: [] - id: '2508.08715' published_date: "2025-08-12" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - gan_based_multilingual_synthesis claims: - claim_id: llm_based_tts_architectures_when_fine_tuned_with_appropriate_speaker role: supports claim: LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS. source: §4.2 evidence: LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS. confidence: high relevance: high - claim_id: child_specific_vocal_characteristics_can_be_transferred_to_multilingual_tts role: supports claim: Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model. source: §2.2, §3.2 evidence: Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model. confidence: high relevance: high - claim_id: foundation_tts_models_originally_trained_on_high_resource_languages_can role: complicates claim: Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility. source: §3.1, §4.3, Table 1 evidence: Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility. confidence: high relevance: high - claim_id: intelligibility_gains_from_modern_tts_architectures_in_low_resource_settings role: complicates claim: Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited. source: §4.3, Table 1 evidence: Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited. confidence: high relevance: medium limitations: - Evaluation uses only 30 speech samples per language rated by 10 listeners each. The small scale means reported MOS differences, while large, should be treated cautiously — and no significance testing distinguishes genuine quality gaps from listener variability. All data is proprietary and not released, preventing independent replication. - The Singaporean-accented Mandarin training set contains only 1,400 utterances from a single child speaker, raising questions about voice diversity and generalisation to other child speakers. The paper does not evaluate speaker similarity to the target child voices, making it difficult to assess how faithfully the age-appropriate characteristics are reproduced. Malay and Tamil models are not evaluated against CosyBase, so it is unclear whether fine-tuning adds value beyond what a capable pretrained model already provides for those languages. All evaluation is limited to Singapore's specific linguistic and cultural context; the approach's generalisability to other low-resource child-speech settings is undemonstrated. caveats: [] - id: '2508.09702' published_date: "2025-08-13" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: quality_mismatch_between_clean_training_corpora_and_real_world_inference role: supports claim: Quality mismatch between clean training corpora and real-world inference prompts is a meaningful source of degradation in zero-shot speech generation systems. source: §1 evidence: Quality mismatch between clean training corpora and real-world inference prompts is a meaningful source of degradation in zero-shot speech generation systems. confidence: high relevance: medium - claim_id: cross_lingual_proxy_prompts_selected_by_linguistic_proximity_can_produce role: supports claim: Cross-lingual proxy prompts selected by linguistic proximity can produce lower synthesis error than native-language prompts of lower audio quality. source: §3.2, Table 3 evidence: Cross-lingual proxy prompts selected by linguistic proximity can produce lower synthesis error than native-language prompts of lower audio quality. confidence: high relevance: high - claim_id: multi_modal_annotation_that_fuses_visual_and_acoustic_signals_yields role: supports claim: Multi-modal annotation that fuses visual and acoustic signals yields substantially higher age and gender labelling accuracy than audio-only or text-only annotation strategies. source: §5.1, Table 2 evidence: Multi-modal annotation that fuses visual and acoustic signals yields substantially higher age and gender labelling accuracy than audio-only or text-only annotation strategies. confidence: high relevance: medium - claim_id: a_curated_high_quality_audio_database_outperforms_generative_face_to role: supports claim: A curated, high-quality audio database outperforms generative face-to-speech synthesis as a source of speaker prompts when audio quality is the binding constraint. source: §5.3, Table 4 evidence: A curated, high-quality audio database outperforms generative face-to-speech synthesis as a source of speaker prompts when audio quality is the binding constraint. confidence: high relevance: low - claim_id: cascaded_interruptible_similarity_retrieval_is_a_viable_approach_to_prompt role: supports claim: Cascaded, interruptible similarity retrieval is a viable approach to prompt selection under strict latency constraints without dedicated hardware. source: §4.2, §5.4, Table 5 evidence: Cascaded, interruptible similarity retrieval is a viable approach to prompt selection under strict latency constraints without dedicated hardware. confidence: high relevance: low limitations: - No subjective listening tests are reported. All quality metrics (UTMOSv2, automatic speaker similarity, Emotion2Vec emotion similarity) are automatic predictors. The relationship between these metrics and perceptual quality in the prompt selection context is not validated. - The evaluation spans only two downstream tasks (TTS and speech translation) and a narrow set of conditions. The claim that M3PDB generalises to other speech generation tasks (interactive systems, voice conversion) is stated as a future direction but not evidenced. The database construction pipeline is described as time-consuming, which limits reproducibility for groups without significant compute. Coverage of 18 languages is broader than most existing resources but still leaves many of the world's languages out. The unseen-language proxy mechanism is validated only on Italian; its reliability for more distant language pairs is unknown. caveats: [] - id: '2508.09767' published_date: "2025-08-13" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: parameter_efficient_fine_tuning_with_lora_can_restore_explicit_pronunciation role: supports claim: Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance. source: §3.2, §5.1 evidence: Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance. confidence: high relevance: high - claim_id: bpe_based_multilingual_tts_models_systematically_underperform_on_phonologically_complex role: supports claim: BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive. source: §1, §5.2 evidence: BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive. confidence: high relevance: high - claim_id: phoneme_tag_injection_via_special_tokens_can_provide_localised_pronunciation role: supports claim: Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans. source: §5.2 evidence: Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans. confidence: high relevance: medium - claim_id: speaker_similarity_in_zero_shot_tts_is_robust_to_targeted role: supports claim: Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data. source: §5.1, Table 1 evidence: Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data. confidence: high relevance: low limitations: - Evaluation uses two custom test sets authored or curated by the single researcher-author, with no publicly available benchmark comparison. The listening test crowd cohort (16 workers for MOS, 15 for accent correctness) is small for a task with high inter-rater variance. Results should be treated as promising but not yet independently validated. - The approach requires users to supply phonemic transcription and accent markup, which demands linguistic expertise most end users lack. The paper targets only standard Tokyo Japanese; dialectal variation and code-switching remain untested, as the author acknowledges. The LoRA module is trained on a single female-heavy corpus (JSUT) and 100-speaker JVS, which may limit generalisation to speaker types not well represented in that data. Applicability to other prosodically complex languages — Mandarin tones, Scandinavian pitch accent — is left for future work. caveats: [] - id: '2508.11273' published_date: "2025-08-15" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: discretizing_self_supervised_speech_features_via_k_means_produces_more role: supports claim: Discretizing self-supervised speech features via k-means produces more stable prosodic control signals than retaining continuous SSL representations in encoder-decoder TTS. source: §5.6, Table 1 evidence: Discretizing self-supervised speech features via k-means produces more stable prosodic control signals than retaining continuous SSL representations in encoder-decoder TTS. confidence: high relevance: medium - claim_id: combining_continuous_spherical_emotion_vectors_with_discrete_ssl_prosody_tokens role: supports claim: Combining continuous spherical emotion vectors with discrete SSL prosody tokens improves both naturalness and intelligibility compared to spherical emotion vectors alone. source: §5.1, §5.3, Tables 1–2 evidence: Combining continuous spherical emotion vectors with discrete SSL prosody tokens improves both naturalness and intelligibility compared to spherical emotion vectors alone. confidence: high relevance: low - claim_id: speaker_independent_prosody_conditioning_via_ssl_tokens_can_generalize_across role: supports claim: Speaker-independent prosody conditioning via SSL tokens can generalize across reference speakers with minimal degradation, enabling robust cross-speaker emotion transfer. source: §5.6, Table 1 evidence: Speaker-independent prosody conditioning via SSL tokens can generalize across reference speakers with minimal degradation, enabling robust cross-speaker emotion transfer. confidence: high relevance: low - claim_id: semantic_text_encoders_contribute_to_emotional_and_prosodic_consistency_in role: supports claim: Semantic text encoders contribute to emotional and prosodic consistency in multilingual TTS even when they do not serve as direct decoder inputs. source: §5.6, Table 1 evidence: Semantic text encoders contribute to emotional and prosodic consistency in multilingual TTS even when they do not serve as direct decoder inputs. confidence: high relevance: high limitations: - '- Evaluations are limited to single-speaker datasets in two languages, making it unclear whether EmoSSLSphere generalises to multi-speaker, low-resource, or unseen-language scenarios. - Subjective listener panels are small, and emotional authenticity is assessed primarily via AVD RMSE as a proxy rather than direct perceptual emotion ratings. - Speaker similarity (SPK-SIM) is not evaluated, making it hard to quantify speaker fidelity claims. - Cross-speaker emotion transfer is described but not formally evaluated; inference always uses same-speaker reference audio. - Separate per-language encoder instances do not scale to many-language settings without significant parameter overhead. - Integration with semi-supervised training (EmoSphere++) and extension to zero-shot speaker scenarios are listed as future work.' caveats: [] - id: interspeech-2025-0143 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: multimodal_fusion_of_data_driven_acoustic_and_word_level_linguistic role: supports claim: Multimodal fusion of data-driven acoustic and word-level linguistic embeddings outperforms unimodal and knowledge-based features for multilingual sentence mode classification. source: §4, Table 2 evidence: Multimodal fusion of data-driven acoustic and word-level linguistic embeddings outperforms unimodal and knowledge-based features for multilingual sentence mode classification. confidence: high relevance: high - claim_id: multilingual_ssl_representations_trained_on_large_corpora_can_transfer_sentence role: supports claim: Multilingual SSL representations trained on large corpora can transfer sentence-mode discriminative information across languages without language-specific training. source: §5 evidence: Multilingual SSL representations trained on large corpora can transfer sentence-mode discriminative information across languages without language-specific training. confidence: high relevance: high - claim_id: state_of_the_art_asr_systems_are_unable_to_reliably role: supports claim: State-of-the-art ASR systems are unable to reliably detect exclamatory sentence mode from speech, producing recall rates below chance level for that class. source: §4, Table 3 evidence: State-of-the-art ASR systems are unable to reliably detect exclamatory sentence mode from speech, producing recall rates below chance level for that class. confidence: high relevance: medium - claim_id: sentence_mode_prediction_performance_degrades_substantially_on_emotional_speech_due role: supports claim: Sentence mode prediction performance degrades substantially on emotional speech due to definitional overlap between exclamatory sentence mode and emotional expressiveness. source: §5, Table 4 evidence: Sentence mode prediction performance degrades substantially on emotional speech due to definitional overlap between exclamatory sentence mode and emotional expressiveness. confidence: high relevance: medium limitations: - '- No end-to-end TTS experiment; the study measures sentence mode prediction accuracy, not synthesized prosody quality. - Labels derived from punctuation marks are a proxy for sentence mode; they may not reflect actual prosodic realization (especially for audiobooks where speakers may monotonize exclamatory passages). - Only three languages evaluated; coverage of typologically diverse languages is absent. - MLP classifier is simple; more powerful sequence models may perform better. - UAR is used rather than accuracy due to class imbalance, but class sizes differ substantially across languages.' caveats: [] - id: interspeech-2025-0469 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: phoneme_based_tts_training_substantially_outperforms_character_based_multilingual_approaches role: supports claim: Phoneme-based TTS training substantially outperforms character-based multilingual approaches in intelligibility for low-resource South Asian languages, as measured by WER, CER, and subjective MOS. source: §4.2, §4.3 evidence: Phoneme-based TTS training substantially outperforms character-based multilingual approaches in intelligibility for low-resource South Asian languages, as measured by WER, CER, and subjective MOS. confidence: high relevance: high - claim_id: script_mismatch_workarounds_such_as_transliteration_introduce_cumulative_errors_that role: supports claim: Script-mismatch workarounds such as transliteration introduce cumulative errors that degrade TTS quality below what a dedicated script-native model achieves. source: §1, §5 evidence: Script-mismatch workarounds such as transliteration introduce cumulative errors that degrade TTS quality below what a dedicated script-native model achieves. confidence: high relevance: medium - claim_id: large_scale_multilingual_tts_models_fail_to_synthesize_numerals_and role: complicates claim: Large-scale multilingual TTS models fail to synthesize numerals and language-specific symbols correctly in languages whose text processing was not explicitly handled at training time, causing systematic intelligibility failures. source: §4.2, §4.3 evidence: Large-scale multilingual TTS models fail to synthesize numerals and language-specific symbols correctly in languages whose text processing was not explicitly handled at training time, causing systematic intelligibility failures. confidence: high relevance: high - claim_id: phoneme_level_evaluation_metrics_drt_mrt_expose_word_final_articulatory role: supports claim: Phoneme-level evaluation metrics (DRT/MRT) expose word-final articulatory weaknesses in TTS systems that aggregate perceptual scores like MOS do not adequately capture. source: §4.3.1 evidence: Phoneme-level evaluation metrics (DRT/MRT) expose word-final articulatory weaknesses in TTS systems that aggregate perceptual scores like MOS do not adequately capture. confidence: high relevance: low - claim_id: for_low_resource_encoder_decoder_tts_simpler_architectures_can_match role: supports claim: For low-resource encoder-decoder TTS, simpler architectures can match or exceed larger ones in transcription accuracy while offering substantially lower inference latency. source: §4.1, §4.2 evidence: For low-resource encoder-decoder TTS, simpler architectures can match or exceed larger ones in transcription accuracy while offering substantially lower inference latency. confidence: high relevance: low limitations: - The Punjabi corpus covers only the Majhi accent; other dialects (Shahpuri, etc.) are not addressed. The Tacotron 2 + WaveGlow system has an RTF of 3.54, making it unsuitable for real-time applications. The MMS comparison is limited since MMS training data details are unavailable, preventing MCD-DTW and SpeechBERTScore computation for MMS. The high MCD-DTW values even for Tacotron models on Punjabi indicate remaining difficulty with position-dependent phonetic transformations (e.g., the aspiration-to-stop neutralization in Majhi accent word-initial position). Subjective tests involved only 20 participants, which is a relatively small panel. caveats: [] - id: interspeech-2025-0762 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: recursive_psychometric_function_estimation_can_establish_a_perceptually_calibrated_mapping role: supports claim: Recursive psychometric function estimation can establish a perceptually calibrated mapping from a continuous TTS conditioning parameter to perceived accentedness, yielding at least 9 distinguishable perceptual levels for English-accented Swedish. source: §3.2, §4.2 evidence: Recursive psychometric function estimation can establish a perceptually calibrated mapping from a continuous TTS conditioning parameter to perceived accentedness, yielding at least 9 distinguishable perceptual levels for English-accented Swedish. confidence: high relevance: medium - claim_id: misalignment_between_transcription_style_and_accentedness_level_is_consistently_dispreferred role: supports claim: Misalignment between transcription style and accentedness level is consistently dispreferred by listeners, regardless of which misaligned combination is presented. source: §3.3, §4.3, Table 1 evidence: Misalignment between transcription style and accentedness level is consistently dispreferred by listeners, regardless of which misaligned combination is presented. confidence: high relevance: medium - claim_id: listener_preference_for_accentedness_in_intrasentential_english_insertions_is_insertion role: supports claim: Listener preference for accentedness in intrasentential English insertions is insertion-type-dependent rather than uniform, requiring fine-grained PEA control beyond binary Swedish/English switching. source: §3.4, §4.4, Table 1 evidence: Listener preference for accentedness in intrasentential English insertions is insertion-type-dependent rather than uniform, requiring fine-grained PEA control beyond binary Swedish/English switching. confidence: high relevance: medium - claim_id: perceptual_sensitivity_to_accent_direction_is_asymmetric_listeners_can_distinguish role: supports claim: 'Perceptual sensitivity to accent direction is asymmetric: listeners can distinguish more granular levels of English-accentedness in Swedish speech than of Swedish-accentedness in English speech.' source: §4.2 evidence: 'Perceptual sensitivity to accent direction is asymmetric: listeners can distinguish more granular levels of English-accentedness in Swedish speech than of Swedish-accentedness in English speech.' confidence: high relevance: medium limitations: - '- Single speaker; generalization to other Swedish voices or to multilingual models with many speakers is unclear. - Listener panels are small (16–32 per experiment), and the paper acknowledges insufficient power for detailed analysis of insertion-type effects. - Only one language pair (Swedish/English); the methodology could generalize to other code-switching scenarios but was not tested. - The recursive psychometric fitting procedure requires significant respondent time per EA comparison set. - The matrix result is missing one sentence due to a technical failure.' caveats: [] - id: interspeech-2025-0973 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: automatic_mos_predictors_trained_primarily_on_english_data_underperform_on role: supports claim: Automatic MOS predictors trained primarily on English data underperform on Spanish TTS, and language-specific fine-tuning provides meaningful improvement even with small datasets. source: §4.2, Table 2 evidence: Automatic MOS predictors trained primarily on English data underperform on Spanish TTS, and language-specific fine-tuning provides meaningful improvement even with small datasets. confidence: high relevance: medium - claim_id: low_level_local_acoustic_features_from_ssl_encoder_layers_are role: supports claim: Low-level local acoustic features from SSL encoder layers are more informative for naturalness prediction than higher-level contextual representations from deeper transformer blocks. source: §4.2, Figure 2 evidence: Low-level local acoustic features from SSL encoder layers are more informative for naturalness prediction than higher-level contextual representations from deeper transformer blocks. confidence: high relevance: medium - claim_id: the_scarcity_of_samples_rated_near_mos_4_0_in role: supports claim: The scarcity of samples rated near MOS 4.0 in evaluation datasets introduces a systematic prediction bias toward the dataset mean, suggesting label distribution matters as much as dataset size for MOS predictor quality. source: §4.2 evidence: The scarcity of samples rated near MOS 4.0 in evaluation datasets introduces a systematic prediction bias toward the dataset mean, suggesting label distribution matters as much as dataset size for MOS predictor quality. confidence: high relevance: low - claim_id: lightweight_downstream_models_trained_on_frozen_ssl_representations_can_achieve role: supports claim: Lightweight downstream models trained on frozen SSL representations can achieve MOS prediction performance comparable to fine-tuned specialist models, despite having fewer than half the parameters. source: §4.2, Table 2 evidence: Lightweight downstream models trained on frozen SSL representations can achieve MOS prediction performance comparable to fine-tuned specialist models, despite having fewer than half the parameters. confidence: high relevance: medium limitations: - Most audio samples received only a single rating, making inter-rater reliability estimates noisy at the individual-item level. MOS scores near 4.0 are underrepresented, creating a systematic bias in model predictions toward the dataset mean. All models exhibit this bias toward predicting mean MOS, suggesting alternative loss functions (e.g., focal loss, distribution-matching losses) may help. The geographic distribution is skewed toward Argentine Spanish, limiting generalization to other Spanish dialects. The dataset does not include modern large-scale zero-shot or LLM-based TTS systems (VALL-E, Voicebox, etc.). caveats: [] - id: interspeech-2025-1034 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: scaling_a_multi_accent_pronunciation_frontend_from_three_to_fourteen role: supports claim: Scaling a multi-accent pronunciation frontend from three to fourteen accents substantially reduces the minimum training data required to achieve ceiling-level performance in a new target accent. source: §4.2, Table 4 evidence: Scaling a multi-accent pronunciation frontend from three to fourteen accents substantially reduces the minimum training data required to achieve ceiling-level performance in a new target accent. confidence: high relevance: medium - claim_id: accent_similarity_quantified_via_levenshtein_distance_over_phoneme_sequences_weakly role: supports claim: Accent similarity, quantified via Levenshtein distance over phoneme sequences, weakly predicts the success of cross-accent pronunciation knowledge transfer, with low-similarity pairs failing at the data threshold where high- and moderate-similarity pairs succeed. source: §4.3, Table 5 evidence: Accent similarity, quantified via Levenshtein distance over phoneme sequences, weakly predicts the success of cross-accent pronunciation knowledge transfer, with low-similarity pairs failing at the data threshold where high- and moderate-similarity pairs succeed. confidence: high relevance: medium - claim_id: joint_scaling_of_target_and_non_target_accent_training_data role: supports claim: Joint scaling of target and non-target accent training data is necessary to prevent overfitting when one accent is severely underrepresented in a multi-accent model. source: §4.2 evidence: Joint scaling of target and non-target accent training data is necessary to prevent overfitting when one accent is severely underrepresented in a multi-accent model. confidence: high relevance: medium - claim_id: bootstrapped_pronunciation_data_derived_from_a_rule_based_tts_frontend role: supports claim: Bootstrapped pronunciation data derived from a rule-based TTS frontend can substitute for hand-transcribed annotations in multi-accent frontend training while maintaining high unseen word accuracy. source: §3.2, §4.1 evidence: Bootstrapped pronunciation data derived from a rule-based TTS frontend can substitute for hand-transcribed annotations in multi-accent frontend training while maintaining high unseen word accuracy. confidence: high relevance: medium limitations: - Accent similarity effect is assessed with only one example per tier (low/moderate/high); replication with additional pairs would strengthen the conclusion. The experiment uses a single target accent (EDI); generalization to non-British accents or non-English languages is not tested. Pronunciation data is bootstrapped (not real human annotation), which may understate difficulties in real-world deployment where annotations are noisy. The paper addresses only the frontend (text-to-phoneme) component of TTS; it does not evaluate full end-to-end accent TTS quality in listening tests. The ethical caveat — that easy accent building could enable appropriation or mockery — is raised but not operationalized into concrete safeguards. caveats: [] - id: interspeech-2025-1344 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: adapter_based_fine_tuning_preserves_zero_shot_tts_capability_in role: supports claim: Adapter-based fine-tuning preserves zero-shot TTS capability in multilingual models more reliably than full parameter fine-tuning on single-speaker low-resource data. source: §4.3, Table 2 evidence: Adapter-based fine-tuning preserves zero-shot TTS capability in multilingual models more reliably than full parameter fine-tuning on single-speaker low-resource data. confidence: high relevance: high - claim_id: cross_lingual_adaptation_of_tts_models_is_feasible_with_fewer role: supports claim: Cross-lingual adaptation of TTS models is feasible with fewer than 15 hours of single-speaker data when a multilingual pre-trained foundation is available. source: §4.1.1, §4.2 evidence: Cross-lingual adaptation of TTS models is feasible with fewer than 15 hours of single-speaker data when a multilingual pre-trained foundation is available. confidence: high relevance: high - claim_id: in_flow_matching_tts_fine_tuning_the_text_conditioning_module role: supports claim: In flow-matching TTS fine-tuning, the text conditioning module requires more parameters to adapt to a typologically distant language than the acoustic generation module. source: §4.4.1 evidence: In flow-matching TTS fine-tuning, the text conditioning module requires more parameters to adapt to a typologically distant language than the acoustic generation module. confidence: high relevance: medium - claim_id: low_rank_adaptation_in_dit_based_tts_models_exhibits_a role: complicates claim: Low-rank adaptation in DiT-based TTS models exhibits a rank-dependent trade-off between linguistic accuracy and speaker generalisation, with optimal rank dependent on dataset scale. source: §4.4.3 evidence: Low-rank adaptation in DiT-based TTS models exhibits a rank-dependent trade-off between linguistic accuracy and speaker generalisation, with optimal rank dependent on dataset scale. confidence: high relevance: low limitations: - Multi-speaker evaluation shows substantially higher WER (21.6%) compared to a large-scale trained system (10.9%), and the system shows hallucination artefacts (word repeating and skipping) not fully resolved by the proposed adapters. Reproducibility is limited to the Korean-specific KSS dataset; generalisation to other low-resource languages is assumed but not tested. - The paper does not address languages with non-Latin scripts that differ even more from F5-TTS's training data (e.g., Arabic, Swahili). The speaker similarity ceiling for multi-speaker scenarios remains low given that adaptation data comes from a single female speaker only. The interaction between DropPath rate and language-pair linguistic distance is not explored, leaving hyperparameter transfer uncertain. Future work identified by the authors includes more robust text conditioning and improved alignment methods to reduce hallucinations. caveats: [] - id: interspeech-2025-1550 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - VC - evaluation architecture: - transformer-enc-dec - VAE - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution - infrastructure current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts - vae_multilingual_latent_models - gan_based_multilingual_synthesis claims: - claim_id: diacritization_of_arabic_text_input_significantly_improves_tts_intelligibility_and role: supports claim: Diacritization of Arabic text input significantly improves TTS intelligibility and subjective quality in Modern Standard Arabic synthesis. source: §4.1, Table 4 evidence: VITS trained with diacritics achieved 20.67% WER versus 35.69% without, and was the only model with a statistically significant preference for the diacritized variant in pairwise preference tests with native speakers. confidence: high relevance: medium - claim_id: multi_speaker_training_improves_synthesis_quality_for_individual_target_speakers role: supports claim: Multi-speaker training improves synthesis quality for individual target speakers relative to speaker-specific training on equivalent data volume. source: §4.1, Figure 2b evidence: Multi-speaker VITS was strongly preferred over single-speaker VITS trained on the same speaker's 2.69 hours; both models were evaluated on the same speaker in a pairwise preference test. confidence: high relevance: low - claim_id: synthetic_speech_data_augmentation_can_improve_tts_intelligibility_even_when role: supports claim: Synthetic speech data augmentation can improve TTS intelligibility even when the augmenting speech is of variable quality. source: §4.1 evidence: Training VITS on all ArVoice parts including synthetic speech (Google Standard and WaveNet TTS) reduced WER by 8.2% absolute versus human-speech-only training, with pairwise preference also favoring the augmented model (44% vs 30%). confidence: high relevance: medium - claim_id: multilingual_llm_based_tts_systems_do_not_reliably_transfer_to role: complicates claim: Multilingual LLM-based TTS systems do not reliably transfer to Arabic without adequate language-specific training data or adaptation, and can produce unintelligible output despite fine-tuning. source: §4.1, Table 4 evidence: Fish-Speech, a dual-autoregressive LLM-based multilingual system, achieved WER above 100% on Arabic after fine-tuning on ArVoice human speech, and was excluded from subjective evaluation due to confirmed low intelligibility. confidence: high relevance: high limitations: - The professionally recorded portions of ArVoice (Parts 1 and 2) are not freely downloadable; access requires signing a formal Data Usage Agreement. Only the ASC-derived and synthetic portions are available under CC BY 4.0. This limits reproducibility for experiments requiring the full human-voice data. - Human speech totals approximately 10 hours across 7 speakers, which is sufficient for fine-tuning but constraining for zero-shot multi-speaker TTS or large-scale model training. Coverage is restricted to Modern Standard Arabic; dialectal varieties are not represented, leaving adaptation to Egyptian, Gulf, or Moroccan Arabic as an open problem. Each preference test uses 25 samples evaluated by at least 10 raters per pair, which is adequate for detecting large differences but may lack power for subtle distinctions. The evaluation of VC systems relies on automated speaker verification similarity scores rather than human perceptual evaluation of voice identity. caveats: [] - id: interspeech-2025-2031 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: for_zero_shot_tts_across_language_families_assigning_g2p_rules role: supports claim: For zero-shot TTS across language families, assigning G2P rules by phonotactic family rather than script family produces substantially more natural and intelligible synthesis. source: §5.3, Tables 1, 2 evidence: Sanskrit synthesis with Dravidian rules (Kannada system, MOS 4.12, WER 4.6%) substantially outperforms synthesis with Indo-Aryan rules (Hindi system, MOS 3.25, WER 28.0%), because schwa deletion applied via IA rules violates Sanskrit's phonotactics despite its Devanagari script. confidence: high relevance: high - claim_id: a_shared_phoneme_inventory_extended_via_phonetic_substitution_can_enable role: supports claim: A shared phoneme inventory extended via phonetic substitution can enable zero-shot synthesis for target languages with sounds absent from all source training sets. source: §4.2 evidence: The CLS was extended to represent nukta-diacritic consonants and borrowed aspiration in Kurukh by mapping them to nearest-neighbour base characters, enabling intelligible synthesis of Kurukh without any Kurukh training data. confidence: high relevance: high - claim_id: dialect_specific_zero_shot_tts_requires_matching_the_source_synthesiser role: supports claim: Dialect-specific zero-shot TTS requires matching the source synthesiser to the dialect's phonotactic affiliation, not its surface script or nominal language grouping. source: §5.3, Table 1 evidence: 'Two Konkani dialects yield opposite synthesiser preferences: Maharashtrian Konkani (IA phonotactics) is best with Marathi at MOS 3.64, while Canara Konkani (DR phonotactics) is best with Kannada at MOS 3.34, despite both dialects being classified under the same language label.' confidence: high relevance: high - claim_id: cross_family_zero_shot_synthesis_degrades_for_target_languages_with role: complicates claim: Cross-family zero-shot synthesis degrades for target languages with phonological features structurally absent from all available source synthesisers. source: §5.3 evidence: Kurukh synthesis struggles specifically with glottal stops across both Hindi and Kannada systems, and question intonation contours were misrendered as assertions in some utterances, because neither source language encodes these features. confidence: high relevance: medium - claim_id: linguistic_proximity_for_zero_shot_tts_transfer_is_captured_more role: refines claim: Linguistic proximity for zero-shot TTS transfer is captured more precisely by phonotactic family than by script family or geographic proximity. source: §5.3, Tables 1, 3 evidence: 'Sanskrit belongs to the IA family but achieves lower MOS with IA synthesisers (3.25) than with DR synthesisers (4.12), and MCD scores independently confirm the same pattern (IA: 8.48 vs. DR: 7.15/6.94). Phonotactic family membership predicts transfer quality more reliably than script or family taxonomy.' confidence: high relevance: medium limitations: - The evaluation relies exclusively on four monolingual source synthesisers (Hindi, Kannada, Marathi, Telugu), and the method is not tested against any large multilingual TTS baseline that might implicitly have learned cross-family phonotactics from scale. For Kurukh, the ground truth audio is Bible recordings with faint background music (cleaned with VoiceFixer), which may depress the GT MOS and make the synthesised system comparisons harder to interpret. The approach requires linguistic expertise to correctly classify the phonotactic family of a new target language, which limits its applicability as an automated pipeline for new low-resource languages. - The anomalous MCD result for Maharashtrian Konkani (higher distortion with Marathi despite better naturalness MOS) is unresolved and may indicate that MCD is less reliable for measuring synthesis quality under phonotactic family mismatch conditions. caveats: [] - id: interspeech-2025-2043 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - VC architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: training_free_voice_conversion_based_on_distribution_matching_in_self role: complicates claim: Training-free voice conversion based on distribution matching in self-supervised embedding subspaces can achieve speaker similarity and content preservation comparable to trained codec-based systems when reference audio is limited to a few seconds. source: §4.4, Table 2 evidence: Training-free voice conversion based on distribution matching in self-supervised embedding subspaces can achieve speaker similarity and content preservation comparable to trained codec-based systems when reference audio is limited to a few seconds. confidence: high relevance: low - claim_id: the_nearest_neighbour_approach_in_knn_style_voice_conversion_degrades role: supports claim: The nearest-neighbour approach in kNN-style voice conversion degrades significantly in cross-lingual settings because phoneme-level proximity in the embedding space conflates content with language-specific pronunciation. source: §1, §4.4, Table 3 evidence: The nearest-neighbour approach in kNN-style voice conversion degrades significantly in cross-lingual settings because phoneme-level proximity in the embedding space conflates content with language-specific pronunciation. confidence: high relevance: high - claim_id: non_uniform_variance_structure_in_self_supervised_speech_representations_makes role: supports claim: Non-uniform variance structure in self-supervised speech representations makes global optimal transport maps suboptimal; factorizing the embedding space by variance before applying transport improves both content preservation and numerical stability. source: §3 evidence: Non-uniform variance structure in self-supervised speech representations makes global optimal transport maps suboptimal; factorizing the embedding space by variance before applying transport improves both content preservation and numerical stability. confidence: high relevance: medium - claim_id: a_content_speaker_similarity_trade_off_is_inherent_to_wavlm role: complicates claim: A content-speaker-similarity trade-off is inherent to WavLM-based voice conversion and can be navigated via the block-size hyperparameter of a factorized transport scheme. source: §4.4, Table 2 evidence: A content-speaker-similarity trade-off is inherent to WavLM-based voice conversion and can be navigated via the block-size hyperparameter of a factorized transport scheme. confidence: high relevance: low limitations: - The Gaussian assumption underlying MKL-VC is specific to WavLM-Large embeddings, as demonstrated empirically. The authors explicitly note that for any new encoder the assumption must be verified from scratch, making the method non-portable without additional analysis effort. - The method is evaluated only against objective metrics (WER, CER, X-vector cosine similarity) on LibriSpeech and FLEURS, and a small subjective ranking with six experts. No MOS or MUSHRA scores are reported, and the subjective evaluation is a ranking rather than an absolute naturalness measure, making it difficult to position MKL-VC on the standard VC evaluation scale. - Diff-VC was evaluated on only 855 of 7800 samples due to compute constraints; its baseline number may therefore be unreliable for fair comparison. SinkVC is a re-implementation rather than the official release, introducing uncertainty about whether the reported gap to MKL-VC reflects the method or the reimplementation fidelity. - No code or demo is linked in the paper; reproducibility depends on the authors releasing code. caveats: [] - id: interspeech-2025-2573 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: - flow-matching - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - infrastructure current_role: active_evidence method_family: - flow_matching_multilingual_tts - gan_based_multilingual_synthesis claims: - claim_id: accurate_grapheme_to_phoneme_conversion_is_a_critical_prerequisite_for role: supports claim: Accurate grapheme-to-phoneme conversion is a critical prerequisite for TTS quality in morphologically complex languages, with phonemizer errors directly degrading perceived naturalness and pronunciation. source: §4.2, Table 2 evidence: The modified espeak-ng phonemizer reduced mispronounced words by 38% (157 to 97) and improved naturalness MOS by 0.17 points for Egyptian dialect; objective MCD scores were unchanged, confirming the perceptual significance of phonemizer accuracy beyond spectrogram-level metrics. confidence: high relevance: high - claim_id: diacritization_vowelization_of_arabic_text_is_essential_for_intelligible_tts role: supports claim: Diacritization (vowelization) of Arabic text is essential for intelligible TTS; training without vowels causes consonant clustering and robotic-sounding output. source: §4.3, Table 3 evidence: MSA naturalness MOS dropped from 3.21 (vowelized) to 2.67 (unvowelized) and pronunciation from 3.95 to 2.53; listening revealed that missing vowels caused consonant clustering that reduced intelligibility and produced unnatural speech. confidence: high relevance: medium - claim_id: code_switching_tts_is_more_difficult_than_monolingual_tts_even role: complicates claim: Code-switching TTS is more difficult than monolingual TTS even when both use the same acoustic model and vocoder trained on equivalent data quantities. source: §4.2, Table 2 evidence: CS pronunciation MOS (4.23 with modified phonemizer) is lower than Egyptian (4.57) and MSA (4.27) despite identical training duration and model architecture, with the phrase-level language-switching strategy introducing boundary artefacts not present in monolingual conditions. confidence: high relevance: medium - claim_id: recording_the_same_speaker_across_multiple_dialects_and_an_l2 role: supports claim: Recording the same speaker across multiple dialects and an L2 language enables controlled disentanglement of speaker identity from dialect-specific acoustic characteristics in TTS datasets. source: §2.2 evidence: SawtArabi records a native Egyptian speaker in EGY, MSA, English, and CS conditions, allowing comparison of L2-speaking characteristics against native dialect production within a constant speaker identity. confidence: high relevance: high limitations: - The corpus is single-speaker (one male voice talent), meaning baseline systems cannot generalise to multi-speaker Arabic TTS and MOS comparisons are specific to this speaker's voice characteristics. Reproducibility of baseline MOS scores by external labs depends on access to similar listener pools proficient in Egyptian Arabic. - The corpus size (4 hours total, 1 hour per condition) is small relative to modern TTS training norms. Matcha-TTS baselines trained on 45 minutes of speech per dialect are unlikely to achieve the quality attainable with larger corpora, which limits the benchmark's utility for measuring state-of-the-art systems. The paper does not include multi-speaker baselines or zero-shot TTS evaluation, leaving open how well the corpus scales to speaker-adaptive or large-model settings. Future work plans to expand to additional Arabic dialects but does not address multi-speaker coverage. caveats: [] - id: interspeech-2025-gourav25_interspeech published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - diffusion - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - diffusion_multilingual_tts - gan_based_multilingual_synthesis claims: - claim_id: inference_time_g2p_modification_can_enable_code_mixed_speech_synthesis role: supports claim: Inference-time G2P modification can enable code-mixed speech synthesis from a multilingual TTS model without retraining or fine-tuning the acoustic model. source: §2 evidence: The authors add a language-detection and chunking layer at the espeak-ng phonemizer; the StyleTTS2 acoustic model and iSTFTNet vocoder are unchanged, and the code-mixed system achieves A-MOS 4.12 matching the monolingual baseline. confidence: high relevance: high - claim_id: multilingual_tts_models_trained_on_diverse_language_data_can_maintain role: supports claim: Multilingual TTS models trained on diverse language data can maintain automated naturalness scores comparable to single-language models when handling code-mixed input. source: §3, Table 1 evidence: Code-Mix StyleTTS2 achieves A-MOS 4.12 ± 0.09 on code-mixed English-Hindi/Spanish/Portuguese inputs, matching monolingual StyleTTS2 at 4.12 ± 0.08 and outperforming VITS (3.69) and YourTTS (2.35). confidence: high relevance: high - claim_id: code_mixed_tts_evaluation_lacks_standardized_benchmarks_and_relies_primarily role: complicates claim: Code-mixed TTS evaluation lacks standardized benchmarks and relies primarily on automated quality metrics rather than human perceptual studies, limiting comparability across systems. source: §3 evidence: The evaluation uses only NISQA A-MOS on a custom 250-utterance dataset with no human listener panel; no comparison to dedicated code-switching TTS baselines from prior work is provided. confidence: high relevance: low - claim_id: practical_code_mixed_tts_systems_are_constrained_to_language_pairs role: complicates claim: Practical code-mixed TTS systems are constrained to language pairs involving a dominant pivot language, limiting generalization to non-English language combinations. source: §4 evidence: The proposed chunking and phonemization pipeline requires English as one of the two input languages; the paper identifies extension to non-English pairs and romanized mixed scripts (Hinglish, Portunish) as explicit future work. confidence: high relevance: medium limitations: - Evaluation uses automated MOS (NISQA) on a custom 250-utterance proprietary test set with CPU inference only. No human listening panel was conducted, and no comparison to existing code-switching TTS systems is made. Results cannot be directly compared to benchmarks in the multilingual TTS literature. - The language-chunking approach is limited to at most two simultaneous languages with English as a mandatory component, ruling out all-native-language pairs. The system does not handle romanized code-mixing (e.g., Hindi written in Latin script) or text that mixes scripts without explicit language boundaries. The extent to which inference time degrades at GPU throughput versus CPU is not reported, and the model's code and weights are not released, making independent replication difficult. caveats: [] - id: interspeech-2025-raju25_interspeech published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - flow_matching_multilingual_tts claims: - claim_id: multilingual_adaptation_of_flow_matching_tts_through_vocabulary_expansion_and role: supports claim: Multilingual adaptation of flow-matching TTS through vocabulary expansion and phoneme-based tokenisation can extend zero-shot speaker preservation to typologically diverse language families not covered by the base model. source: §2.2.2 evidence: F5-TTS fine-tuned on 1,626 hours across English and 9 Indian languages with an expanded vocabulary and phoneme-based tokenisation achieves zero-shot speaker-conditioned synthesis across all supported languages. confidence: high relevance: high - claim_id: end_to_end_automated_dubbing_pipelines_integrating_asr_machine_translation role: supports claim: End-to-end automated dubbing pipelines integrating ASR, machine translation, and zero-shot TTS can substantially reduce manual effort in cross-lingual content localisation. source: §2.1, §3 evidence: The platform automates six sequential stages from audio extraction to output re-encoding, supports 9 Indian languages, and has been adopted by content creators and educators without requiring specialist voice talent. confidence: high relevance: high - claim_id: duration_alignment_for_lip_synchronisation_in_automated_dubbing_requires_a role: complicates claim: Duration alignment for lip-synchronisation in automated dubbing requires a user-controlled trade-off between temporal precision and naturalness that cannot be resolved automatically. source: §2.3 evidence: Duration alignment is a toggleable feature; when active it constrains synthesis duration to source timestamps, potentially affecting naturalness; when disabled, utterances are produced sequentially with a fixed 1-second pause between them. confidence: high relevance: medium limitations: - 'No quantitative evaluation is reported, making it impossible to verify speaker similarity, naturalness, or ASR accuracy in the dubbed output. The paper does not compare against any TTS or dubbing baseline. Coverage is uneven: the in-house training data covers only six of the ten supported languages, and the paper does not break out performance by language. Cross-lingual expressivity is flagged as a limitation by the authors, with the acknowledgement that nuanced prosodic transfer across typologically distant language pairs remains unsolved.' caveats: [] - id: '2508.15442' published_date: "2025-08-21" entry_date: '2026-07-29' year: 2025 venue: EMNLP task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models claims: - claim_id: token_level_entropy_is_a_statistically_significant_predictor_of_hallucination role: supports claim: Token-level entropy is a statistically significant predictor of hallucination in autoregressive LM-based TTS, enabling unsupervised detection of likely error regions before synthesis is complete. source: §2.1, §2.2 evidence: Token-level entropy is a statistically significant predictor of hallucination in autoregressive LM-based TTS, enabling unsupervised detection of likely error regions before synthesis is complete. confidence: high relevance: medium - claim_id: reformulating_autoregressive_speech_token_generation_as_trajectory_flow_optimization_via role: supports claim: Reformulating autoregressive speech token generation as trajectory flow optimization via GFlowNets enables reward-proportional sampling that reduces hallucination without modifying the inference procedure. source: §3.1, §3.2 evidence: Reformulating autoregressive speech token generation as trajectory flow optimization via GFlowNets enables reward-proportional sampling that reduces hallucination without modifying the inference procedure. confidence: high relevance: medium - claim_id: an_intrinsic_reward_derived_from_the_backbone_lm_s_own role: supports claim: An intrinsic reward derived from the backbone LM's own probability distribution is sufficient to drive distribution alignment in post-training, without requiring human annotations or an external reward model. source: §3.2.2 evidence: An intrinsic reward derived from the backbone LM's own probability distribution is sufficient to drive distribution alignment in post-training, without requiring human annotations or an external reward model. confidence: high relevance: medium - claim_id: fine_grained_subtrajectory_optimization_is_essential_for_hallucination_suppression_in role: supports claim: Fine-grained subtrajectory optimization is essential for hallucination suppression in long and complex utterances; degrading to trajectory-level balance significantly reduces the error reduction. source: §4.4.5, Table 2 evidence: Fine-grained subtrajectory optimization is essential for hallucination suppression in long and complex utterances; degrading to trajectory-level balance significantly reduces the error reduction. confidence: high relevance: medium - claim_id: post_training_alignment_for_hallucination_suppression_in_one_language_generalizes role: supports claim: Post-training alignment for hallucination suppression in one language generalizes to unseen languages, with cross-lingual models retaining meaningful gains over the untuned baseline. source: §4.4.4, Table 1 evidence: Post-training alignment for hallucination suppression in one language generalizes to unseen languages, with cross-lingual models retaining meaningful gains over the untuned baseline. confidence: high relevance: high limitations: - '- Evaluated on a single backbone (CosyVoice 2); generalization to other LM-based TTS architectures (VALL-E, BASE TTS, LLaSA) is asserted but not demonstrated. - The uncertainty-hallucination correlation explains only part of hallucination causes (correlation 0.636–0.649); prosody errors show no consistent uncertainty pattern and are not effectively addressed. - The internal reward assumes the backbone LM assigns higher probability to higher-quality sequences — an assumption the authors acknowledge is a hypothesis. - Training still requires H100 GPU access (~70 GB VRAM per GPU), which is non-trivial despite the small data requirement. - Reward temperature is a sensitive hyperparameter; the sweet spot (0.825) was found empirically and may differ for other backbones or domains. - No subjective listening evaluation (MOS/MUSHRA) is reported; quality improvements are measured entirely through ASR-based error rates and the automatic UTMOS predictor.' caveats: [] - id: '2508.17494' published_date: "2025-08-24" entry_date: '2026-07-29' year: 2025 venue: workshop task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: cascaded_task_decomposition_separating_structural_tag_prediction_from_numerical_parameter role: supports claim: Cascaded task decomposition — separating structural tag prediction from numerical parameter regression — substantially outperforms joint LLM generation for SSML-based prosody control. source: §4.4, §5.4, Table 4, Table 5 evidence: Cascaded task decomposition — separating structural tag prediction from numerical parameter regression — substantially outperforms joint LLM generation for SSML-based prosody control. confidence: high relevance: low - claim_id: prompt_only_llms_zero_shot_and_few_shot_systematically_under role: supports claim: Prompt-only LLMs (zero-shot and few-shot) systematically under-generate prosodic markup tags relative to gold annotations, and this failure persists across architectures and scales. source: §5.3, Figure 3 evidence: Prompt-only LLMs (zero-shot and few-shot) systematically under-generate prosodic markup tags relative to gold annotations, and this failure persists across architectures and scales. confidence: high relevance: medium - claim_id: prosody_enhancement_via_ssml_yields_substantial_perceptual_gains_over_neutral role: supports claim: Prosody enhancement via SSML yields substantial perceptual gains over neutral commercial TTS voices, even when the underlying synthesiser is not retrained. source: §5.1 evidence: Prosody enhancement via SSML yields substantial perceptual gains over neutral commercial TTS voices, even when the underlying synthesiser is not retrained. confidence: high relevance: low - claim_id: french_tts_prosody_normalised_relative_to_a_synthetic_baseline_captures role: supports claim: French TTS prosody normalised relative to a synthetic baseline captures linguistically meaningful patterns — phrase-final pitch rises, deliberate pacing — without requiring manual annotation. source: §3, Appendix A evidence: French TTS prosody normalised relative to a synthetic baseline captures linguistically meaningful patterns — phrase-final pitch rises, deliberate pacing — without requiring manual annotation. confidence: high relevance: low limitations: - The entire pipeline is calibrated and evaluated against a single commercial TTS voice (Azure Henri, fr-FR). SSML tag semantics — the acoustic realisation of percentage pitch and rate adjustments — are implementation-dependent and voice-dependent. Transfer to any other engine or voice requires voice-specific recalibration, limiting the method's out-of-the-box generalisability. - The dataset is 14 hours of proprietary French podcasts; generalisation to other French domains, other speech styles (spontaneous, informal, unpunctuated text), or other languages is unvalidated. The paper's pipeline assumes that punctuation and syntactic cues reliably predict prosodic boundaries — an assumption that breaks down for social media text or transcribed spontaneous speech. Fine-tuning each Qwen 2.5-7B stage requires approximately 15 GB of GPU memory at 4-bit quantisation, which constrains deployment in low-resource settings. The perceptual test involved 18 listeners, a sample size sufficient for statistical significance but small for robust effect-size estimation across listener backgrounds. - An open question is whether a single end-to-end model trained with a combined structural and regression loss could match or exceed the two-stage pipeline — the authors note this as future work. caveats: [] - id: '2508.18006' published_date: "2025-08-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - gan_based_multilingual_synthesis claims: - claim_id: adapter_based_fine_tuning_prevents_catastrophic_forgetting_in_tts_models role: supports claim: Adapter-based fine-tuning prevents catastrophic forgetting in TTS models while achieving better naturalness and speaker similarity than full fine-tuning. source: §4.3, Table 1; §4.4, Table 2 evidence: Adapter-based fine-tuning prevents catastrophic forgetting in TTS models while achieving better naturalness and speaker similarity than full fine-tuning. confidence: high relevance: low - claim_id: in_gan_based_tts_language_agnostic_speaker_representations_are_encoded role: supports claim: In GAN-based TTS, language-agnostic speaker representations are encoded primarily in the vocoder rather than the acoustic model, as evidenced by the outsized impact of vocoder-only adapters on accent nativeness. source: §4.4, Table 2 evidence: In GAN-based TTS, language-agnostic speaker representations are encoded primarily in the vocoder rather than the acoustic model, as evidenced by the outsized impact of vocoder-only adapters on accent nativeness. confidence: high relevance: low - claim_id: the_number_and_diversity_of_speakers_in_target_language_pre role: supports claim: The number and diversity of speakers in target-language pre-training data is a critical prerequisite for high-quality cross-lingual speaker adaptation; single-speaker pre-training leads to severe degradation. source: §4.4, Table 2 evidence: The number and diversity of speakers in target-language pre-training data is a critical prerequisite for high-quality cross-lingual speaker adaptation; single-speaker pre-training leads to severe degradation. confidence: high relevance: high - claim_id: phoneme_substitution_rate_derived_from_a_mispronunciation_detection_model_provides role: supports claim: Phoneme substitution rate, derived from a mispronunciation detection model, provides a reliable and reproducible objective proxy for accent nativeness in cross-lingual TTS, correlating well with human MUSHRA accent judgements. source: §3.3 evidence: Phoneme substitution rate, derived from a mispronunciation detection model, provides a reliable and reproducible objective proxy for accent nativeness in cross-lingual TTS, correlating well with human MUSHRA accent judgements. confidence: high relevance: high limitations: - All evaluations are purely objective — no subjective listening test (MOS, MUSHRA) is reported for the proposed systems. The NORESQA-MOS (TorchAudio-SQUIM) values used as "MOS" are automatic predictions, not human ratings; the paper's decision to discard subjective evaluation limits the strength of quality claims. - The study is restricted to two languages (British English and Castilian Spanish) and a single source speaker per adaptation direction. It is unclear whether the findings generalise to phonologically more distant language pairs, accents, or speakers with more unusual voice characteristics. The backbone model is proprietary and not publicly released, limiting reproducibility. Adapter hyperparameters (bottleneck dimension 16, kernel sizes [3, 5, 3]) were chosen via early experiments but no systematic search is reported. caveats: [] - id: '2509.01391' published_date: "2025-09-01" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - transformer_encoder_decoder_multilingual_tts claims: - claim_id: ssl_derived_discrete_tokens_can_substitute_phoneme_sequences_as_input role: supports claim: SSL-derived discrete tokens can substitute phoneme sequences as input representations in a TTS front-end without a substantial naturalness penalty in automatic evaluation. source: §V.B, Table II evidence: SSL-derived discrete tokens can substitute phoneme sequences as input representations in a TTS front-end without a substantial naturalness penalty in automatic evaluation. confidence: high relevance: low - claim_id: g2p_free_tts_pipelines_that_learn_text_to_token_mappings role: supports claim: G2P-free TTS pipelines that learn text-to-token mappings from paired speech data avoid the language-specific resource burden of phoneme dictionaries and morphological analysers. source: §I, §III evidence: G2P-free TTS pipelines that learn text-to-token mappings from paired speech data avoid the language-specific resource burden of phoneme dictionaries and morphological analysers. confidence: high relevance: high - claim_id: in_an_ssl_token_based_tts_pipeline_the_spectral_predictor role: supports claim: In an SSL-token-based TTS pipeline, the spectral predictor contributes more to naturalness differences than the text-to-token mapping stage. source: §V.B, Table II evidence: In an SSL-token-based TTS pipeline, the spectral predictor contributes more to naturalness differences than the text-to-token mapping stage. confidence: high relevance: medium - claim_id: ssl_based_speech_representations_preserve_sufficient_acoustic_quality_through_a role: supports claim: SSL-based speech representations preserve sufficient acoustic quality through a discretise-then-synthesise pipeline to remain competitive with G2P-derived representations on codec-style quality metrics. source: §V.C, Table II evidence: SSL-based speech representations preserve sufficient acoustic quality through a discretise-then-synthesise pipeline to remain competitive with G2P-derived representations on codec-style quality metrics. confidence: high relevance: high limitations: - All evaluations use automatic metrics only (UTMOS, CER, WARP-Q, SDR) on 100 utterances from a single speaker subset of JVS. No subjective MOS or preference tests are reported; the conclusions about naturalness parity are therefore tentative. - The system is demonstrated exclusively on Japanese and does not yet extend to multilingual settings. The T5 tokenizer is language-specific (tohoku-BERTv3), which the authors acknowledge as a barrier to multilingual scalability. Future directions involve BPE tokenizers (mT5, ByT5) to reduce this dependency. The oracle's counter-intuitive lower UTMOS than the proposed system is left unexplained and may indicate an interaction between SSL token sequence statistics and FastSpeech 2's duration predictor. Individual contribution analysis of duration, pitch, and accent inputs to FastSpeech 2 is identified as missing. caveats: [] - id: '2509.04702' published_date: "2025-09-04" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - SCA architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: constructing_large_scale_conversational_speech_corpora_with_fine_grained_speaker role: supports claim: Constructing large-scale conversational speech corpora with fine-grained speaker diarization and word-level timestamps is feasible from publicly available broadcast sources when specialized alignment pipelines are used in place of general-purpose ASR. source: §2.4, §3 evidence: OleSpeech-IV Tier IV assembles 5,000+ hours from English podcasts, talk shows, and teleconferences with human-sourced speaker labels and Olign-produced word-level timestamps and confidence scores. confidence: high relevance: low - claim_id: end_to_end_attention_and_ctc_based_speech_aligners_introduce role: complicates claim: End-to-end attention- and CTC-based speech aligners introduce systematic boundary errors and overconfident posterior scores that degrade the reliability of automatically annotated training data. source: §3.1.1.2, §3.1.2 evidence: Whisper-based E2E aligners produce boundary smearing (segment endings placed at the onset of the next word) and assign high posterior probabilities (>0.8) to misrecognized words; Olign assigns zero duration to hallucinated insertions and provides calibrated per-word confidence scores. confidence: high relevance: medium - claim_id: speaker_overlap_in_naturalistic_multi_party_speech_remains_an_open role: complicates claim: Speaker overlap in naturalistic multi-party speech remains an open challenge for current alignment pipelines, with quantitative accuracy benchmarks largely unreported. source: §3.1.1.3 evidence: Olign supports per-speaker word-level timestamps in overlapping segments, but the paper explicitly notes that benchmarking for this capability is ongoing and no numerical accuracy figures are available. confidence: high relevance: low - claim_id: calibrated_word_level_confidence_scores_in_speech_annotation_pipelines_can role: supports claim: Calibrated word-level confidence scores in speech annotation pipelines can reduce human review effort by directing annotators to mid-confidence ambiguous segments rather than requiring complete audio re-examination. source: §3.1.2 evidence: The Olign pipeline provides per-word confidence scores that highlight words likely to contain errors, enabling annotators to focus effort without reviewing every word in a recording. confidence: high relevance: medium limitations: - Olign, the central pipeline component, is proprietary and not open-sourced, making the data curation methodology unreproducible. No quantitative accuracy benchmarks for Olign are provided; the paper explicitly states these are work in progress. - The open-source release covers only 100 of the 5,000+ hours in the Tier IV collection, limiting reproducibility of experiments at scale. The paper does not evaluate any downstream speech generation or understanding model trained on OleSpeech-IV, so the practical utility of the dataset for TTS or spoken conversational agent training remains undemonstrated. The dataset currently supports full Olign alignment only for English; Spanish, Japanese, and Mandarin support is listed as under development. caveats: [] - id: '2509.05863' published_date: "2025-09-06" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: preference_based_alignment_dpo_improves_intelligibility_in_multilingual_autoregressive_tts role: supports claim: Preference-based alignment (DPO) improves intelligibility in multilingual autoregressive TTS systems beyond what supervised fine-tuning achieves. source: §4.3, §5.2, Table 2, Table 3 evidence: LatinX (DPO) reduces WER across nearly all 30 cross-lingual language pairs compared to the supervised fine-tuned baseline, and outperforms XTTSv2 in most pairs; Romanian-source conditions show particularly large gains (e.g., ro-to-es at 0.45% WER). confidence: high relevance: high - claim_id: automated_speaker_similarity_metrics_based_on_speaker_encoder_embeddings_do role: complicates claim: Automated speaker similarity metrics based on speaker encoder embeddings do not reliably reflect human perceptual judgments of voice identity in zero-shot TTS. source: §5.2, §6, Table 4, Table 5 evidence: XTTSv2 achieves higher Sim-O scores than both LatinX models, yet human evaluators strongly prefer LatinX speaker similarity (SMOS 3.63/3.54 vs. 3.24); the paper explicitly flags this as a divergence between objective and subjective evaluation. confidence: high relevance: low - claim_id: dpo_alignment_in_tts_involves_a_trade_off_optimizing_for role: complicates claim: 'DPO alignment in TTS involves a trade-off: optimizing for intelligibility and objective similarity can reduce naturalness MOS and, in some language conditions, perceptual similarity relative to the fine-tuned baseline.' source: §5.2, §6, Table 5, Table 6 evidence: LatinX (DPO) improves WER and Sim-E over the fine-tuned model but achieves lower average MOS (3.35 vs. 3.41) and lower SMOS in several cross-lingual conditions; the paper attributes this partly to the codec introducing artifacts that cap perceptual quality. confidence: high relevance: medium - claim_id: lossy_neural_audio_codecs_set_a_perceptual_quality_ceiling_in role: complicates claim: Lossy neural audio codecs set a perceptual quality ceiling in codec-based TTS that preference alignment cannot overcome, because the model learns to replicate codec artifacts introduced during reference encoding. source: §6 evidence: The paper notes that the VQ-VAE codec is lossy and the model learns to reproduce its artifacts, limiting the maximum perceptual quality achievable regardless of post-training alignment method. confidence: high relevance: low limitations: - The evaluation uses an internal test set of unseen speakers with no publicly named benchmark, and the human rating pool is predominantly English and Portuguese native speakers. Conclusions about multilingual naturalness and similarity, especially for French, Italian, and Romanian, should be treated with caution. - The DPO preference signal is constructed solely from WER and speaker similarity; no prosody, naturalness, or rhythm metric is incorporated, which likely explains the MOS regression relative to the fine-tuned baseline. The preference labeling is fully automated with no human verification of winner/loser assignments. The real-time factor of 4.85 makes the system unsuitable for real-time applications, and the authors note that non-autoregressive architectures are a necessary direction. The Romanian evaluation suffers from very small rater counts and predominantly non-native listeners, undermining the interpretation of the unusually high SMOS scores that exceed real audio. caveats: [] - id: '2509.14270' published_date: "2025-09-15" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: structured_keyphrase_infusion_via_multi_step_llm_prompting_substantially_reduces role: supports claim: Structured keyphrase infusion via multi-step LLM prompting substantially reduces semantic repetition in generated TTS training corpora compared to single-prompt domain generation. source: §3.1, §4.1 evidence: SpeechWeave's keyphrase pipeline achieves 45.8%/44.4% lower grouped mean pairwise similarity in English/Spanish and 17.4%/9.7% higher diphone coverage relative to direct domain prompting with Mistral-7b-Instruct, even at high temperature (Table 3). confidence: high relevance: medium - claim_id: generating_semiotic_class_entities_with_embedded_normalization_rules_at_source role: supports claim: Generating semiotic class entities with embedded normalization rules at source achieves higher normalization accuracy for TTS training text than applying a post-hoc normalizer to LLM-generated output. source: §3.2, §4.2.1 evidence: At-source entity normalization reaches 0.97 accuracy in English and 0.94 in Spanish, compared to 0.67 and 0.54 for NeMo's WFST text normalizer on the same 500-sentence evaluation sets per language (Table 4). confidence: high relevance: medium - claim_id: fine_tuning_a_pre_trained_tts_model_on_synthetic_pipeline role: supports claim: Fine-tuning a pre-trained TTS model on synthetic pipeline-generated data with controlled diversity and normalization can meaningfully improve downstream speech intelligibility. source: §4.2.3 evidence: StyleTTS 2 fine-tuned on SpeechWeave data shows 40% relative WER reduction in English (15.37% to 9.36%) and 27% in Spanish (85.05% to 48.44%) on LibriSpeech test sets; the Spanish result also demonstrates cross-lingual adaptation from a monolingual English checkpoint (Table 6). confidence: high relevance: medium - claim_id: automated_speech_quality_metrics_from_predictive_models_should_not_be role: complicates claim: Automated speech quality metrics from predictive models should not be treated as equivalent to human MOS scores when evaluating synthetic TTS training data. source: §4.2.2, Table 5 evidence: SpeechWeave reports MOS of 4.95 (English) and 4.87 (Spanish) estimated by the NISQA neural quality predictor rather than human raters; the paper notes this explicitly. Automated MOS predictors are known to diverge from human judgments, particularly for TTS systems that differ from the predictor's training distribution. confidence: high relevance: medium limitations: - The evaluation covers only English and Spanish, and improvements in normalization accuracy and diversity may not hold for morphologically rich or lower-resource languages, which the paper acknowledges as future work. - The entity sampler supports nine semiotic classes; any entity type not included in the recipes will not be normalized at source, and the post-processor may handle them incorrectly. The extent of this gap depends on how often such entities appear in target domains. - The pipeline depends on Mistral-7b-Instruct for text generation and OpenVoice V2 for voice cloning, and performance is expected to vary with model substitution. No ablation is provided over LLM or TTS model choice, so the contribution of each component to the overall result is not fully isolated. - The downstream evaluation uses WER from an ASR model to assess TTS quality, which measures intelligibility rather than naturalness or speaker similarity. Prosody, expressiveness, and voice consistency are not evaluated beyond the automated MOS and SNR scores. caveats: [] - id: '2509.14579' published_date: "2025-09-18" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - VC architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: cross_lingual_zero_shot_voice_cloning_in_flow_matching_tts role: supports claim: Cross-lingual zero-shot voice cloning in flow-matching TTS can be achieved without audio prompt transcripts by using forced alignment to partition training data at word boundaries. source: §4.3, Table 3 evidence: CL-F5 achieves WER 2.496% on a cross-lingual English test set drawn from four unseen languages (German, French, Hindi, Korean), while the F5-TTS baseline cannot perform cross-lingual cloning at all due to its transcript dependency. confidence: high relevance: high - claim_id: multi_granularity_speaking_rate_predictors_trained_with_ordinal_aware_gaussian role: supports claim: Multi-granularity speaking rate predictors trained with ordinal-aware Gaussian loss provide reliable duration estimation for transcript-free flow-matching TTS. source: §4.1, §4.2, Table 1, Table 2 evidence: The phoneme-level predictor (M1) achieves MAE=0.759s on LibriSpeech-PC and enables WER 2.079%, improving over the length-ratio baseline (2.205%); the syllable-level predictor (M2) is preferred for Chinese synthesis on SeedTTS test-zh. confidence: high relevance: medium - claim_id: removing_audio_prompt_transcripts_from_flow_matching_tts_incurs_a role: complicates claim: Removing audio prompt transcripts from flow-matching TTS incurs a small but consistent cost in speaker similarity. source: §4.2, Table 2, §5 evidence: CL-F5 M1 achieves SIM-o 0.663 vs. 0.668 for the transcript-conditioned baseline on LibriSpeech-PC; the authors also report reduced capacity for transferring accent and emotion relative to the original F5-TTS. confidence: high relevance: low - claim_id: the_optimal_linguistic_granularity_for_acoustic_speaking_rate_prediction_is role: refines claim: The optimal linguistic granularity for acoustic speaking rate prediction is language-dependent rather than universal. source: §4.1, Table 1 evidence: Phoneme-level (M1) outperforms syllable-level (M2) for English duration prediction, while M2 outperforms M1 for Chinese on SeedTTS test-zh; word-level (M3) is consistently the weakest across all languages and test sets. confidence: high relevance: medium limitations: - The speaking rate predictors are trained exclusively on English and Chinese data. While cross-lingual generalisation to German, French, Hindi, and Korean is demonstrated empirically, the scope of generalisation to genuinely low-resource or typologically distant languages remains untested. - 'Speaker characteristic transfer is degraded compared to the transcript-conditioned baseline: the authors note reduced accuracy in preserving accents and emotional cues. This points to an open question about how much speaker-identity information is carried by the reference transcript versus the audio signal alone. Future work suggested by the paper includes compensating for missing linguistic information from transcripts to recover expressiveness.' caveats: [] - id: '2509.14784' published_date: "2025-09-18" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: aligning_autoregressive_decoder_representations_with_pre_trained_asr_encoder_semantics role: supports claim: Aligning autoregressive decoder representations with pre-trained ASR encoder semantics improves content consistency in continuous-feature TTS. source: §2.3, §3.2, Table 1, Table 2 evidence: Representation alignment reduces WER 6.3 to 5.3 on LibriTTS ablation and produces 25%, 40%, and 30% relative CER/WER reductions on seed-tts-eval test-zh, test-en, and test-hard at 170K-hour scale. Training convergence accelerates by over 3.3x. confidence: high relevance: medium - claim_id: using_the_target_acoustic_domain_mel_spectrogram_as_an_intermediate role: contradicts claim: Using the target acoustic domain (mel-spectrogram) as an intermediate alignment target helps continuous AR TTS systems. source: §3.2, Table 1 evidence: Aligning transformer decoder outputs to mel-spectrograms (Exp 2, Table 1) raises WER from 6.3 to 6.7 and degrades SS1/SS2, while alignment to SenseVoice-Large ASR representations (Exp 1) reduces WER to 5.3. confidence: high relevance: medium - claim_id: continuous_feature_ar_tts_achieves_competitive_content_accuracy_but_lags role: complicates claim: Continuous-feature AR TTS achieves competitive content accuracy but lags behind discrete-token systems on speaker similarity in zero-shot voice cloning. source: §3.3, Table 2 evidence: MELA-TTS achieves best CER on seed-tts-eval test-zh but lower SS1 (0.59 vs. F5-TTS 0.65) and SS2 (0.68 vs. F5-TTS 0.74) on test-en; attributed to the diffusion module's local context constraint versus discrete-token systems that condition on all prior tokens and the full prompt. confidence: high relevance: low - claim_id: unified_interleaved_training_on_streaming_and_offline_sequences_enables_continuous role: supports claim: Unified interleaved training on streaming and offline sequences enables continuous AR TTS to operate in both modes at comparable quality without separate model variants. source: §2.4, §3.2, §3.3, Table 1, Table 2 evidence: Streaming mode achieves WER 5.0 versus offline 5.2, and SS1/SS2 parity, on LibriTTS. At 170K-hour scale, streaming WER is 2.5 versus offline 2.4 on test-en. confidence: high relevance: low - claim_id: continuous_feature_tts_without_speech_tokenization_scales_effectively_with_training role: supports claim: Continuous-feature TTS without speech tokenization scales effectively with training data, reducing content error rates and approaching discrete-token system performance. source: §3.3, Table 2 evidence: Scaling from LibriTTS (585h) to 170K-hour in-house data reduces English WER from 5.2 to 2.4 and improves SS1 from 0.48 to 0.59, bringing MELA-TTS close to CosyVoice 2.0 (2.6 WER) and CosyVoice 3.0 (2.5 WER). confidence: high relevance: medium limitations: - The diffusion module operates on local context only (the immediately adjacent latent vector and the previous mel chunk), preventing it from attending to the full prompt speech. This is identified as the likely structural cause of the consistent speaker similarity gap versus discrete-token systems across test-en and test-hard. The limitation is left as future work with no proposed fix. - Main comparison results rely on a proprietary 170K-hour multilingual dataset that is not publicly available, limiting reproducibility of the top-line numbers. Ablations are conducted on LibriTTS only (585 hours), which may not reflect behavior at full scale. No comparison to DiTAR on test-hard is available, making it unclear whether MELA-TTS's robustness advantage on challenging inputs holds relative to the closest continuous AR baseline. Code and demo are not released with the preprint. caveats: [] - id: '2509.17765' published_date: "2025-09-22" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_multilingual_codec_models - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: replacing_a_block_wise_diffusion_vocoder_with_a_lightweight_causal role: supports claim: Replacing a block-wise diffusion vocoder with a lightweight causal convolutional decoder, driven by a multi-codebook autoregressive token predictor, can substantially reduce first-packet latency in streaming speech generation without sacrificing competitiveness on content-consistency metrics. source: §2.4, §2.5, Table 1, Table 13 evidence: The Talker's multi-codebook AR scheme plus a 200M-parameter causal ConvNet Code2Wav stage achieves a 234ms end-to-end first-packet latency at 1x concurrency and the lowest reported content-consistency error on SEED test-en (1.39) among all compared zero-shot TTS systems including flow-matching and diffusion-based baselines. confidence: high relevance: low - claim_id: mixing_unimodal_and_cross_modal_training_data_from_the_earliest role: supports claim: Mixing unimodal and cross-modal training data from the earliest stage of pretraining allows a language model to add new input/output modalities without degrading its original text, vision, or audio-specific capabilities relative to matched unimodal baselines. source: §6, Table 16 evidence: A controlled comparison of parameter-matched text-only, vision-only, and Omni models trained on identical corpora, schedules, and compute shows the Omni model matches or exceeds the unimodal baselines on general, math/STEM, coding, and multilingual text benchmarks as well as vision and video benchmarks. confidence: high relevance: medium - claim_id: strong_zero_shot_voice_cloning_performance_in_one_or_two role: complicates claim: Strong zero-shot voice-cloning performance in one or two conditioning languages does not guarantee comparable speaker-similarity performance uniformly across all supported languages. source: §5.2.2, Table 14 evidence: Against MiniMax-Speech and ElevenLabs Multilingual v2 on a 10-language multilingual test set, the system leads by a substantial margin on Chinese, English, and French but reports only "competitive," non-leading, speaker-similarity or content-consistency scores on several other languages such as Portuguese and Russian. confidence: high relevance: low - claim_id: a_large_scale_purpose_built_supervised_audio_encoder_trained_from role: supports claim: A large-scale, purpose-built supervised audio encoder trained from scratch for a multimodal LLM's audio pathway can outperform reusing a general pretrained ASR encoder (e.g., Whisper) as the perceptual front-end for both speech understanding and downstream speech generation. source: §1, §2.2, Table 6, Table 7 evidence: Replacing the Whisper-based audio encoder from the predecessor system with AuT, trained from scratch on 20 million hours of supervised audio at a 12.5 Hz token rate, is cited as a key driver of gains across ASR, lyric-ASR, and voice-interaction benchmarks relative to Qwen2.5-Omni. confidence: high relevance: medium limitations: - Speech generation quality is evaluated exclusively with automatic metrics (WER/CER-style content consistency, embedding-based speaker similarity, BLEU for translation); no human MOS or listening-test results are reported anywhere in the paper for the Talker's synthesized speech, so claims of "stable, naturalistic speech synthesis" in the conclusion are not directly supported by subjective evidence in this report. - The paper acknowledges suboptimal performance on long-video benchmarks, attributed to limited positional extrapolation and restricted context length, as an explicit architectural limitation left for future work. The reported 234ms first-packet latency is described as "theoretical," measured under a specific vLLM/torch.compile/CUDA-Graph deployment configuration rather than as an end-user-measured figure across arbitrary hardware or network conditions, and latency degrades substantially under higher concurrency (up to 1172ms at 6-way concurrency in the audio case). Several baselines used for comparison (ElevenLabs, MiniMax-Speech, Gemini-2.5-Pro, GPT-4o variants) are closed proprietary systems, so exact reproduction of the comparative numbers is not possible outside the authors' own evaluation pipeline. The non-degradation ablation study, while methodologically rigorous, was run at limited model scales due to computational cost, and the authors explicitly caution that they could not sweep across all model sizes. caveats: [] - id: '2509.18060' published_date: "2025-09-22" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - infrastructure current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: routing_hidden_representations_through_dialect_specific_sub_networks_inside_the role: supports claim: Routing hidden representations through dialect-specific sub-networks inside the Transformer FFN captures fine-grained dialectal acoustic variation more effectively than conditioning through a shared-parameter network alone. source: §3.3, Table 2 evidence: Ablating DSDR-Net (replacing it with a standard shared FFN) while keeping the dialect fusion embedding drops dialect classification accuracy from 80.25% to 60.12% and dialect embedding cosine similarity from 78.3% to 58.6%. confidence: high relevance: high - claim_id: injecting_dialect_style_conditioning_early_into_both_the_encoder_and role: supports claim: Injecting dialect/style conditioning early, into both the encoder and the generative (flow-prediction) network, produces more dialect-consistent speech than injecting it only at the output stage via separate per-dialect vocoders. source: §3.2, Table 1 evidence: Compared to a prior Tibetan multi-dialect design that shares a mel-decoder and uses separate per-dialect WaveNet vocoders, TMD-TTS's early dialect fusion module plus DSDR-Net reaches up to 88.09% DECS and 87.78% DCA versus a Matcha-TTS baseline extended with the late-fusion design (65.2–65.8% on the same metrics). confidence: high relevance: high - claim_id: synthetic_speech_generated_by_a_dialect_conditioned_tts_system_when role: supports claim: Synthetic speech generated by a dialect-conditioned TTS system, when filtered for dialect consistency and perceptual quality, is usable as training data for a downstream speech-to-speech conversion task and can outperform a small, manually collected parallel corpus. source: §3.5, Table 4 evidence: A voice conversion model (DurFlex-EVC) trained on the TMDD dataset synthesized by TMD-TTS achieves higher MOS (3.23/3.63 at 16k/22kHz) than the same model trained on the prior manually collected baseline dataset (3.07/3.54). confidence: high relevance: high - claim_id: gains_in_speaker_style_consistency_metrics_from_dialect_or_attribute role: complicates claim: Gains in speaker/style-consistency metrics from dialect- or attribute-specific conditioning mechanisms can come with a measurable inference-speed cost relative to simpler architectures. source: §3.2 evidence: TMD-TTS's real-time factor (~0.031–0.032) is roughly 50% higher than VITS2's (~0.020–0.021), even though TMD-TTS still meets real-time synthesis requirements. confidence: high relevance: high limitations: - The evaluation is confined to a single language (Tibetan) and its three dialects; no experiments test whether DSDR-Net's routing mechanism generalizes to other dialect-continuum languages or to conventional multilingual TTS. The dialect classifier used for DCA/DECS metrics is itself a pretrained model whose own accuracy is not independently validated in this paper, so objective dialect-fidelity numbers are only as trustworthy as that classifier. Code and demo availability are not stated in the paper. The comparison baselines were reimplemented by the authors rather than run from official checkpoints, which is a reasonable and disclosed choice given the lack of existing public multi-dialect Tibetan TTS baselines, but leaves some residual uncertainty about baseline tuning. caveats: [] - id: '2501.04561' published_date: "2025-09-23" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: progressive_text_pivoted_alignment_across_modality_pairs_can_substitute_for role: supports claim: Progressive, text-pivoted alignment across modality pairs can substitute for paired tri-modal training data without sacrificing downstream omnimodal task performance. source: §4.2, Table 1 evidence: OpenOmni trains only on speech-text and image-text pairs (no image-speech-text triples) yet outperforms VITA, which is trained on 5M tri-modal samples, by 4 points on OmniBench while using a 7B rather than 7×8B language model and roughly 5x fewer training samples. confidence: high relevance: medium - claim_id: non_autoregressive_discrete_unit_speech_decoding_trades_generation_quality_for role: complicates claim: Non-autoregressive discrete-unit speech decoding trades generation quality for latency relative to autoregressive decoding. source: §4.2, §D "AR mode" evidence: The paper reports that its AR mode (NTP loss, 16K-unit vocabulary) yields higher speech generation quality but slower streaming, while the NAR mode (CTC loss, 6K-unit vocabulary) achieves under-1-second latency for up to 30 seconds of speech (5x faster) at the cost of "slightly worse" generation quality. confidence: high relevance: low - claim_id: direct_preference_optimization_can_be_adapted_to_discrete_unit_ctc role: supports claim: Direct preference optimization can be adapted to discrete-unit, CTC-trained speech generators to improve emotional coherence without an auxiliary emotion-control module. source: §4.2, Table 4 evidence: CTC-DPO training on the 9K-pair EO2S-9K preference dataset (Plutchik-based emotion categories, CosyVoice-synthesized positive/negative pairs) raises Emotion2Vec-classified accuracy from 57.9% to 70.4% on Chinese and 62.6% to 65.4% on English test speech. confidence: high relevance: low - claim_id: mixture_of_experts_capacity_is_necessary_not_merely_beneficial_for role: complicates claim: Mixture-of-experts capacity is necessary, not merely beneficial, for stabilizing CTC-loss training of multilingual non-autoregressive speech decoders. source: §3.4, §C, Table 6 evidence: Ablations show a single feed-forward decoder layer (1 expert) fails to converge on bilingual WeNetSpeech/LibriSpeech data (CER/WER of 113.6/129.7/87.8/96.5), while increasing to a 4-expert MoE layer brings these down to single digits (8.5/8.4/4.2/4.7); the paper states that "without this layer, the speech decoder fails to train effectively." confidence: high relevance: high limitations: - The system is trained and validated only on Chinese and English; the authors explicitly note that multilingual speech data beyond these two languages was not used due to resource constraints, leaving the generalization of the alignment and speech-generation strategy to other languages untested (§Limitation). Separately, the paper acknowledges that because the speech decoder conditions on the LLM's internal hidden states rather than its final decoded text, occasional mismatches between the LLM's actual text answer and the conditioning features used for speech generation can still occur despite the text-guided fusion module designed to mitigate this (§Appendix D). Evaluation of emotional and omnimodal quality relies on automated classifiers (Emotion2Vec, Whisper-based WER) and the authors' own benchmarks rather than independent human listening tests, so subjective perceptual quality of the generated speech is not directly reported. caveats: [] - id: '2509.20086' published_date: "2025-09-24" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: combining_large_multilingual_lexica_with_a_principled_statistical_fallback_for role: supports claim: Combining large multilingual lexica with a principled statistical fallback for out-of-vocabulary compound segmentation reduces phoneme error rate compared to purely rule-based or purely neural G2P systems. source: §4.1, Table 2 evidence: OLaPh achieves 5.32% average PER across English, French, German, and Spanish on WikiPron, versus 9.84% (eSpeak-NG), 10.29% (Gruut), and 12.52% (ByT5 tiny) *(§4.1, Table 2)*. confidence: high relevance: high - claim_id: a_statistical_subword_segmentation_fallback_can_perform_as_a_robust role: supports claim: A statistical subword segmentation fallback can perform as a robust primary G2P mechanism for compound-rich, out-of-vocabulary words, not merely as a weak last resort. source: §4.1, Table 3 evidence: On Spanish, which has the highest OOV rate in the benchmark (30.69%), OLaPh's OOV-only PER (2.06%) is lower than its PER on the full dataset (2.58%), and its overall OOV-subset average PER (5.47%) closely tracks its full-dataset average (5.32%) *(§4.1, Table 3)*. confidence: high relevance: high - claim_id: a_neural_g2p_model_distilled_from_a_deterministic_teacher_s role: refines claim: A neural G2P model distilled from a deterministic teacher's synthetic output can exceed the teacher on a meaningful share of ambiguous or out-of-vocabulary cases, even while trailing it in overall accuracy. source: §4.2, Table 4 evidence: The OLaPh LLM's average PER (9.74%) trails the OLaPh framework's (5.32%), but in a sample-wise comparison on OOV words the LLM wins outright on 22.49% of English samples, 10.16% of French, 9.01% of German, and 2.02% of Spanish samples *(§4.2, Table 4)*. confidence: high relevance: high - claim_id: cross_system_phoneme_error_rate_comparisons_for_languages_with_deep role: complicates claim: Cross-system phoneme error rate comparisons for languages with deep orthography can be inflated by benchmark transcription inconsistencies rather than reflecting true differences in phonemization capability. source: §4.1, §5 evidence: Every evaluated system (eSpeak-NG, Gruut, ByT5, OLaPh, OLaPh LLM) shows its highest PER on English; manual sample audits attributed much of this increase to allophonic variation and British/American English mixups in the WikiPron reference labels rather than fundamental phonemic mapping failures *(§4.1, §5)*. confidence: high relevance: medium limitations: - The evaluation measures phoneme error rate against WikiPron reference transcriptions only; the paper does not measure the downstream effect of these differences on synthesized speech quality or intelligibility in an actual TTS pipeline, and the authors explicitly flag this as unresolved future work. - The framework currently supports and is evaluated on only four languages (English, German, French, Spanish), despite lexica being prepared for six additional languages awaiting integration. The WikiPron benchmark itself is acknowledged to contain transcription inconsistencies, particularly for English, that were not corrected due to the manual effort required at scale, which caps how precisely the reported PER gaps can be interpreted. The LLM extension is trained and evaluated only on synthetic data generated by OLaPh itself, so its generalization is bounded by the coverage and consistency of that teacher signal rather than by human-labeled data; the authors note that scaling parameter count and training diversity may be needed before it can exceed the deterministic framework outright. caveats: [] - id: '2509.22727' published_date: "2025-09-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - flow_matching_multilingual_tts claims: - claim_id: joint_training_across_multiple_dialects_induces_style_averaging_where_dialect role: supports claim: Joint training across multiple dialects induces "style averaging," where dialect-specific prosodic identity is diluted unless the model has a mechanism to route dialect-specific information separately. source: §4.4.2, Table 3 evidence: Ablating the dialect-aware MoE module (removing it while keeping the IPA frontend) degrades both MOS (e.g., 2.46 vs. 3.15 on Xi'an) and WER (45.08% vs. 33.00% on Xi'an) relative to the full model, isolating the MoE's contribution to preserving per-dialect quality in a jointly trained model. confidence: high relevance: high - claim_id: a_standardized_phonetic_representation_ipa_resolves_grapheme_to_phoneme_ambiguity role: supports claim: A standardized phonetic representation (IPA) resolves grapheme-to-phoneme ambiguity better than orthography-based inputs (e.g., pinyin) when a single model must serve multiple dialects with divergent character-to-sound mappings. source: §4.4.2, Table 3 evidence: Replacing the IPA frontend with pinyin input (keeping the MoE module) collapses MOS to roughly 1.0-1.2 and raises WER to roughly 90-93% across four dialects, a far larger degradation than removing the MoE alone. confidence: high relevance: high - claim_id: parameter_efficient_fine_tuning_lora_plus_lightweight_adapters_on_a role: supports claim: Parameter-efficient fine-tuning (LoRA plus lightweight adapters) on a frozen multidialect backbone can extend zero/few-shot speech synthesis to a new, extremely low-resource dialect using only a few hours of data. source: §3.3, §4.1, Table 2 evidence: Stage 3 adapts the frozen Stage-2 model to Peking Opera (Jingbai/Yunbai) and Nanjing dialect using only ~3 hours of data per target style, training only LoRA (rank 16) and Conditioning Adapters while the backbone stays frozen, producing usable synthesis (e.g., Jingbai MOS 3.04, UTMOSv2 1.75) for styles with no commercial TTS baseline available for comparison. confidence: high relevance: high - claim_id: open_modest_scale_multidialect_training_data_can_substantially_underperform_proprietary role: complicates claim: Open, modest-scale multidialect training data can substantially underperform proprietary, large-scale commercial systems on standard intelligibility metrics even when subjective naturalness scores are comparable. source: §4.4.1, Table 1, Table 2 evidence: Trained on ~1.1k total hours versus CosyVoice2's ~150k hours and Qwen-TTS's ~3,000k hours, DiaMoE-TTS's WER trails the baseline average by wide margins in several dialects (e.g., 44.61 points worse on Cantonese, 29.69 points worse on Chengdu), even where its UTMOSv2 score is on par with or better than the commercial average. confidence: high relevance: high limitations: - Word error rate is substantially worse than commercial baselines across nearly every dialect evaluated (e.g., 76.59% vs. a baseline average of 31.98% on Cantonese), and the paper does not report a controlled comparison isolating whether this gap stems from training data scale, IPA/G2P errors, or the MoE routing itself. - 'The evaluation compares against commercial black-box systems (Edge TTS, CosyVoice2, Qwen-TTS) whose training data, architectures, and exact evaluation conditions are not fully known, making some of the reported gaps difficult to attribute precisely. Coverage is also uneven: several dialects and styles (Southern Min, Shijiazhuang, Nanjing, Peking Opera) have no baseline comparison at all, so the paper''s claim of being "almost comparable" to commercial systems rests on a subset of dialects where baselines exist. The MoE''s number of experts and their assignment to specific dialects are not detailed in the text, and no comparison is given for how the dialect-classification auxiliary loss weight (λ) was chosen or how sensitive results are to it.' caveats: [] - id: '2509.22062' published_date: "2025-09-26" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models claims: - claim_id: injecting_explicit_linguistic_structure_into_the_primary_codebook_of_a role: supports claim: Injecting explicit linguistic structure into the primary codebook of a neural speech codec reduces the downstream language model's learning burden and improves synthesis intelligibility. source: §4.3, Table 4 evidence: Removing the semantic distillation loss during codec-conditioned TTS training raises WER from 3.31% to 3.97% on SeedTTS-test, 9.74% to 11.83% on PGC-Hard, and 16.57% to 18.34% on PGC-Poly, with SIM and UTMOS also degrading. confidence: high relevance: low - claim_id: an_automatic_speech_recognition_model_can_serve_as_an_effective role: supports claim: An automatic speech recognition model can serve as an effective semantic teacher for codec distillation, as an alternative to self-supervised speech representation models. source: §E.2, Table 5 evidence: S3Codec distills Whisper encoder embeddings (rather than HuBERT/SSL features) into the first RVQ level; a small model trained with S3Codec reaches 3.30% WER on SeedTTS-test vs. 4.21% for an otherwise identical model using undistilled DAC tokens. confidence: high relevance: low - claim_id: fully_autoregressive_tts_systems_that_omit_an_explicit_continuous_acoustic role: complicates claim: Fully autoregressive TTS systems that omit an explicit continuous acoustic-feature conditioning stage (e.g., mel-spectrogram or speaker-similarity-vector guidance) tend to underperform hybrid AR+NAR or flow-matching systems on speaker similarity even when intelligibility is competitive. source: §4.2, Table 2-3 evidence: CaT-TTS reports SIM of 0.668-0.678 across test sets versus 0.71-0.80 for Seed-TTS, CosyVoice 2/3, and F5-TTS, despite comparable or better WER among AR-only baselines. confidence: high relevance: low - claim_id: test_time_parallel_decoding_with_learned_input_dependent_aggregation_weights role: refines claim: Test-time parallel decoding with learned, input-dependent aggregation weights can reduce autoregressive error accumulation at near-zero added latency, but the achievable robustness gain is bounded by how many parallel streams are used, trading GPU utilization against benefit. source: §4.3, §3.3, Figure 3-4 evidence: MAPI ablation across increasing parallel-stream counts shows WER improving and becoming more stable across 10 repeated inferences per sample, while the authors note GPU resource utilization rises correspondingly and stream count must be tuned per deployment scenario. confidence: high relevance: low limitations: - Training relies on an unreleased proprietary corpus (~200k hours, ~85% Chinese / ~15% English), and neither code nor a demo is available, which limits independent verification of the reported results. - The semantic-distillation ablation (removing the loss) and the MAPI ablation are both run on smaller sub-datasets and reduced-size "CaT-TTS-small" models rather than the full 0.4B system, so it is not established that the same magnitude of gains transfers to the full-scale model. Speaker similarity remains a clear weak point relative to hybrid and NAR baselines, which the authors attribute to the deliberate absence of continuous acoustic conditioning rather than treat as a target for improvement. The evaluation is also dominated by Chinese-language and Chinese-out-of-domain test sets (PGC-Hard, PGC-Poly, Seed-TTS test-zh/test-hard), with comparatively less English-language evidence. caveats: [] - id: '2509.23147' published_date: "2025-09-27" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: neural_ctc_based_forced_aligners_can_achieve_substantial_inference_speedups role: supports claim: Neural CTC-based forced aligners can achieve substantial inference speedups over classical HMM-GMM aligners while retaining competitive recall at practical tolerance levels. source: §3, Table 1, Table 2 evidence: The proposed aligner processes a 10-second clip in ~0.2-0.25s versus roughly 45-60 minutes for a comparable Buckeye clip under MFA, a measured 45-242x total-time speedup, while matching or exceeding MFA's recall at 40ms and 60ms tolerance on both TIMIT and Buckeye. confidence: high relevance: medium - claim_id: aligners_that_predict_both_phoneme_onset_and_offset_boundaries_trade role: complicates claim: Aligners that predict both phoneme onset and offset boundaries trade precision for recall relative to onset-only baselines, because doubling the predicted boundary count increases the chance of spurious matches against onset-only ground truth. source: §4 "End Boundary", Table 2 evidence: At 20ms tolerance on TIMIT, the proposed aligner's onset precision drops to 55.6% versus MFA's 81.2%, even though its recall is comparable, because the model predicts roughly double the boundary count that onset-only annotations penalize as insertions. confidence: high relevance: medium - claim_id: multilingual_phoneme_representations_trained_without_target_language_data_can_generalize role: supports claim: Multilingual phoneme representations trained without target-language data can generalize to unseen-language alignment tasks with only modest performance degradation. source: §4 "Cross-lingual Performance", Table 2 evidence: Aligner variants trained exclusively on non-English multilingual corpora (7-language and 35-language configurations) achieve recall within 1-11 percentage points of an English-only trained variant when evaluated on English test sets, with the 7-language variant nearly matching it (71.0% vs. 71.4% recall at 20ms on TIMIT). confidence: high relevance: high - claim_id: natural_speech_contains_measurable_silent_gaps_between_phonemes_that_conventional role: refines claim: Natural speech contains measurable silent gaps between phonemes that conventional onset-only forced alignment discards by assuming acoustic contiguity. source: §4 "End Boundary", Table 3 evidence: Explicit dual boundary prediction shows 30-40% of phonemes on both TIMIT and Buckeye have a measurable inter-phoneme gap, with median gap durations of 39-41ms on TIMIT and 39-40ms on Buckeye. confidence: high relevance: medium limitations: - Cross-lingual generalization is validated only on English test sets. The multilingual and universal variants are trained on non-English corpora but are never evaluated on non-English alignment accuracy in this paper, so the claim that the universal phoneme representation "generalizes across languages" rests on training-data composition, not on measured performance in those languages. - The dual boundary formulation roughly doubles predicted boundary count relative to onset-only ground truth annotations, which mechanically depresses precision scores at tight tolerances and complicates direct precision comparison with onset-only aligners. Evaluation is restricted to two English-language benchmarks (TIMIT, a clean read-speech corpus, and Buckeye, spontaneous conversational speech); performance on tonal languages and other typologically distant languages is left to future work, as the authors themselves note. caveats: [] - id: '2509.25131' published_date: "2025-09-29" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_multilingual_codec_models - flow_matching_multilingual_tts - hybrid_multilingual_semantic_acoustic_systems claims: - claim_id: chunking_text_into_aligned_segments_with_a_short_token_delay role: supports claim: Chunking text into aligned segments with a short token-delay before speech decoding reduces error accumulation in long-form autoregressive speech generation. source: §4.2, Table 6 evidence: Removing chunk-based decoding raises Long-TTS-Eval error rates above those of concurrent long-form TTS baselines, and with it enabled MGM-Omni-TTS-2B achieves EN-hard WER 26.26 versus 42.48-98.61 for CosyVoice2, MOSS-TTSD-v0.5, and Higgs-Audio-v2. confidence: high relevance: medium - claim_id: multi_token_parallel_decoding_is_not_restricted_to_rvq_speech role: supports claim: Multi-token parallel decoding is not restricted to RVQ speech tokenizers and can be applied effectively to finite scalar quantization (FSQ) tokenizers. source: §4.2, Table 6 evidence: Increasing parallel decoding size on the CosyVoice2 FSQ tokenizer maintains TTS quality on Seed-TTS-Eval while cutting inference RTF by roughly 3x at parallel size 4. confidence: high relevance: medium - claim_id: increasing_the_parallel_decoding_size_trades_off_synthesis_error_rate role: complicates claim: Increasing the parallel decoding size trades off synthesis error rate against inference speed rather than improving both simultaneously. source: §4.2 evidence: Larger parallel sizes in the ablation slightly raise audio error rate even as they substantially accelerate inference, leading the authors to select a parallel size of 4 as a balance point. confidence: high relevance: medium - claim_id: separating_multimodal_reasoning_from_speech_synthesis_into_distinct_model_components role: refines claim: Separating multimodal reasoning from speech synthesis into distinct model components can improve long-form audio understanding without sacrificing speech generation efficiency. source: §4.1.1, §4.1.3, Figure 5, Table 5b evidence: The dual-track brain-mouth design lets the MLLM handle needle-in-the-haystack audio inputs up to 4,500 seconds while the SpeechLM independently achieves the lowest RTF among compared long-form TTS systems. confidence: high relevance: medium limitations: - 'The long-form evaluation itself is partly self-authored: Long-TTS-Eval is introduced by this paper, and while its construction and normalized-text scoring procedure are documented, results on it cannot yet be cross-checked against independent replications. The comparison in Table 5b is limited to three baseline systems, and the qualitative long-speech examples in the appendix (a classical Chinese poem and a code-switched English-Chinese poem) are illustrative rather than a systematic error analysis. The paper does not report results on emotion or prosody control, nor does it evaluate robustness to reference audio recorded in noisy or far-field conditions. The 32B MLLM variant''s long-form and vision-speech results are mixed relative to the 7B variant (e.g., lower TextVQA-Speech and EN-hard performance context is not directly reported for 32B TTS), suggesting scaling benefits are not uniform across all sub-tasks.' caveats: [] claim_clusters: - id: shared_multilingual_training_improves_low_resource_tts claim: Shared multilingual training improves synthesis for low-resource languages by transferring linguistic and acoustic structure from better-resourced languages. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - 2025.naacl-short.69 - '2508.14049' - '2508.07302' - interspeech-2025-0143 - interspeech-2025-0469 contradicting_papers: [] refining_papers: - interspeech-2025-1550 caveats: - Transfer depends on linguistic relatedness, data balance, and front-end compatibility. last_reviewed: '2026-07-29' - id: training_imbalance_biases_multilingual_quality claim: Imbalanced multilingual corpora bias synthesis quality toward high-resource languages and dominant speakers or accents. status: strongly_supported confidence: high supporting_papers: - 2025.naacl-short.69 - '2507.16875' - 2025.acl-short.81 - 2025.unlp-1.11 - '2508.14049' - '2508.07302' - '2508.08715' - interspeech-2025-0469 - interspeech-2025-1034 - '2509.22727' contradicting_papers: [] refining_papers: - '2507.16875' - '2508.14049' - '2508.08715' caveats: - Reported averages can hide large per-language and demographic gaps. last_reviewed: '2026-07-29' - id: cross_lingual_cloning_trades_identity_against_language_fidelity claim: Cross-lingual voice cloning must balance speaker identity against target-language pronunciation, accent, and intelligibility. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - '2406.04904' - 2025.coling-main.685 - '2508.07302' - interspeech-2025-raju25_interspeech - '2509.14579' contradicting_papers: [] refining_papers: - '2406.04904' caveats: - Identity and accent are entangled in both listener judgments and speaker-verification embeddings. last_reviewed: '2026-07-29' - id: explicit_language_conditioning_reduces_confusion claim: Explicit language embeddings, identifiers, prompts, or routing reduce language confusion in shared multilingual models. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - 2025.findings-acl.687 - 2025.ccl-1.80 contradicting_papers: [] refining_papers: - 2025.findings-acl.687 caveats: - Language labels do not by themselves capture dialect continua or code-switching. last_reviewed: '2026-07-29' - id: shared_phonetic_representations_enable_transfer claim: Shared phonetic representations such as IPA and universal phone inventories enable cross-lingual transfer beyond orthographic overlap. status: strongly_supported confidence: high supporting_papers: - '2312.01479' - '2411.01156' - 2025.naacl-short.69 - 2025.acl-industry.42 - 2025.unlp-1.11 - '2508.09767' - interspeech-2025-0469 - interspeech-2025-2031 - interspeech-2025-2573 - interspeech-2025-gourav25_interspeech - interspeech-2025-raju25_interspeech - '2509.01391' - '2509.22727' contradicting_papers: [] refining_papers: [] caveats: - Universal inventories can erase language-specific allophony and prosodic distinctions. last_reviewed: '2026-07-29' - id: code_switching_requires_joint_language_and_context_modeling claim: Natural code-switching requires joint modeling of language identity, linguistic context, pronunciation, and speaker continuity. status: emerging confidence: medium supporting_papers: - 2025.ccl-1.80 contradicting_papers: [] refining_papers: - interspeech-2025-2573 caveats: - Code-switching benchmarks are small and often cover only a few language pairs. last_reviewed: '2026-07-29' - id: text_front_ends_remain_a_low_resource_bottleneck claim: Text normalization, grapheme-to-phoneme conversion, tokenization, and alignment remain bottlenecks for multilingual and low-resource TTS. status: strongly_supported confidence: high supporting_papers: - '2312.01479' - '2411.01156' - '2412.02612' - '2502.05512' - '2507.16875' - '2507.18119' - 2025.acl-industry.42 - 2025.findings-acl.687 - 2025.unlp-1.11 - 2025.ccl-1.80 - '2508.07302' - '2508.09767' - interspeech-2025-0469 - interspeech-2025-0762 - interspeech-2025-1034 - interspeech-2025-2031 - interspeech-2025-2043 - interspeech-2025-2573 - interspeech-2025-gourav25_interspeech - interspeech-2025-raju25_interspeech - '2508.15442' - '2508.18006' - '2509.01391' - '2509.05863' - '2509.22727' - '2509.23147' contradicting_papers: [] refining_papers: - '2507.16875' - 2025.findings-acl.75 - '2509.05863' - '2509.20086' caveats: - End-to-end models can shift rather than eliminate errors caused by sparse linguistic coverage. last_reviewed: '2026-07-29' - id: zero_shot_models_generalize_to_unseen_languages_imperfectly claim: Large multilingual models show zero-shot transfer to unseen languages, but pronunciation and intelligibility remain sensitive to language coverage. status: strongly_supported confidence: high supporting_papers: - '2312.01479' - '2406.04904' - '2407.05361' - 2025.coling-main.685 - '2505.17589' - 2025.acl-long.313 - 2025.findings-acl.687 - '2503.11026' - '2508.07302' - '2508.07426' - interspeech-2025-1344 - interspeech-2025-2031 - interspeech-2025-raju25_interspeech - '2508.15442' - '2509.14579' contradicting_papers: [] refining_papers: - '2406.04904' - '2407.05361' - '2507.16875' - interspeech-2025-2031 caveats: - Unseen-language demonstrations often use related languages or external phonetic resources. last_reviewed: '2026-07-29' - id: continued_pretraining_and_fine_tuning_extend_language_coverage claim: Continued pretraining and language-specific fine-tuning efficiently extend multilingual TTS to new low-resource languages. status: strongly_supported confidence: high supporting_papers: - '2406.04904' - '2508.07302' - '2508.08715' - '2508.09767' - interspeech-2025-1344 - '2509.22727' contradicting_papers: [] refining_papers: - '2508.08715' caveats: - Adaptation can cause forgetting or reduce performance on the original language set. last_reviewed: '2026-07-29' - id: parameter_efficient_modules_support_language_extension claim: Adapters, LoRA, vocabulary expansion, and lightweight language modules support parameter-efficient multilingual extension. status: strongly_supported confidence: high supporting_papers: - interspeech-2025-1344 - interspeech-2025-raju25_interspeech - '2509.22727' contradicting_papers: [] refining_papers: [] caveats: - Small modules may not supply missing acoustic or phonetic coverage in the base model. last_reviewed: '2026-07-29' - id: multilingual_data_scale_improves_generalization claim: Larger and more diverse multilingual corpora improve language coverage, robustness, and cross-lingual generalization. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - '2407.05361' - 2025.coling-main.685 - '2507.21138' - 2025.acl-long.681 - '2508.07426' - interspeech-2025-1344 - '2508.18006' - '2509.17765' contradicting_papers: [] refining_papers: - 2025.coling-main.685 - '2508.14049' - interspeech-2025-0469 caveats: - Scale gains are confounded by filtering, transcription quality, and proprietary data. last_reviewed: '2026-07-29' - id: language_and_accent_leakage_persist claim: Shared multilingual models can leak source-language accent or average language-specific speaking styles. status: emerging confidence: medium supporting_papers: - '2509.22727' contradicting_papers: [] refining_papers: - '2508.14049' caveats: - Accent judgments depend strongly on listener language background and reference choice. last_reviewed: '2026-07-29' - id: duration_and_alignment_are_language_sensitive claim: Duration modeling and text–speech alignment behave differently across languages, scripts, and prosodic systems. status: strongly_supported confidence: high supporting_papers: - '2507.16875' - '2507.18119' - 2025.findings-acl.687 - '2508.07302' - interspeech-2025-0762 - '2508.15442' - '2509.05863' - '2509.14579' contradicting_papers: [] refining_papers: - '2507.16875' - 2025.findings-acl.75 - '2509.05863' caveats: - Many comparisons change tokenization and acoustic modeling alongside alignment. last_reviewed: '2026-07-29' - id: multilingual_evaluation_requires_per_language_reporting claim: Multilingual TTS evaluation requires per-language subjective and objective reporting because aggregate metrics conceal language-specific failures. status: strongly_supported confidence: high supporting_papers: - '2508.07426' - interspeech-2025-0762 - interspeech-2025-0973 contradicting_papers: [] refining_papers: - '2509.18060' caveats: - Listener proficiency and uneven test-set difficulty limit cross-language score comparability. last_reviewed: '2026-07-29' - id: codec_language_models_enable_multilingual_generation claim: Codec-token language models provide a shared generative interface for multilingual and cross-lingual speech synthesis. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - '2306.12925' - '2412.02612' - '2505.17589' - 2025.acl-short.81 - 2025.findings-acl.75 - '2508.07302' - '2508.15442' contradicting_papers: [] refining_papers: [] caveats: - Tokenizers may encode unequal phonetic and acoustic detail across languages. last_reviewed: '2026-07-29' - id: flow_and_diffusion_support_multilingual_scaling claim: Flow-matching and diffusion models provide scalable parallel acoustic generation for multilingual TTS. status: strongly_supported confidence: high supporting_papers: - iclr-2025-hQvX9MBowC - 2025.naacl-short.69 - 2025.acl-long.313 - 2025.acl-long.598 - 2025.findings-acl.687 - '2503.11026' - '2508.14049' - '2508.07302' - interspeech-2025-1344 - interspeech-2025-raju25_interspeech - '2509.14579' contradicting_papers: [] refining_papers: [] caveats: - Matched multilingual comparisons against autoregressive systems remain scarce. last_reviewed: '2026-07-29' - id: dialect_identity_requires_explicit_modeling claim: Dialect identity and regional prosody require explicit representations or routing to avoid averaging in shared multilingual models. status: strongly_supported confidence: high supporting_papers: - interspeech-2025-2031 - interspeech-2025-2573 - '2509.22727' contradicting_papers: [] refining_papers: - '2501.06282' caveats: - Dialect boundaries are socially and linguistically continuous rather than fixed labels. last_reviewed: '2026-07-29' method_families: - id: autoregressive_multilingual_codec_models name: Autoregressive multilingual codec models summary: Autoregressive language models generate codec or acoustic tokens across languages using shared parameters, language prompts, and speaker references. papers: - '2303.03926' - '2306.12925' - '2406.04904' - '2407.05407' - '2409.00750' - '2411.01156' - '2412.02612' - '2501.06282' - '2502.05512' - '2503.14345' - '2505.07916' - '2505.13000' - '2505.17589' - '2507.21138' - '2507.18119' - 2025.acl-long.598 - 2025.acl-long.681 - 2025.acl-short.81 - 2025.findings-acl.71 - '2507.17527' - 2025.ccl-1.80 - '2508.14049' - '2508.04195' - '2508.07302' - '2508.08715' - '2508.09767' - '2508.15442' - '2509.05863' - '2509.17765' - '2501.04561' - '2509.20086' - '2509.22062' - '2509.25131' open_questions: - How can shared token modeling avoid language confusion while retaining cross-lingual speaker and prosodic transfer? - id: flow_matching_multilingual_tts name: Flow-matching multilingual TTS summary: Flow-matching systems generate multilingual acoustic representations in parallel with language, speaker, and phonetic conditioning. papers: - '2407.05407' - '2412.02612' - '2501.06282' - '2503.14345' - 2025.naacl-short.69 - '2505.07916' - '2505.17589' - '2507.16875' - '2507.18119' - 2025.acl-long.313 - 2025.acl-long.598 - 2025.findings-acl.687 - '2503.11026' - '2508.14049' - '2508.04195' - '2508.07302' - '2508.08715' - interspeech-2025-0762 - interspeech-2025-1344 - interspeech-2025-2573 - interspeech-2025-raju25_interspeech - '2509.14579' - '2509.18060' - '2509.22727' - '2509.25131' open_questions: - Do flow-matching advantages persist for low-resource languages when data scale and front-end quality are controlled? - id: transformer_encoder_decoder_multilingual_tts name: Transformer encoder–decoder multilingual TTS summary: Encoder–decoder models share text and acoustic representations across languages through multilingual encoders, language embeddings, and adapters. papers: - '2212.04356' - iclr-2025-hQvX9MBowC - 2025.americasnlp-1.1 - 2025.acl-industry.42 - 2025.acl-short.81 - 2025.findings-acl.75 - 2025.unlp-1.11 - '2508.05385' - '2508.07426' - '2508.11273' - interspeech-2025-0469 - interspeech-2025-1034 - interspeech-2025-1550 - interspeech-2025-2031 - interspeech-2025-2043 - '2509.01391' open_questions: - Which encoder sharing pattern best balances positive transfer against high-resource-language dominance? - id: gan_based_multilingual_synthesis name: GAN-based multilingual synthesis summary: Adversarial acoustic models and vocoders share waveform generation across languages while conditioning on language, speaker, or phonetic features. papers: - '2206.04658' - '2312.01479' - '2406.04904' - '2411.01156' - '2502.05512' - 2025.americasnlp-1.1 - 2025.acl-industry.42 - '2508.07426' - '2508.08715' - interspeech-2025-1550 - interspeech-2025-2573 - interspeech-2025-gourav25_interspeech - '2508.18006' open_questions: - How much language-specific vocoder adaptation is needed for unseen phonation, prosody, and recording domains? - id: hybrid_multilingual_semantic_acoustic_systems name: Hybrid multilingual semantic–acoustic systems summary: Hybrid systems combine linguistic, semantic, acoustic, codec, speaker, and language pathways for multilingual generation. papers: - '2407.05407' - '2501.06282' - 2025.acl-long.598 - 2025.ccl-1.80 - '2508.17494' - '2509.14784' - '2509.17765' - '2501.04561' - '2509.25131' open_questions: - Can semantic and acoustic sharing be separated without losing language-specific rhythm, tone, and speaker identity? - id: vae_multilingual_latent_models name: VAE multilingual latent models summary: VAE-based systems learn continuous or factorized latent spaces for multilingual speaker, style, and acoustic transfer. papers: - '2312.01479' - '2505.07916' - 2025.findings-acl.687 - '2508.07426' - interspeech-2025-1550 open_questions: - How should latent regularization separate language from speaker and style without suppressing cross-lingual transfer? - id: diffusion_multilingual_tts name: Diffusion multilingual TTS summary: Diffusion systems synthesize multilingual acoustic targets conditioned on language, reference speech, and shared text representations. papers: - iclr-2025-hQvX9MBowC - '2507.15272' - interspeech-2025-gourav25_interspeech open_questions: - Can diffusion systems maintain intelligibility and speaker similarity for unseen languages at practical inference cost? reassessment_queue: - id: cross_lingual_cloning_trades_identity_against_language_fidelity type: claim_status reason: Speaker, accent, and language fidelity are measured with inconsistent listener and embedding protocols. trigger: Matched multilingual cloning studies jointly calibrate identity, accent, and intelligibility with listeners. due: 2026-10 current_assessment: strongly_supported watch_for: - Cross-lingual identity listening tests - Accent-aware speaker metrics - id: zero_shot_models_generalize_to_unseen_languages_imperfectly type: claim_status reason: Unseen-language evidence often uses related languages or external phonetic resources. trigger: Independent tests cover typologically distant languages without target-language supervision. due: 2026-10 current_assessment: strongly_supported watch_for: - Typologically distant zero-shot tests - No-resource language transfer - id: shared_phonetic_representations_enable_transfer type: claim_status reason: IPA and universal inventories may obscure language-specific phones and prosody. trigger: Matched studies separate inventory sharing from model and data scale across diverse scripts. due: 2026-10 current_assessment: strongly_supported watch_for: - Shared-versus-language-specific phone ablations - Tone-language evaluations - id: multilingual_evaluation_requires_per_language_reporting type: benchmark_validity reason: Listener proficiency and test-set difficulty differ across languages. trigger: Balanced multilingual benchmarks standardize listener qualification and per-language difficulty. due: 2026-10 current_assessment: strongly_supported watch_for: - Balanced multilingual listening panels - Per-language benchmark calibration - id: dialect_identity_requires_explicit_modeling type: claim_status reason: Dialect evidence remains concentrated in a small set of language families. trigger: Explicit routing or representation gains replicate across unrelated dialect continua. due: 2026-10 current_assessment: strongly_supported watch_for: - Cross-family dialect benchmarks - Dialect routing ablations open_questions: - How should multilingual training data be balanced across languages, speakers, scripts, and recording domains? - Which text representation best supports transfer while preserving language-specific phonology, tone, and prosody? - How can cross-lingual cloning preserve speaker identity without importing source-language accent? - When should a new language use zero-shot prompting, continued pretraining, adapters, or full fine-tuning? - How should multilingual evaluation normalize listener proficiency and test-set difficulty across languages? - Can shared multilingual models preserve dialect continua and code-switching without discrete, potentially reductive labels? trend_notes: - Multilingual TTS has shifted from language-specific models toward shared zero-shot and few-shot foundation systems. - Codec language models and flow matching became prominent multilingual generation backbones in 2024–2025. - Recent systems increasingly combine shared representations with explicit language, speaker, and dialect conditioning. - Parameter-efficient language extension through adapters, LoRA, and vocabulary expansion is growing. - Evaluation is moving toward per-language reporting, cross-lingual speaker identity, accent fidelity, and low-resource fairness.