concept: speaker-adaptation last_updated: '2026-07-29' paper_count: 79 papers: - id: '1609.03499' published_date: "2016-09-12" entry_date: '2026-07-29' year: 2016 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: foundational method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: direct_generation_of_raw_audio_waveforms_without_intermediate_vocoder_parameters role: supports claim: Direct generation of raw audio waveforms, without intermediate vocoder parameters, produces substantially higher naturalness than parametric or concatenative synthesis pipelines as judged by human listeners. source: §3.2, Table 1 evidence: Direct generation of raw audio waveforms, without intermediate vocoder parameters, produces substantially higher naturalness than parametric or concatenative synthesis pipelines as judged by human listeners. confidence: high relevance: medium - claim_id: dilated_causal_convolutions_enable_autoregressive_audio_models_to_achieve_receptive role: supports claim: Dilated causal convolutions enable autoregressive audio models to achieve receptive fields orders of magnitude larger than standard causal convolutions with comparable computational cost. source: §2.1, Figure 3 evidence: Dilated causal convolutions enable autoregressive audio models to achieve receptive fields orders of magnitude larger than standard causal convolutions with comparable computational cost. confidence: high relevance: medium - claim_id: a_single_autoregressive_model_conditioned_on_speaker_identity_can_represent role: supports claim: A single autoregressive model conditioned on speaker identity can represent many voices with shared internal structure, and multi-speaker training improves per-speaker quality relative to single-speaker training. source: §3.1 evidence: A single autoregressive model conditioned on speaker identity can represent many voices with shared internal structure, and multi-speaker training improves per-speaker quality relative to single-speaker training. confidence: high relevance: high - claim_id: receptive_field_size_is_a_binding_constraint_for_prosodic_naturalness role: supports claim: 'Receptive field size is a binding constraint for prosodic naturalness: when the receptive field is insufficient to cover phrase-level F0 contours, prosody degrades even when segmental quality remains high.' source: §3.2 evidence: 'Receptive field size is a binding constraint for prosodic naturalness: when the receptive field is insufficient to cover phrase-level F0 contours, prosody degrades even when segmental quality remains high.' confidence: high relevance: low - claim_id: autoregressive_raw_waveform_generation_achieves_high_naturalness_at_the_cost role: complicates claim: Autoregressive raw-waveform generation achieves high naturalness at the cost of sequential sample-level inference, creating a fundamental speed-quality trade-off that constrains deployment in real-time applications. source: §4, §3.2 evidence: Autoregressive raw-waveform generation achieves high naturalness at the cost of sequential sample-level inference, creating a fundamental speed-quality trade-off that constrains deployment in real-time applications. confidence: high relevance: medium limitations: - Inference is strictly sequential at the sample level, requiring approximately one computation step per generated sample. At the reported generation rates (roughly 1.5× real-time compute), WaveNet is not suitable for real-time TTS deployment without hardware-specific optimisation or a parallel decoding approximation. - Evaluation is conducted on proprietary Google TTS databases, making direct replication by external researchers impossible. The MOS comparison is fair internally (same data, same test sentences for all systems) but cannot be directly compared to numbers from other published evaluations. - The receptive field of 240 milliseconds covers 3-4 phonemes. Long-range prosodic structure above the syllable and phrase level requires either an external F0 model or a substantially larger receptive field than dilated convolutions alone provide efficiently. - WaveNet as presented requires high-quality linguistic features derived from a text analysis front-end. It is not end-to-end trainable from text to waveform, deferring the alignment and duration prediction problems to external modules. caveats: [] - id: '1904.02882' published_date: "2019-04-05" entry_date: '2026-07-29' year: 2019 venue: arXiv task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: [] claims: - claim_id: training_tts_models_at_24_khz_rather_than_16_khz role: supports claim: Training TTS models at 24 kHz rather than 16 kHz produces measurably higher subjective naturalness scores even when the underlying model architecture is held constant. source: §5.2, Table 4 evidence: Training TTS models at 24 kHz rather than 16 kHz produces measurably higher subjective naturalness scores even when the underlying model architecture is held constant. confidence: high relevance: medium - claim_id: sentence_boundary_segmentation_and_preserved_punctuation_are_theoretically_beneficial_for role: supports claim: Sentence-boundary segmentation and preserved punctuation are theoretically beneficial for TTS training, but their effect is difficult to isolate empirically because corpus size and acoustic quality covary when comparing corpus versions. source: §5.2 evidence: Sentence-boundary segmentation and preserved punctuation are theoretically beneficial for TTS training, but their effect is difficult to isolate empirically because corpus size and acoustic quality covary when comparing corpus versions. confidence: high relevance: medium - claim_id: snr_based_filtering_of_audiobook_recordings_improves_corpus_quality_at role: complicates claim: 'SNR-based filtering of audiobook recordings improves corpus quality at the cost of significantly reducing yield: approximately 25% of clean-subset sentences are removed by a 20 dB threshold.' source: §4, Table 3 evidence: 'SNR-based filtering of audiobook recordings improves corpus quality at the cost of significantly reducing yield: approximately 25% of clean-subset sentences are removed by a 20 dB threshold.' confidence: high relevance: medium - claim_id: gender_disparities_in_tts_naturalness_persist_under_multi_speaker_training role: supports claim: Gender disparities in TTS naturalness persist under multi-speaker training on large open corpora, with male voices showing larger gaps to natural speech than female voices. source: §5.2, Table 4 evidence: Gender disparities in TTS naturalness persist under multi-speaker training on large open corpora, with male voices showing larger gaps to natural speech than female voices. confidence: high relevance: medium limitations: - The experimental validation uses a single model architecture (GMVAE-Tacotron) and evaluates only six speakers from the train-clean subsets. The corpus size is approximately half that of LibriSpeech due to stricter filtering, which means some model comparisons are confounded by training data volume. - The stricter filtering pipeline produces an imbalanced per-speaker duration distribution with higher variance than LibriSpeech, which may affect the quality of low-duration speakers disproportionately. The text normalisation and sentence splitting rely on proprietary Google tools, creating a dependency that cannot be reproduced externally if regeneration is needed. The paper does not evaluate the impact of contextual sentence information, leaving that as an open research question. The significant naturalness gap for male speakers is noted but not resolved, and is attributed tentatively to model configuration rather than corpus issues. caveats: [] - id: '2301.02111' published_date: "2023-01-05" entry_date: '2026-07-29' year: 2023 venue: arXiv task: - TTS - codec architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: foundational method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: treating_tts_as_conditional_language_modeling_over_discrete_codec_tokens role: supports claim: Treating TTS as conditional language modeling over discrete codec tokens enables zero-shot speaker generalisation as in-context learning, without speaker-specific fine-tuning or engineered speaker encoders. source: §4.1, §5.2 evidence: Treating TTS as conditional language modeling over discrete codec tokens enables zero-shot speaker generalisation as in-context learning, without speaker-specific fine-tuning or engineered speaker encoders. confidence: high relevance: high - claim_id: training_on_large_scale_semi_supervised_speech_data_even_with role: supports claim: Training on large-scale semi-supervised speech data, even with noisy transcriptions and diverse acoustic conditions, yields stronger generalisation to unseen speakers than training on smaller clean corpora. source: §1, §5.2 evidence: Training on large-scale semi-supervised speech data, even with noisy transcriptions and diverse acoustic conditions, yields stronger generalisation to unseen speakers than training on smaller clean corpora. confidence: high relevance: medium - claim_id: the_hierarchical_structure_of_residual_vector_quantization_supports_a_two role: supports claim: The hierarchical structure of residual vector quantization supports a two-stage AR+NAR generation pipeline in which first-codebook tokens carry speaker identity and subsequent codebooks refine fine acoustic detail. source: §4.2 evidence: The hierarchical structure of residual vector quantization supports a two-stage AR+NAR generation pipeline in which first-codebook tokens carry speaker identity and subsequent codebooks refine fine acoustic detail. confidence: high relevance: high - claim_id: speaker_similarity_in_zero_shot_codec_tts_improves_monotonically_with role: supports claim: Speaker similarity in zero-shot codec TTS improves monotonically with acoustic prompt length, suggesting that speaker identity modelling does not saturate within a few seconds. source: §5.3, Table 6 evidence: Speaker similarity in zero-shot codec TTS improves monotonically with acoustic prompt length, suggesting that speaker identity modelling does not saturate within a few seconds. confidence: high relevance: high - claim_id: stochastic_sampling_in_autoregressive_codec_generation_introduces_output_diversity_varying role: supports claim: Stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation. source: §4.3, §5.4 evidence: Stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation. confidence: high relevance: low limitations: - 'Synthesis robustness is a material constraint: the autoregressive first-stage LM exhibits attention alignment failures that cause word deletions, insertions, and repetitions. WER on LibriSpeech test-clean is 5.9%, nearly three times the ground-truth rate of 2.2%. This limits deployment in applications requiring high intelligibility and is the principal limitation acknowledged by the authors.' - Training data is entirely English audiobook speech, limiting coverage of accented, spontaneous, or conversational speech styles. The two-model architecture (AR + NAR) adds inference complexity; the authors note that a single universal model is a natural future direction. Model parameter count is not reported, making compute comparisons with non-codec TTS systems difficult. The evaluation covers two benchmarks only; no noise-robustness or cross-domain experiments are included. Zero-shot voice cloning from 3 seconds of audio raises misuse risks that the paper acknowledges but does not experimentally mitigate. caveats: [] - id: '2305.07243' published_date: "2023-05-12" entry_date: '2026-07-29' year: 2023 venue: arXiv task: - TTS architecture: - autoregressive-LM - diffusion - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_prompt_conditioned_adaptation - diffusion_based_speaker_adaptation - vae_latent_speaker_adaptation claims: - claim_id: conditioning_a_diffusion_decoder_on_the_continuous_latent_activations_of role: supports claim: Conditioning a diffusion decoder on the continuous latent activations of an autoregressive model rather than its discrete token outputs substantially improves output quality in a cascaded AR-diffusion TTS pipeline. source: §2.2.2, Appendix B.4 evidence: Conditioning a diffusion decoder on the continuous latent activations of an autoregressive model rather than its discrete token outputs substantially improves output quality in a cascaded AR-diffusion TTS pipeline. confidence: high relevance: medium - claim_id: contrastive_re_ranking_of_multiple_autoregressive_candidates_using_a_text role: supports claim: Contrastive re-ranking of multiple autoregressive candidates using a text-speech discriminator measurably improves the final output quality of a TTS system without requiring the expensive decoder to process every candidate. source: §2.3, §4 evidence: Contrastive re-ranking of multiple autoregressive candidates using a text-speech discriminator measurably improves the final output quality of a TTS system without requiring the expensive decoder to process every candidate. confidence: high relevance: medium - claim_id: applying_image_generation_scaling_techniques_large_scale_self_supervised_data role: supports claim: Applying image-generation scaling techniques (large-scale self-supervised data, generalist transformer architectures, multi-stage AR-then-diffusion generation) to speech synthesis yields high-expressiveness multi-speaker TTS even when trained by a single researcher on commodity hardware. source: §7 evidence: Applying image-generation scaling techniques (large-scale self-supervised data, generalist transformer architectures, multi-stage AR-then-diffusion generation) to speech synthesis yields high-expressiveness multi-speaker TTS even when trained by a single researcher on commodity hardware. confidence: high relevance: medium - claim_id: building_a_large_scale_tts_training_corpus_by_scraping_and role: supports claim: Building a large-scale TTS training corpus by scraping and filtering internet audio (audiobooks, podcasts) with automatic transcription is a viable path to tens-of-thousands-of-hours datasets without manual labelling. source: §5, Appendix A evidence: Building a large-scale TTS training corpus by scraping and filtering internet audio (audiobooks, podcasts) with automatic transcription is a viable path to tens-of-thousands-of-hours datasets without manual labelling. confidence: high relevance: medium limitations: - No formal listening test or MOS table is reported. The primary quality claim rests on informal sample comparisons; the paper's own evaluation suite (CLVP-FID) is not a standard benchmark. This makes it difficult to place TorToise on the same scale as systems evaluated under controlled conditions. - 'Additional limitations noted in the paper include: slow inference due to multi-pass DDIM sampling (64 steps, large candidate sets) making real-time use impractical; fixed positional encodings in the AR model limiting maximum utterance length; the CLVP re-ranking model was only trained on sequences up to 13 seconds, degrading on longer outputs; and the VQVAE codebook embedding dimension was not constrained, which subsequent work showed to be a missed performance gain. Training was resource-constrained to 8 consumer GPUs, so model scale is limited relative to what the paper''s own training curves suggest would still improve results.' caveats: [] - id: '2406.04904' published_date: "2024-06-07" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_prompt_conditioned_adaptation - gan_based_speaker_transfer claims: - claim_id: multilingual_zero_shot_tts_training_degrades_speaker_similarity_compared_to role: complicates claim: Multilingual zero-shot TTS training degrades speaker similarity compared to monolingual training on the same data, reflecting a fundamental trade-off in cross-lingual speaker conditioning. source: §4.1, Table 2, Table 3 evidence: Multilingual zero-shot TTS training degrades speaker similarity compared to monolingual training on the same data, reflecting a fundamental trade-off in cross-lingual speaker conditioning. confidence: high relevance: high - claim_id: a_perceiver_resampler_based_speaker_conditioning_encoder_producing_multiple_fixed role: supports claim: A Perceiver Resampler-based speaker conditioning encoder, producing multiple fixed-length embeddings from variable-length reference audio, improves voice cloning robustness in massively multilingual autoregressive TTS over single-embedding approaches. source: §2 evidence: A Perceiver Resampler-based speaker conditioning encoder, producing multiple fixed-length embeddings from variable-length reference audio, improves voice cloning robustness in massively multilingual autoregressive TTS over single-embedding approaches. confidence: high relevance: high - claim_id: evaluating_multilingual_tts_models_against_monolingual_baselines_on_the_same role: supports claim: Evaluating multilingual TTS models against monolingual baselines on the same language produces misleading comparisons, because the multilingual model's per-language training data is substantially reduced. source: §3.2, §4.1 evidence: Evaluating multilingual TTS models against monolingual baselines on the same language produces misleading comparisons, because the multilingual model's per-language training data is substantially reduced. confidence: high relevance: low - claim_id: a_small_amount_of_target_speaker_fine_tuning_data_approximately role: supports claim: A small amount of target-speaker fine-tuning data (approximately 10 minutes) substantially improves speaker similarity in cross-lingual zero-shot synthesis, including extreme prosody styles such as whispering. source: §5 evidence: A small amount of target-speaker fine-tuning data (approximately 10 minutes) substantially improves speaker similarity in cross-lingual zero-shot synthesis, including extreme prosody styles such as whispering. confidence: high relevance: high - claim_id: low_frequency_codec_codebook_entries_can_be_pruned_without_quality role: supports claim: Low-frequency codec codebook entries can be pruned without quality loss and improve expressiveness in multilingual discrete-token TTS. source: §2 evidence: Low-frequency codec codebook entries can be pruned without quality loss and improve expressiveness in multilingual discrete-token TTS. confidence: high relevance: low limitations: - Speaker similarity lags behind monolingual specialists in English, and the multilingual evaluation uses cross-lingual prompting (English speaker references for non-English languages), which may understate true within-language similarity. No human listening test was conducted for multilingual outputs beyond subjective English comparisons. - The paper lacks ablations isolating the contribution of the Perceiver Resampler versus the larger reference representation alone. The VQ-VAE compression (21.53 Hz, 1 codebook) is highly compact compared to EnCodec at 75 Hz with 8 codebooks; the quality ceiling this imposes is not characterised against higher-fidelity codecs. Arabic and CJK language results remain weakest by CER (Table 4), and the causes, whether limited training data, romanisation quality, or tokeniser coverage, are not analysed. The authors acknowledge future intent to disentangle speaker and prosody for cross-speaker prosody transfer, which the current architecture does not support. caveats: [] - id: '2406.05370' published_date: "2024-06-08" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: adaptive_sampling_that_detects_and_breaks_token_repetition_loops_can role: supports claim: Adaptive sampling that detects and breaks token repetition loops can stabilise autoregressive codec LM decoding without requiring forced-alignment auxiliary data. source: §3.4.1, Table 1 evidence: Adaptive sampling that detects and breaks token repetition loops can stabilise autoregressive codec LM decoding without requiring forced-alignment auxiliary data. confidence: high relevance: low - claim_id: grouping_codec_codes_into_multi_token_ar_steps_reduces_effective role: supports claim: Grouping codec codes into multi-token AR steps reduces effective sequence length and simultaneously improves long-context modelling quality at moderate group sizes. source: §3.1, §4.2.1, Table 1 evidence: Grouping codec codes into multi-token AR steps reduces effective sequence length and simultaneously improves long-context modelling quality at moderate group sizes. confidence: high relevance: low - claim_id: autoregressive_codec_tts_can_match_or_exceed_ground_truth_speech role: supports claim: Autoregressive codec TTS can match or exceed ground-truth speech on robustness and speaker similarity metrics when evaluated on clean English audiobook benchmarks. source: §4.2.2, Table 2; §4.3.2, Table 5 evidence: Autoregressive codec TTS can match or exceed ground-truth speech on robustness and speaker similarity metrics when evaluated on clean English audiobook benchmarks. confidence: high relevance: high - claim_id: prompt_availability_in_both_the_ar_and_nar_stages_is role: supports claim: Prompt availability in both the AR and NAR stages is independently necessary for preserving speaker identity; removing either prompt degrades speaker similarity substantially. source: §4.2.3, Table 3; §4.3.3, Table 6 evidence: Prompt availability in both the AR and NAR stages is independently necessary for preserving speaker identity; removing either prompt degrades speaker similarity substantially. confidence: high relevance: high - claim_id: inference_time_multiple_sampling_followed_by_metric_based_selection_can role: supports claim: Inference-time multiple sampling followed by metric-based selection can substantially close the single-sample robustness gap, but at proportional computational cost. source: §4.1.3, Table 1 evidence: Inference-time multiple sampling followed by metric-based selection can substantially close the single-sample robustness gap, but at proportional computational cost. confidence: high relevance: medium limitations: - Human parity is claimed solely from results on LibriSpeech test-clean and VCTK; both benchmarks are read speech from controlled or semi-controlled recording conditions. Generalisation to spontaneous, noisy, or low-resource speech is undemonstrated and the authors explicitly flag this caveat. - The model is English-only and trained on audiobook data (Libriheavy), leaving multilingual and conversational speech scenarios unexplored. No model size is reported, making it difficult to assess parameter efficiency against competing approaches. Code is not released, limiting reproducibility. The paper does not compare against contemporaneous diffusion or flow-matching zero-shot systems such as NaturalSpeech 3 or Voicebox on the same test sets, so cross-paradigm positioning is unclear. The stability benefit of Repetition Aware Sampling comes at the cost of a non-deterministic inference procedure, which may complicate deployment in latency-sensitive applications. caveats: [] - id: '2406.07855' published_date: "2024-06-12" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: phoneme_monotonic_alignment_in_decoder_only_autoregressive_tts_can_close role: supports claim: Phoneme monotonic alignment in decoder-only autoregressive TTS can close most of the robustness gap caused by unconstrained attention, achieving near-ground-truth WER without encoder-decoder architectural changes. source: §3.2, Table 1 evidence: Phoneme monotonic alignment in decoder-only autoregressive TTS can close most of the robustness gap caused by unconstrained attention, achieving near-ground-truth WER without encoder-decoder architectural changes. confidence: high relevance: medium - claim_id: downsampling_only_the_first_rvq_layer_of_a_neural_codec role: supports claim: Downsampling only the first RVQ layer of a neural codec at inference time reduces autoregressive steps and latency by more than half, with negligible impact on PESQ and STOI. source: §3.1, Table 5 evidence: Downsampling only the first RVQ layer of a neural codec at inference time reduces autoregressive steps and latency by more than half, with negligible impact on PESQ and STOI. confidence: high relevance: low - claim_id: robustness_improvements_that_route_additional_phoneme_tokens_through_the_autoregressive role: complicates claim: Robustness improvements that route additional phoneme tokens through the autoregressive stream (as in ELLA-V) improve WER but increase inference time, illustrating a robustness-efficiency trade-off in codec LM TTS. source: §5.3, Table 4 evidence: Robustness improvements that route additional phoneme tokens through the autoregressive stream (as in ELLA-V) improve WER but increase inference time, illustrating a robustness-efficiency trade-off in codec LM TTS. confidence: high relevance: low - claim_id: explicit_phoneme_level_alignment_in_a_codec_lm_enables_independent role: supports claim: Explicit phoneme-level alignment in a codec LM enables independent control of prosody and timbre by substituting preset phoneme sequences at inference, enabling a form of voice conversion. source: §3.2.3, Table 3 evidence: Explicit phoneme-level alignment in a codec LM enables independent control of prosody and timbre by substituting preset phoneme sequences at inference, enabling a form of voice conversion. confidence: high relevance: low limitations: - All evaluations use LibriSpeech (clean English read speech). Robustness gains from monotonic alignment and codec-merging quality preservation have not been tested on noisy, expressive, or multilingual speech. - The model size is not explicitly reported, though the architecture (12-layer Transformer, 1024-dim hidden, 16 heads) matches the VALL-E reference scale. Code and model weights are not publicly released, limiting reproducibility. The prosody control evaluation uses MCD-DTW-SL, a proxy metric; perceptual validation of prosody cloning quality is absent. The merging rate of 2x is validated by reconstruction metrics but its downstream effect on naturalness under diverse speaker and content conditions is not fully explored. RALL-E (chain-of-thought prompting for robustness) is included only in the efficiency comparison, not in the WER robustness comparison, making head-to-head robustness assessment with concurrent work incomplete. caveats: [] - id: '2406.18009' published_date: "2024-06-26" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - flow_matching_speaker_adaptation claims: - claim_id: explicit_phoneme_alignment_supervision_in_non_autoregressive_tts_while_useful role: supports claim: Explicit phoneme alignment supervision in non-autoregressive TTS, while useful for early convergence, imposes a naturalness ceiling that joint character-level training can exceed. source: §3.4, Table 2 evidence: Explicit phoneme alignment supervision in non-autoregressive TTS, while useful for early convergence, imposes a naturalness ceiling that joint character-level training can exceed. confidence: high relevance: medium - claim_id: flow_matching_mel_spectrogram_generators_can_learn_alignment_implicitly_from role: supports claim: Flow-matching mel spectrogram generators can learn alignment implicitly from raw characters and filler tokens, eliminating the need for grapheme-to-phoneme converters, phoneme aligners, and duration models. source: §2.1, §2.3 evidence: Flow-matching mel spectrogram generators can learn alignment implicitly from raw characters and filler tokens, eliminating the need for grapheme-to-phoneme converters, phoneme aligners, and duration models. confidence: high relevance: medium - claim_id: unsupervised_pretraining_on_unlabeled_speech_improves_downstream_zero_shot_tts role: supports claim: Unsupervised pretraining on unlabeled speech improves downstream zero-shot TTS performance in both intelligibility and speaker similarity. source: §3.4, Table 1 evidence: Unsupervised pretraining on unlabeled speech improves downstream zero-shot TTS performance in both intelligibility and speaker similarity. confidence: high relevance: high - claim_id: zero_shot_tts_speaker_similarity_scores_at_inference_can_exceed role: supports claim: Zero-shot TTS speaker similarity scores at inference can exceed those of ground-truth recordings on standard speaker verification metrics, suggesting the metrics reward consistency within a generation rather than perceptual identity. source: §3.4, Table 2 evidence: Zero-shot TTS speaker similarity scores at inference can exceed those of ground-truth recordings on standard speaker verification metrics, suggesting the metrics reward consistency within a generation rather than perceptual identity. confidence: high relevance: high - claim_id: non_autoregressive_tts_systems_trained_jointly_on_duration_and_acoustics role: supports claim: Non-autoregressive TTS systems trained jointly on duration and acoustics scale predictably with training data volume without requiring architectural changes. source: §3.4, Table 1 evidence: Non-autoregressive TTS systems trained jointly on duration and acoustics scale predictably with training data volume without requiring architectural changes. confidence: high relevance: medium limitations: - 'The CMOS evaluation covers only 39 samples from 39 LibriSpeech speakers read English. The naturalness finding ("indistinguishable from ground truth") is thus narrow in scope: spontaneous speech, non-native speakers, expressive or prosodically complex content, and cross-lingual settings are untested.' - E2 TTS requires specifying a target duration at inference. A separate regression-based duration model (trained on Voicebox's approach) is used for fair baseline comparison, but this reintroduces a pipeline dependency. The paper does not propose a learned duration estimator integrated into the main model. - 'Inference requires the audio prompt length to be controlled carefully: the model must automatically identify the prompt/synthesis boundary from context, and very long prompts could confuse this. The analysis in §3.6.2 is reassuring for prompts up to 10 seconds but does not test extremes.' - The X1 extension degrades SIM-o modestly (0.664 vs 0.675 without pretraining), indicating some speaker identity leaks through the audio prompt's transcription. Whether this matters in practice depends on the application. caveats: [] - id: '2407.08551' published_date: "2024-07-11" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: continuous_mel_spectrogram_representations_preserve_more_speaker_relevant_acoustic_information role: supports claim: Continuous mel-spectrogram representations preserve more speaker-relevant acoustic information than vector-quantized codec codes at standard compression rates. source: §5.1, Table 1 evidence: Continuous mel-spectrogram representations preserve more speaker-relevant acoustic information than vector-quantized codec codes at standard compression rates. confidence: high relevance: low - claim_id: autoregressive_tts_models_trained_to_predict_continuous_frames_can_achieve role: supports claim: Autoregressive TTS models trained to predict continuous frames can achieve naturalness comparable to human speech while avoiding the silence and repetition failures endemic to discrete codec language models. source: §5.2, Table 3 evidence: Autoregressive TTS models trained to predict continuous frames can achieve naturalness comparable to human speech while avoiding the silence and repetition failures endemic to discrete codec language models. confidence: high relevance: low - claim_id: variational_sampling_in_the_continuous_latent_space_is_more_effective role: supports claim: Variational sampling in the continuous latent space is more effective than top-p discrete sampling for improving output diversity and speaker similarity in autoregressive TTS. source: §5.3, Table 4 evidence: Variational sampling in the continuous latent space is more effective than top-p discrete sampling for improving output diversity and speaker similarity in autoregressive TTS. confidence: high relevance: high - claim_id: a_reduction_factor_that_predicts_multiple_frames_per_autoregressive_step role: supports claim: A reduction factor that predicts multiple frames per autoregressive step can substantially reduce inference time with only modest degradation in speaker similarity. source: §5.4, Table 5 evidence: A reduction factor that predicts multiple frames per autoregressive step can substantially reduce inference time with only modest degradation in speaker similarity. confidence: high relevance: high limitations: - The subjective evaluation rests on only 40 samples from a single English corpus (LibriSpeech test-clean). The naturalness and speaker similarity advantages may not generalize to noisier prompts, non-native accents, or other languages. - The model's output quality is bounded by the HiFi-GAN vocoder, which was trained on only 585 hours of LibriTTS. Voicebox, which used a proprietary 60K-hour vocoder, showed higher SIM in part for this reason. Replacing or scaling the vocoder is identified as the most direct path to improvement. - The evaluation is English-only. Multilingual extension analogous to VALL-E X is deferred to future work. The paper also leaves open whether other continuous representations (VAE latent spaces, flow-based representations) would outperform mel-spectrograms as the target token. The model size is not reported, limiting cost comparisons with VALL-E 2 or Voicebox. caveats: [] - id: '2409.03283' published_date: "2024-09-05" entry_date: '2026-07-29' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation - hybrid_semantic_acoustic_adaptation claims: - claim_id: separating_the_waveform_generation_stage_into_a_low_sampling_rate role: supports claim: Separating the waveform generation stage into a low-sampling-rate Mel decoder and a super-resolution vocoder allows a system trained predominantly on low-sampling-rate data to produce high-fidelity 48kHz output. source: §3.3 evidence: Separating the waveform generation stage into a low-sampling-rate Mel decoder and a super-resolution vocoder allows a system trained predominantly on low-sampling-rate data to produce high-fidelity 48kHz output. confidence: high relevance: medium - claim_id: few_shot_fine_tuning_of_a_large_foundation_tts_model role: supports claim: Few-shot fine-tuning of a large foundation TTS model substantially outperforms zero-shot in-context learning for highly expressive, distinctive target voices, even with only one hour of data. source: §5.2.1, Table 5 evidence: Few-shot fine-tuning of a large foundation TTS model substantially outperforms zero-shot in-context learning for highly expressive, distinctive target voices, even with only one hour of data. confidence: high relevance: high - claim_id: prompt_audio_enhancement_improves_voice_cloning_quality_for_noisy_prompts role: supports claim: Prompt audio enhancement improves voice cloning quality for noisy prompts but can slightly degrade performance when prompts are already clean. source: §5.2.2, Table 6 evidence: Prompt audio enhancement improves voice cloning quality for noisy prompts but can slightly degrade performance when prompts are already clean. confidence: high relevance: medium - claim_id: instruction_tuning_with_a_small_domain_specific_dataset_dramatically_improves role: supports claim: Instruction tuning with a small domain-specific dataset dramatically improves emotion controllability in a pre-trained TTS language model, raising accuracy from near-chance to near-ceiling. source: §5.3, Table 7 evidence: Instruction tuning with a small domain-specific dataset dramatically improves emotion controllability in a pre-trained TTS language model, raising accuracy from near-chance to near-ceiling. confidence: high relevance: low - claim_id: autoregressive_tts_systems_trained_on_predominantly_one_language_show_markedly role: supports claim: Autoregressive TTS systems trained on predominantly one language show markedly higher pronunciation error rates on under-represented languages, even at large data scales. source: §5.1.2, Table 3 evidence: Autoregressive TTS systems trained on predominantly one language show markedly higher pronunciation error rates on under-represented languages, even at large data scales. confidence: high relevance: low limitations: - All evaluations are conducted on proprietary internal test sets with no publicly released benchmarks, data, or model weights. This makes direct comparison with other systems difficult to reproduce and limits the generalisability of the reported numbers. - The streamable decoder incurs a measurable quality penalty (0.07 CoMOS) and the paper notes that Mel codec quality is a bottleneck, which the authors flag for future work. The English and code-switch pronunciation error rates remain high (12% and 8.5%), driven by limited language diversity in training data. The paralinguistic behaviour framework currently supports 13 types targeting primarily Chinese conversational speech; coverage of other languages and more complex prosodic phenomena is not addressed. caveats: [] - id: '2502.18924' published_date: "2025-02-26" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - diffusion - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - diffusion_based_speaker_adaptation - vae_latent_speaker_adaptation claims: - claim_id: providing_coarse_stochastic_phoneme_anchors_rather_than_fully_expanded_forced role: supports claim: Providing coarse stochastic phoneme anchors rather than fully expanded forced alignments improves both naturalness and robustness simultaneously in latent diffusion TTS. source: §3.2, §4.3, Table 4, Table 7 evidence: Providing coarse stochastic phoneme anchors rather than fully expanded forced alignments improves both naturalness and robustness simultaneously in latent diffusion TTS. confidence: high relevance: medium - claim_id: compact_continuous_latent_representations_at_very_low_token_rates_enable role: supports claim: Compact continuous latent representations at very low token rates enable higher zero-shot TTS quality than discrete codecs at higher bit rates when used as the target space for diffusion. source: §4.5, Table 5, Table 6 evidence: Compact continuous latent representations at very low token rates enable higher zero-shot TTS quality than discrete codecs at higher bit rates when used as the target space for diffusion. confidence: high relevance: high - claim_id: piecewise_rectified_flow_distillation_reduces_inference_steps_from_25_to role: supports claim: Piecewise rectified flow distillation reduces inference steps from 25 to 8 with negligible degradation in speaker similarity and intelligibility. source: §3.2, §4.2, Table 1 evidence: Piecewise rectified flow distillation reduces inference steps from 25 to 8 with negligible degradation in speaker similarity and intelligibility. confidence: high relevance: high - claim_id: decoupled_text_and_speaker_guidance_scales_in_classifier_free_guidance role: supports claim: Decoupled text and speaker guidance scales in classifier-free guidance provide a continuous accent intensity control axis without requiring accent labels. source: §3.2, §4.4, Table 3 evidence: Decoupled text and speaker guidance scales in classifier-free guidance provide a continuous accent intensity control axis without requiring accent labels. confidence: high relevance: medium - claim_id: latent_diffusion_tts_systems_exhibit_strong_data_and_model_scaling role: supports claim: Latent diffusion TTS systems exhibit strong data and model scaling behaviour, with both speaker similarity and intelligibility improving consistently as training data grows from 2k to 600k hours and model size grows from 0.5B to 7B parameters. source: Appendix D, Table 8 evidence: Latent diffusion TTS systems exhibit strong data and model scaling behaviour, with both speaker similarity and intelligibility improving consistently as training data grows from 2k to 600k hours and model size grows from 0.5B to 7B parameters. confidence: high relevance: high limitations: - The main results are reported on LibriSpeech test-clean, a read-speech corpus recorded in controlled conditions. The scaling and cross-domain results (Appendix D) use an internal test set of 400 samples, limiting external reproducibility for those claims. - Language coverage is restricted to English and Chinese despite the 600k-hour multilingual training corpus. The sparse alignment mechanism still depends on an external forced aligner (Montreal Forced Aligner) at training time, which requires a transcription pipeline and does not eliminate the dependency on alignment tools, merely relaxing it. The relationship between alignment anchor density and generation quality is explored only qualitatively; no principled analysis determines the optimal sparsity level. Code and checkpoints are not publicly available at time of writing. caveats: [] - id: iclr-2025-cuFzE8Jlvb published_date: "2025-04-24" entry_date: '2026-07-29' year: 2025 venue: ICLR task: - TTS architecture: - autoregressive-LM - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - vae_latent_speaker_adaptation claims: - claim_id: continuous_latent_representations_can_replace_discrete_vector_quantization_in_autoregressive role: supports claim: Continuous latent representations can replace discrete vector quantization in autoregressive TTS without sacrificing generation quality. source: §5.1, Table 1, Table 2 evidence: GMM-LM trained on continuous GMM-VAE encoder features outperforms VALL-E (RVQ-based) on WER, speaker similarity, Q-MOS, and S-MOS on LibriSpeech test-clean across all prompt lengths, while using 10.3% of VALL-E's parameter count. confidence: high relevance: medium - claim_id: longer_audio_prompts_do_not_uniformly_improve_zero_shot_speaker role: complicates claim: Longer audio prompts do not uniformly improve zero-shot speaker cloning across AR architectures. source: §5.1, Table 2 evidence: VALL-E's WER increases monotonically with prompt length (6.04% at 3s, 7.54% at 8s, 9.68% at 15s), suggesting that simple cross-attention cannot leverage extended speaker context in AR decoding; the proposed GMM-LM shows the opposite trend, consistently benefiting from longer prompts. confidence: high relevance: high - claim_id: strict_monotonic_alignment_substantially_reduces_word_error_rate_in_autoregressive role: supports claim: Strict monotonic alignment substantially reduces word error rate in autoregressive TTS compared to standard cross-attention and soft monotonic variants. source: Appendix A.1, Table 6 evidence: Among alignment strategies tested on the same GMM-LM architecture, stochastic monotonic alignment with ST-Gumbel achieves WER 2.72% vs. 6.6% for cross-attention alone; even monotonic attention with Gumbel (without the stochastic binary forward pass) scores 3.34%. confidence: high relevance: medium - claim_id: continuous_speech_representations_improve_downstream_autoregressive_model_performance_relative_to role: supports claim: Continuous speech representations improve downstream autoregressive model performance relative to discrete counterparts, independent of the alignment mechanism. source: Appendix A.6, Table 10 evidence: A head-to-head ablation comparing GMM-LM (continuous) against discrete AR models (VQ-VAE single codebook and DAC multi-codebook with delayed prediction), all using the proposed monotonic alignment, shows GMM-LM achieves WER 2.72% vs. 5.35% and 5.87% for the discrete variants. confidence: high relevance: medium - claim_id: increasing_the_number_of_gaussian_components_in_continuous_ar_modeling role: complicates claim: Increasing the number of Gaussian components in continuous AR modeling yields diminishing returns and can reduce quality through overfitting. source: §5.4, Table 4, Table 5, Appendix A.3 evidence: GMM-LM with 6 diagonal-covariance Gaussians (WER 2.72%, SIM 0.91) outperforms 3-Gaussian (WER 2.89%, SIM 0.85), but 10-Gaussian degrades to WER 5.21%, SIM 0.71; the 6-mixture GMM-VAE also shows worse evaluation-set reconstruction than the 3-mixture model despite better training-set fit. confidence: high relevance: medium limitations: - The evaluation compares against VALL-E (2023), a dated AR baseline. Stronger AR systems published by the time of this paper's submission are not included, limiting the strength of the superiority claim for continuous AR over discrete AR in general. - The model is evaluated exclusively on English audiobook speech (LibriLight training, LibriSpeech evaluation). Generalisation to conversational speech, noisy in-the-wild data, or other languages is not demonstrated, though noise robustness experiments (Appendix A.4) show the method degrades gracefully under additive noise in prompts. - The GMM-VAE introduces an additional 76.5M-parameter component, partially offsetting the parameter savings claimed relative to VALL-E's RVQ codec (which uses 16.7M codec-related params). The total system size (GMM-VAE + GMM-LM-Mini) is 76.5M + 51.5M = 128M, larger than the headline 51.5M figure. - The stochastic monotonic alignment requires sequential per-step alignment computation (Algorithm 1), which may limit training throughput compared to fully parallelisable attention. The paper does not report training wall-clock times or throughput comparisons. - Code and pre-trained models are planned for release but were not available at submission time. caveats: [] - id: 2025.findings-naacl.130 published_date: "2025-04-29" entry_date: '2026-07-29' year: 2025 venue: NAACL task: - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation - gan_based_speaker_transfer claims: - claim_id: in_video_to_speech_synthesis_discrete_acoustic_unit_intermediate_representations role: supports claim: In video-to-speech synthesis, discrete acoustic-unit intermediate representations preserve speech content but discard speaker-identifying acoustic detail relative to continuous Mel-spectrogram representations. source: §3, Table 1, Figure 1 evidence: With ground-truth input held fixed, Unit-HiFiGAN scores SECS 0.555 / EER 40.52 versus HiFi-GAN's SECS 0.894 / EER 22.96, and a Mel-spectrogram visual comparison shows the unit-based vocoder output diverges from ground truth in the frequency domain. confidence: high relevance: medium - claim_id: audio_visual_pre_trained_visual_encoders_can_supply_enough_speaker role: supports claim: Audio-visual pre-trained visual encoders can supply enough speaker-identity information from silent video to make explicit speaker embeddings unnecessary in video-to-speech synthesis. source: §5.2, Table 4 evidence: DiVISe attains the best or near-best SECS scores on LRS2 (0.609) and LRS3 (0.624) among all compared methods, including several that use audio speaker embeddings during training or inference (SVTS, Multi-Task). confidence: high relevance: medium - claim_id: gains_in_objective_intelligibility_metrics_from_architectural_changes_do_not role: complicates claim: Gains in objective intelligibility metrics from architectural changes do not necessarily transfer to subjective audio-quality preference, especially when speaker-identity cues are removed from the listening context. source: Appendix C, Table 16 evidence: In an audio-only MOS test without speaker reference images, Unit-HiFiGAN scores higher (4.37±0.11) than the Mel-based HiFi-GAN (4.24±0.12), the inverse of the speaker-matching and intelligibility rankings observed elsewhere in the paper. confidence: high relevance: medium - claim_id: the_benefit_of_audio_visual_pre_training_for_video_to role: refines claim: The benefit of audio-visual pre-training for video-to-speech synthesis is not uniform across model components; it most strongly aids the component responsible for output representation choice rather than uniformly improving all downstream metrics. source: §5.4.2, Table 7 evidence: Removing pre-training degrades DiVISe across all four reported metrics (SECS, EER, ESTOI, WER), but for ReVISE removing pre-training mainly hurts intelligibility (WER 36.03 to 77.24) while leaving speaker metrics comparatively unaffected (SECS 0.5384 to 0.5304). confidence: high relevance: medium limitations: - Reported WER remains high (35.68-36.24% in the full-resource setting), and the paper's own subjective MOS audio-quality test (without speaker context) ranks the proposed Mel-based vocoder below the unit-based baseline, indicating the speaker-preservation gains are not accompanied by a clear audio-quality win in isolation. - The paper requires the same heavy mouth-region preprocessing pipeline as AV-HuBERT, which the authors note limits real-time applicability (§8, Limitations). Evaluation is restricted to English-only corpora (LRS2, LRS3); the authors explicitly flag multilingual generalization as untested due to compute constraints. The model's WER substantially trails dedicated ASR or text-conditioned TTS systems, reflecting the inherent difficulty of inferring content purely from lip movements without textual or acoustic priors. Latency analysis (§6) shows the conformer module, while improving intelligibility, reduces throughput relative to a no-conformer variant, an explicit accuracy/speed trade-off the paper surfaces but does not resolve. caveats: [] - id: 2025.findings-naacl.279 published_date: "2025-04-29" entry_date: '2026-07-29' year: 2025 venue: NAACL task: - TTS architecture: - autoregressive-LM - GAN - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - gan_based_speaker_transfer - hybrid_semantic_acoustic_adaptation claims: - claim_id: few_shot_fine_tuning_on_a_small_amount_of_target role: supports claim: Few-shot fine-tuning on a small amount of target-speaker studio audio substantially improves naturalness, clarity, and speaker similarity over a zero-shot pretrained multilingual TTS model, even when the zero-shot model was pretrained on thousands of hours of in-language speech. source: §4, Table 1 evidence: Fine-tuning on 20 minutes per speaker (4 speakers, 10 epochs) raises SMOS from 4.456 (BnTTS-0, zero-shot) to 4.624 (BnTTS-n, few-shot) and SECS (reference) from 0.529 to 0.548 on BnStudioEval. confidence: high relevance: high - claim_id: expressive_conversationally_prosodic_tts_output_can_score_worse_on_asr role: complicates claim: Expressive, conversationally-prosodic TTS output can score worse on ASR-based intelligibility metrics than flatter-sounding commercial systems, even when subjective listening scores favor the expressive system. source: §4, "High CER in Text Generation", Tables 1–2 evidence: BnTTS-n and BnTTS-0 show higher CER than Google Cloud TTS and Azure TTS on both BnStudioEval and BnTTSTextEval despite BnTTS-n achieving the highest SMOS, Naturalness, and Clarity scores overall; the authors attribute this to ASR systems being better suited to standardized, less expressive speech patterns. confidence: high relevance: medium - claim_id: autoregressive_codec_based_tts_models_trained_primarily_on_longer_utterances role: complicates claim: Autoregressive codec-based TTS models trained primarily on longer utterances can fail to generate short sequences reliably under default sampling settings. source: §4, "Effect of Sampling and Prompt Length on Short Speech Generation", Table 3 evidence: Under default generation settings (T=0.85, TopK=50), the model achieves only 0.699 DurationEquality and 0.081 CER on a 23-sample short-text subset (under 30 characters); adjusting prompt length and sampling temperature/top-K improves this to 0.827 DurationEquality and 0.015 CER. confidence: high relevance: low - claim_id: adapting_a_multilingual_zero_shot_tts_architecture_to_a_new role: refines claim: Adapting a multilingual zero-shot TTS architecture to a new low-resource language via continual pretraining can match or exceed existing language-specific commercial and open-source systems without requiring training from scratch. source: §3 "BnTTS model", §7 "Limitations", Tables 1–2 evidence: BnTTS-n, initialized from the pretrained XTTS checkpoint and continually pretrained on 3.85k hours of Bengali data, outperforms IndicTTS, Google Cloud TTS, and Azure TTS on SMOS, Naturalness, and Clarity across both evaluation datasets, while the authors note training from scratch was not attempted due to resource constraints. confidence: high relevance: high limitations: - The system struggles to adapt to speakers with unique vocal traits absent from training, and fails to generate short sequences (under two words or 20 characters) in some cases even after mitigation; these failures stem from limitations inherited from the underlying XTTS foundation model rather than being fully resolved by the Bangla adaptation. - The authors did not train XTTS from scratch for Bangla, relying instead on continual pretraining due to resource constraints, leaving the ceiling of a from-scratch Bangla model untested. A substantial portion of the pretraining corpus (3636 of 3856 hours) is pseudo-labeled and not manually reviewed, which may introduce label noise not fully captured by the reported metrics. The evaluation is limited to four in-house speakers for the few-shot speaker-adaptation setting, and dialectal variation within Bangla is not addressed. caveats: [] - id: 2025.naacl-short.65 published_date: "2025-04-29" entry_date: '2026-07-29' year: 2025 venue: NAACL task: - TTS architecture: - hybrid - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation - gan_based_speaker_transfer claims: - claim_id: ssl_feature_spaces_from_pre_trained_models_encode_cross_speaker role: supports claim: SSL feature spaces from pre-trained models encode cross-speaker structure that enables zero-shot voice transfer through nearest-neighbor retrieval, without speaker-specific training data. source: §2.1, Table 1 evidence: kNN-TTS uses WavLM-Large layer 6 features, where frames from different speakers that are linearly close share phonetic information while preserving speaker identity; kNN retrieval over these features achieves SECS 0.72 and competitive MOS scores trained only on 24h of single-speaker LJSpeech data. confidence: high relevance: high - claim_id: zero_shot_multi_speaker_tts_competitive_with_large_multi_speaker role: supports claim: Zero-shot multi-speaker TTS competitive with large multi-speaker end-to-end systems can be achieved with single-speaker transcribed training data by delegating speaker identity to inference-time retrieval. source: §4, Table 1 evidence: GlowkNN-TTS (24h training, single speaker) achieves N-MOS and S-MOS within the confidence intervals of HierSpeech++ (2,796h, 7299 speakers) and XTTS (27,282h, multi-speaker) on LibriSpeech test-clean. confidence: high relevance: high - claim_id: retrieval_based_zero_shot_tts_requires_substantially_more_reference_audio role: complicates claim: Retrieval-based zero-shot TTS requires substantially more reference audio from the target speaker than embedding-based approaches to achieve sufficient quality. source: §Limitations, Figure 3b evidence: kNN-TTS requires approximately 30 seconds of target speaker audio for suitable intelligibility and around 1 minute for speaker similarity to plateau, whereas competing embedding-based systems show diminishing returns beyond 10-30 seconds of reference audio. confidence: high relevance: high - claim_id: frame_level_knn_speaker_transfer_does_not_address_speaker_specific role: complicates claim: Frame-level kNN speaker transfer does not address speaker-specific duration and rhythm, leaving prosodic timing patterns fixed to the training speaker. source: §Limitations "Rhythmic variations" evidence: In kNN-TTS, utterance duration is determined entirely by the single-speaker Text-to-SSL model; frame-by-frame retrieval substitutes voice quality but does not adapt speaking rate or rhythm to the target speaker. confidence: high relevance: medium limitations: - 'The reference audio requirement is a practical limitation: kNN-TTS needs approximately 30 seconds of target speaker audio for usable intelligibility, which is notably higher than embedding-based competitors that can function with shorter clips. This restricts applicability in truly few-shot or single-utterance zero-shot scenarios.' - Duration adaptation to the target speaker is not addressed; the speaking rate and rhythm of the output always reflect the training speaker (LJSpeech). The paper proposes Urhythmic-style rhythm modeling as future work. Evaluation is English-only, and while the authors note potential for cross-lingual transfer (via kNN-VC cross-lingual capabilities), this is not demonstrated. Using mel-spectrogram features as an alternative to SSL features was ablated and found completely ineffective, confirming the dependency on WavLM's particular representational structure. caveats: [] - id: 2025.americasnlp-1.1 published_date: "2025-05-04" entry_date: '2026-07-29' year: 2025 venue: AmericasNLP 2025 task: - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation - gan_based_speaker_transfer claims: - claim_id: transfer_learning_from_a_closely_related_language_can_enable_intelligible role: supports claim: Transfer learning from a closely related language can enable intelligible TTS for an indigenous language with fewer than 5 hours of training data. source: §4.1, §5.1 evidence: Transfer learning from a closely related language can enable intelligible TTS for an indigenous language with fewer than 5 hours of training data. confidence: high relevance: low - claim_id: subjective_evaluation_by_native_speakers_is_necessary_when_objective_metrics role: supports claim: Subjective evaluation by native speakers is necessary when objective metrics such as PESQ and POLQA cannot be applied due to absent high-quality reference recordings. source: §5.3 evidence: Subjective evaluation by native speakers is necessary when objective metrics such as PESQ and POLQA cannot be applied due to absent high-quality reference recordings. confidence: high relevance: low - claim_id: pronunciation_drift_in_a_native_speaker_living_outside_the_language role: supports claim: Pronunciation drift in a native speaker living outside the language community can cause natural speech to score lower on intelligibility than synthetic speech on standardised evaluation. source: §5.4 evidence: Pronunciation drift in a native speaker living outside the language community can cause natural speech to score lower on intelligibility than synthetic speech on standardised evaluation. confidence: high relevance: low - claim_id: gan_based_vocoders_initialized_from_a_universal_pretrained_model_converge role: supports claim: GAN-based vocoders initialized from a universal pretrained model converge rapidly when the target speaker is similar to the pretraining distribution. source: §4.2, §5.2 evidence: GAN-based vocoders initialized from a universal pretrained model converge rapidly when the target speaker is similar to the pretraining distribution. confidence: high relevance: high limitations: - '- Corpus is only 4h 37m from a single female speaker, limiting prosodic and phonetic diversity and voice coverage. - Transfer learning from Spanish may introduce Spanish phonotactic biases that are not present in Shipibo-Konibo. - Evaluation relied entirely on subjective metrics (no PESQ/POLQA due to lack of reference data); subjectivity and evaluator bias are acknowledged. - Two writing conventions for Shipibo-Konibo exist; the 2015 standard was used, limiting applicability in communities using older orthographies. - The speaker''s extended residence in Lima introduced pronunciation drift that complicates the natural vs. synthetic comparison. - The model has not been tested on adaptation to other Panoan or Amazonian languages yet, though the framework is intended to generalize.' caveats: [] - id: '2505.07916' published_date: "2025-05-12" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation - vae_latent_speaker_adaptation claims: - claim_id: jointly_training_a_speaker_encoder_with_an_autoregressive_tts_model role: supports claim: Jointly training a speaker encoder with an autoregressive TTS model yields better intelligibility and competitive speaker similarity compared to using a fixed speaker verification embedding. source: §3.6, Table 4 evidence: Jointly training a speaker encoder with an autoregressive TTS model yields better intelligibility and competitive speaker similarity compared to using a fixed speaker verification embedding. confidence: high relevance: high - claim_id: zero_shot_voice_cloning_conditioned_on_untranscribed_reference_audio_produces role: complicates claim: Zero-shot voice cloning conditioned on untranscribed reference audio produces lower word error rates than one-shot conditioning with a paired text-audio exemplar, at the cost of slightly reduced speaker similarity. source: §3.2, Table 1; §3.5, Table 3 evidence: Zero-shot voice cloning conditioned on untranscribed reference audio produces lower word error rates than one-shot conditioning with a paired text-audio exemplar, at the cost of slightly reduced speaker similarity. confidence: high relevance: high - claim_id: augmenting_a_vae_with_a_normalizing_flow_on_the_latent role: supports claim: Augmenting a VAE with a normalizing flow on the latent space (Flow-VAE) improves both waveform reconstruction quality and downstream TTS speaker similarity over a standard VAE with the same architecture. source: §3.7, Tables 5–6 evidence: Augmenting a VAE with a normalizing flow on the latent space (Flow-VAE) improves both waveform reconstruction quality and downstream TTS speaker similarity over a standard VAE with the same architecture. confidence: high relevance: high - claim_id: speaker_encoder_representations_trained_without_text_dependencies_support_cross_lingual role: supports claim: Speaker encoder representations trained without text dependencies support cross-lingual synthesis with high intelligibility, outperforming prompt-based systems that require transcribed reference audio in cross-lingual scenarios. source: §3.5, Table 3 evidence: Speaker encoder representations trained without text dependencies support cross-lingual synthesis with high intelligibility, outperforming prompt-based systems that require transcribed reference audio in cross-lingual scenarios. confidence: high relevance: high - claim_id: disentangled_speaker_embeddings_from_a_task_specific_encoder_enable_parameter role: supports claim: Disentangled speaker embeddings from a task-specific encoder enable parameter-efficient per-speaker adaptation by fine-tuning only the speaker embedding vector, preserving base model generalization. source: §4.3 evidence: Disentangled speaker embeddings from a task-specific encoder enable parameter-efficient per-speaker adaptation by fine-tuning only the speaker embedding vector, preserving base model generalization. confidence: high relevance: high limitations: - The training dataset is proprietary and the model size is not reported, which makes independent replication impossible. All results are from internal or public leaderboard evaluations only; no code release is indicated. - The leaderboard evaluation (Artificial Analysis Arena) uses preference judgements that may be influenced by sample selection; the Arena snapshot is from a single date (May 12, 2025) and rankings can shift as more models are added. The claim of "top position" is time-bound. - WER metrics for several tested languages (Cantonese, French, Hindi) remain substantially higher than for well-resourced languages, indicating multilingual coverage is uneven. The cross-lingual evaluation is limited to seven target languages using Chinese source speakers, leaving broader cross-lingual generalization untested. - Emotion control via LoRA requires separate training runs per emotion category and discrete emotion labels, which limits granularity. The text-to-voice extension relies on structured attribute tags alongside free-text descriptions, constraining the range of expressible timbres. No human evaluation of the LoRA emotion control or text-to-voice outputs is reported. caveats: [] - id: '2506.13053' published_date: "2025-06-16" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: compact_flow_matching_tts_models_can_match_the_speech_quality role: supports claim: Compact flow-matching TTS models can match the speech quality of models two to eight times larger when architectural components are selected for per-parameter efficiency rather than chosen by default from the diffusion literature. source: §V.A, Table I evidence: Compact flow-matching TTS models can match the speech quality of models two to eight times larger when architectural components are selected for per-parameter efficiency rather than chosen by default from the diffusion literature. confidence: high relevance: medium - claim_id: in_non_autoregressive_tts_without_explicit_duration_prediction_a_simple role: supports claim: In non-autoregressive TTS without explicit duration prediction, a simple uniform-duration upsampling assumption provides substantially better initial alignment than filler-token padding, yielding large intelligibility improvements without additional parameters. source: §II.D, §V.C, Table IV evidence: In non-autoregressive TTS without explicit duration prediction, a simple uniform-duration upsampling assumption provides substantially better initial alignment than filler-token padding, yielding large intelligibility improvements without additional parameters. confidence: high relevance: medium - claim_id: flow_distillation_that_conditions_the_student_on_cfg_strength_eliminates role: complicates claim: Flow distillation that conditions the student on CFG strength eliminates the doubled forward-pass cost of classifier-free guidance while preserving its quality benefit, outperforming consistency distillation and ReFlow at 4 NFEs. source: §II.E, §V.E, Table VI evidence: Flow distillation that conditions the student on CFG strength eliminates the doubled forward-pass cost of classifier-free guidance while preserving its quality benefit, outperforming consistency distillation and ReFlow at 4 NFEs. confidence: high relevance: medium - claim_id: the_u_net_style_multi_resolution_structure_and_convolutional_modules role: supports claim: The U-Net-style multi-resolution structure and convolutional modules in Zipformer transfer the inductive biases beneficial for diffusion-family models from ASR into TTS vector field estimation, with ablation evidence showing each structural element independently contributes to intelligibility and naturalness. source: §II.C, §V.D, Table V evidence: The U-Net-style multi-resolution structure and convolutional modules in Zipformer transfer the inductive biases beneficial for diffusion-family models from ASR into TTS vector field estimation, with ablation evidence showing each structural element independently contributes to intelligibility and naturalness. confidence: high relevance: medium limitations: - Speaker similarity (SIM-o) for ZipVoice-Distill is consistently below F5-TTS and the larger NAR baselines on all three test sets (Table I), despite quality and intelligibility advantages. This trade-off is acknowledged but not explained mechanistically; it is unclear whether it stems from the uniform-duration alignment assumption, the distillation objective, or the compact model size. - The paper uses UTMOS rather than human MOS for most evaluations; CMOS/SMOS human scores are reported only against a limited set of baselines, making the subjective quality advantage difficult to verify broadly. Average upsampling assumes uniform token durations within a sentence, which is a poor model of natural prosody, potentially limiting expressiveness for prosodically varied speech. The model operates on mel features and requires Vocos for waveform synthesis; end-to-end codec-native generation is not explored. caveats: [] - id: '2507.00808' published_date: "2025-07-02" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: iterative_natural_language_feedback_can_progressively_refine_the_speaking_style role: supports claim: Iterative natural language feedback can progressively refine the speaking style of synthesized speech without accumulating naturalness degradation. source: §4.2, §4.4, Figure 4, Figure 6 evidence: Over three interaction sessions, Iterative (ours) significantly outperformed the Identical baseline on a 5-point style refinement MOS, and naturalness MOS showed no significant difference between Iterative and the oracle condition across all style groups. confidence: high relevance: low - claim_id: global_speech_embedding_based_conditioning_cannot_accurately_reflect_fine_grained role: complicates claim: Global speech embedding-based conditioning cannot accurately reflect fine-grained positional or linguistic instructions in expressive TTS. source: §5.1, §5.2, Table 4 evidence: Low-scoring examples in the test set contained directions targeting specific word positions ("at the beginning", "at the end", "for the part of...") or linguistic modifications ("hold your breath", "place just a slight pause between words"), which the speech embedding manipulation approach could not handle. confidence: high relevance: medium - claim_id: semantic_similarity_of_style_direction_text_not_exact_wording_governs role: supports claim: Semantic similarity of style direction text, not exact wording, governs how well listeners perceive style refinement as aligned with the instruction. source: §4.3, Figure 5 evidence: In the style refinement accuracy evaluation, directions semantically similar to the one used for refinement (Random Similar) yielded scores comparable to the Matched condition, while semantically dissimilar directions scored significantly lower across all style groups. confidence: high relevance: medium - claim_id: holistic_subjective_evaluation_scales_may_not_adequately_capture_fine_grained role: complicates claim: Holistic subjective evaluation scales may not adequately capture fine-grained stylistic alignment in iterative TTS refinement tasks. source: §4.2, §5.3 evidence: Even the Actor-Guided oracle condition scored around 3 out of 5 on the iterative style refinement MOS, which the authors attribute to the evaluation task not fully discriminating subtle style differences; similar evaluation difficulties have been noted in text-to-image/video generation research. confidence: high relevance: low limitations: - All training and evaluation data is proprietary in-house Japanese speech from two voice actors. No public dataset is used, and no results are reported outside this setup. Reproducibility and generalization are untested. - 'The style refiner is speaker-dependent; the authors plan to extend to speaker-independent operation as future work. The current model refines only paralinguistic information (speaking style via global embeddings), not linguistic content, so instructions involving pauses, stress, or pitch accent placement cannot be followed. The directions cover only two of four practical categories from actual recording sessions (paralinguistic and text-expressible linguistic instructions), omitting demonstrative and gestural instructions entirely. The evaluation task design is also noted as an open problem: the relatively low absolute scores even under oracle conditions suggest that existing MOS paradigms do not cleanly measure this type of fine-grained iterative alignment, and more sensitive evaluation methods are needed.' caveats: [] - id: '2507.02380' published_date: "2025-07-03" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: minor method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation claims: - claim_id: routing_llm_hidden_states_into_the_tts_module_s_embedding role: supports claim: Routing LLM hidden states into the TTS module's embedding space can enable voice cloning in an end-to-end spoken chatbot without a dedicated speaker encoder. source: §1.2 evidence: JoyTTS projects Qwen-7B hidden states via an MLP into the CosyVoice2-based LLM-TTS embedding, achieving SS of 0.73 on seed-tts-zh using prompt audio as the only speaker reference. confidence: high relevance: high - claim_id: jointly_training_an_llm_chat_module_with_a_tts_module role: complicates claim: Jointly training an LLM-Chat module with a TTS module in a chatbot pipeline can degrade intelligibility relative to running the TTS component standalone, even when speaker similarity improves. source: §4, Table 1 evidence: JoyTTS achieves WER of 5.09 compared to 1.45 for standalone CosyVoice2 on seed-tts-zh, despite closing the speaker similarity gap (JoyTTS SS 0.73 vs. CosyVoice2 SS 0.748). confidence: high relevance: high - claim_id: end_to_end_spoken_chatbot_systems_pairing_a_7b_parameter role: supports claim: End-to-end spoken chatbot systems pairing a 7B-parameter LLM with an autoregressive TTS module can achieve sub-2-second response latency on a single consumer GPU without specialised inference optimisations. source: §4 evidence: JoyTTS reports 1.8-second end-to-end latency on a single NVIDIA 4090D with no engineering optimisation applied. confidence: high relevance: low limitations: - The evaluation is limited to a single Chinese benchmark (seed-tts-zh), leaving performance on English, multilingual, or spontaneous conversational speech uncharacterised. The WER gap between JoyTTS (5.09) and standalone CosyVoice2 (1.45) is large and unexplained; the paper does not ablate whether the regression originates from the joint training procedure, the conversational data distribution, or the MLP projection coupling. No subjective listening tests are reported, making it impossible to assess naturalness or perceived quality beyond intelligibility and speaker similarity metrics. Emotion control, identified as a target for future work, is not implemented in the current system. caveats: [] - id: '2507.08319' published_date: "2025-07-11" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: active_learning_with_model_informed_informativeness_criteria_produces_more_data role: supports claim: Active learning with model-informed informativeness criteria produces more data-efficient TTS corpora than model-agnostic static data selection at equal corpus size. source: §III.B, Table I evidence: The proposed method achieves 71.0% synthesizable speaker ratio at 3,943 utterances versus 66.6% for the static baseline of the same size, measured by pseudo-MOS exceedance rate. confidence: high relevance: medium - claim_id: data_selection_methods_that_optimize_synthesizable_speaker_coverage_may_reduce role: complicates claim: Data selection methods that optimize synthesizable speaker coverage may reduce synthesis quality for speakers already near the quality threshold. source: §III.B, Figure 3 evidence: The proposed method underperforms the static baseline for speakers with pseudo-MOS above 2.6, because the informativeness criterion explicitly excludes speakers the current model can already synthesize well. confidence: high relevance: medium - claim_id: tts_corpus_quality_estimators_remain_effective_when_trained_on_a role: supports claim: TTS corpus quality estimators remain effective when trained on a small fraction of candidate data rather than requiring full candidate set pre-processing. source: §III.B, Figure 7 evidence: The data quality estimator trained on 10% of the YouTube candidate pool shows a strong correlation with the estimator trained on 100% of the data, with no significant change in selection behavior. confidence: high relevance: medium - claim_id: diffusion_models_better_capture_complex_speaker_embedding_distributions_than_gaussian role: supports claim: Diffusion models better capture complex speaker embedding distributions than Gaussian mixture models for speaker generation tasks with limited training samples. source: §III.B, Figure 5 evidence: The diffusion model achieves Wasserstein distance to held-out test speakers that is more than twice the standard deviation lower than GMMs across all cluster sizes (M=1..10). confidence: high relevance: high limitations: - All experiments are conducted on Japanese YouTube data only. Generalizability to other languages, recording conditions, or web data sources is untested, and the quality threshold calibration requires a language-matched studio corpus. - The active learning loop is evaluated with only two iterations (K=2). Whether additional iterations continue to improve coverage or reach diminishing returns is not explored. The main baseline comparison is the authors' own prior static selection method; no comparison against other active learning or subset selection approaches from the computer vision or NLP literature is included. The evaluation uses pseudo-MOS (an automatic quality predictor) rather than subjective listening tests, so the absolute synthesis quality of the final corpora is not independently verified. caveats: [] - id: '2507.09282' published_date: "2025-07-12" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - diffusion - VAE relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - diffusion_based_speaker_adaptation - vae_latent_speaker_adaptation claims: - claim_id: the_normalising_tendency_of_zero_shot_tts_models_can_be role: supports claim: The normalising tendency of zero-shot TTS models can be deliberately exploited to filter atypical speech characteristics while preserving general voice timbre. source: §2.2, §4, Table 1 evidence: ClaritySpeech applies XTTSv2 to dementia-affected speech without fine-tuning, reducing adversarial dementia detection F1 by 16% on ADReSS (total mean 0.70 to 0.59) while maintaining speaker similarity of 0.50, as the model smooths prosodic irregularities learned from diverse fluent training data. confidence: high relevance: high - claim_id: speaker_identity_preservation_and_speech_naturalness_trade_off_against_each role: complicates claim: Speaker identity preservation and speech naturalness trade off against each other when zero-shot TTS is applied to atypical speech. source: §4, Table 1 evidence: StyleTTS2 achieves UTMOS 2.86 and WER 0.04 on ADReSS but speaker similarity of only 0.36, while XTTSv2 achieves UTMOS 2.15 and speaker similarity 0.50, showing that models that normalise most aggressively sacrifice voice fidelity. confidence: high relevance: high - claim_id: text_level_content_correction_and_acoustic_re_synthesis_provide_complementary role: supports claim: Text-level content correction and acoustic re-synthesis provide complementary privacy protection that neither provides alone in disordered speech obfuscation. source: §4, Table 2 evidence: Ablation on ADReSS shows that removing text obfuscation raises text-modality F1 by 10% (0.59 to 0.68), while removing zero-shot TTS collapses speaker similarity to 0.12 and degrades privacy protection; the full pipeline requires both components. confidence: high relevance: medium - claim_id: pipeline_based_speech_obfuscation_systems_that_chain_asr_text_processing role: complicates claim: Pipeline-based speech obfuscation systems that chain ASR, text processing, and TTS carry latency that precludes real-time deployment on standard hardware. source: §4, Table 4 evidence: ClaritySpeech achieves a mean real-time factor of 6.07 on CPU for 3-6 second audio samples; ASR alone contributes RTF 3.01 and TTS contributes RTF 1.71, making live speech obfuscation impractical without GPU acceleration. confidence: high relevance: low limitations: - 'Evaluation uses small, low-diversity datasets (ADReSS: 690 samples, ADReSSo: 947 samples) from a single task (cookie-theft description). Generalisability to other dementia types, languages, or conversational speech is untested.' - The adaptive adversary setting, in which the detection model trains on both original and obfuscated samples, remains difficult to defeat — mean F1 drops are modest (roughly 5 percentage points) under adaptive conditions. No comparison with speaker-dependent fine-tuned TTS models is provided, leaving open whether fine-tuning would offer a better privacy-utility trade-off despite the privacy risks. The system is English-only. Latency (RTF 6.07) rules out real-time applications without hardware acceleration. The effect on the speaker's perceived identity from the listener's perspective is measured only automatically via ECAPA-TDNN embeddings, not through subjective listening tests. caveats: [] - id: '2506.18296' published_date: "2025-07-15" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - VC - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: non_anonymous_speaker_corpora_with_publicly_identifiable_voices_enable_more role: supports claim: Non-anonymous speaker corpora with publicly identifiable voices enable more rigorous subjective evaluations of speaker similarity in TTS and VC systems. source: §1, §3.1 evidence: JIS assigns stage names to 169 Japanese live idol speakers, allowing experiment designers to recruit listeners familiar with the speakers, enabling discrimination of subtle voice characteristics that anonymous corpus evaluations cannot capture. confidence: high relevance: high - claim_id: automatic_mos_predictors_trained_on_tts_generated_speech_may_underestimate role: complicates claim: Automatic MOS predictors trained on TTS-generated speech may underestimate audio quality when applied to recordings of non-professional speakers, even under studio conditions. source: §4.2.1 evidence: JIS Speech A (studio-recorded) achieves a mean UTMOS of 3.4 compared to JVS parallel100's 3.7, with the gap attributed to linguistic mismatch in UTMOS training data and speech hesitations inherent to non-professional speakers rather than recording quality differences. confidence: high relevance: medium - claim_id: speaking_style_and_communicative_context_introduce_systematic_variation_in_speaker role: supports claim: Speaking style and communicative context introduce systematic variation in speaker embeddings that interacts with speaker identity, presenting a challenge for style-robust speaker representation. source: §4.2.2 evidence: ECAPA-TDNN embeddings of JIS speakers show that specific speaking styles (energetic post-performance greetings, intimate photo-session speech) produce cross-speaker clusters in t-SNE, partially overriding individual speaker identity, while speech expressing personal individuality is more dispersed. confidence: high relevance: high limitations: - The corpus is small (17 hours, 169 speakers) and covers only young Japanese female voices, restricting direct use for general-purpose or multilingual TTS training. Distribution requires a signed agreement and is limited to non-commercial basic research. Speech B recording conditions are unspecified and variable across groups, introducing inconsistencies in audio quality. No TTS or VC model is trained on JIS in this work, leaving empirical validation of the corpus's utility for model development open. caveats: [] - id: '2507.15272' published_date: "2025-07-21" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - diffusion_based_speaker_adaptation claims: - claim_id: conditioning_duration_prediction_on_reference_audio_from_the_same_speaker role: supports claim: Conditioning duration prediction on reference audio from the same speaker can improve speaker similarity in zero-shot diffusion TTS without modifying the training objective. source: §2.2, Table 2 evidence: Cross-attention over an unrelated reference mel-spectrogram (same speaker, different content) improves SIM-O over a speaker-embedding-conditioned Grad-TTS baseline across all seven Indian languages tested. confidence: high relevance: high - claim_id: classifier_free_guidance_with_a_dataset_mean_unconditional_condition_improves role: supports claim: Classifier-free guidance with a dataset-mean unconditional condition improves speaker consistency at inference time in diffusion-based TTS for unseen speakers. source: §2.3, Table 2 evidence: A2TTS applies CFG at inference, following UnitSpeech, to amplify speaker conditioning without changing the training procedure; speaker similarity gains are observed across all languages evaluated. confidence: high relevance: medium - claim_id: diffusion_based_zero_shot_tts_can_achieve_reasonable_speaker_similarity role: complicates claim: Diffusion-based zero-shot TTS can achieve reasonable speaker similarity, but intelligibility on out-of-domain benchmarks degrades substantially relative to ground-truth speech. source: §Appendix A, Tables 3-9 evidence: A2TTS TTS WER on FLEURS, CommonVoice, and IndicTTS is 1.5-4x higher than the corresponding ASR baseline on the same test sets across Hindi, Marathi, Punjabi, Bengali, Gujarati, Tamil, and Malayalam. confidence: high relevance: high - claim_id: language_specific_diffusion_tts_models_can_be_trained_on_standard role: supports claim: Language-specific diffusion TTS models can be trained on standard benchmark speech corpora to achieve multi-speaker zero-shot synthesis in low-resource settings. source: §3.1, Table 2 evidence: A2TTS trains separate models for seven Indian languages using only the IndicSUPERB dataset (except Malayalam and Tamil which also use IndicTTS), achieving SIM-O above 0.70 for all languages on the IndicSUPERB evaluation set. confidence: high relevance: high limitations: - No naturalness evaluation (MOS or automated predictors) is reported. Speaker similarity (SIM-O) and CER are the only metrics, so claims about speech quality and naturalness cannot be verified from the paper's own experiments. - The model is only compared to a single baseline (Grad-TTS with speaker embeddings), without evaluating against UnitSpeech (from which the speaker encoder is borrowed), YourTTS, or other zero-shot multilingual systems. This makes it difficult to assess how much of the improvement comes from the cross-attention duration predictor versus the overall system design. - The paper trains language-specific models rather than a single multilingual model, which limits scalability to new languages. The authors acknowledge that out-of-domain speakers and languages not covered by IndicSUPERB would require fine-tuning. High training cost (2,000 epochs) is also noted as a practical constraint. - The elevated TTS WER relative to ASR baselines on CommonVoice and FLEURS suggests intelligibility issues on speakers and recording conditions not well represented in IndicSUPERB training data. caveats: [] - id: '2507.16875' published_date: "2025-07-22" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - flow_matching_speaker_adaptation claims: - claim_id: duration_prediction_strategy_choice_in_zero_shot_tts_presents_language role: complicates claim: Duration prediction strategy choice in zero-shot TTS presents language-dependent trade-offs between intelligibility and speaker similarity that are not consistent across multilingual settings. source: §5, Table 3, Table 5 evidence: The speaker-prompted (PFlow-style) predictor improves Tamil intelligibility (WER 0.295 vs. 0.312) and speaker similarity (SMOS 4.56 vs. 4.41) but degrades Marathi intelligibility substantially (WER 0.365 vs. 0.224 for infill), while Bengali human evaluation favours infill on both dimensions. confidence: high relevance: high - claim_id: speaker_conditioned_audio_prompting_enables_duration_prediction_without_forced_alignment role: supports claim: Speaker-conditioned audio prompting enables duration prediction without forced alignment, yielding competitive or superior speaker similarity in low-resource multilingual TTS. source: §3.4.2, §5.1, Table 4, Table 5 evidence: The 84M-parameter speaker-prompted predictor, conditioning on a 3-second mel spectrogram via cross-attention, achieves higher Sim-o scores than the infill predictor for Tamil, Telugu, and Bengali, and substantially higher human SMOS for Hindi (4.14 vs. 3.57), without using explicit duration context at inference. confidence: high relevance: high - claim_id: pooled_low_resource_training_corpora_for_multilingual_tts_contain_significant role: complicates claim: Pooled low-resource training corpora for multilingual TTS contain significant quality variance that requires aggressive filtering before flow-matching models can be reliably trained. source: §4.2.2 evidence: WER-based filtering using IndicWhisper reduced the Hindi training set from 1.3M to 1M utterances; a partial recovery via CTC alignment scores added 30K back, indicating that roughly 23% of the raw data was too noisy or misaligned to be usable. confidence: high relevance: medium - claim_id: the_relative_benefit_of_speaker_prompted_duration_conditioning_over_text role: refines claim: The relative benefit of speaker-prompted duration conditioning over text-and-alignment conditioning is not uniform and depends on the phonological and prosodic characteristics of the target language. source: §5.1, §5.2, Table 3, Table 4, Table 5 evidence: Tamil, with complex phoneme structures and high prosodic variability, benefits consistently from speaker prompting across both objective WER and human SMOS; Marathi shows a clear regression; Hindi and Bengali exhibit mixed patterns depending on test subset. confidence: high relevance: low limitations: - All experiments are conducted on speech infilling (50% sentence masking), not full-sentence TTS generation. Intelligibility and speaker similarity results may not transfer directly to the more common single-utterance TTS evaluation setting. - The cross-sentence completion task is described but no quantitative results are reported in the paper text. Comparisons are against ground-truth durations and the two proposed predictors only; no external TTS baselines from the Indian language literature are included. Training data for some languages is below 1,000 hours, and the five languages evaluated represent a small fraction of India's linguistic diversity. All models train in FP32, which is resource-intensive and may limit reproducibility for lower-compute settings. No code or model checkpoints are publicly released. caveats: [] - id: '2505.15670' published_date: "2025-07-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: full_duplex_spoken_dialogue_systems_can_be_built_without_speech role: supports claim: Full-duplex spoken dialogue systems can be built without speech-text pretraining by routing user audio through a pretrained streaming encoder rather than requiring the LLM to learn audio representations end-to-end. source: §3, §6.1, §6.2, Tables 2–3 evidence: SALM-Duplex skips speech pretraining entirely and instead uses a 100M streaming CTC encoder for user input; it still outperforms Moshi on barge-in success rate (94.5% vs. 55.1%) and reasoning GPT scores across all five evaluation sets. confidence: high relevance: low - claim_id: asymmetric_duplex_architectures_that_separate_user_and_agent_speech_pathways role: supports claim: Asymmetric duplex architectures that separate user and agent speech pathways enable independent specialization, including speaker-specific codec fine-tuning without affecting user comprehension. source: §3.2, §6.3, Table 4 evidence: Personalized 0.6 kbps NanoCodec (fine-tuned on 21k target-speaker utterances) outperforms Moshi's Mimi at 1.1 kbps and untuned NanoCodec at 1.2 kbps on MOS, CER, and SECS, while operating at roughly half the bitrate. confidence: high relevance: low - claim_id: end_to_end_speech_to_speech_models_do_not_consistently role: complicates claim: End-to-end speech-to-speech models do not consistently match optimal cascaded systems in reasoning quality, even when the cascaded oracle has access to ground-truth ASR transcriptions of user speech. source: §6.2, Table 3 evidence: SALM-Duplex outperforms GT+LLM on Roleplay and ASR-QA but underperforms on UltraChat (3.5 vs. 6.4), Topic, and Alpaca; the gap reflects compounding ASR error and limited backbone reasoning capacity at 1.1B parameters. confidence: high relevance: medium - claim_id: barge_in_success_rate_and_latency_together_constitute_more_discriminative role: supports claim: Barge-in success rate and latency together constitute more discriminative signals for evaluating full-duplex systems than speech quality metrics such as UTMOS. source: §5.2, §6.1, Table 2 evidence: On the Impatient set, SALM-Duplex and Moshi have zero false alarms each and UTMOS within 0.2 points (4.0 vs. 3.8), yet differ by 39.4 percentage points in barge-in success rate, making success rate the dominant differentiating metric. confidence: high relevance: low limitations: - All agent speech in training data is synthesized using a TTS model with a fixed speaker. Generalization to diverse agent voices or real conversational speech has not been demonstrated. - The 1.1B TinyLlama backbone limits reasoning ceiling; the gap to the GT+LLM oracle on complex dialogue tasks suggests ASR error compounds with limited LLM capacity. The quantitative comparison is restricted to Moshi; other contemporaneous duplex systems (OmniFlatten, SALMONN-Omni, MinMo) are discussed in related work but not benchmarked. The hardcoded 0.64s post-user-turn silence used to suppress unexpected agent barge-in may not transfer to naturally paced conversation. Evaluation datasets are largely synthetic, so performance on real recorded conversational speech remains an open question. caveats: [] - id: 2025.acl-industry.42 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: - GAN - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - gan_based_speaker_transfer - transformer_encoder_decoder_adaptation claims: - claim_id: accurate_grapheme_to_phoneme_conversion_is_the_single_most_important role: supports claim: Accurate grapheme-to-phoneme conversion is the single most important preprocessing component for intelligibility in tonal-language TTS, contributing more to WER reduction than tokenisation or pause prediction combined. source: §6, Table 3 evidence: Accurate grapheme-to-phoneme conversion is the single most important preprocessing component for intelligibility in tonal-language TTS, contributing more to WER reduction than tokenisation or pause prediction combined. confidence: high relevance: low - claim_id: proprietary_general_purpose_tts_systems_degrade_substantially_on_domain_specific role: supports claim: Proprietary general-purpose TTS systems degrade substantially on domain-specific vocabulary, while systems trained with targeted vertical-domain data show much smaller performance drops in specialised settings. source: §6, Table 2 evidence: Proprietary general-purpose TTS systems degrade substantially on domain-specific vocabulary, while systems trained with targeted vertical-domain data show much smaller performance drops in specialised settings. confidence: high relevance: medium - claim_id: low_resource_tts_systems_can_achieve_competitive_speaker_similarity_in role: supports claim: Low-resource TTS systems can achieve competitive speaker similarity in zero-shot voice cloning relative to voice conversion models when trained with a style embedding module, even without speaker-specific fine-tuning. source: §6, Table 4 evidence: Low-resource TTS systems can achieve competitive speaker similarity in zero-shot voice cloning relative to voice conversion models when trained with a style embedding module, even without speaker-specific fine-tuning. confidence: high relevance: high - claim_id: contextual_prosody_modelling_via_a_tonal_phoneme_aware_bert_encoder role: supports claim: Contextual prosody modelling via a tonal phoneme-aware BERT encoder improves naturalness scores in tonal-language TTS beyond what standard duration and pitch predictors achieve. source: §5 evidence: Contextual prosody modelling via a tonal phoneme-aware BERT encoder improves naturalness scores in tonal-language TTS beyond what standard duration and pitch predictors achieve. confidence: high relevance: low limitations: - '- The approach has been validated only on Thai; generalization to other tonal or low-resource languages is claimed but not demonstrated. - Model size and full architecture details are not reported, limiting reproducibility. - The zero-shot comparison is against OpenVoice (a VC model) rather than other zero-shot TTS systems. - Data collection from TikTok and YouTube raises questions about license and long-term data availability.' caveats: [] - id: 2025.acl-long.790 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - VC architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: self_consistency_training_enables_shortcut_flow_matching_to_match_full role: supports claim: Self-consistency training enables shortcut flow matching to match full-step quality in voice conversion with as few as two inference steps. source: §4.2, Table 1 evidence: R-VC at NFE=2 matches NFE=10 across all quality metrics (SECS 0.930 vs 0.931, UTMOS 4.1 vs 4.1, QMOS 4.03 vs 4.05, SMOS 4.11 vs 4.12) while reducing inference time by 2.83x; vanilla CFM degrades sharply below 10 steps. confidence: high relevance: medium - claim_id: explicit_rhythm_modeling_via_a_target_conditioned_duration_model_substantially role: supports claim: Explicit rhythm modeling via a target-conditioned duration model substantially improves emotion style transfer in zero-shot VC. source: §4.3, §4.5, Table 2, Table 4 evidence: Removing the duration module from R-VC drops the emotion score from 0.59 to 0.425 on the ESD dataset, while also increasing WER slightly; baselines that preserve source rhythm score 0.395-0.489. confidence: high relevance: high - claim_id: fine_grained_duration_prediction_in_non_autoregressive_models_introduces_instability role: complicates claim: Fine-grained duration prediction in non-autoregressive models introduces instability in voice conversion that coarser duration strategies do not fully resolve. source: §7, Table 4 evidence: R-VC's masked transformer duration model produces occasional over-extended pronunciations; sentence-level duration as a fallback yields worse WER (9.86 vs 6.95) and UTMOS (3.58 vs 3.85), offering no stability improvement in practice. confidence: high relevance: medium - claim_id: data_perturbation_before_discrete_content_tokenisation_reduces_timbre_leakage_more role: supports claim: Data perturbation before discrete content tokenisation reduces timbre leakage more effectively than relying on the self-supervised representation alone. source: §4.5, Table 4 evidence: Removing pitch perturbation before HuBERT token extraction degrades WER from 3.51 to 7.28 and speaker similarity from 0.930 to 0.869, confirming that perturbation actively suppresses content-irrelevant speaker information. confidence: high relevance: medium limitations: - 'The masked transformer duration model has a known instability: inaccurate predictions cause over-extended pronunciations. Sentence-level duration as an alternative proved worse in both stability and quality, leaving robust duration modeling as an unresolved challenge.' - The system is evaluated only on English (MLS, LibriSpeech, ESD) and English Seed-TTS subsets; generalisation to cross-lingual or multilingual VC is untested. Training data (20k hours) is smaller than top competitors such as CosyVoice-VC (171k hours), which makes speaker similarity comparisons somewhat favourable to R-VC but also means that high-similarity performance on out-of-distribution accents or recording conditions is unknown. The RTF of 0.12 using 2-step inference is faster than most flow-matching competitors but still 20% slower than non-diffusion methods (FACodec-VC RTF 0.10). caveats: [] - id: 2025.acl-long.87 published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: ACL task: - VC architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: combining_asr_derived_phonetic_features_and_quantized_self_supervised_representations role: supports claim: Combining ASR-derived phonetic features and quantized self-supervised representations via adaptive fusion reduces timbre leakage while preserving paralinguistic content in zero-shot voice conversion. source: §5.3, Table 3 evidence: Removing the PPG branch (WavLM-only) causes SMOS to drop from 4.11 to 3.07 and SECS from 0.71 to 0.45 on LibriTTS, indicating that SSL features alone carry substantial timbre leakage; removing the SSL branch degrades NMOS and WER, confirming PPGs alone lose paralinguistic richness. confidence: high relevance: high - claim_id: flow_matching_provides_faster_inference_than_diffusion_based_voice_conversion role: supports claim: Flow matching provides faster inference than diffusion-based voice conversion systems without sacrificing speaker similarity or naturalness. source: §5.1, Table 1 evidence: Takin-VC achieves RTF 0.154, lower than DiffVC (0.294), NS2VC (0.347), and SeedVC (0.341), while simultaneously outperforming these baselines on NMOS, SMOS, and SECS. confidence: high relevance: high - claim_id: global_time_invariant_speaker_embeddings_are_insufficient_for_robust_timbre role: complicates claim: Global, time-invariant speaker embeddings are insufficient for robust timbre modeling in expressive zero-shot voice conversion. source: §5.3, Table 4 evidence: Removing the context-aware cross-attention module (which aligns source content with target timbre dynamically) drops SMOS from 4.11 to 3.61 and SECS from 0.71 to 0.58, while the memory-augmented module removal drops SECS to 0.52. Both modules provide content-sensitive timbre conditioning beyond a static speaker embedding alone. confidence: high relevance: high - claim_id: cross_gender_voice_conversion_consistently_yields_lower_speaker_similarity_than role: complicates claim: Cross-gender voice conversion consistently yields lower speaker similarity than same-gender conversion even in well-trained systems. source: §5.2, Table 2 evidence: 'On the large-scale multilingual dataset, same-gender pairs (F2F: SECS 0.74; M2M: 0.73) outperform cross-gender pairs (F2M: 0.71; M2F: 0.70) in speaker embedding cosine similarity, a gap that persists across all conversion directions.' confidence: high relevance: high - claim_id: quantizing_self_supervised_speech_features_before_content_encoding_reduces_timbre role: refines claim: Quantizing self-supervised speech features before content encoding reduces timbre leakage more effectively than using continuous SSL representations directly. source: §3.2, §5.3, Table 3 evidence: The RVQ quantizer (codebook size 8,200) applied to WavLM features is the key mechanism for timbre suppression in the hybrid encoder; ablation with WavLM-only (continuous features without adaptive fusion) shows SECS drops to 0.45 compared to 0.71 for the full model, consistent with timbre leakage from unquantized SSL features. confidence: high relevance: medium limitations: - All large-scale training data (500k hours) and the 100-speaker evaluation set are proprietary and not publicly available. The large-scale results cannot be reproduced by external researchers, and it is unclear how much of the gain over competitive baselines is attributable to data scale rather than the proposed modules. - The paper does not include targeted evaluation of paralinguistic preservation (breathing, crying, emotion transfer), despite listing this as a primary contribution. NMOS and SMOS measure general naturalness and speaker similarity but are not designed to capture expressive fidelity specifically. Speech editing under zero-shot conditions is acknowledged as out of scope and a direction for future work. Ethical risks from voice impersonation are noted but no technical mitigations are proposed. caveats: [] - id: '2507.20140' published_date: "2025-07-27" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: machine_unlearning_via_randomization_based_training_objectives_can_selectively_suppress role: supports claim: Machine unlearning via randomization-based training objectives can selectively suppress specific speaker identities in zero-shot TTS while preserving synthesis quality for retained speakers. source: §5.2, Table 1 evidence: TGU achieves SIM-F of 0.169 (within the measured cross-speaker similarity range of 0.02-0.17) and retain-set SPK-SIM of 0.631, a drop of only 2.8% from the original model's 0.649, with WER-F of 2.4% comparable to the original's 2.1% on the retain set. confidence: high relevance: high - claim_id: excluding_target_speakers_from_the_fine_tuning_dataset_is_insufficient role: complicates claim: Excluding target speakers from the fine-tuning dataset is insufficient for voice privacy protection in zero-shot TTS, because these models generalize at inference time to replicate unseen speakers via in-context learning. source: §5.2, Table 1 evidence: Exact Unlearning (retraining from scratch on the retain set) and Fine Tuning on the retain set yield SIM-F of 0.687 and 0.675 respectively, nearly identical to the original model's 0.708, confirming the model continues to clone forgotten speakers not present in the fine-tuning data. confidence: high relevance: high - claim_id: gradient_reversal_and_kl_divergence_based_unlearning_methods_degrade_speech role: complicates claim: Gradient-reversal and KL-divergence-based unlearning methods degrade speech intelligibility rather than achieving genuine speaker forgetting in voice-conditioned generative models, due to entanglement between speaker style and linguistic content. source: §5.2, Table 1 evidence: Negative Gradient achieves WER-F of 5.0% and KL divergence maximization achieves WER-F of 47.2%, compared to 2.4% for TGU; the paper attributes this to the model learning to generate incoherent audio rather than truly unlearning speaker identity, as speaker style and linguistic content are jointly encoded in VoiceBox's pre-training. confidence: high relevance: medium - claim_id: speaker_similarity_metrics_alone_are_insufficient_to_verify_effective_machine role: supports claim: Speaker similarity metrics alone are insufficient to verify effective machine unlearning in generative speech models, as consistent failure patterns can yield low similarity scores without achieving the randomness needed to resist voice reconstruction. source: §4.3, §5.2, Table 1 evidence: Negative Gradient and KL methods achieve low SIM-F scores (0.402 and 0.179) but spk-ZRF-F values of 0.842 and 0.810, at or below the original model's baseline of 0.846, revealing patterned outputs that could be reverse-engineered. TGU achieves both low SIM-F (0.169) and elevated spk-ZRF-F (0.871). confidence: high relevance: high - claim_id: a_pre_trained_generative_model_can_serve_as_its_own role: supports claim: A pre-trained generative model can serve as its own teacher for unlearning by providing diverse speaker-identity targets, eliminating the need for aligned cross-speaker audio pairs. source: §4.2 evidence: TGU generates per-step training targets by running VoiceBox conditioned only on text (no audio prompt), which produces a different speaker identity at each Gaussian initialization; these unconditional outputs replace paired cross-speaker audio as forget-set targets in the modified CFM loss. confidence: high relevance: medium limitations: - All experiments are conducted on a single TTS architecture (VoiceBox with mel-spectrogram representation). Transferability of TGU to codec-based autoregressive ZS-TTS systems is not evaluated, which limits the generalizability of the findings given that the broader field has largely shifted to codec-based generation. - Machine unlearning of voice identity raises boundary cases when remain speakers have similar vocal characteristics to forget speakers. The robustness experiment (Appendix H) shows only a weak positive correlation (r = 0.14) between a remain speaker's similarity to forget speakers and retain-set performance degradation, suggesting TGU is largely robust; however, the effect is not fully absent. - TGU introduces a trade-off in speech diversity, measured by Frechet Speech Distance (FSD), which increases from 170.2 to 177.8 on LibriSpeech test-other. This suggests the reduced effective training distribution from unlearning modestly narrows generative variety. - The paper explores adversarial recovery only through direct fine-tuning on forget speaker audio (Appendix J). More sophisticated inversion attacks or embedding-space reconstruction are left as open questions. caveats: [] - id: '2503.11026' published_date: "2025-07-30" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: conditioning_a_flow_matching_mel_spectrogram_generator_on_rich_multimodal role: supports claim: Conditioning a flow matching mel-spectrogram generator on rich multimodal speaker representations produces more consistent speaker identity in zero-shot cross-lingual speech synthesis than injecting a single speaker embedding at the vocoder stage. source: §5.4, Table 1 evidence: MAVFlow achieves an average 36% improvement in speaker similarity (SS) over AV2AV across four language pairs on MuAViC, using OT-CFM conditioned on x-vector speaker embeddings plus facial emotion embeddings, while AV2AV uses d-vector conditioning in the vocoder only. confidence: high relevance: high - claim_id: higher_quality_intermediate_mel_spectrogram_synthesis_propagates_benefits_to_downstream role: supports claim: Higher-quality intermediate mel-spectrogram synthesis propagates benefits to downstream talking-face generation even when the face decoder itself is unchanged. source: §5.5, Table 5 evidence: MAVFlow improves LSE-C by +0.87, LSE-D by -0.49, and FID by -0.61 relative to AV2AV on LRS3 visual evaluation, despite using the same Wav2Lip face decoder, suggesting that the mel quality bottleneck affects face sync accuracy. confidence: high relevance: medium - claim_id: visual_emotion_conditioning_is_insufficient_on_its_own_to_improve role: complicates claim: Visual emotion conditioning is insufficient on its own to improve emotion expression in synthesized speech and requires concurrent audio speaker conditioning to be effective. source: §5.6, Table 8 evidence: Adding only visual guidance to the CFM model marginally maintains speaker similarity (SS 0.056 vs 0.057 without guidance) but reduces emotion accuracy from 28.66% to 26.83% on CREMA-D; the combination of audio and visual guidance is needed to reach 36.46%. confidence: high relevance: low - claim_id: paralinguistic_and_linguistic_generation_objectives_are_compatible_in_zero_shot role: supports claim: Paralinguistic and linguistic generation objectives are compatible in zero-shot cross-lingual speech synthesis; improving speaker fidelity does not require sacrificing translation accuracy. source: §5.4, Tables 1 and 3 evidence: MAVFlow maintains competitive ASR-BLEU scores (26.97 Es-En vs 26.57 for AV2AV and 28.66–30.55 for cascaded systems) while substantially improving speaker similarity, using the same unit translation module as AV2AV. confidence: high relevance: high - claim_id: emotion_recognition_accuracy_in_synthesized_cross_lingual_speech_remains_far role: complicates claim: Emotion recognition accuracy in synthesized cross-lingual speech remains far below ground-truth levels even with multimodal conditioning, indicating that paralinguistic preservation is an unsolved challenge. source: §5.4, Table 2; §5.6, Table 7 evidence: MAVFlow achieves 36.46% audio emotion accuracy vs a GT upper bound of 81.95% on CREMA-D, even with dual audio-visual guidance; additional training on an emotion-rich dataset (CREMA-D) raises this to 51.46% but still far below GT. confidence: high relevance: low limitations: - The system relies on emotional cues from facial video alone; audio-side paralinguistics such as prosody and timbre variation are not used as emotion conditioning signals, which may limit emotion transfer when visual input is unavailable or low quality. The unit extractor and unit-to-unit translation modules are adopted unchanged from AV2AV, so improvements in semantic translation quality require addressing those upstream components separately. The Duration Length Regulator interpolates to match source duration, which may introduce length-related artifacts when source and translated speech have substantially different natural lengths. Evaluation is limited to five European languages with English as the target; generalization to typologically distant language pairs (e.g., tonal languages, right-to-left scripts) is untested. caveats: [] - id: 2025.ccl-1.77 published_date: "2025-08-01" entry_date: '2026-07-29' year: 2025 venue: workshop task: - TTS architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_speaker_adaptation claims: - claim_id: cross_modal_facial_features_can_provide_sufficient_speaker_identity_signal role: supports claim: Cross-modal facial features can provide sufficient speaker identity signal for zero-shot voice generation without reference audio. source: §3.2, Table 1, Table 2 evidence: The cross-modal biometric unseen speaker modelling module maps a face image to a speaker embedding via ResNet50 and a trainable visual network; Id.Acc of 66.87/68.99 on LRS2/GRID exceeds all baselines that require reference audio at inference. confidence: high relevance: high - claim_id: diffusion_based_denoising_in_a_multimodal_tts_dubbing_pipeline_improves role: supports claim: Diffusion-based denoising in a multimodal TTS/dubbing pipeline improves prosodic diversity relative to deterministic autoregressive or attention-based baselines. source: §4.2.4, Figure 4 evidence: Running HFSD-V2C 10 times per speaker produces diverse F0 contours capturing individual accent patterns; Neural Dubber and HPMDubbing produce fixed prosodic distributions. confidence: high relevance: medium - claim_id: hierarchical_multimodal_prosody_modelling_video_audio_and_text_levels_jointly role: supports claim: Hierarchical multimodal prosody modelling (video, audio, and text levels jointly) improves emotion accuracy over methods relying on fewer modalities. source: §4.2.1, §4.2.5, Table 1, Table 2 evidence: Emo.Acc reaches 65.63 on LRS2 and 66.32 on GRID, against 61.46 and 63.66 for HPMDubbing; the ablation (w/o UP) drops Emo.Acc to 22.08/27.64, confirming the prosody module's role. confidence: high relevance: low - claim_id: zero_shot_visual_voice_cloning_accuracy_remains_significantly_below_ground role: complicates claim: Zero-shot visual voice cloning accuracy remains significantly below ground-truth speaker identity, indicating that cross-modal biometric embeddings do not fully replace reference audio. source: §4.2.1, Table 1 evidence: GT Id.Acc on LRS2 is 91.52 vs. HFSD-V2C's 66.87; GT MOS is 4.72 vs. 4.29 for HFSD-V2C, a gap of 0.43 that persists after hierarchical multimodal conditioning. confidence: high relevance: high limitations: - The evaluation is restricted to LRS2 and GRID, both of which contain constrained speaking styles (BBC broadcasts and phonetically structured lab speech), leaving generalisation to spontaneous conversational video undemonstrated. The subjective MOS is collected on only 20 clips rated by 20 evaluators, which is a limited sample for drawing robust conclusions. The model trains on both LRS2 and GRID but the zero-shot claim means unseen speakers at test time, not unseen datasets; the extent of genuine out-of-domain generalisation is not assessed. Code and demos are not released, limiting reproducibility. Finally, the method depends on visible, well-lit face images, which may not be robust in natural video production environments. caveats: [] - id: '2508.14049' published_date: "2025-08-05" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation claims: - claim_id: large_multilingual_tts_systems_built_on_semantic_token_intermediaries_transfer role: supports claim: Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models. source: §2.1, §5.1 evidence: Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models. confidence: high relevance: medium - claim_id: decoupling_the_text_to_semantic_and_semantic_to_acoustic_stages role: supports claim: Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining. source: §2, §4.1 evidence: Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining. confidence: high relevance: medium - claim_id: flow_matching_is_a_viable_replacement_for_diffusion_in_the role: supports claim: Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity. source: §2.3, §5.1 evidence: Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity. confidence: high relevance: medium - claim_id: intelligibility_in_low_resource_languages_with_limited_training_data_remains role: complicates claim: Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system. source: §5.1, Table 2 evidence: Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system. confidence: high relevance: medium limitations: - The evaluation relies exclusively on WER over 10 sentences per language with no MOS, SMOS, or naturalness scores. This makes it impossible to assess audio quality, expressiveness, or speaker similarity relative to baselines — core dimensions for a TTS system. - 'English dominates the training set at 58%, which may explain strong English results but raises questions about whether true cross-lingual transfer or data dominance is responsible. The system lacks prosody and pace control conditioning in M1, which the authors flag as future work. Zero-shot speaker fidelity for M2 is acknowledged as limited compared to infilling-based approaches like Seamless. Fine-tuning introduces hallucination that requires careful intervention (freezing classification heads only), suggesting the LM component is sensitive to distribution shift. Languages with fewer training hours (Assamese: 48h, Dogri: 8h, Rajasthani: 20h) show substantially weaker results.' caveats: [] - id: '2508.04141' published_date: "2025-08-06" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - hybrid_semantic_acoustic_adaptation claims: - claim_id: generating_semantic_and_acoustic_tokens_simultaneously_in_a_single_autoregressive role: supports claim: Generating semantic and acoustic tokens simultaneously in a single autoregressive forward pass, rather than cascading semantic prediction before acoustic prediction, reduces word error rate and improves naturalness in zero-shot TTS. source: §V.A, Table I evidence: Generating semantic and acoustic tokens simultaneously in a single autoregressive forward pass, rather than cascading semantic prediction before acoustic prediction, reduces word error rate and improves naturalness in zero-shot TTS. confidence: high relevance: high - claim_id: combining_specialist_ssl_models_for_distinct_speech_attributes_semantic_content role: supports claim: Combining specialist SSL models for distinct speech attributes (semantic content, acoustic texture, speaker identity) as frozen feature extractors enables more effective token-level disentanglement than using a single encoder for all attributes. source: §III.A, Tables III–IV evidence: Combining specialist SSL models for distinct speech attributes (semantic content, acoustic texture, speaker identity) as frozen feature extractors enables more effective token-level disentanglement than using a single encoder for all attributes. confidence: high relevance: high - claim_id: a_hybrid_ar_nar_design_that_enforces_independence_at_the role: supports claim: A hybrid AR+NAR design that enforces independence at the coarse token level and interdependence at the fine-grained level distributes modeling complexity more effectively than an all-AR or all-NAR approach. source: §V.B, Tables III–IV evidence: A hybrid AR+NAR design that enforces independence at the coarse token level and interdependence at the fine-grained level distributes modeling complexity more effectively than an all-AR or all-NAR approach. confidence: high relevance: medium - claim_id: parallel_semantic_acoustic_modeling_improves_naturalness_and_intelligibility_without_fully role: supports claim: Parallel semantic-acoustic modeling improves naturalness and intelligibility without fully closing the speaker similarity gap relative to systems with dedicated speaker embedding refinement. source: §V.A, Table I evidence: Parallel semantic-acoustic modeling improves naturalness and intelligibility without fully closing the speaker similarity gap relative to systems with dedicated speaker embedding refinement. confidence: high relevance: high limitations: - Speaker similarity lags behind CosyVoice (SMOS gap ~0.15–0.2 on English), suggesting the parallel architecture does not yet fully leverage speaker conditioning. UTMOS scores, while competitive, do not reach ground-truth levels. Model size is not reported, making compute comparisons difficult. The internal Chinese dataset and preprocessing pipeline (Emilia + NCSSD) are not publicly released, limiting reproducibility on that front. Extending the framework to prosody control, emotion conditioning, or cross-lingual voice conversion is not explored. The subjective decoupling evaluation (Section V.C) relies on 90% evaluator agreement rather than a standardized metric, leaving quantitative disentanglement assessment as an open question. caveats: [] - id: '2508.04996' published_date: "2025-08-07" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - VC architecture: - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: ssl_features_improve_paralinguistic_expressiveness_in_voice_conversion_but_introduce role: supports claim: SSL features improve paralinguistic expressiveness in voice conversion but introduce timbre leakage and noise sensitivity that require explicit mitigation. source: §I, §II.B evidence: SSL features improve paralinguistic expressiveness in voice conversion but introduce timbre leakage and noise sensitivity that require explicit mitigation. confidence: high relevance: medium - claim_id: random_feature_erasure_at_training_time_can_reduce_a_model role: supports claim: Random feature erasure at training time can reduce a model's over-reliance on information-rich but noise-sensitive representations without information bottleneck machinery. source: §II.B evidence: Random feature erasure at training time can reduce a model's over-reliance on information-rich but noise-sensitive representations without information bottleneck machinery. confidence: high relevance: medium - claim_id: implicit_alignment_borrowed_from_non_autoregressive_tts_can_improve_noise role: supports claim: Implicit alignment borrowed from non-autoregressive TTS can improve noise robustness in voice conversion by preventing the model from over-reconstructing noise-carrying source frames. source: §II.C evidence: Implicit alignment borrowed from non-autoregressive TTS can improve noise robustness in voice conversion by preventing the model from over-reconstructing noise-carrying source frames. confidence: high relevance: medium - claim_id: shortcut_models_reduce_flow_matching_inference_steps_by_an_order role: supports claim: Shortcut Models reduce flow-matching inference steps by an order of magnitude with only marginal quality loss in voice conversion. source: §II.D, Table I evidence: Shortcut Models reduce flow-matching inference steps by an order of magnitude with only marginal quality loss in voice conversion. confidence: high relevance: medium - claim_id: asr_based_bottleneck_features_and_ssl_representations_are_complementary_the role: supports claim: 'ASR-based bottleneck features and SSL representations are complementary: the former provides noise-robust linguistic content, the latter contributes paralinguistic fidelity that ASR training suppresses.' source: §I, §II.A evidence: 'ASR-based bottleneck features and SSL representations are complementary: the former provides noise-robust linguistic content, the latter contributes paralinguistic fidelity that ASR training suppresses.' confidence: high relevance: medium limitations: - The model cannot synthesise arbitrarily long utterances. The implicit alignment mechanism introduces a maximum-length constraint analogous to that in E2TTS-style TTS systems. The authors flag this as a known limitation without providing an upper bound or workaround. - 'Evaluation scale is small: 100 clean and 50 noisy test utterances is insufficient to draw strong conclusions about generalisation across noise types or speaking styles. The noisy set recording conditions are not fully documented. Comparison to other noise-robust VC systems such as NORO ([[2411.19770|NORO]]) is absent — only Seed-VC and a VITS-VC internal baseline are used.' - 'The prosody preservation trade-off is acknowledged: REF-VC preserves source prosody well, but users may prefer target speaker style transfer instead. Future work is needed to support simultaneous prosody preservation and style transfer.' - Singing voice conversion is mentioned as a capability but receives no quantitative evaluation. caveats: [] - id: '2508.07426' published_date: "2025-08-10" entry_date: '2026-07-29' year: 2025 venue: ASRU task: - TTS architecture: - transformer-enc-dec - GAN - VAE relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation - gan_based_speaker_transfer - vae_latent_speaker_adaptation claims: - claim_id: accent_label_quality_exerts_a_larger_effect_on_accent_synthesis role: supports claim: Accent label quality exerts a larger effect on accent synthesis fidelity than raw data volume, and filtering noisy self-reported labels consistently outperforms using them directly. source: §V-C, Table VI evidence: Accent label quality exerts a larger effect on accent synthesis fidelity than raw data volume, and filtering noisy self-reported labels consistently outperforms using them directly. confidence: high relevance: medium - claim_id: speaker_accent_disentanglement_in_low_resource_settings_benefits_from_voice role: supports claim: Speaker-accent disentanglement in low-resource settings benefits from voice conversion-based timbre augmentation, even when the augmented data introduces acoustic degradation. source: §V-D, Table VIII evidence: Speaker-accent disentanglement in low-resource settings benefits from voice conversion-based timbre augmentation, even when the augmented data introduces acoustic degradation. confidence: high relevance: medium - claim_id: automatic_accent_identification_metrics_and_human_perceptual_evaluations_of_accent role: supports claim: Automatic accent identification metrics and human perceptual evaluations of accent plausibility frequently diverge, limiting the reliability of either metric alone as a benchmark. source: §V-B, §V-E evidence: Automatic accent identification metrics and human perceptual evaluations of accent plausibility frequently diverge, limiting the reliability of either metric alone as a benchmark. confidence: high relevance: medium - claim_id: large_zero_shot_tts_systems_can_produce_competitive_accented_speech role: supports claim: Large zero-shot TTS systems can produce competitive accented speech through style copying without explicit accent conditioning, often matching or exceeding dedicated accent-controllable models on naturalness. source: §V-B, Table VII evidence: Large zero-shot TTS systems can produce competitive accented speech through style copying without explicit accent conditioning, often matching or exceeding dedicated accent-controllable models on naturalness. confidence: high relevance: high - claim_id: geolocation_based_accent_label_discovery_is_extensible_to_languages_and role: supports claim: Geolocation-based accent label discovery is extensible to languages and accent varieties without existing labeled speech, but precision varies substantially across accent regions depending on broadcast training data coverage. source: §III-A, Table III evidence: Geolocation-based accent label discovery is extensible to languages and accent varieties without existing labeled speech, but precision varies substantially across accent regions depending on broadcast training data coverage. confidence: high relevance: medium limitations: - The paper evaluates English accent synthesis only, despite XTTS-v2 supporting multiple languages. The claim that geolocation-based label discovery generalises to any accent in any language is theoretical; precision on Canada (13.3%), Scotland (9.4%), Germany (15.9%), Ireland (20.4%), and Malaysia (7.1%) is poor, limiting practical applicability to a subset of the 11 accents studied. - 'Human evaluation is expensive and was conducted for only 4 of the 11 or more accents included in training, leaving uncertain whether objective metric rankings carry over to human preference for the remaining accents. The kNN-VC augmentation results are inconsistent: human evaluators found no clear benefit for augmented versus non-augmented systems in Australian and Scottish evaluations, contradicting objective metric improvements. The WER gap between the proposed systems and CosyVoice2 (13.1% vs. 7.1%) is notable and unexplained in depth. The geolocation model was trained on radio broadcasts, which may introduce domain mismatch when applied to crowd-sourced microphone speech.' caveats: [] - id: '2508.08399' published_date: "2025-08-11" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - codec architecture: - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - vae_latent_speaker_adaptation claims: - claim_id: fully_discrete_disentanglement_of_phonetic_prosodic_and_speaker_information_in role: complicates claim: Fully discrete disentanglement of phonetic, prosodic, and speaker information in a speech codec is achievable without phoneme labels or F0 supervision, at the cost of a small reconstruction quality degradation. source: §III, §IV.B, Table II evidence: Fully discrete disentanglement of phonetic, prosodic, and speaker information in a speech codec is achievable without phoneme labels or F0 supervision, at the cost of a small reconstruction quality degradation. confidence: high relevance: low - claim_id: quantizing_speaker_vectors_into_discrete_codes_reduces_speaker_identity_fidelity role: complicates claim: Quantizing speaker vectors into discrete codes reduces speaker identity fidelity compared to continuous speaker representations, posing a fundamental trade-off between LLM compatibility and speaker retention. source: §IV.B, Table III evidence: Quantizing speaker vectors into discrete codes reduces speaker identity fidelity compared to continuous speaker representations, posing a fundamental trade-off between LLM compatibility and speaker retention. confidence: high relevance: high - claim_id: instance_normalization_of_ssl_residual_features_provides_a_label_free role: supports claim: Instance normalization of SSL residual features provides a label-free mechanism to separate time-invariant speaker statistics from time-variant prosodic content. source: §III.B evidence: Instance normalization of SSL residual features provides a label-free mechanism to separate time-invariant speaker statistics from time-variant prosodic content. confidence: high relevance: medium - claim_id: fully_discrete_speech_codecs_can_match_conventional_voice_conversion_methods role: supports claim: Fully discrete speech codecs can match conventional voice conversion methods on intelligibility and naturalness while enabling attribute manipulation through codebook-level operations. source: §IV.B, Table III evidence: Fully discrete speech codecs can match conventional voice conversion methods on intelligibility and naturalness while enabling attribute manipulation through codebook-level operations. confidence: high relevance: medium limitations: - All experiments use LibriSpeech clean speech (16 kHz, studio conditions); performance on noisy, spontaneous, or out-of-domain speech is untested. The one-shot VC evaluation uses only two reference speakers (one male, one female), limiting statistical confidence in the speaker similarity results. - The model is not tested on any downstream application (TTS, ASR, speech LM), despite this being the stated motivation. Whether the disentangled discrete tokens actually improve over non-disentangled tokens on downstream tasks remains an open question — the paper acknowledges this as future work. The GRVQ codebook dimensionality analysis shows a clear trade-off between bitrate and speaker identity, but optimal bitrate allocation across the three streams is not systematically explored. Prosody quantization codebook interpretability beyond F0 correlation (e.g., energy, duration, speaking rate) is not investigated. caveats: [] - id: '2508.08715' published_date: "2025-08-12" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation - gan_based_speaker_transfer claims: - claim_id: llm_based_tts_architectures_when_fine_tuned_with_appropriate_speaker role: supports claim: LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS. source: §4.2 evidence: LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS. confidence: high relevance: low - claim_id: child_specific_vocal_characteristics_can_be_transferred_to_multilingual_tts role: supports claim: Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model. source: §2.2, §3.2 evidence: Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model. confidence: high relevance: medium - claim_id: foundation_tts_models_originally_trained_on_high_resource_languages_can role: complicates claim: Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility. source: §3.1, §4.3, Table 1 evidence: Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility. confidence: high relevance: low - claim_id: intelligibility_gains_from_modern_tts_architectures_in_low_resource_settings role: complicates claim: Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited. source: §4.3, Table 1 evidence: Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited. confidence: high relevance: medium limitations: - Evaluation uses only 30 speech samples per language rated by 10 listeners each. The small scale means reported MOS differences, while large, should be treated cautiously — and no significance testing distinguishes genuine quality gaps from listener variability. All data is proprietary and not released, preventing independent replication. - The Singaporean-accented Mandarin training set contains only 1,400 utterances from a single child speaker, raising questions about voice diversity and generalisation to other child speakers. The paper does not evaluate speaker similarity to the target child voices, making it difficult to assess how faithfully the age-appropriate characteristics are reproduced. Malay and Tamil models are not evaluated against CosyBase, so it is unclear whether fine-tuning adds value beyond what a capable pretrained model already provides for those languages. All evaluation is limited to Singapore's specific linguistic and cultural context; the approach's generalisability to other low-resource child-speech settings is undemonstrated. caveats: [] - id: '2508.09702' published_date: "2025-08-13" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: quality_mismatch_between_clean_training_corpora_and_real_world_inference role: supports claim: Quality mismatch between clean training corpora and real-world inference prompts is a meaningful source of degradation in zero-shot speech generation systems. source: §1 evidence: Quality mismatch between clean training corpora and real-world inference prompts is a meaningful source of degradation in zero-shot speech generation systems. confidence: high relevance: high - claim_id: cross_lingual_proxy_prompts_selected_by_linguistic_proximity_can_produce role: supports claim: Cross-lingual proxy prompts selected by linguistic proximity can produce lower synthesis error than native-language prompts of lower audio quality. source: §3.2, Table 3 evidence: Cross-lingual proxy prompts selected by linguistic proximity can produce lower synthesis error than native-language prompts of lower audio quality. confidence: high relevance: low - claim_id: multi_modal_annotation_that_fuses_visual_and_acoustic_signals_yields role: supports claim: Multi-modal annotation that fuses visual and acoustic signals yields substantially higher age and gender labelling accuracy than audio-only or text-only annotation strategies. source: §5.1, Table 2 evidence: Multi-modal annotation that fuses visual and acoustic signals yields substantially higher age and gender labelling accuracy than audio-only or text-only annotation strategies. confidence: high relevance: medium - claim_id: a_curated_high_quality_audio_database_outperforms_generative_face_to role: supports claim: A curated, high-quality audio database outperforms generative face-to-speech synthesis as a source of speaker prompts when audio quality is the binding constraint. source: §5.3, Table 4 evidence: A curated, high-quality audio database outperforms generative face-to-speech synthesis as a source of speaker prompts when audio quality is the binding constraint. confidence: high relevance: medium - claim_id: cascaded_interruptible_similarity_retrieval_is_a_viable_approach_to_prompt role: supports claim: Cascaded, interruptible similarity retrieval is a viable approach to prompt selection under strict latency constraints without dedicated hardware. source: §4.2, §5.4, Table 5 evidence: Cascaded, interruptible similarity retrieval is a viable approach to prompt selection under strict latency constraints without dedicated hardware. confidence: high relevance: low limitations: - No subjective listening tests are reported. All quality metrics (UTMOSv2, automatic speaker similarity, Emotion2Vec emotion similarity) are automatic predictors. The relationship between these metrics and perceptual quality in the prompt selection context is not validated. - The evaluation spans only two downstream tasks (TTS and speech translation) and a narrow set of conditions. The claim that M3PDB generalises to other speech generation tasks (interactive systems, voice conversion) is stated as a future direction but not evidenced. The database construction pipeline is described as time-consuming, which limits reproducibility for groups without significant compute. Coverage of 18 languages is broader than most existing resources but still leaves many of the world's languages out. The unseen-language proxy mechanism is validated only on Italian; its reliability for more distant language pairs is unknown. caveats: [] - id: '2508.09767' published_date: "2025-08-13" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: parameter_efficient_fine_tuning_with_lora_can_restore_explicit_pronunciation role: supports claim: Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance. source: §3.2, §5.1 evidence: Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance. confidence: high relevance: medium - claim_id: bpe_based_multilingual_tts_models_systematically_underperform_on_phonologically_complex role: supports claim: BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive. source: §1, §5.2 evidence: BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive. confidence: high relevance: medium - claim_id: phoneme_tag_injection_via_special_tokens_can_provide_localised_pronunciation role: supports claim: Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans. source: §5.2 evidence: Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans. confidence: high relevance: medium - claim_id: speaker_similarity_in_zero_shot_tts_is_robust_to_targeted role: supports claim: Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data. source: §5.1, Table 1 evidence: Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data. confidence: high relevance: high limitations: - Evaluation uses two custom test sets authored or curated by the single researcher-author, with no publicly available benchmark comparison. The listening test crowd cohort (16 workers for MOS, 15 for accent correctness) is small for a task with high inter-rater variance. Results should be treated as promising but not yet independently validated. - The approach requires users to supply phonemic transcription and accent markup, which demands linguistic expertise most end users lack. The paper targets only standard Tokyo Japanese; dialectal variation and code-switching remain untested, as the author acknowledges. The LoRA module is trained on a single female-heavy corpus (JSUT) and 100-speaker JVS, which may limit generalisation to speaker types not well represented in that data. Applicability to other prosodically complex languages — Mandarin tones, Scandinavian pitch accent — is left for future work. caveats: [] - id: interspeech-2025-0047 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: mixing_high_quality_audio_only_data_with_audio_visual_corpora role: supports claim: Mixing high-quality audio-only data with audio-visual corpora via contrastive modality alignment substantially improves speech naturalness in face-driven TTS systems. source: §2.3, Table 1 evidence: Mixing high-quality audio-only data with audio-visual corpora via contrastive modality alignment substantially improves speech naturalness in face-driven TTS systems. confidence: high relevance: medium - claim_id: neural_style_transfer_augmentation_of_training_face_images_reduces_the role: supports claim: Neural style transfer augmentation of training face images reduces the domain gap between photorealistic faces and artistic portraits, improving both naturalness and face-matching scores. source: §2.2, Table 1 evidence: Neural style transfer augmentation of training face images reduces the domain gap between photorealistic faces and artistic portraits, improving both naturalness and face-matching scores. confidence: high relevance: medium - claim_id: in_context_prompting_with_a_user_selected_speech_sample_enables role: supports claim: In-context prompting with a user-selected speech sample enables consistent voice generation across multiple utterances from the same face, addressing the inherent one-to-many ambiguity of face-to-voice mapping. source: §2.4 evidence: In-context prompting with a user-selected speech sample enables consistent voice generation across multiple utterances from the same face, addressing the inherent one-to-many ambiguity of face-to-voice mapping. confidence: high relevance: medium - claim_id: contrastive_pre_training_of_face_and_audio_encoders_primarily_improves role: supports claim: Contrastive pre-training of face and audio encoders primarily improves face-voice association rather than overall speech naturalness. source: §3.4.1, Table 1 evidence: Contrastive pre-training of face and audio encoders primarily improves face-voice association rather than overall speech naturalness. confidence: high relevance: medium - claim_id: objective_acoustic_metrics_speaking_rate_si_sdr_c50_pitch_standard role: supports claim: Objective acoustic metrics (speaking rate, SI-SDR, C50, pitch standard deviation) shift predictably with natural language descriptors, confirming that text-driven style control transfers to measurable acoustic properties. source: §3.4.3, Table 3 evidence: Objective acoustic metrics (speaking rate, SI-SDR, C50, pitch standard deviation) shift predictably with natural language descriptors, confirming that text-driven style control transfers to measurable acoustic properties. confidence: high relevance: low limitations: - '- Voice consistency score (VCS) for RV-TTS (3.96) is below the audio-driven YourTTS (4.42), indicating residual inconsistency in pure face-conditioned generation. - The artistic portrait test uses only 20 copyright-free images; broader and more controlled evaluation is needed. - Descriptive text label generation for LRS3/VoxCeleb2 relies on Data-Speech auto-annotation rather than human labels. - Face encoder is ResNet50; more powerful vision-language models may improve face-voice matching. - No multilingual evaluation.' caveats: [] - id: interspeech-2025-0596 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: a_teacher_student_speaker_encoder_architecture_where_the_teacher_conditions role: supports claim: A teacher-student speaker encoder architecture, where the teacher conditions the TTS backbone during training and the student handles inference-time pathological inputs, substantially reduces phoneme articulation errors compared to single-encoder approaches in dysarthric TTS. source: §4.2, Table 3 evidence: A teacher-student speaker encoder architecture, where the teacher conditions the TTS backbone during training and the student handles inference-time pathological inputs, substantially reduces phoneme articulation errors compared to single-encoder approaches in dysarthric TTS. confidence: high relevance: high - claim_id: progressive_curriculum_learning_which_gradually_reduces_the_duration_of_student role: supports claim: Progressive curriculum learning, which gradually reduces the duration of student encoder inputs during training, outperforms random cropping and substantially improves phoneme error rate over training without structured audio augmentation. source: §4.3, Figure 3 evidence: Progressive curriculum learning, which gradually reduces the duration of student encoder inputs during training, outperforms random cropping and substantially improves phoneme error rate over training without structured audio augmentation. confidence: high relevance: medium - claim_id: zero_shot_multi_speaker_tts_trained_on_normal_speech_can role: supports claim: Zero-shot multi-speaker TTS trained on normal speech can generalize to dysarthric speakers at inference time without any dysarthric training data, provided the speaker encoder is made robust to articulation distortions and short reference lengths. source: §1, §3 evidence: Zero-shot multi-speaker TTS trained on normal speech can generalize to dysarthric speakers at inference time without any dysarthric training data, provided the speaker encoder is made robust to articulation distortions and short reference lengths. confidence: high relevance: high - claim_id: speaker_similarity_and_phoneme_intelligibility_can_be_simultaneously_improved_through role: complicates claim: Speaker similarity and phoneme intelligibility can be simultaneously improved through knowledge anchoring, avoiding the trade-off seen in prior hybrid approaches where intelligibility gains came at the cost of speaker identity. source: §4.1, Table 1 evidence: Speaker similarity and phoneme intelligibility can be simultaneously improved through knowledge anchoring, avoiding the trade-off seen in prior hybrid approaches where intelligibility gains came at the cost of speaker identity. confidence: high relevance: high limitations: - The system is English-only. The backbone is non-autoregressive (FastSpeech2) with mel-spectrogram output, which imposes a quality ceiling compared to codec-based or flow-matching systems. The approach requires labeled phoneme data to compute PER, which is available for UASpeech but may not generalize. Speaker similarity (0.619) remains below what might be needed for truly personalized assistive use. The method has not been tested on languages with non-Latin scripts or very different phoneme inventories. Future work could explore curriculum learning with codec-based or diffusion-based TTS backbones. caveats: [] - id: interspeech-2025-0787 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: speaker_embedding_manipulation_can_transfer_a_distinctive_speaking_style_lombard role: supports claim: Speaker embedding manipulation can transfer a distinctive speaking style (Lombard speech) to unseen speakers using only a small feedforward network trained on minimal paired data, without retraining the underlying TTS model. source: §2.3, §4.1 evidence: Speaker embedding manipulation can transfer a distinctive speaking style (Lombard speech) to unseen speakers using only a small feedforward network trained on minimal paired data, without retraining the underlying TTS model. confidence: high relevance: high - claim_id: perceptual_plausibility_of_artificially_generated_lombard_speech_depends_substantially_on role: supports claim: Perceptual plausibility of artificially generated Lombard speech depends substantially on modelling pitch and duration changes, not only level amplification as recommended by ITU-T P.1150. source: §4.3, Table 1 evidence: Perceptual plausibility of artificially generated Lombard speech depends substantially on modelling pitch and duration changes, not only level amplification as recommended by ITU-T P.1150. confidence: high relevance: medium - claim_id: spherical_interpolation_between_plain_and_lombard_speaker_embeddings_when_calibrated role: supports claim: Spherical interpolation between plain and Lombard speaker embeddings, when calibrated to a physical noise-level scale, enables continuous and controllable Lombard intensity without requiring recordings at each intermediate level. source: §2.5 evidence: Spherical interpolation between plain and Lombard speaker embeddings, when calibrated to a physical noise-level scale, enables continuous and controllable Lombard intensity without requiring recordings at each intermediate level. confidence: high relevance: medium - claim_id: embedding_space_style_transfer_trained_on_in_distribution_data_can role: supports claim: Embedding-space style transfer trained on in-distribution data can generalise to out-of-distribution speakers with a speaker similarity loss of approximately 0.5 MOS relative to an ideal upper bound. source: §4.3 evidence: Embedding-space style transfer trained on in-distribution data can generalise to out-of-distribution speakers with a speaker similarity loss of approximately 0.5 MOS relative to an ideal upper bound. confidence: high relevance: high limitations: - '- Metavoice 1B is English-only; multilingual or language-adaptive Lombard generation is unaddressed. - Only 10 listeners in the listening test; statistical significance is limited. - The level calibration used ITU-T P.1150, which participants judged as too quiet at the highest noise levels — the standard may underestimate real Lombard level increases. - The feedforward embedding transform was trained on Lombard Grid (slow, scripted utterances) and tested on ITU-P.501 (different style); domain mismatch effects are partially visible. - Dynamic/time-varying background noise scenarios are left as future work.' caveats: [] - id: interspeech-2025-0816 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - singing - VC architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - diffusion_based_speaker_adaptation claims: - claim_id: converting_speech_timbre_to_singing_requires_cross_modal_speaker_embedding role: supports claim: Converting speech timbre to singing requires cross-modal speaker embedding alignment, and standard singer-identity conditioning generalises poorly across the speech-singing domain boundary. source: §1, §2.1 evidence: Converting speech timbre to singing requires cross-modal speaker embedding alignment, and standard singer-identity conditioning generalises poorly across the speech-singing domain boundary. confidence: high relevance: high - claim_id: cycle_training_strategies_that_simulate_paired_cross_domain_data_can role: supports claim: Cycle training strategies that simulate paired cross-domain data can compensate for the scarcity of matched speech-singing corpora in voice conversion training. source: §2.3 evidence: Cycle training strategies that simulate paired cross-domain data can compensate for the scarcity of matched speech-singing corpora in voice conversion training. confidence: high relevance: medium - claim_id: zero_shot_singing_voice_conversion_with_speech_prompts_achieves_lower role: supports claim: Zero-shot singing voice conversion with speech prompts achieves lower timbre similarity scores than same-domain (singing-to-singing) conversion, indicating that the cross-modal gap is not fully closed by embedding alignment alone. source: §3.3, Table 1, Table 2 evidence: Zero-shot singing voice conversion with speech prompts achieves lower timbre similarity scores than same-domain (singing-to-singing) conversion, indicating that the cross-modal gap is not fully closed by embedding alignment alone. confidence: high relevance: high - claim_id: automated_speaker_similarity_metrics_capture_relative_improvements_from_cross_domain role: supports claim: Automated speaker similarity metrics capture relative improvements from cross-domain adaptation that are not clearly reflected in small-panel subjective timbre similarity ratings. source: §3.2, §3.3, Table 3 evidence: Automated speaker similarity metrics capture relative improvements from cross-domain adaptation that are not clearly reflected in small-panel subjective timbre similarity ratings. confidence: high relevance: high limitations: - The subjective evaluation relies on only 10 volunteers, producing confidence intervals that overlap between all three systems on both MOS-n and MOS-ts. The claimed superiority of SSANSVC-stage2 over CoMoSVC in naturalness and similarity is not statistically robust at this sample size. - The model's loss function addresses only mel reconstruction; the authors note that timbre loss and lyrics recognition loss (to reduce CER) are absent and represent the primary direction for future improvement. The two-stage training procedure also introduces significant complexity and training cost compared to the CoMoSVC baseline. Evaluation is restricted to Mandarin speech and singing datasets, so generalisation to other languages and vocal styles is untested. The dependency on NUS-48E — one of few available paired speech-singing corpora — limits reproducibility in languages where such data does not exist. caveats: [] - id: interspeech-2025-0854 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - GAN - flow-matching - transformer-enc-dec relevance: low evidence_role: - historical_context - control_evidence - evaluation_caution current_role: minor method_family: - gan_based_speaker_transfer - flow_matching_speaker_adaptation - transformer_encoder_decoder_adaptation claims: - claim_id: generative_postprocessing_models_for_spectrogram_enhancement_lose_effectiveness_in_low role: supports claim: Generative postprocessing models for spectrogram enhancement lose effectiveness in low-resource settings because the acoustic model produces stronger artifacts that are underrepresented in the available training pairs. source: §1, §2.1 evidence: Generative postprocessing models for spectrogram enhancement lose effectiveness in low-resource settings because the acoustic model produces stronger artifacts that are underrepresented in the available training pairs. confidence: high relevance: medium - claim_id: simulating_low_resource_speaker_conditions_from_subsampled_high_resource_data role: supports claim: Simulating low-resource speaker conditions from subsampled high-resource data provides a viable source of artifact-realistic training pairs for spectrogram postprocessors, yielding consistent naturalness improvements across both GAN and flow-matching architectures. source: §2.2, §4, Table 2 evidence: Simulating low-resource speaker conditions from subsampled high-resource data provides a viable source of artifact-realistic training pairs for spectrogram postprocessors, yielding consistent naturalness improvements across both GAN and flow-matching architectures. confidence: high relevance: medium - claim_id: flow_matching_postprocessors_achieve_higher_absolute_naturalness_than_gan_based role: supports claim: Flow-matching postprocessors achieve higher absolute naturalness than GAN-based postprocessors on low-resource speakers, but show smaller and statistically insignificant gains from the proposed data generation strategy compared to GAN-based postprocessors. source: §4, Table 2 evidence: Flow-matching postprocessors achieve higher absolute naturalness than GAN-based postprocessors on low-resource speakers, but show smaller and statistically insignificant gains from the proposed data generation strategy compared to GAN-based postprocessors. confidence: high relevance: medium - claim_id: noise_augmentation_of_the_acoustic_model_does_not_improve_spectrogram role: supports claim: Noise augmentation of the acoustic model does not improve spectrogram naturalness when ground-truth prosody is used at evaluation, as its primary benefit is prosody prediction rather than spectrogram texture quality. source: §4, Table 2 evidence: Noise augmentation of the acoustic model does not improve spectrogram naturalness when ground-truth prosody is used at evaluation, as its primary benefit is prosody prediction rather than spectrogram texture quality. confidence: high relevance: low - claim_id: multi_stimulus_hidden_reference_tests_alone_may_be_insufficient_to role: supports claim: Multi-stimulus hidden-reference tests alone may be insufficient to detect when a TTS system exceeds vocoded ground-truth quality; ranking-by-elimination tests can reveal preferences for synthetic over reference conditions that MUSHRA-style scoring cannot capture. source: §3.3, §4 evidence: Multi-stimulus hidden-reference tests alone may be insufficient to detect when a TTS system exceeds vocoded ground-truth quality; ranking-by-elimination tests can reveal preferences for synthetic over reference conditions that MUSHRA-style scoring cannot capture. confidence: high relevance: medium limitations: - The method requires training multiple acoustic models (4 in the experiments), increasing compute cost proportional to the number of high-resource speakers used. Evaluation uses ground-truth prosody, making it unclear how the approach fares in full end-to-end synthesis with predicted prosody. Only two low-resource speakers (one male, one female) are tested; generalization to different accent or language scenarios is not validated. The authors themselves acknowledge the noise-augmentation baseline performs worse here because it improves prosody — the interactions between prosody control and spectrogram naturalness are not fully resolved. caveats: [] - id: interspeech-2025-1034 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: minor method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: scaling_a_multi_accent_pronunciation_frontend_from_three_to_fourteen role: supports claim: Scaling a multi-accent pronunciation frontend from three to fourteen accents substantially reduces the minimum training data required to achieve ceiling-level performance in a new target accent. source: §4.2, Table 4 evidence: Scaling a multi-accent pronunciation frontend from three to fourteen accents substantially reduces the minimum training data required to achieve ceiling-level performance in a new target accent. confidence: high relevance: medium - claim_id: accent_similarity_quantified_via_levenshtein_distance_over_phoneme_sequences_weakly role: supports claim: Accent similarity, quantified via Levenshtein distance over phoneme sequences, weakly predicts the success of cross-accent pronunciation knowledge transfer, with low-similarity pairs failing at the data threshold where high- and moderate-similarity pairs succeed. source: §4.3, Table 5 evidence: Accent similarity, quantified via Levenshtein distance over phoneme sequences, weakly predicts the success of cross-accent pronunciation knowledge transfer, with low-similarity pairs failing at the data threshold where high- and moderate-similarity pairs succeed. confidence: high relevance: medium - claim_id: joint_scaling_of_target_and_non_target_accent_training_data role: supports claim: Joint scaling of target and non-target accent training data is necessary to prevent overfitting when one accent is severely underrepresented in a multi-accent model. source: §4.2 evidence: Joint scaling of target and non-target accent training data is necessary to prevent overfitting when one accent is severely underrepresented in a multi-accent model. confidence: high relevance: medium - claim_id: bootstrapped_pronunciation_data_derived_from_a_rule_based_tts_frontend role: supports claim: Bootstrapped pronunciation data derived from a rule-based TTS frontend can substitute for hand-transcribed annotations in multi-accent frontend training while maintaining high unseen word accuracy. source: §3.2, §4.1 evidence: Bootstrapped pronunciation data derived from a rule-based TTS frontend can substitute for hand-transcribed annotations in multi-accent frontend training while maintaining high unseen word accuracy. confidence: high relevance: medium limitations: - Accent similarity effect is assessed with only one example per tier (low/moderate/high); replication with additional pairs would strengthen the conclusion. The experiment uses a single target accent (EDI); generalization to non-British accents or non-English languages is not tested. Pronunciation data is bootstrapped (not real human annotation), which may understate difficulties in real-world deployment where annotations are noisy. The paper addresses only the frontend (text-to-phoneme) component of TTS; it does not evaluate full end-to-end accent TTS quality in listening tests. The ethical caveat — that easy accent building could enable appropriation or mockery — is raised but not operationalized into concrete safeguards. caveats: [] - id: interspeech-2025-1344 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: adapter_based_fine_tuning_preserves_zero_shot_tts_capability_in role: supports claim: Adapter-based fine-tuning preserves zero-shot TTS capability in multilingual models more reliably than full parameter fine-tuning on single-speaker low-resource data. source: §4.3, Table 2 evidence: Adapter-based fine-tuning preserves zero-shot TTS capability in multilingual models more reliably than full parameter fine-tuning on single-speaker low-resource data. confidence: high relevance: high - claim_id: cross_lingual_adaptation_of_tts_models_is_feasible_with_fewer role: supports claim: Cross-lingual adaptation of TTS models is feasible with fewer than 15 hours of single-speaker data when a multilingual pre-trained foundation is available. source: §4.1.1, §4.2 evidence: Cross-lingual adaptation of TTS models is feasible with fewer than 15 hours of single-speaker data when a multilingual pre-trained foundation is available. confidence: high relevance: medium - claim_id: in_flow_matching_tts_fine_tuning_the_text_conditioning_module role: supports claim: In flow-matching TTS fine-tuning, the text conditioning module requires more parameters to adapt to a typologically distant language than the acoustic generation module. source: §4.4.1 evidence: In flow-matching TTS fine-tuning, the text conditioning module requires more parameters to adapt to a typologically distant language than the acoustic generation module. confidence: high relevance: low - claim_id: low_rank_adaptation_in_dit_based_tts_models_exhibits_a role: complicates claim: Low-rank adaptation in DiT-based TTS models exhibits a rank-dependent trade-off between linguistic accuracy and speaker generalisation, with optimal rank dependent on dataset scale. source: §4.4.3 evidence: Low-rank adaptation in DiT-based TTS models exhibits a rank-dependent trade-off between linguistic accuracy and speaker generalisation, with optimal rank dependent on dataset scale. confidence: high relevance: high limitations: - Multi-speaker evaluation shows substantially higher WER (21.6%) compared to a large-scale trained system (10.9%), and the system shows hallucination artefacts (word repeating and skipping) not fully resolved by the proposed adapters. Reproducibility is limited to the Korean-specific KSS dataset; generalisation to other low-resource languages is assumed but not tested. - The paper does not address languages with non-Latin scripts that differ even more from F5-TTS's training data (e.g., Arabic, Swahili). The speaker similarity ceiling for multi-speaker scenarios remains low given that adaptation data comes from a single female speaker only. The interaction between DropPath rate and language-pair linguistic distance is not explored, leaving hyperparameter transfer uncertain. Future work identified by the authors includes more robust text conditioning and improved alignment methods to reduce hallucinations. caveats: [] - id: interspeech-2025-1434 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - VC architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_speaker_adaptation claims: - claim_id: full_utterance_time_reversal_can_serve_as_an_effective_signal role: supports claim: Full-utterance time reversal can serve as an effective signal-level data augmentation for speaker representation learning in voice conversion, as it suppresses phonemic content while retaining speaker-discriminative tonal features. source: §3.1, Table 1 evidence: A perceptual study shows 80.3% speaker identification accuracy from time-reversed speech; Table 1 confirms complete reversal achieves 100% WER (full linguistic removal) alongside the highest cosine speaker similarity score (0.96), higher than any short-time reversal window. confidence: high relevance: high - claim_id: fusing_speaker_embeddings_from_augmented_training_signals_with_conventional_embeddings role: supports claim: Fusing speaker embeddings from augmented training signals with conventional embeddings improves speaker similarity in zero-shot diffusion-based voice conversion. source: §4.3, Table 2 evidence: Adding reversed-speech speaker embeddings via a weighted fusion layer (α = β = 0.5) improves objective speaker similarity by 4.16% on average across DiffHierVC and DDDM-VC; DDDM-VC objective SPK-SIM rises from 0.70 to 0.79 and subjective MUSHRA from 77.46 to 78.61. confidence: high relevance: high - claim_id: the_effectiveness_of_speaker_embedding_augmentation_in_voice_conversion_varies role: complicates claim: The effectiveness of speaker embedding augmentation in voice conversion varies substantially across backbone architectures, complicating claims of generalisability. source: §4.3, Table 2 evidence: For DiffVC, the augmentation improves subjective speaker similarity (50.12 to 53.42) but reduces objective similarity (0.75 to 0.71), while DiffHierVC shows objective improvement but negligible subjective change; only DDDM-VC shows consistent gains on both metrics. confidence: high relevance: high - claim_id: improving_speaker_disentanglement_through_augmentation_does_not_necessarily_trade_off role: supports claim: Improving speaker disentanglement through augmentation does not necessarily trade off against generated speech quality in diffusion-based voice conversion. source: §4.3, Table 2 evidence: DDDM-VC+Ours improves both WV-MOS (3.84 to 3.91) and UTMOS (3.21 to 3.55) alongside speaker similarity gains, indicating that stronger speaker conditioning from the STR augmentation does not degrade synthesis quality. confidence: high relevance: high limitations: - 'Objective and subjective speaker similarity disagree for DiffVC: the augmentation reduces objective similarity (0.75 to 0.71) while improving subjective similarity (50.12 to 53.42). This discrepancy limits confidence in the metric-level generalisation claim across all diffusion backbones. *(§4.3, Table 2)*' - 'The perceptual study supporting the STR principle is small: 25 participants and 6 speakers, all in English. Whether the tonal-pattern preservation property holds equally for tonal languages (Mandarin, Thai) or heavily inflected languages is untested. The approach has been evaluated only on diffusion-based VC systems; compatibility with flow-matching and codec-based VC architectures remains unexplored. No code is publicly released, limiting reproducibility. The weighted fusion coefficients (α and β) are set empirically to 0.5; the paper does not explore learned dynamic weighting conditioned on the input utterance.' caveats: [] - id: interspeech-2025-1440 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - codec architecture: - VAE relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - vae_latent_speaker_adaptation claims: - claim_id: self_supervised_disentanglement_of_speech_into_content_speaker_and_prosody role: supports claim: Self-supervised disentanglement of speech into content, speaker, and prosody streams can match or exceed supervised codec quality at significantly lower bitrate. source: §4.1, Table 1, Table 2 evidence: Self-supervised disentanglement of speech into content, speaker, and prosody streams can match or exceed supervised codec quality at significantly lower bitrate. confidence: high relevance: low - claim_id: codec_coding_efficiency_is_more_sensitive_to_information_factorisation_than role: supports claim: Codec coding efficiency is more sensitive to information factorisation than to raw model capacity or bitrate allocation. source: §4.1, Table 1 evidence: Codec coding efficiency is more sensitive to information factorisation than to raw model capacity or bitrate allocation. confidence: high relevance: low - claim_id: routing_wavlm_supervision_to_the_decoder_rather_than_the_encoder role: supports claim: Routing WavLM supervision to the decoder rather than the encoder during training improves speaker-content disentanglement for voice conversion. source: §2.5, §4.2 evidence: Routing WavLM supervision to the decoder rather than the encoder during training improves speaker-content disentanglement for voice conversion. confidence: high relevance: medium - claim_id: ultra_low_bitrate_codecs_below_0_5_kbps_can_achieve role: supports claim: Ultra-low-bitrate codecs (below 0.5 kbps) can achieve MUSHRA scores competitive with codecs operating at 2–3 kbps when disentanglement is applied to reduce frame-level redundancy. source: §4.1, Table 2 evidence: Ultra-low-bitrate codecs (below 0.5 kbps) can achieve MUSHRA scores competitive with codecs operating at 2–3 kbps when disentanglement is applied to reduce frame-level redundancy. confidence: high relevance: medium limitations: - The demo and code availability are not confirmed in the paper or metadata. Reproducibility relies on external checkpoints for baselines — FACodec and SpeechTokenizer results are inferred from official checkpoints under potentially different conditions than the re-trained TiCodec and DAC baselines. - Evaluation is restricted to English (LibriSpeech and VCTK). Generalisation to other languages, accents, or spontaneous-speech domains is untested. The prosody encoder's low-mel-bin design is validated empirically via t-SNE visualisation but without a formal mutual information analysis. It is unclear how much prosody actually remains once the speaker and content encoders are also active during decoding — partial speaker clustering in Fig. 2 suggests the separation is not complete. caveats: [] - id: interspeech-2025-1478 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation claims: - claim_id: replacing_resnet_18_av_hubert_visual_encoders_with_mobile_video role: supports claim: Replacing ResNet-18/AV-HuBERT visual encoders with mobile video networks reduces lip-to-speech system complexity by an order of magnitude at the cost of intelligibility degradation. source: §4.1, Table 1 evidence: LightL2S uses MoViNet-A0 instead of ResNet-18-based AV-HuBERT, reducing inference from 32–34 GMacs to 0.8 GMacs on LRS3 while WER increases from 27–30% (AV-HuBERT-based methods) to 64.8%. confidence: high relevance: medium - claim_id: multi_resolution_spectrogram_discriminators_substantially_improve_speech_naturalness_in_ddsp role: supports claim: Multi-resolution spectrogram discriminators substantially improve speech naturalness in DDSP-based lip-to-speech synthesis, beyond spectral regression alone. source: §4.3, Table 3 evidence: Removing the adversarial GAN loss from LightL2S collapses UTMOS from 2.93 to 1.58 and SECS from 0.72 to 0.63 on LRS3, while computational cost remains identical at 0.8 GMacs. confidence: high relevance: medium - claim_id: objective_quality_metrics_can_diverge_from_human_perceptual_judgments_in role: complicates claim: Objective quality metrics can diverge from human perceptual judgments in lip-to-speech synthesis, making WER and UTMOS insufficient as sole quality signals. source: §4.2, Table 2 evidence: NaturalL2S exceeds Ground Truth on UTMOS (3.66 vs. 3.59) but scores lower in subjective naturalness MOS (4.10 vs. 4.51); LightL2S achieves the highest speaker similarity MOS (3.72) despite being outranked on SECS by several baselines. confidence: high relevance: medium - claim_id: efficient_transformer_variants_zipformer_can_substitute_standard_conformer_backbones_in role: refines claim: Efficient transformer variants (Zipformer) can substitute standard Conformer backbones in visual speech modelling with simultaneous improvements in quality and computational cost. source: §4.3, Table 3 evidence: Replacing Zipformer with a Conformer backbone in LightL2S increases GMacs from 0.80 to 1.09 while reducing UTMOS from 2.93 to 2.74 and raising WER from 64.8% to 66.3% on LRS3. confidence: high relevance: medium limitations: - 'Intelligibility remains a major limitation: LightL2S achieves WER 64.8% on LRS3, more than double the 27–30% of AV-HuBERT-based methods. This gap is attributed to the absence of large-scale pre-trained visual-acoustic features and the lack of text supervision, and is explicitly left as an open problem. At this WER level, practical deployment for communication assistance — the paper''s stated motivation — requires further work.' - Evaluation is confined to English TED/TEDx video from LRS3; generalisation to other languages, accents, spontaneous speech, and noisy real-world conditions is untested. The system uses a reference speaker embedding extracted at inference time, meaning an enrolment clip must be available, which may be impractical for truly in-the-wild edge scenarios. Parameter count is not reported, making direct comparison with published efficient TTS systems difficult. caveats: [] - id: interspeech-2025-1536 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: zero_shot_tts_systems_trained_on_typical_speech_systematically_over role: supports claim: Zero-shot TTS systems trained on typical speech systematically over-normalise intelligibility when cloning dysarthric speech, producing cleaner output than the reference at the cost of dysarthric feature preservation. source: §4.1, §4.2, Table 3 evidence: F5-TTS yields delta-WER DI of 0.59 for high-severity speakers and delta-CER DI of 0.75, both rated "poor," while SIM-o DI remains 0.85 and AutoPCP DI 0.90, indicating intelligibility divergence is the primary failure mode rather than loss of speaker or prosodic similarity. confidence: high relevance: high - claim_id: augmenting_dysarthric_asr_training_data_with_zero_shot_voice_cloned role: complicates claim: Augmenting dysarthric ASR training data with zero-shot voice-cloned samples improves performance for mild cases but degrades it for severe cases when the cloning system is biased toward intelligibility normalisation. source: §4.3, Table 4 evidence: Adding F5-TTS synthetic data alongside reference audio reduces WER by 44.6% for low-severity speakers but increases WER by 5.7% and 7.7% for mid- and high-severity speakers respectively, compared to the reference-only baseline. confidence: high relevance: high - claim_id: fairness_metrics_disparate_impact_and_parity_difference_applied_to_objective role: supports claim: Fairness metrics (Disparate Impact and Parity Difference) applied to objective TTS evaluation measures reveal severity-level disparities that aggregate quality scores would obscure. source: §2.3, §4.2, Table 3 evidence: Per-severity DI analysis on TORGO shows near-zero intelligibility bias for low severity (delta-WER DI=0.97) escalating to severe bias for high severity (DI=0.59), a pattern invisible in mean quality scores and only apparent through disaggregated fairness analysis. confidence: high relevance: low - claim_id: bias_in_zero_shot_dysarthric_speech_cloning_varies_across_gender role: complicates claim: Bias in zero-shot dysarthric speech cloning varies across gender as well as severity, with male and female speakers affected on different dimensions. source: §4.2, Table 3 evidence: Male speakers exhibit higher intelligibility bias (delta-WER PD=0.37, DI=0.68) while female speakers show higher prosody bias (AutoPCP PD=0.36), demonstrating that severity-level aggregation masks a secondary gender-based disparity axis. confidence: high relevance: high limitations: - The study uses a single TTS system (F5-TTS) and a single dataset (TORGO, restricted sentences only). Whether the intelligibility-bias pattern generalises to other zero-shot TTS systems, to other dysarthric datasets, or to connected speech is untested. Results from a small speaker pool (8 dysarthric, 7 healthy) should be interpreted with caution. - The fairness metrics (DI, PD) are borrowed from sociotechnical fairness literature and treat the healthy group as a reference. This framing may not be universally appropriate for disability-related speech variability, where defining a "privileged" reference group raises its own ethical questions. No human listening studies were conducted, so the relationship between the measured objective biases and perceptual quality from the perspective of dysarthric users remains unexamined. Future work on fairness-aware data augmentation, dysarthria severity-conditioned synthesis, or severity-adaptive training objectives is suggested but not pursued here. caveats: [] - id: interspeech-2025-1538 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - VC architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: conditioning_acoustic_generation_on_explicitly_predicted_text_tokens_reduces_intelligibility role: supports claim: Conditioning acoustic generation on explicitly predicted text tokens reduces intelligibility errors in autoregressive voice conversion relative to purely acoustic-domain approaches. source: §3.3.3, Table 1 evidence: Removing text token generation from StarVC raises WER from 6.27% to 7.30% and SECS-WavLM drops from 0.472 to 0.382; StarVC achieves the lowest WER and CER among all compared systems including diffusion-based CosyVoice (8.24%/4.27%). confidence: high relevance: medium - claim_id: multi_stage_training_that_initializes_voice_conversion_with_asr_pretraining role: supports claim: Multi-stage training that initializes voice conversion with ASR pretraining improves both content preservation and speaker similarity relative to single-stage training. source: §3.3.3, Table 1 evidence: Removing multi-stage training degrades SECS-Res from 0.835 to 0.812 and raises WER from 6.27% to 7.24%; multi-stage training is the single largest contributor in the ablation study. confidence: high relevance: high - claim_id: objective_speaker_embedding_metrics_and_perceptual_speaker_similarity_ratings_can role: complicates claim: Objective speaker embedding metrics and perceptual speaker similarity ratings can diverge for codec-based voice conversion systems trained with strong linguistic objectives. source: §3.3.1, §3.3.2, Tables 1-2 evidence: StarVC scores marginally below CosyVoice on SECS-Res (0.835 vs. 0.839) and SECS-WavLM (0.472 vs. 0.478), yet exceeds CosyVoice on subjective SMOS (3.98 vs. 3.94), suggesting embedding-based metrics underestimate perceived similarity for this system class. confidence: high relevance: high - claim_id: autoregressive_voice_conversion_systems_can_produce_explicit_transcription_output_alongside role: refines claim: Autoregressive voice conversion systems can produce explicit transcription output alongside converted audio at negligible additional cost, enabling inline content verification without separate ASR inference. source: §3.3.1, Table 1 evidence: StarVC generates text tokens with WER-Text of 4.95% and CER-Text of 1.51% as a byproduct of the VC decoding process, providing word-level content verification as part of the conversion pipeline. confidence: high relevance: medium limitations: - Subjective MOS evaluation involves only 20 listeners and 20 source-target pairs, making the reported SMOS and NMOS advantages over CosyVoice and OpenVoice V2 (all within overlapping confidence intervals) difficult to interpret as significant. - The evaluation covers English only on a single clean corpus (LibriTTS test-clean). Generalization to cross-lingual conversion, noisy conditions, or longer conversational utterances is untested. The three-stage training pipeline requires 180 GPU-hours on 8 H100s, representing a substantial compute cost that may limit practical adoption. Data augmentation relies on OpenVoice V2-synthesized speech, which could propagate artifacts from that system into StarVC's training distribution. Whether the text-before-speech decoding constraint generalizes to expressive or emotional speech conversion, where prosody is not captured by a pure transcription, remains an open question. caveats: [] - id: interspeech-2025-1638 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation claims: - claim_id: treating_emotion_as_a_disentangled_parallel_synthesis_branch_improves_emotional role: supports claim: Treating emotion as a disentangled parallel synthesis branch improves emotional expressiveness over global style conditioning in zero-shot TTS. source: §3.6, Table 2 evidence: Ablation removing emotion decoupling reduces EMOS from 3.96 to 3.45 on LibriTTS test, with Emotion Discrepancy rising from 0.571 to 0.939, confirming that explicit emotion isolation drives the expressiveness gains. confidence: high relevance: high - claim_id: llm_based_emotion_transformation_conditioned_on_target_text_semantics_reduces role: supports claim: LLM-based emotion transformation conditioned on target text semantics reduces emotion-content mismatch in zero-shot speech synthesis. source: §3.6, Table 2 evidence: Removing the emotion-aware LLM raises ED from 0.571 to 1.056 and drops EMOS from 3.96 to 3.73 on LibriTTS test, with both components contributing independently to emotional consistency. confidence: high relevance: high - claim_id: direct_reference_emotion_transfer_in_zero_shot_tts_produces_emotionally role: complicates claim: Direct reference emotion transfer in zero-shot TTS produces emotionally inconsistent speech when the reference and target texts differ in emotional valence. source: §3.3, §3.5, Table 1 evidence: All five comparison zero-shot TTS baselines (YourTTS, TransferTTS, VALL-E, E2-TTS, CosyVoice) show Emotion Discrepancy scores of 0.802 to 1.399 versus EATS-Speech at 0.571, suggesting that without explicit emotion adaptation, standard conditioning mechanisms misalign emotion to content. confidence: high relevance: high - claim_id: evaluation_of_emotion_expressiveness_with_metrics_derived_from_the_same role: complicates claim: Evaluation of emotion expressiveness with metrics derived from the same model used in training introduces circularity that limits the reliability of reported Emotion Discrepancy scores. source: §2.1, §3.5 evidence: ED is computed using utterance-level emotion2vec embeddings, the same pre-trained model whose frame-level features the Emotion Tokenizer builds on; this means the metric and the system share the same representational basis, potentially inflating reported gains. confidence: high relevance: low limitations: - The Emotion Discrepancy metric is computed using emotion2vec embeddings, the same model used as the feature extractor inside EATS-Speech's Emotion Tokenizer. This circularity means the objective metric may overstate the system's emotion alignment advantage relative to baselines that do not use emotion2vec internally. - Evaluation is limited to LibriTTS, an audiobook corpus with naturally constrained emotional variability. How well the framework transfers to genuinely expressive speech (acted emotion corpora, spontaneous speech) is not assessed. The paper does not report how reference and target samples are matched for emotional contrast in the test set, making it difficult to assess task difficulty. - The model size is not reported; the system combines a flow transformer, an emotion tokenizer, and a GPT-2-based LLM across three sequential training stages, suggesting non-trivial resource requirements that are not characterized. Only English is evaluated; extension to multilingual settings with different prosodic and emotional conventions is left open. caveats: [] - id: interspeech-2025-1726 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation claims: - claim_id: fine_tuning_large_scale_tts_models_on_minimal_scripted_read role: supports claim: Fine-tuning large-scale TTS models on minimal scripted read-aloud data (order of minutes) produces personalised voices with perceived AAC suitability comparable to fine-tuning on hour-scale spontaneous recordings. source: §3.2, Table 1 evidence: FT-Read (12 min of read-aloud sentences, SECS 0.643) and FT-Spont-Short (1h spontaneous, SECS 0.706) show no significant difference in MUSHRA-rated listener preference (p=0.213) for a Hungarian stroke survivor case study. confidence: high relevance: medium - claim_id: zero_shot_tts_adaptation_from_dysarthric_reference_audio_achieves_perceptual role: supports claim: Zero-shot TTS adaptation from dysarthric reference audio achieves perceptual suitability for assistive communication comparable to adaptation from typical speech, enabling personalisation for users who lack pre-morbid recordings. source: §3.2 evidence: ZS-Atypical and ZS-Typical show no significant difference in MUSHRA-rated AAC suitability (p=0.683), despite a large objective speaker similarity gap (SECS 0.324 vs 0.635). confidence: high relevance: high - claim_id: embedding_based_objective_speaker_similarity_metrics_may_not_reliably_predict role: refines claim: Embedding-based objective speaker similarity metrics may not reliably predict perceived suitability in assistive communication contexts, where other voice qualities matter more than timbre fidelity. source: §3.1, §3.2, Table 1 evidence: ZS-Atypical achieves the lowest SECS (0.324) of all evaluated systems yet is rated comparably to ZS-Typical (SECS 0.635) by human listeners in the MUSHRA-like AAC suitability test (p=0.683). confidence: high relevance: high - claim_id: using_dysarthric_speech_as_a_prosodic_reference_input_for_a role: supports claim: Using dysarthric speech as a prosodic reference input for a fine-tuned TTS system yields measurable alignment with the speaker's intended prosodic pattern while preserving intelligibility. source: §5.2 evidence: Dysarthric-prompted synthesis achieved 29.3% higher SSIM in scalogram comparison of prosodic peaks versus unprompted synthesis (0.0548 vs 0.0424, p=0.0013 by binomial test); CER increased only from 2.9% to 2.3% (actually improved) when switching to dysarthric-prompted synthesis. confidence: high relevance: medium limitations: - The entire evaluation rests on a single research participant (one Hungarian stroke survivor). Generalisability across speakers, languages, and impairment types is untested, and the authors acknowledge this directly. - The interface requires typed text input, making it inaccessible for AAC users with motor impairments. The paper treats the interface as an evaluation platform rather than a functional device; eye-tracking compatibility is noted as future work. The prosody adaptation mechanism relies on the fact that for identical lexical content, prosodic patterns partially transfer through the audio prompt, which may not hold for diverse sentence types or severe dysarthria cases. The study also evaluates only XTTS-v2 and does not compare against other zero-shot or fine-tuning approaches that have emerged in parallel. caveats: [] - id: interspeech-2025-1747 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - VC architecture: - diffusion - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_speaker_adaptation - gan_based_speaker_transfer claims: - claim_id: performing_knowledge_distillation_in_the_conversion_process_rather_than_the role: supports claim: Performing knowledge distillation in the conversion process rather than the reconstruction process is necessary to avoid identity-mapping collapse when the content encoder is simultaneously trainable. source: §4.2, Table 1, Table 2 evidence: Baseline reconstruction-distillation with trainable CNN (FastVoiceGrad + pϕ) achieves UTMOS 3.45 and SECS 0.718; switching to conversion-based ADCD raises these to 4.03 and 0.853, respectively. Direct distillation (an alternative that avoids conversion-process training) degrades to UTMOS 3.35 with a 1-layer encoder, confirming the mechanism failure. confidence: high relevance: medium - claim_id: simultaneously_distilling_both_the_diffusion_backbone_and_the_content_encoder role: supports claim: Simultaneously distilling both the diffusion backbone and the content encoder of a one-step diffusion VC model can reduce inference cost substantially with little or no quality degradation relative to distilling only the diffusion module. source: §4.3, Table 3, Table 5 evidence: FasterVoiceGrad (joint distillation) achieves UTMOS 4.03 and SECS 0.853 vs. FastVoiceGrad's 3.96 and 0.847, while delivering 6.6x GPU RTF and 1.8x CPU RTF improvement across both VCTK and LibriTTS evaluations. confidence: high relevance: medium - claim_id: neural_speaker_encoder_based_similarity_metrics_may_not_reflect_perceptual role: complicates claim: Neural speaker-encoder-based similarity metrics may not reflect perceptual speaker identity as judged by human listeners, particularly when source speaker residuals remain after conversion. source: §4.3, Table 3, Table 4 evidence: FasterVoiceGrad scores higher SECS (0.853 vs. 0.847) than FastVoiceGrad but lower sMOS (2.66 vs. 2.84); the authors attribute the discrepancy to the CNN content encoder not explicitly removing source speaker characteristics, which humans but not speaker encoders detect. confidence: high relevance: high - claim_id: contrastive_speaker_emphasis_via_inverse_score_distillation_mitigates_the_trade role: refines claim: Contrastive speaker emphasis via inverse score distillation mitigates the trade-off between content preservation and speaker conversion when joint distillation is used. source: §4.2, Table 1 evidence: Adding inverse score distillation on top of reconversion-based distillation improves SECS from 0.844 to 0.853 without degrading CER (1.2) or UTMOS (4.03), recovering the speaker similarity lost by reconversion's content preservation bias. confidence: high relevance: medium limitations: - The CNN content encoder does not explicitly separate source speaker identity from linguistic content. Subjective sMOS scores (2.66) are measurably lower than FastVoiceGrad's (2.84), suggesting residual source voice characteristics persist and limit perceptual speaker conversion quality despite competitive automatic metrics. - Subjective evaluation involved only 11 participants and 90 speaker/sentence pairs, which constrains the statistical power of the sMOS comparisons. The GPU speedup (6.6x) is larger than the CPU speedup (1.8x), reflecting architectural factors that may affect deployment on edge hardware differently. The study does not evaluate against modern flow-matching or large-scale VC baselines, leaving the system's position in the broader landscape uncharacterised. Future directions suggested include accent conversion and real-time VC applications. caveats: [] - id: interspeech-2025-2679 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: natural_language_prompt_conditioning_provides_insufficient_fine_grained_control_for role: complicates claim: Natural language prompt conditioning provides insufficient fine-grained control for voice reconstruction tasks requiring consistent intelligibility and speaker identity. source: §5.2, §5.3, §6 evidence: WER for synthesised speech remains approximately 20-25% across all prompted intelligibility levels (from "extremely poor" to "extremely good") and across all six test speakers; speaker similarity scores are similarly insensitive to prompt variation; expert listening confirms stochastic sampling across traits despite fixed prompts. confidence: high relevance: high - claim_id: large_instruction_conditioned_tts_models_can_be_adapted_to_dysarthric role: supports claim: Large instruction-conditioned TTS models can be adapted to dysarthric speech through fine-tuning on small curated datasets. source: §4.1, §5.1 evidence: Parler TTS Mini fine-tuned on ~23 hours of combined SAP dysarthric and MLS healthy speech learns to generate from the combined distribution; WER for fine-tuning data correlates with intelligibility ratings, confirming the model internalises the distribution. confidence: high relevance: medium - claim_id: automatic_evaluation_metrics_calibrated_on_healthy_synthetic_speech_do_not role: complicates claim: Automatic evaluation metrics calibrated on healthy synthetic speech do not generalise reliably to dysarthric speech and require domain validation before use. source: §5.1 evidence: UTMOS scores fall consistently below 3 for all SAP dysarthric speakers, even those with mild impairment (rating 1-2), indicating a domain mismatch that makes UTMOS unusable in this context; Resemblyzer speaker similarity shows comparable ranges across healthy and dysarthric conditions, demonstrating metric-level robustness varies substantially. confidence: high relevance: low limitations: - UTMOS is found unusable for evaluating dysarthric speech synthesis in this work. Papers that report UTMOS for atypical speech without a domain validation step should be interpreted with caution. - The dataset is limited to 22 speakers across two etiologies (ALS and Cerebral Palsy), with no speakers at the most severe intelligibility levels (ratings 6-7). Generalisation to other dysarthric conditions or more severe impairment is undemonstrated. Subjective evaluation was restricted to expert listening rather than a formal MOS study, which makes comparison to other voice reconstruction and personalised TTS systems difficult. - 'The paper attributes the controllability failure to Parler''s architecture: the entire prompt is embedded by a pre-trained NLP model and attended to by the speech LM, which may be too indirect for categorical control signals like speaker identity. The authors suggest that fine-grained, direct conditioning mechanisms for speaker identity and intelligibility are needed. They also propose data augmentation via acoustic-to-articulatory inversion as a path to larger training sets, and note that proper evaluation will require involvement of VOCA users and speech therapists, for which no established protocol currently exists.' caveats: [] - id: interspeech-2025-2765 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: cmos_and_mushra_scores_can_overestimate_naturalness_relative_to_human role: supports claim: CMOS and MUSHRA scores can overestimate naturalness relative to human deception rates, because listeners in preference tests match stimuli to reference recordings rather than assess absolute human-likeness. source: §4.1, §4.2, Table 2 evidence: CMOS and MUSHRA scores can overestimate naturalness relative to human deception rates, because listeners in preference tests match stimuli to reference recordings rather than assess absolute human-likeness. confidence: high relevance: medium - claim_id: evaluating_tts_systems_on_benchmarks_with_low_expressive_variation_sets role: supports claim: Evaluating TTS systems on benchmarks with low expressive variation sets an artificially low bar, as human recordings from such datasets are themselves infrequently identified as human. source: §4.1 evidence: Evaluating TTS systems on benchmarks with low expressive variation sets an artificially low bar, as human recordings from such datasets are themselves infrequently identified as human. confidence: high relevance: medium - claim_id: commercial_tts_systems_achieve_near_human_deception_rates_in_zero role: supports claim: Commercial TTS systems achieve near-human deception rates in zero-shot speaker adaptation to expressive conversational speech, while leading open-source systems remain substantially below this threshold. source: §4.2, Table 2 evidence: Commercial TTS systems achieve near-human deception rates in zero-shot speaker adaptation to expressive conversational speech, while leading open-source systems remain substantially below this threshold. confidence: high relevance: high - claim_id: fine_tuning_on_high_quality_expressive_data_improves_but_does role: supports claim: Fine-tuning on high-quality expressive data improves but does not close the gap between open-source TTS naturalness and human speech in deception-based evaluations. source: §4.3, Table 3 evidence: Fine-tuning on high-quality expressive data improves but does not close the gap between open-source TTS naturalness and human speech in deception-based evaluations. confidence: high relevance: medium - claim_id: digital_voice_quality_artifacts_and_flat_prosody_are_the_primary role: supports claim: Digital voice quality artifacts and flat prosody are the primary perceptual cues that allow listeners to reliably identify synthetic speech from open-source systems. source: §4.4, Table 4 evidence: Digital voice quality artifacts and flat prosody are the primary perceptual cues that allow listeners to reliably identify synthetic speech from open-source systems. confidence: high relevance: low limitations: - The study evaluates only US-English native listeners on English speech; findings about human deception rates may not generalise to other languages, accents, or listener populations with different familiarity with synthetic speech. - The evaluation is limited to ten systems due to budget constraints, excluding many recent strong models. HFR is a binary measure that does not distinguish degrees of unnaturalness, and — like all subjective tests — is subject to listener variability and session-level fatigue effects. The fine-tuning experiments use only 40 hours of Expresso; whether larger or more diverse fine-tuning data would close the open-source/commercial gap remains an open question. SAFFRON's reproducibility in other institutional contexts (ethics approval, crowdsourcing platform) has not been demonstrated. caveats: [] - id: interspeech-2025-2815 published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: [] claims: - claim_id: zero_shot_tts_enables_personalized_voice_output_in_assistive_speech role: supports claim: Zero-shot TTS enables personalized voice output in assistive speech applications by conditioning synthesis on short user-supplied reference audio, without requiring model retraining or large speaker-specific corpora. source: §2.1 evidence: XTTS is integrated into the AAC platform so that users record or upload a short voice clip; synthesis is then conditioned on this reference to produce speech in the target voice for any input text, with no fine-tuning step. confidence: high relevance: high - claim_id: integrating_cloud_based_generative_ai_into_assistive_applications_with_privacy role: complicates claim: Integrating cloud-based generative AI into assistive applications with privacy-sensitive users creates tension between expressive capability and data security when user speech or conversational content must be transmitted to third-party services. source: §2.3, §2.4 evidence: The paper flags that GPT-4o-based text expansion sends conversational content to third-party servers, and recommends replacing it with a locally run LLM before broad deployment; the same privacy concern applies to cloud-based image generation components. confidence: high relevance: medium limitations: - No user study or controlled evaluation is included, leaving the system's effectiveness relative to current AAC solutions undemonstrated. Long-term effects on communicative skill development, social engagement, and generalization across linguistic and cultural settings are identified as open questions for future longitudinal research. The text expansion feature relies on GPT-4o, creating a privacy exposure that the authors explicitly defer to future work for remediation. Use of DALL-E 3, which may be trained on data sourced without permission, raises ethical concerns that the modular design aims to accommodate but does not resolve. caveats: [] - id: interspeech-2025-raju25_interspeech published_date: "2025-08-17" entry_date: '2026-07-29' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - flow_matching_speaker_adaptation claims: - claim_id: multilingual_adaptation_of_flow_matching_tts_through_vocabulary_expansion_and role: supports claim: Multilingual adaptation of flow-matching TTS through vocabulary expansion and phoneme-based tokenisation can extend zero-shot speaker preservation to typologically diverse language families not covered by the base model. source: §2.2.2 evidence: F5-TTS fine-tuned on 1,626 hours across English and 9 Indian languages with an expanded vocabulary and phoneme-based tokenisation achieves zero-shot speaker-conditioned synthesis across all supported languages. confidence: high relevance: high - claim_id: end_to_end_automated_dubbing_pipelines_integrating_asr_machine_translation role: supports claim: End-to-end automated dubbing pipelines integrating ASR, machine translation, and zero-shot TTS can substantially reduce manual effort in cross-lingual content localisation. source: §2.1, §3 evidence: The platform automates six sequential stages from audio extraction to output re-encoding, supports 9 Indian languages, and has been adopted by content creators and educators without requiring specialist voice talent. confidence: high relevance: high - claim_id: duration_alignment_for_lip_synchronisation_in_automated_dubbing_requires_a role: complicates claim: Duration alignment for lip-synchronisation in automated dubbing requires a user-controlled trade-off between temporal precision and naturalness that cannot be resolved automatically. source: §2.3 evidence: Duration alignment is a toggleable feature; when active it constrains synthesis duration to source timestamps, potentially affecting naturalness; when disabled, utterances are produced sequentially with a fixed 1-second pause between them. confidence: high relevance: medium limitations: - 'No quantitative evaluation is reported, making it impossible to verify speaker similarity, naturalness, or ASR accuracy in the dubbed output. The paper does not compare against any TTS or dubbing baseline. Coverage is uneven: the in-house training data covers only six of the ten supported languages, and the paper does not break out performance by language. Cross-lingual expressivity is flagged as a limitation by the authors, with the acknowledgement that nuanced prosodic transfer across typologically distant language pairs remains unsolved.' caveats: [] - id: '2508.15565' published_date: "2025-08-21" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - VC architecture: - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - gan_based_speaker_transfer claims: - claim_id: anonymizing_adversarial_utterances_toward_a_pseudo_speaker_derived_from_the role: supports claim: Anonymizing adversarial utterances toward a pseudo-speaker derived from the batch mean of adversarial embeddings improves identity unlinkability without requiring a designated real speaker. source: §IV-A, Table I evidence: Anonymizing adversarial utterances toward a pseudo-speaker derived from the batch mean of adversarial embeddings improves identity unlinkability without requiring a designated real speaker. confidence: high relevance: medium - claim_id: in_feedforward_adversarial_anonymization_training_with_a_composite_loss_covering role: supports claim: In feedforward adversarial anonymization, training with a composite loss covering untargeted attack, perceptual quality, and unlinkability objectives achieves better balance between de-identification and identity unlinkability than untargeted attack alone. source: §IV-A, Table I evidence: In feedforward adversarial anonymization, training with a composite loss covering untargeted attack, perceptual quality, and unlinkability objectives achieves better balance between de-identification and identity unlinkability than untargeted attack alone. confidence: high relevance: medium - claim_id: adversarial_perturbation_based_voice_anonymization_degrades_substantially_under_strong_adaptive role: supports claim: Adversarial perturbation-based voice anonymization degrades substantially under strong adaptive attacks such as quantization, revealing a fundamental vulnerability that is independent of the training strategy. source: §V-F, Table III evidence: Adversarial perturbation-based voice anonymization degrades substantially under strong adaptive attacks such as quantization, revealing a fundamental vulnerability that is independent of the training strategy. confidence: high relevance: medium - claim_id: speaker_adversarial_perturbations_transfer_less_effectively_to_black_box_speaker role: supports claim: Speaker adversarial perturbations transfer less effectively to black-box speaker extractors with strong verification capability, indicating that white-box EER gains overstate real-world privacy protection. source: §V-G, Table II evidence: Speaker adversarial perturbations transfer less effectively to black-box speaker extractors with strong verification capability, indicating that white-box EER gains overstate real-world privacy protection. confidence: high relevance: medium limitations: - De-identification and identity unlinkability EERs drop sharply under adaptive attacks (quantization reduces de-id EER from 46.79% to 7.63%), and black-box transfer to the ResNet100 extractor similarly reduces EERs to single digits. The method's privacy guarantees are therefore limited to white-box threat models, which may not reflect realistic deployment conditions. - Identity unlinkability generalises less well than de-identification to out-of-domain datasets, particularly LibriSpeech and AIShell, suggesting the batch mean loss is sensitive to the speaker distribution at training time. The method also degrades speech intelligibility (WER increases across all ASR test sets compared to original speech), which limits applicability in voice assistant or captioning contexts. Future directions include improving transferability against black-box extractors, enhancing resilience to adaptive attacks, and strengthening out-of-domain unlinkability. caveats: [] - id: '2508.15931' published_date: "2025-08-21" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - evaluation architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: pairwise_comparison_framing_reduces_annotation_subjectivity_and_label_sparsity_in role: supports claim: Pairwise comparison framing reduces annotation subjectivity and label sparsity in perceptual attribute modelling more effectively than scalar labelling alone. source: §2.2, §3.1 evidence: Pairwise comparison framing reduces annotation subjectivity and label sparsity in perceptual attribute modelling more effectively than scalar labelling alone. confidence: high relevance: medium - claim_id: differential_attention_that_subtracts_shared_information_between_two_representations_improves role: supports claim: Differential attention that subtracts shared information between two representations improves cross-speaker generalisation in timbre attribute discrimination. source: §3.2, Table 1 evidence: Differential attention that subtracts shared information between two representations improves cross-speaker generalisation in timbre attribute discrimination. confidence: high relevance: high - claim_id: transitivity_based_data_augmentation_over_sparse_pairwise_labels_significantly_expands role: supports claim: Transitivity-based data augmentation over sparse pairwise labels significantly expands training coverage and improves model robustness without requiring additional human annotation. source: §3.1, Table 2 evidence: Transitivity-based data augmentation over sparse pairwise labels significantly expands training coverage and improves model robustness without requiring additional human annotation. confidence: high relevance: medium - claim_id: acoustic_signal_features_and_human_perceptual_judgements_of_timbre_attributes role: supports claim: Acoustic signal features and human perceptual judgements of timbre attributes are systematically inconsistent, limiting the effectiveness of handcrafted feature approaches. source: §2.2, Figure 1 evidence: Acoustic signal features and human perceptual judgements of timbre attributes are systematically inconsistent, limiting the effectiveness of handcrafted feature approaches. confidence: high relevance: medium limitations: - Evaluation is confined to VCTK-RVA, a dataset of 110 clean studio-recorded English speakers. Generalisation to spontaneous speech, noisy conditions, or non-English timbre descriptors is entirely unvalidated. - The model targets classification accuracy but does not demonstrate downstream impact on any generation task. Whether improved timbre attribute detection actually produces better controllable TTS or voice editing remains an open question. The FACodec encoder is frozen throughout; it is unclear whether joint fine-tuning would further improve performance. The 34 timbre attributes in VCTK-RVA are English-centric and human-annotated by a small group, raising questions about attribute definition consistency and cultural transferability. caveats: [] - id: '2508.17031' published_date: "2025-08-23" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation - gan_based_speaker_transfer claims: - claim_id: cross_modal_attention_between_phoneme_and_audio_representations_is_more role: supports claim: Cross-modal attention between phoneme and audio representations is more effective than global speaker embeddings for preserving localised prosodic and acoustic context in speech insertion tasks. source: §4.3, Table 3 evidence: Cross-modal attention between phoneme and audio representations is more effective than global speaker embeddings for preserving localised prosodic and acoustic context in speech insertion tasks. confidence: high relevance: medium - claim_id: non_autoregressive_tts_architectures_with_duration_prediction_can_infer_variable role: supports claim: Non-autoregressive TTS architectures with duration prediction can infer variable-length insertion segments without explicit phoneme-level alignment at inference time. source: §3.1, §3.2 evidence: Non-autoregressive TTS architectures with duration prediction can infer variable-length insertion segments without explicit phoneme-level alignment at inference time. confidence: high relevance: medium - claim_id: combining_local_and_global_adversarial_losses_with_a_triplet_based role: supports claim: Combining local and global adversarial losses with a triplet-based style matching loss reduces perceptual artefacts in synthesised speech segments that L1 reconstruction loss alone cannot eliminate. source: §3.3, §4.3 evidence: Combining local and global adversarial losses with a triplet-based style matching loss reduces perceptual artefacts in synthesised speech segments that L1 reconstruction loss alone cannot eliminate. confidence: high relevance: medium - claim_id: speech_insertion_quality_degrades_more_gracefully_with_insertion_length_when role: supports claim: Speech insertion quality degrades more gracefully with insertion length when the model conditions directly on local audio context rather than full-utterance speaker embeddings. source: §4.2, Table 1 evidence: Speech insertion quality degrades more gracefully with insertion length when the model conditions directly on local audio context rather than full-utterance speaker embeddings. confidence: high relevance: medium limitations: - Code and demos are not publicly released with the preprint, and model size is not reported, making independent evaluation and comparison difficult. - 'Evaluation is limited to LibriTTS dev-clean and dev-other; both are clean read-speech corpora, so generalisation to noisy or spontaneous speech is untested. The user study comprises only 15 utterances rated by 6 annotators — a small sample that limits statistical power. MCD is used as the primary objective metric, but it is an indirect measure of naturalness quality and is known to correlate imperfectly with perceptual preference. The comparison set is narrow: SpeechPainter is referenced in the related work but not included in the quantitative evaluation because it uses a fixed-duration formulation, leaving the question of how well RephraseTTS would perform relative to more recent neural audio inpainting approaches unanswered. Duration quality of the inserted segment relative to ground truth is not measured directly.' caveats: [] - id: '2508.18006' published_date: "2025-08-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - gan_based_speaker_transfer claims: - claim_id: adapter_based_fine_tuning_prevents_catastrophic_forgetting_in_tts_models role: supports claim: Adapter-based fine-tuning prevents catastrophic forgetting in TTS models while achieving better naturalness and speaker similarity than full fine-tuning. source: §4.3, Table 1; §4.4, Table 2 evidence: Adapter-based fine-tuning prevents catastrophic forgetting in TTS models while achieving better naturalness and speaker similarity than full fine-tuning. confidence: high relevance: high - claim_id: in_gan_based_tts_language_agnostic_speaker_representations_are_encoded role: supports claim: In GAN-based TTS, language-agnostic speaker representations are encoded primarily in the vocoder rather than the acoustic model, as evidenced by the outsized impact of vocoder-only adapters on accent nativeness. source: §4.4, Table 2 evidence: In GAN-based TTS, language-agnostic speaker representations are encoded primarily in the vocoder rather than the acoustic model, as evidenced by the outsized impact of vocoder-only adapters on accent nativeness. confidence: high relevance: low - claim_id: the_number_and_diversity_of_speakers_in_target_language_pre role: supports claim: The number and diversity of speakers in target-language pre-training data is a critical prerequisite for high-quality cross-lingual speaker adaptation; single-speaker pre-training leads to severe degradation. source: §4.4, Table 2 evidence: The number and diversity of speakers in target-language pre-training data is a critical prerequisite for high-quality cross-lingual speaker adaptation; single-speaker pre-training leads to severe degradation. confidence: high relevance: low - claim_id: phoneme_substitution_rate_derived_from_a_mispronunciation_detection_model_provides role: supports claim: Phoneme substitution rate, derived from a mispronunciation detection model, provides a reliable and reproducible objective proxy for accent nativeness in cross-lingual TTS, correlating well with human MUSHRA accent judgements. source: §3.3 evidence: Phoneme substitution rate, derived from a mispronunciation detection model, provides a reliable and reproducible objective proxy for accent nativeness in cross-lingual TTS, correlating well with human MUSHRA accent judgements. confidence: high relevance: medium limitations: - All evaluations are purely objective — no subjective listening test (MOS, MUSHRA) is reported for the proposed systems. The NORESQA-MOS (TorchAudio-SQUIM) values used as "MOS" are automatic predictions, not human ratings; the paper's decision to discard subjective evaluation limits the strength of quality claims. - The study is restricted to two languages (British English and Castilian Spanish) and a single source speaker per adaptation direction. It is unclear whether the findings generalise to phonologically more distant language pairs, accents, or speakers with more unusual voice characteristics. The backbone model is proprietary and not publicly released, limiting reproducibility. Adapter hyperparameters (bottleneck dimension 16, kernel sizes [3, 5, 3]) were chosen via early experiments but no systematic search is reported. caveats: [] - id: '2411.19770' published_date: "2025-08-28" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - VC architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_speaker_adaptation claims: - claim_id: contrastive_training_with_noise_augmented_views_enforces_noise_invariant_speaker role: supports claim: Contrastive training with noise-augmented views enforces noise-invariant speaker representations and substantially improves one-shot VC robustness at low SNR without adding inference cost. source: §III.A, Tables I-II evidence: Noro's dual-branch reference encoding module and noise-agnostic contrastive speaker loss hold SECS at 80.09 and CER at 4.66 in 0-5 dB noise, versus 77.28 and 7.26 for the baseline; weight sharing means inference architecture is unchanged. confidence: high relevance: high - claim_id: standard_one_shot_voice_conversion_systems_degrade_substantially_when_reference role: complicates claim: Standard one-shot voice conversion systems degrade substantially when reference speech contains background noise, even after data augmentation training. source: §III.A.2, Table II evidence: The diffusion-based baseline trained on LibriLight with no noise-robustness mechanism shows CMOS dropping from 3.29 to 2.09 and SMOS from 3.02 to 2.75 under 0-5 dB noisy reference conditions. confidence: high relevance: high - claim_id: voice_conversion_reference_encoders_trained_on_large_scale_speech_data role: supports claim: Voice conversion reference encoders trained on large-scale speech data develop speaker representations competitive with dedicated self-supervised speaker models. source: §III.B.2, Table III evidence: VC-SPK2VEC (the Noro baseline reference encoder repurposed as a speaker encoder, 72.4M params, trained on LibriLight 60k hr) achieves 5.32% EER on VoxCeleb1 under SUPERB, outperforming wav2vec 2.0 Base (6.02%), wav2vec 2.0 Large (5.65%), and HuBERT Large (5.98%). confidence: high relevance: medium - claim_id: speaker_noise_disentanglement_in_voice_conversion_benefits_from_training_objectives role: refines claim: Speaker-noise disentanglement in voice conversion benefits from training objectives that explicitly align clean and noisy representations of the same speaker, beyond simple noise augmentation. source: §III.A.2, Figure 2 evidence: t-SNE visualisations show that the baseline (trained with augmentation but no contrastive alignment) produces clearly separated clean/noisy representation clusters, while Noro's contrastive loss causes them to mix, correlating with the performance gap under noise. confidence: high relevance: medium limitations: - The evaluation uses reference speech from VCTK (studio-recorded English) with synthetically added noise from DEMAND; real-world recordings that mix noise with reverberation, codec compression, or far-field capture may behave differently. The test set is small (ten source-reference pairs per condition for subjective evaluation, 150 for objective), and the paper does not report statistical significance for differences between Noro and the baseline in clean conditions where they are nearly equal. The contrastive loss relies on speaker labels during training, so the approach does not extend directly to fully unsupervised training on unlabelled data. The secondary VC-SPK2VEC finding is evaluated only under one SUPERB protocol; performance on other speaker tasks (diarization, speaker counting) is not explored. caveats: [] - id: '2509.00675' published_date: "2025-08-31" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_adaptation claims: - claim_id: speaker_specific_phrasing_behaviour_is_a_substantive_source_of_variance role: complicates claim: Speaker-specific phrasing behaviour is a substantive source of variance in RP insertion that generic multi-speaker models fail to capture, and modeling it explicitly improves both objective and subjective phrasing quality. source: §5.1.2, Table 3, Table 5 evidence: Speaker-specific phrasing behaviour is a substantive source of variance in RP insertion that generic multi-speaker models fail to capture, and modeling it explicitly improves both objective and subjective phrasing quality. confidence: high relevance: medium - claim_id: phoneme_level_language_models_outperform_subword_level_models_on_phrase role: supports claim: Phoneme-level language models outperform subword-level models on phrase break prediction, even at smaller model sizes, because phoneme representations carry acoustic information more directly relevant to pause insertion than subword tokens. source: §5.1.3, Table 4 evidence: Phoneme-level language models outperform subword-level models on phrase break prediction, even at smaller model sizes, because phoneme representations carry acoustic information more directly relevant to pause insertion than subword tokens. confidence: high relevance: low - claim_id: scaling_subword_plms_from_base_to_large_yields_diminishing_returns role: supports claim: Scaling subword PLMs from BASE to LARGE yields diminishing returns for phrasing tasks, suggesting a representational ceiling specific to this task modality. source: §5.1.3, Table 4 evidence: Scaling subword PLMs from BASE to LARGE yields diminishing returns for phrasing tasks, suggesting a representational ceiling specific to this task modality. confidence: high relevance: medium - claim_id: pre_trained_speaker_verification_embeddings_capture_prosodic_and_fluency_related role: supports claim: Pre-trained speaker verification embeddings capture prosodic and fluency-related characteristics that transfer to phrasing models via few-shot adaptation without fine-tuning, enabling reasonable generalization to unseen speakers. source: §5.2.2, Table 7 evidence: Pre-trained speaker verification embeddings capture prosodic and fluency-related characteristics that transfer to phrasing models via few-shot adaptation without fine-tuning, enabling reasonable generalization to unseen speakers. confidence: high relevance: high - claim_id: f0_5_score_and_naturalness_mos_can_diverge_for_phrasing role: supports claim: F0.5 score and naturalness MOS can diverge for phrasing models using different PLMs, indicating that objective phrasing accuracy does not fully predict perceived speech naturalness. source: §5.2.3, Table 8 evidence: F0.5 score and naturalness MOS can diverge for phrasing models using different PLMs, indicating that objective phrasing accuracy does not fully predict perceived speech naturalness. confidence: high relevance: medium limitations: - Training data is exclusively from LibriTTS-R audiobook readings. The resulting phrasing models are likely miscalibrated for spontaneous speech, conversational TTS, or out-of-domain styles; generalization is explicitly flagged by the authors as an open problem. - 'Additional limitations: the paper cannot disentangle the contributions of phoneme vs. subword information within phoneme-level PLMs (since their pre-training also includes grapheme-level objectives), leaving the mechanism of improvement partially unclear. The embedding adapter for unseen speakers assumes that the mapping from PSVM embeddings to trained embeddings is approximately injective and learnable with a small network — this assumption may not hold for speakers whose acoustic characteristics fall outside the training distribution.' caveats: [] - id: '2509.04667' published_date: "2025-09-04" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - VC architecture: - GAN - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - gan_based_speaker_transfer - hybrid_semantic_acoustic_adaptation claims: - claim_id: limited_lookahead_in_causal_speech_encoders_substantially_improves_linguistic_content role: supports claim: Limited lookahead in causal speech encoders substantially improves linguistic content preservation with minimal latency penalty compared to purely causal encoders. source: §V.A, §V.B, Tables I–II evidence: Wave+CL accuracy improves from 53.16% at zero lookahead to 78.99% at 140ms, while end-to-end latency increases from 84.3ms to 203ms; extending to 280ms adds only 0.7pp accuracy with 120ms additional delay. confidence: high relevance: low - claim_id: better_content_encoding_in_streaming_anonymization_introduces_a_fundamental_tension role: complicates claim: 'Better content encoding in streaming anonymization introduces a fundamental tension: improved linguistic clarity reduces speaker anonymization strength under adversarial threat models.' source: §V.E, Table III evidence: Adding the contextual layer drops lazy-informed EER from 36.61% to 20.35% at zero lookahead, meaning the cleaner representations are more discriminative for speaker recognition attacks, directly trading anonymization quality for intelligibility. confidence: high relevance: low - claim_id: token_quantization_via_k_means_clustering_achieves_near_chance_speaker role: supports claim: Token quantization via k-means clustering achieves near-chance speaker verification performance in streaming anonymization by removing fine-grained speaker cues from content representations. source: §V.C, §V.D, §V.E, Tables IV–V evidence: Applying a 256-centroid k-means bottleneck raises lazy-informed EER from ~12% to ~47% (Wave+CL, 140ms lookahead), at a cost of WER rising from 2.09% to 9.52% and MOS falling from 3.79 to 3.22. confidence: high relevance: high - claim_id: streaming_voice_anonymization_systems_can_approach_offline_pipeline_anonymization_performance role: supports claim: Streaming voice anonymization systems can approach offline-pipeline anonymization performance when evaluated under the lazy-informed threat scenario, while retaining real-time latency. source: §V.F, Table VI evidence: DarkStream achieves 22.68% semi-informed EER in streaming mode (140ms lookahead), matching VoicePrivacy 2024 baselines B3 (26.28%) and B5a (22.09%) that require full-utterance processing. confidence: high relevance: low - claim_id: direct_waveform_synthesis_in_streaming_voice_conversion_maintains_acceptable_naturalness role: complicates claim: Direct waveform synthesis in streaming voice conversion maintains acceptable naturalness without mel-spectrogram intermediate representations, but k-means quantization introduced for privacy causes perceivable quality degradation beyond what objective metrics capture. source: §V.D, Table V evidence: MOS drops from 3.79 (Wave+CL) to 3.22 (Wave+CL+KMeans) with quantization; WER degrades only modestly, indicating that intelligibility metrics underestimate the perceptual impact of quantization artifacts. confidence: high relevance: low limitations: - DarkStream does not explicitly disentangle static speaker traits (accent, age, sex) from dynamic attributes (emotion, speaking style), leaving indirect identity cues potentially intact. The semi-informed EER of 22.68% remains well above chance, indicating meaningful residual linkability for well-resourced adversaries. - 'The privacy/quality trade-off exposed by the quantization ablation is steep: each MOS point recovered (by disabling k-means) costs roughly 30pp EER under the lazy-informed scenario. Systems requiring both high quality and robust anonymization against semi-informed attackers have no current solution in this architecture. Comparison of perceptual quality against the offline VoicePrivacy baselines is not reported, so whether DarkStream''s naturalness advantage over batch-processing pipelines is real remains an open question. Evaluation is limited to English (LibriTTS), and generalization to accented or code-switched speech is untested.' caveats: [] - id: '2509.04702' published_date: "2025-09-04" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - SCA architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: constructing_large_scale_conversational_speech_corpora_with_fine_grained_speaker role: supports claim: Constructing large-scale conversational speech corpora with fine-grained speaker diarization and word-level timestamps is feasible from publicly available broadcast sources when specialized alignment pipelines are used in place of general-purpose ASR. source: §2.4, §3 evidence: OleSpeech-IV Tier IV assembles 5,000+ hours from English podcasts, talk shows, and teleconferences with human-sourced speaker labels and Olign-produced word-level timestamps and confidence scores. confidence: high relevance: medium - claim_id: end_to_end_attention_and_ctc_based_speech_aligners_introduce role: complicates claim: End-to-end attention- and CTC-based speech aligners introduce systematic boundary errors and overconfident posterior scores that degrade the reliability of automatically annotated training data. source: §3.1.1.2, §3.1.2 evidence: Whisper-based E2E aligners produce boundary smearing (segment endings placed at the onset of the next word) and assign high posterior probabilities (>0.8) to misrecognized words; Olign assigns zero duration to hallucinated insertions and provides calibrated per-word confidence scores. confidence: high relevance: medium - claim_id: speaker_overlap_in_naturalistic_multi_party_speech_remains_an_open role: complicates claim: Speaker overlap in naturalistic multi-party speech remains an open challenge for current alignment pipelines, with quantitative accuracy benchmarks largely unreported. source: §3.1.1.3 evidence: Olign supports per-speaker word-level timestamps in overlapping segments, but the paper explicitly notes that benchmarking for this capability is ongoing and no numerical accuracy figures are available. confidence: high relevance: medium - claim_id: calibrated_word_level_confidence_scores_in_speech_annotation_pipelines_can role: supports claim: Calibrated word-level confidence scores in speech annotation pipelines can reduce human review effort by directing annotators to mid-confidence ambiguous segments rather than requiring complete audio re-examination. source: §3.1.2 evidence: The Olign pipeline provides per-word confidence scores that highlight words likely to contain errors, enabling annotators to focus effort without reviewing every word in a recording. confidence: high relevance: medium limitations: - Olign, the central pipeline component, is proprietary and not open-sourced, making the data curation methodology unreproducible. No quantitative accuracy benchmarks for Olign are provided; the paper explicitly states these are work in progress. - The open-source release covers only 100 of the 5,000+ hours in the Tier IV collection, limiting reproducibility of experiments at scale. The paper does not evaluate any downstream speech generation or understanding model trained on OleSpeech-IV, so the practical utility of the dataset for TTS or spoken conversational agent training remains undemonstrated. The dataset currently supports full Olign alignment only for English; Spanish, Japanese, and Mandarin support is listed as under development. caveats: [] - id: '2509.07376' published_date: "2025-09-09" entry_date: '2026-07-29' year: 2025 venue: EMNLP task: - TTS architecture: - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - vae_latent_speaker_adaptation claims: - claim_id: hierarchical_locally_aware_facial_feature_extraction_substantially_improves_voice_face role: supports claim: Hierarchical, locally-aware facial feature extraction substantially improves voice-face identity congruence compared to global visual representations in face-to-speech synthesis. source: §5.3, Table 5 evidence: Progressive Facial Granularity Aggregation (FGA) — hierarchical patch decomposition with progressive convolutional aggregation — yields SECS of 79.96 (seen) vs. 66.25 when replaced with a single vanilla transformer over the full image, a gap of 13.71 points. confidence: high relevance: medium - claim_id: cross_modal_multi_task_supervision_over_demographic_attributes_improves_face role: supports claim: Cross-modal multi-task supervision over demographic attributes improves face-voice alignment in end-to-end synthesis systems. source: §5.2, Table 4 evidence: Bilateral attribute enhancement (predicting gender and ethnicity from both the facial embedding and synthesized audio) produces the highest SECS configuration at 79.96 seen / 71.39 unseen; applying supervision to only one modality yields lower scores (77.76 / 70.27 for visual-only, 75.84 / 69.52 for audio-only). confidence: high relevance: medium - claim_id: multi_view_face_augmentation_pairing_multiple_facial_frames_per_speaker role: supports claim: Multi-view face augmentation — pairing multiple facial frames per speaker across varying poses and lighting — is critical for robust speaker identity generalization in face-to-speech systems. source: §5.3, Table 5 evidence: Removing the multi-view augmentation strategy causes the largest single-component SECS drop in ablation studies (from 79.96 to 64.43, a decrease of 15.53 on seen speakers and 9.35 on unseen), exceeding the contribution of any individual architectural module. confidence: high relevance: high - claim_id: end_to_end_face_to_voice_synthesis_without_audio_space role: complicates claim: End-to-end face-to-voice synthesis without audio-space alignment targets still requires explicit cross-modal semantic supervision to maintain speaker identity coherence. source: §5.2, §5.3, Table 4, Table 5 evidence: Training the progressive facial encoder end-to-end without audio-alignment targets is the paper's stated improvement over prior multi-stage pipelines, but removing bilateral attribute supervision (even partially) consistently degrades SECS — visual attribute enhancement removal alone costs 5.23 SECS points. confidence: high relevance: high - claim_id: face_to_voice_synthesis_systems_trained_on_demographically_skewed_corpora role: complicates claim: Face-to-voice synthesis systems trained on demographically skewed corpora may produce uneven speaker similarity across underrepresented groups. source: §Appendix B, Table 6, Table 7 evidence: The LRS3-TED training set is 74.66% male and 72.23% Caucasian; the proposed model achieves SECS of 75.97 for male and 75.03 for female speakers, while Middle Eastern (69.25) and African American (71.65) speakers show measurably lower scores despite the bilateral attribute supervision. confidence: high relevance: high limitations: - 'The framework focuses on overall vocal timbre and does not capture emotional prosody: facial expressions that correlate with affective speech are excluded from the current conditioning design. Age-conditioned synthesis is also limited by dataset skew; the LRS3-TED distribution is not representative of older speakers, leaving age alignment as partial.' - Demographic coverage in the training data is heavily imbalanced, with Middle Eastern and African American speakers showing noticeably lower SECS scores despite the bilateral supervision. Whether this gap can be closed by dataset rebalancing or additional attribute supervision is an open question. - The evaluation uses 40 speakers and 21 annotators, which, while reasonable for an FTV study, limits statistical power for cross-demographic comparisons. The comparison baselines do not include the most recent large-scale zero-shot TTS systems, so absolute quality relative to the broader field is unclear. caveats: [] - id: '2506.04077' published_date: "2025-09-11" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: speaker_conditioned_tts_synthesis_when_conditioned_on_reference_embeddings_from role: supports claim: Speaker-conditioned TTS synthesis, when conditioned on reference embeddings from real non-native learners, can produce augmentation data that preserves proficiency-level-appropriate disfluencies and prosodic variability. source: §2.2, Figure 1 evidence: XTTSv2 conditioned on 512-dimensional speaker embeddings from real learner recordings produces speech retaining speaking rate, rhythm, and characteristic disfluencies; this contrasts with standard single-speaker TTS, which generates uniformly fluent output incompatible with non-native learner characteristics. confidence: high relevance: medium - claim_id: synthetic_speech_from_standard_tts_pipelines_cannot_substitute_for_authentic role: complicates claim: Synthetic speech from standard TTS pipelines cannot substitute for authentic learner recordings in isolation, even when speaker conditioning is applied to mimic non-native characteristics. source: §3.4, Table 2 evidence: Training exclusively on synthesised data achieves only 42.22% seen-test accuracy versus 73.33% for a real-data-only baseline, demonstrating that the acoustic domain gap between synthetic and authentic speech is not fully bridged by speaker-conditioned voice cloning alone. confidence: high relevance: medium - claim_id: mixed_real_plus_synthetic_training_with_adaptive_importance_reweighting_outperforms role: supports claim: Mixed real-plus-synthetic training with adaptive importance reweighting outperforms training on real data alone for low-resource speech understanding tasks. source: §3.3, Table 1 evidence: Adding synthesised augmentation with dynamic real-sample upweighting raises seen-test overall accuracy from 73.33% to 76.67% and pass-fail accuracy from 77.78% to 81.11% on the LTTC GEPT dataset, outperforming both single-modality and multimodal baselines trained on real data only. confidence: high relevance: medium - claim_id: gains_from_generative_data_augmentation_in_speech_assessment_diminish_substantially role: complicates claim: Gains from generative data augmentation in speech assessment diminish substantially on out-of-distribution prompt types, limiting the generalisation benefit of synthetic speech augmentation. source: §3.3, Table 1 evidence: Overall accuracy on unseen-prompt test data improves by only 0.67 percentage points (63.33% to 64.00%) compared to 3.34 points on seen prompts, indicating that augmentation-driven gains do not transfer robustly across different prompt domains. confidence: high relevance: medium limitations: - The evaluation is restricted to a single proprietary dataset (LTTC GEPT Intermediate-Advanced) in English only, with 1,200 samples total. The extent to which the augmentation approach generalises across languages, accent groups, or assessment rubrics is not tested. - Accent and hesitation mismatches between synthesised and real non-native speech persist despite speaker conditioning, as acknowledged in the paper. The skewed mid-range score distribution in the dataset also impairs discrimination at extreme proficiency levels. The paper does not report any measure of the TTS output's perceptual quality (e.g., MOS or speaker similarity scores), making it difficult to assess whether the voice cloning pipeline actually produces convincing non-native-sounding speech or whether the scoring model is robust to low-fidelity synthesis. caveats: [] - id: '2509.09631' published_date: "2025-09-11" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - flow_matching_speaker_adaptation - transformer_encoder_decoder_adaptation claims: - claim_id: discrete_flow_matching_defined_directly_over_factorized_speech_token_subspaces role: supports claim: Discrete flow matching defined directly over factorized speech token subspaces achieves competitive naturalness and superior prosody reconstruction compared to continuous-space flow and diffusion TTS baselines trained on comparable data. source: §4.2, Table 1, Table 2 evidence: Discrete flow matching defined directly over factorized speech token subspaces achieves competitive naturalness and superior prosody reconstruction compared to continuous-space flow and diffusion TTS baselines trained on comparable data. confidence: high relevance: low - claim_id: factorized_multi_head_velocity_prediction_separate_prediction_heads_for_distinct role: supports claim: Factorized multi-head velocity prediction — separate prediction heads for distinct speech attribute subspaces within a single discrete flow model — improves prosody and acoustic fidelity over a single-head alternative. source: §4.3, Table 4 evidence: Factorized multi-head velocity prediction — separate prediction heads for distinct speech attribute subspaces within a single discrete flow model — improves prosody and acoustic fidelity over a single-head alternative. confidence: high relevance: low - claim_id: global_speaker_embedding_conditioning_via_adaln_is_insufficient_for_reliable role: supports claim: Global speaker embedding conditioning via AdaLN is insufficient for reliable zero-shot speaker similarity in discrete token-space TTS; perceptual similarity judgements diverge from embedding-based automatic metrics (SIM-O) in this setting. source: §4.2, Table 1, Table 2 evidence: Global speaker embedding conditioning via AdaLN is insufficient for reliable zero-shot speaker similarity in discrete token-space TTS; perceptual similarity judgements diverge from embedding-based automatic metrics (SIM-O) in this setting. confidence: high relevance: high - claim_id: non_autoregressive_discrete_flow_models_with_compact_dit_backbones_can role: supports claim: Non-autoregressive discrete flow models with compact DiT backbones can match the inference latency of single-step flow matching systems while operating at higher NFE, without compromising data efficiency. source: §4.2, Table 3 evidence: Non-autoregressive discrete flow models with compact DiT backbones can match the inference latency of single-step flow matching systems while operating at higher NFE, without compromising data efficiency. confidence: high relevance: low limitations: - 'The model''s weakest dimension is automatic speaker similarity (SIM-O): simple global AdaLN speaker conditioning is insufficient for reliable timbre reproduction, especially in zero-shot settings. The authors acknowledge this and suggest cross-attention over local timbre embeddings as future work.' - Because FACodec separates speaker identity via a timbre embedding external to the discrete token streams, integrating speaker conditioning into a DFM framework is non-trivial; alternative codecs such as EnCodec implicitly embed speaker information within VQ codebooks, potentially enabling stronger speaker adaptation in future discrete-space models. - The evaluation uses a single language (English) and a moderately sized training set (470h), leaving open questions about multilingual scalability and behavior at larger data scales. The gap between SIM-O ranking (third among five) and perceptual similarity MOS ranking (best) also points to known deficiencies in embedding-based automatic speaker similarity metrics, and a better evaluation framework for zero-shot speaker identity remains an open research direction. caveats: [] - id: '2509.14270' published_date: "2025-09-15" entry_date: '2026-07-29' year: 2025 venue: ACL task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: structured_keyphrase_infusion_via_multi_step_llm_prompting_substantially_reduces role: supports claim: Structured keyphrase infusion via multi-step LLM prompting substantially reduces semantic repetition in generated TTS training corpora compared to single-prompt domain generation. source: §3.1, §4.1 evidence: SpeechWeave's keyphrase pipeline achieves 45.8%/44.4% lower grouped mean pairwise similarity in English/Spanish and 17.4%/9.7% higher diphone coverage relative to direct domain prompting with Mistral-7b-Instruct, even at high temperature (Table 3). confidence: high relevance: medium - claim_id: generating_semiotic_class_entities_with_embedded_normalization_rules_at_source role: supports claim: Generating semiotic class entities with embedded normalization rules at source achieves higher normalization accuracy for TTS training text than applying a post-hoc normalizer to LLM-generated output. source: §3.2, §4.2.1 evidence: At-source entity normalization reaches 0.97 accuracy in English and 0.94 in Spanish, compared to 0.67 and 0.54 for NeMo's WFST text normalizer on the same 500-sentence evaluation sets per language (Table 4). confidence: high relevance: medium - claim_id: fine_tuning_a_pre_trained_tts_model_on_synthetic_pipeline role: supports claim: Fine-tuning a pre-trained TTS model on synthetic pipeline-generated data with controlled diversity and normalization can meaningfully improve downstream speech intelligibility. source: §4.2.3 evidence: StyleTTS 2 fine-tuned on SpeechWeave data shows 40% relative WER reduction in English (15.37% to 9.36%) and 27% in Spanish (85.05% to 48.44%) on LibriSpeech test sets; the Spanish result also demonstrates cross-lingual adaptation from a monolingual English checkpoint (Table 6). confidence: high relevance: medium - claim_id: automated_speech_quality_metrics_from_predictive_models_should_not_be role: complicates claim: Automated speech quality metrics from predictive models should not be treated as equivalent to human MOS scores when evaluating synthetic TTS training data. source: §4.2.2, Table 5 evidence: SpeechWeave reports MOS of 4.95 (English) and 4.87 (Spanish) estimated by the NISQA neural quality predictor rather than human raters; the paper notes this explicitly. Automated MOS predictors are known to diverge from human judgments, particularly for TTS systems that differ from the predictor's training distribution. confidence: high relevance: medium limitations: - The evaluation covers only English and Spanish, and improvements in normalization accuracy and diversity may not hold for morphologically rich or lower-resource languages, which the paper acknowledges as future work. - The entity sampler supports nine semiotic classes; any entity type not included in the recipes will not be normalized at source, and the post-processor may handle them incorrectly. The extent of this gap depends on how often such entities appear in target domains. - The pipeline depends on Mistral-7b-Instruct for text generation and OpenVoice V2 for voice cloning, and performance is expected to vary with model substitution. No ablation is provided over LLM or TTS model choice, so the contribution of each component to the overall result is not fully isolated. - The downstream evaluation uses WER from an ASR model to assess TTS quality, which measures intelligibility rather than naturalness or speaker similarity. Prosody, expressiveness, and voice consistency are not evaluated beyond the automated MOS and SNR scores. caveats: [] - id: '2412.16846' published_date: "2025-09-17" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - VAE relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - vae_latent_speaker_adaptation claims: - claim_id: distributional_training_objectives_for_continuous_ar_speech_modeling_achieve_higher role: supports claim: Distributional training objectives for continuous AR speech modeling achieve higher intelligibility than regression-based alternatives. source: §TTS Evaluation, Table 2; §Ablation Study, Table 5 evidence: KALL-E with KL divergence loss achieves WER 1.94 / CER 0.96 on Seed-TTS test sets, below all discrete-token and regression-based baselines; ablation replacing Flow-VAE with Stable Audio VAE (near-zero KL weight, approaching a plain autoencoder) collapses CER from 2.79 to 40.09 at the same latent dimension. confidence: high relevance: medium - claim_id: low_frame_rate_continuous_representations_reduce_autoregressive_tts_inference_compute role: supports claim: Low frame-rate continuous representations reduce autoregressive TTS inference compute by over an order of magnitude without sacrificing synthesis quality. source: §TTS Evaluation, Table 3, Table 4 evidence: KALL-E at 12.5 Hz requires 7,947 GFLOPs to synthesize 10 seconds vs. 122,170 for Llasa-1B at 50 Hz, while achieving higher MOS (4.17 vs. 3.92) and lower WER (1.94 vs. 3.6) on the same test set. confidence: high relevance: medium - claim_id: objective_speaker_similarity_metrics_for_zero_shot_tts_are_unreliable role: complicates claim: Objective speaker similarity metrics for zero-shot TTS are unreliable for cross-system comparisons when decoder architectures differ in their use of reference audio. source: §TTS Evaluation, Table 2, Table 3 evidence: Discrete-token systems (Seed-TTS SIM 0.796, FireRedTTS SIM 0.635) score differently on objective SPK-SIM than KALL-E (SIM 0.646/0.568), but KALL-E receives higher listener naturalness ratings; the authors attribute the gap to those systems conditioning the waveform decoder on the reference utterance at decode time, which inflates the metric independent of perceived speaker fidelity. confidence: high relevance: high - claim_id: test_time_adaptation_from_a_single_reference_utterance_improves_speaker role: supports claim: Test-time adaptation from a single reference utterance improves speaker similarity in continuous-representation AR TTS without requiring full model retraining. source: §Test Time Training; §TTS Evaluation, Table 2 evidence: KALL-E (TTT) improves SPK-SIM from 0.568 to 0.611 on test-en using N=200 latent sequences sampled from the reference utterance's Flow-VAE distribution, with WER remaining stable at 1.90. confidence: high relevance: high - claim_id: increasing_vae_kl_regularization_weight_trades_reconstruction_fidelity_for_a role: complicates claim: Increasing VAE KL regularization weight trades reconstruction fidelity for a latent space structure that is more suitable for downstream generative modeling. source: §VAE Evaluation, Table 1; §Ablation Study, Table 5 evidence: Flow-VAE uses KL weight 32 and scores PESQ-WB 3.26 at 512-dim/12.5 Hz, below Stable Audio VAE (3.11) at the same frame rate with near-zero KL weight; however, Stable Audio VAE's latent space causes CER to collapse when used as the AR LM encoder, demonstrating that reconstruction quality and generation compatibility impose conflicting constraints on VAE training. confidence: high relevance: medium limitations: - 'Objective speaker similarity remains below discrete-token systems that condition their decoders on the reference audio, suggesting the Flow-VAE''s information bottleneck trades some speaker detail for a more LM-friendly latent space. The TTT procedure assumes the transcript of the reference utterance is available, which may not hold in all deployment settings. Overfitting risk in TTT is real: CER rises after N=200 in ablation, limiting the effective adaptation set size. Evaluations are conducted solely on the Seed-TTS test sets; generalization to other benchmarks, out-of-distribution speakers, or noisy acoustic conditions is not assessed. Training data composition differs from the most directly comparable system (Llasa-1B), making it difficult to fully isolate architecture from data quality as the source of WER gains.' caveats: [] - id: '2509.15253' published_date: "2025-09-18" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: [] claims: - claim_id: end_to_end_speech_generation_pipelines_that_chain_multiple_vision role: complicates claim: End-to-end speech generation pipelines that chain multiple vision and language components suffer from compounding errors that substantially limit overall system accuracy. source: §3.5 evidence: Speaker attribution accuracy in Setting C is 64.8% and emotion classification micro-F1 is 42.9%, but joint speaker-and-emotion accuracy is only 20.4%, illustrating how independent module errors multiply at the pipeline level. confidence: high relevance: low - claim_id: llm_based_dialogue_attribution_for_tts_struggles_in_spatially_ambiguous role: complicates claim: LLM-based dialogue attribution for TTS struggles in spatially ambiguous layouts where dialogue text is not co-located with the speaking character. source: §3.3, Table 1 evidence: Setting C achieves 79.2% speaker accuracy on easy cases but only 20.5% on hard cases (spatially separated dialogue), showing that the LLM falls back to proximity heuristics that fail when layout breaks co-location assumptions. confidence: high relevance: low - claim_id: imbalanced_emotion_datasets_make_fine_grained_emotion_conditioning_for_tts role: complicates claim: Imbalanced emotion datasets make fine-grained emotion conditioning for TTS unreliable, particularly for the neutral category which dominates real-world speech. source: §2.4, §3.4, Table 3 evidence: 'KangaiSet''s extreme class imbalance (neutral: 35%, disgust: 0.47%) required binary reformulation of 7-class emotion classification; the neutral class still achieves only 34.6% recall in 5-way LLM-based emotion prediction.' confidence: high relevance: low - claim_id: standard_accuracy_metrics_for_emotion_classification_may_underestimate_the_perceptual role: complicates claim: Standard accuracy metrics for emotion classification may underestimate the perceptual quality of emotion-conditioned speech when ground-truth emotion labels are inherently ambiguous. source: §3.4 evidence: Manual inspection of Setting C mispredictions reveals that many misclassified emotions still produce perceptually appropriate voice outputs, and the authors attribute this to inherent visual and contextual ambiguity in comic-style emotional expression. confidence: high relevance: low limitations: - No perceptual evaluation of synthesised speech is conducted. The pipeline's audio output quality, naturalness, and emotional appropriateness are not assessed with listening tests or any objective metric (MOS, UTMOS, WER). The speech synthesis module cannot be independently compared to other TTS systems. - The TTS component's architecture and training setup are not described, making the synthesis stage unreproducible. The pipeline is evaluated exclusively on Japanese manga (Manga109/KangaiSet), leaving generalisation to Western comics, webtoons, or other visual styles untested. Character identification at 62.9% accuracy with only 40 reference images per character introduces substantial upstream error that propagates through speaker attribution. Emotion classification accuracy remains low even with LLM-based reasoning, and the 20.4% joint accuracy suggests significant headroom before deployment quality is reached. The evaluation covers only the first 10 pages per comic, which may not capture the full distribution of visual complexity encountered in complete volumes. caveats: [] - id: '2509.15462' published_date: "2025-09-18" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - codec - VC architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation - flow_matching_speaker_adaptation claims: - claim_id: factorising_speech_into_semantic_components_content_style_timbre_enables_task role: supports claim: Factorising speech into semantic components (content, style, timbre) enables task-relevant transmission at lower bitrates than general-purpose codecs that encode all features uniformly. source: §3.3, Table 1 evidence: Content-style tokens at 650 bps achieve WER 0.15 and sentiment accuracy 59%, within 1% of EnCodec at 1.5 kbps (WER 0.20, accuracy 60%), while using approximately half the bitrate. confidence: high relevance: medium - claim_id: reusing_a_compressed_speaker_reference_transmitted_once_per_speaker_can role: supports claim: Reusing a compressed speaker reference transmitted once per speaker can maintain speaker similarity at lower average bitrate than encoding full audio continuously. source: §3.4, Table 1 evidence: Vevo with Zonos speaker embedding (1 sec timbre) achieves SpkrSim 0.54, matching EnCodec at 1.5 kbps (0.54), while long-run average bitrate approaches 650 bps as registered speakers accumulate. confidence: high relevance: high - claim_id: generative_reconstruction_from_semantic_tokens_improves_perceptual_quality_scores_but role: complicates claim: Generative reconstruction from semantic tokens improves perceptual quality scores but degrades low-level signal fidelity metrics relative to waveform-level codecs. source: §3.4, Table 1 evidence: Vevo configurations outperform EnCodec on UTMOS and NISQA across all bitrates, but score lower on PESQ and STOI because the flow-matching decoder was not trained to preserve signal-level characteristics. confidence: high relevance: medium - claim_id: semantic_codec_approaches_face_a_latency_and_error_propagation_trade role: complicates claim: Semantic codec approaches face a latency and error-propagation trade-off that general-purpose codecs avoid. source: §2.2, §4 evidence: Timbre transmission introduces per-speaker latency L = d_sample + d_transmit; errors in that one-time transmission cause permanent voice reconstruction inaccuracies until a correction is transmitted. confidence: high relevance: low - claim_id: neural_semantic_token_representations_are_moderately_robust_to_channel_bit role: supports claim: Neural semantic token representations are moderately robust to channel bit errors, maintaining acceptable downstream task performance at realistic noise levels. source: §3.5, Table 2 evidence: At 0.1% bit-flip rate the system shows no measurable degradation; at 1% BER sentiment classification (0.63) and speaker verification (0.77) still exceed Opus at 5 kbps. Performance collapses only above 10% BER. confidence: high relevance: medium limitations: - 'Timbre transmission errors are permanent: a corrupted one-time speaker embedding results in incorrect voice reconstruction for all subsequent utterances from that speaker until a retransmission is triggered. The paper does not propose an error-recovery mechanism.' - The system does not handle overlapping speakers; the paper acknowledges this would require sender-side speaker separation at additional bitrate cost. Evaluation is restricted to English and does not address non-English languages, which are explicitly noted as out-of-distribution for several downstream models. The test set of 1,000 clips from VoxCeleb1 is relatively small for claims about general-purpose voice communication performance. The real-time latency introduced by speaker embedding generation and transmission is uncharacterised across different network conditions. caveats: [] - id: '2509.15629' published_date: "2025-09-19" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - VC - singing - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: dynamic_temporal_features_in_singing_vibrato_glissando_breathy_voice_are role: supports claim: Dynamic temporal features in singing (vibrato, glissando, breathy voice) are substantially harder to convert than static spectral features, and current systems cannot model them reliably. source: §4.1 evidence: Style accuracy scores for vibrato (43.9%), glissando (42.6%), and breathy (37.3%) were markedly lower than those for falsetto (48.8%), mixed voice (48.3%), and pharyngeal (44.8%); no system approached human-level performance on the dynamic styles. confidence: high relevance: medium - claim_id: jointly_converting_singer_identity_and_singing_style_simultaneously_degrades_naturalness role: supports claim: Jointly converting singer identity and singing style simultaneously degrades naturalness compared to identity-only conversion. source: §4.1 evidence: Top SVCC 2025 systems achieved approximately 3.7 MOS naturalness, below the approximately 3.9 MOS ground truth, whereas prior SVCC 2023 top systems reached human-level naturalness on the simpler identity-only task. confidence: high relevance: medium - claim_id: autoregressive_language_model_architectures_with_flow_matching_decoders_are_competitive role: supports claim: Autoregressive language model architectures with flow-matching decoders are competitive with specialized diffusion systems for singing voice conversion. source: §4.1 evidence: Systems based on Vevo1.5 (ARLM + flow-matching) achieved the highest singing style similarity scores in the challenge, outperforming pure diffusion-based systems using SeedVC and Serenade. confidence: high relevance: low - claim_id: neural_mos_prediction_models_show_stronger_correlation_with_subjective_naturalness role: supports claim: Neural MOS prediction models show stronger correlation with subjective naturalness scores for singing voice than traditional signal-based metrics. source: §4.2 evidence: SHEETSSQA and SingMOS achieved Spearman rank correlation coefficients above 0.6 with subjective naturalness scores; speaker and singer embeddings were the strongest predictors of identity and style similarity (SRCC above 0.75). confidence: high relevance: medium - claim_id: singer_identity_similarity_evaluation_in_challenges_is_confounded_by_evaluation role: complicates claim: 'Singer identity similarity evaluation in challenges is confounded by evaluation design: even ground truth recordings achieve only approximately 70% similarity scores with a single-sample reference.' source: §4.1 evidence: Ground truth samples reached only approximately 70% singer identity similarity in SVCC 2025 listening tests; the challenge responded by presenting three reference samples per singer (instead of one in prior challenges) to reduce this floor effect. confidence: high relevance: low limitations: - Objective metrics show weak-to-moderate correlation with subjective scores across most dimensions, with only neural MOS predictors and embedding-based similarity reaching SRCC above 0.6. Objective-only comparisons for singing style conversion should be treated with caution until better proxy metrics are established. - The challenge attracted only 7 participating teams (26 systems including ablations), limiting the diversity of architectural approaches evaluated. The dataset is derived exclusively from GTSinger, restricting style diversity to the 7 included categories and to the singers chosen for Task 1 and Task 2. The zero-shot task evaluation is further complicated by the inherent ambiguity of singer identity evaluation (GT achieving only approximately 70% similarity), which may inflate the apparent success of some zero-shot systems. Open questions include how to model the time-varying noise and F0 dynamics that characterize breathy, vibrato, and glissando styles, and whether objective metrics can be developed with sufficiently high correlation to replace or supplement crowd-sourced listening tests at scale. caveats: [] - id: '2509.15969' published_date: "2025-09-19" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_prompt_conditioned_adaptation claims: - claim_id: fully_autoregressive_streaming_tts_can_achieve_first_packet_latencies_under role: supports claim: Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation. source: §4, Table 1, Table 3 evidence: Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation. confidence: high relevance: low - claim_id: training_data_scale_is_the_primary_driver_of_speaker_similarity role: supports claim: Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive. source: §4, Table 1, Table 2 evidence: Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive. confidence: high relevance: high - claim_id: borrowing_frozen_depth_transformer_weights_from_a_large_scale_pretrained role: supports claim: Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training. source: §3, Table 4 evidence: Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training. confidence: high relevance: medium - claim_id: full_stream_input_processing_introduces_only_marginal_quality_degradation_relative role: supports claim: Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS. source: §4, Table 1 evidence: Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS. confidence: high relevance: low - claim_id: non_autoregressive_flow_matching_decoders_used_in_chunk_based_streaming role: supports claim: Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents. source: §4, Table 3 evidence: Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents. confidence: high relevance: low limitations: - Speaker similarity remains lower than large-scale systems trained on hundreds of thousands of hours, particularly in full-stream mode (SPK-SIM 0.458 on LibriSpeech test-clean vs. 0.587 for CosyVoice2 trained on 167k hours). The system is English-only; multilingual extension is not addressed. Prosody and speaking rate are not explicitly controllable at inference. The use of a frozen CSM depth transformer introduces a dependency on an external large-scale model. Long-form streaming beyond 10-15 second utterances is identified as future work. Performance in adverse or spontaneous speech conditions (outside the training domain) is untested. caveats: [] - id: '2509.16010' published_date: "2025-09-19" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS - VC architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation claims: - claim_id: disentangling_speaker_identity_from_expressive_style_into_separate_adapter_modules role: supports claim: Disentangling speaker identity from expressive style into separate adapter modules improves both speaker similarity and style expressivity in personalised voice cloning. source: §3.3, Table 2 evidence: Ablations on GPT-SOVITS-V4 show removing the private ID-LoRA drops SS from 0.645 to 0.507, while removing the Style-LoRA drops SE from 0.704 to 0.588; a single unified adapter cannot optimise both objectives without mutual degradation. confidence: high relevance: high - claim_id: collaborative_style_learning_across_clients_in_a_federated_setting_can role: supports claim: Collaborative style learning across clients in a federated setting can overcome per-client data scarcity and produce higher speaker similarity than purely local fine-tuning. source: §3.2, Table 2 evidence: Fed-PISA exceeds Local FT (LoRA) in both SS (0.645 vs 0.529) and nMOS (4.08 vs 3.36), demonstrating that sharing expressive style information across clients compensates for limited per-client data. confidence: high relevance: high - claim_id: personalized_aggregation_weighted_by_stylistic_similarity_between_clients_outperforms_uniform role: supports claim: Personalized aggregation weighted by stylistic similarity between clients outperforms uniform averaging for federated style learning. source: §3.3, Table 2 evidence: Under identical GPT-SOVITS-V4 backbone and LoRA configuration, attention-based personalized aggregation achieves SE 0.704 and SS 0.645 versus FedAvg's SE 0.476 and SS 0.523, confirming that style-averaged updates homogenise client models and harm expressivity. confidence: high relevance: medium - claim_id: in_speaker_adaptation_stylisation_steps_beyond_a_small_fraction_of role: complicates claim: In speaker adaptation, stylisation steps beyond a small fraction of total training degrade speaker identity and peak naturalness, establishing a ceiling on achievable style transfer without identity loss. source: §3.4, Figure 2 evidence: Holding total steps at 100, speaker similarity decreases monotonically as stylisation steps increase, while naturalness peaks at 20% stylisation (m=20); excessive stylisation corrupts identity and degrades nMOS. confidence: high relevance: high - claim_id: fair_cross_framework_comparison_in_federated_tts_is_obstructed_when role: complicates claim: Fair cross-framework comparison in federated TTS is obstructed when competing methods are architecturally coupled to legacy backbones and cannot be reimplemented on modern foundations. source: §3.1.2 evidence: FedSpeech and Federated Dynamic Transformer use FastSpeech2-XL and Transformer-TTS respectively and are incompatible with LoRA; the evaluation must compare them on their original configurations, making it impossible to isolate the contribution of the federation strategy from the backbone quality. confidence: high relevance: medium limitations: - The federated baselines (FedSpeech, Federated Dynamic Transformer) run on architecturally distinct and weaker backbones; the performance gap reflects backbone modernization in addition to the proposed federation strategy. No ablation on GPT-SOVITS-V4 with a standard FedAvg baseline using the same LoRA parameter budget is reported to cleanly isolate the aggregation contribution. - Evaluation uses 60 speakers treated as 60 independent clients from four relatively small emotion datasets, which may not represent the statistical diversity or scale of real-world federated deployments. Naturalness is evaluated by 22 in-lab researchers rather than crowd-sourced listeners, which may limit generalisability of the nMOS scores. The paper does not address communication round latency or asynchronous client availability, both of which matter in practical on-device deployment. How the framework scales to many more clients or to cross-lingual speaker sets remains untested. caveats: [] - id: '2509.22718' published_date: "2025-09-24" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - singing architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_adaptation claims: - claim_id: synchronized_visual_cues_from_a_speaker_s_or_singer_s role: supports claim: Synchronized visual cues from a speaker's or singer's mouth region can substitute for explicit phoneme-level duration annotations in duration-based speech and singing synthesis pipelines. source: §3.5, §5.1, Table 1 evidence: Replacing externally supplied phoneme durations with a duration predictor conditioned on lip-cue-fused content features (via VCFM) allows the duration predictor and Length Regulator to operate without duration inputs at inference, while achieving lower MCD and FFE than a duration-free StyleSinger baseline that has no visual input. confidence: high relevance: medium - claim_id: fusion_mechanisms_designed_for_one_cross_modal_alignment_problem_do role: complicates claim: Fusion mechanisms designed for one cross-modal alignment problem do not transfer directly to singing, because singing's rhythmic and prosodic complexity differs from speech. source: §5.2, Table 3 evidence: Adopting StyleDubber's Stepwise Monotonic Multi-head Attention (originally designed for movie-dubbing lip-speech alignment) for textual-visual fusion in this SVS setting produced worse MCD, COS, and LSE-D than the no-visual baseline, whereas the proposed VCFM improved on the same baseline across all metrics. confidence: high relevance: medium - claim_id: staged_training_that_first_establishes_a_strong_acoustic_model_before role: supports claim: Staged training that first establishes a strong acoustic model before introducing an auxiliary modality improves quality over jointly training all components from the start. source: §5.2, Table 3 evidence: The two-stage schedule (stage one trains phoneme/pitch encoders and decoder; stage two adds the frozen visual encoder and VCFM) outperforms single-stage training on every objective metric (MCD 3.1125 vs. 3.1307, FFE 0.3921 vs. 0.4022, COS 0.9206 vs. 0.9195, LSE-C 1.4270 vs. 1.3888, LSE-D 10.2782 vs. 10.3146). confidence: high relevance: medium - claim_id: visual_dubbing_architectures_built_for_spoken_dialogue_do_not_generalize role: complicates claim: Visual dubbing architectures built for spoken dialogue do not generalize to singing without substantial adaptation, even when augmented with pitch information. source: §4.1, §5.1, Table 1 evidence: HPMDubbing, a lip-synchronized speech dubbing model, and its pitch-augmented variant HPMDubbing-P both perform far below SVS-specific baselines on every metric (e.g., MOS-Q of 1.45–1.46 vs. 3.54–3.71 for singing-specific systems), despite HPMDubbing-P adding phoneme-level pitch as input. confidence: high relevance: low limitations: - All results are reported on a single self-constructed dataset (9 singers, 69 songs, ~3 hours, Chinese language only), evaluated against two baselines that were adapted from other tasks or settings rather than SVS systems natively designed for duration-free, multimodal input. There is no evaluation on an external or multi-lingual benchmark, so the generality of the reported gains beyond this dataset is untested. - The visual encoder is frozen during stage-two training rather than fine-tuned jointly with the rest of the pipeline, which may limit how well lip-cue features adapt to the singing domain specifically (the encoder was pretrained for general audio-visual speech tasks, not singing). The paper also does not report model parameter counts or inference latency, so the practical cost of adding a video branch relative to text-only SVS systems is not established. Manual TextGrid annotation required 40-45 human-hours per hour of audio, indicating that scaling VisualSinger-style datasets to more singers or languages will be labor-intensive. caveats: [] - id: '2509.22727' published_date: "2025-09-25" entry_date: '2026-07-29' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - flow_matching_speaker_adaptation claims: - claim_id: joint_training_across_multiple_dialects_induces_style_averaging_where_dialect role: supports claim: Joint training across multiple dialects induces "style averaging," where dialect-specific prosodic identity is diluted unless the model has a mechanism to route dialect-specific information separately. source: §4.4.2, Table 3 evidence: Ablating the dialect-aware MoE module (removing it while keeping the IPA frontend) degrades both MOS (e.g., 2.46 vs. 3.15 on Xi'an) and WER (45.08% vs. 33.00% on Xi'an) relative to the full model, isolating the MoE's contribution to preserving per-dialect quality in a jointly trained model. confidence: high relevance: medium - claim_id: a_standardized_phonetic_representation_ipa_resolves_grapheme_to_phoneme_ambiguity role: supports claim: A standardized phonetic representation (IPA) resolves grapheme-to-phoneme ambiguity better than orthography-based inputs (e.g., pinyin) when a single model must serve multiple dialects with divergent character-to-sound mappings. source: §4.4.2, Table 3 evidence: Replacing the IPA frontend with pinyin input (keeping the MoE module) collapses MOS to roughly 1.0-1.2 and raises WER to roughly 90-93% across four dialects, a far larger degradation than removing the MoE alone. confidence: high relevance: medium - claim_id: parameter_efficient_fine_tuning_lora_plus_lightweight_adapters_on_a role: supports claim: Parameter-efficient fine-tuning (LoRA plus lightweight adapters) on a frozen multidialect backbone can extend zero/few-shot speech synthesis to a new, extremely low-resource dialect using only a few hours of data. source: §3.3, §4.1, Table 2 evidence: Stage 3 adapts the frozen Stage-2 model to Peking Opera (Jingbai/Yunbai) and Nanjing dialect using only ~3 hours of data per target style, training only LoRA (rank 16) and Conditioning Adapters while the backbone stays frozen, producing usable synthesis (e.g., Jingbai MOS 3.04, UTMOSv2 1.75) for styles with no commercial TTS baseline available for comparison. confidence: high relevance: high - claim_id: open_modest_scale_multidialect_training_data_can_substantially_underperform_proprietary role: complicates claim: Open, modest-scale multidialect training data can substantially underperform proprietary, large-scale commercial systems on standard intelligibility metrics even when subjective naturalness scores are comparable. source: §4.4.1, Table 1, Table 2 evidence: Trained on ~1.1k total hours versus CosyVoice2's ~150k hours and Qwen-TTS's ~3,000k hours, DiaMoE-TTS's WER trails the baseline average by wide margins in several dialects (e.g., 44.61 points worse on Cantonese, 29.69 points worse on Chengdu), even where its UTMOSv2 score is on par with or better than the commercial average. confidence: high relevance: medium limitations: - Word error rate is substantially worse than commercial baselines across nearly every dialect evaluated (e.g., 76.59% vs. a baseline average of 31.98% on Cantonese), and the paper does not report a controlled comparison isolating whether this gap stems from training data scale, IPA/G2P errors, or the MoE routing itself. - 'The evaluation compares against commercial black-box systems (Edge TTS, CosyVoice2, Qwen-TTS) whose training data, architectures, and exact evaluation conditions are not fully known, making some of the reported gaps difficult to attribute precisely. Coverage is also uneven: several dialects and styles (Southern Min, Shijiazhuang, Nanjing, Peking Opera) have no baseline comparison at all, so the paper''s claim of being "almost comparable" to commercial systems rests on a subset of dialects where baselines exist. The MoE''s number of experts and their assignment to specific dialects are not detailed in the text, and no comparison is given for how the dialect-classification auxiliary loss weight (λ) was chosen or how sensitive results are to it.' caveats: [] claim_clusters: - id: multi_speaker_training_learns_shared_voice_structure claim: Multi-speaker training learns shared acoustic structure that improves synthesis and generalization across speakers. status: strongly_supported confidence: high supporting_papers: - '1609.03499' - '1904.02882' - 2025.naacl-short.65 - '2507.15272' - interspeech-2025-0596 contradicting_papers: [] refining_papers: - '2509.00675' caveats: - Dataset balance, recording quality, and speaker coverage often vary together. last_reviewed: '2026-07-29' - id: reference_conditioning_enables_unseen_speaker_adaptation claim: Reference-audio conditioning enables synthesis or conversion for unseen speakers without full speaker-specific retraining. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2406.04904' - '2505.07916' - '2507.08319' - '2506.18296' - '2507.15272' - interspeech-2025-0787 - interspeech-2025-0816 - interspeech-2025-1434 - interspeech-2025-1726 - interspeech-2025-2815 - '2509.09631' contradicting_papers: [] refining_papers: - 2025.naacl-short.65 - '2505.07916' - 2025.acl-long.87 - interspeech-2025-1538 caveats: - Performance depends on reference duration, cleanliness, content, and match to the training domain. last_reviewed: '2026-07-29' - id: learned_speaker_encoders_outperform_fixed_global_embeddings claim: Task-trained or locally conditioned speaker representations can preserve identity better than fixed global verification embeddings. status: strongly_supported confidence: high supporting_papers: - '2505.07916' - '2506.18296' - interspeech-2025-0816 - interspeech-2025-1434 - '2509.07376' - '2509.09631' contradicting_papers: [] refining_papers: - 2025.acl-long.87 - interspeech-2025-1538 caveats: - Dedicated encoders may improve synthesis while becoming less comparable to verification embeddings. last_reviewed: '2026-07-29' - id: prompt_length_and_quality_control_cloning_quality claim: Speaker-cloning quality is sensitive to prompt duration, acoustic quality, and where the prompt enters the generation pipeline. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2406.05370' - '2409.03283' - '2507.16875' - interspeech-2025-2815 - '2411.19770' contradicting_papers: [] refining_papers: - 2025.naacl-short.65 - '2505.07916' caveats: - Longer references do not uniformly help across architectures, and enhancement can damage already-clean prompts. last_reviewed: '2026-07-29' - id: few_shot_fine_tuning_improves_distinctive_voice_matching claim: Few-shot speaker-specific fine-tuning generally improves identity and naturalness for distinctive, low-resource, or out-of-domain voices. status: strongly_supported confidence: high supporting_papers: - '2406.04904' - '2409.03283' - 2025.findings-naacl.279 - 2025.acl-long.87 - interspeech-2025-0816 - interspeech-2025-1344 - interspeech-2025-1726 - interspeech-2025-2765 - '2508.18006' - '2506.04077' - '2412.16846' - '2509.16010' - '2509.22727' contradicting_papers: [] refining_papers: - 2025.findings-naacl.279 - '2507.20140' - '2509.16010' caveats: - Benefits depend on data quality and can trade against the base model's generalization. last_reviewed: '2026-07-29' - id: parameter_efficient_adaptation_limits_forgetting claim: Adapters, LoRA, speaker vectors, and other parameter-efficient updates can personalize voices while limiting catastrophic forgetting. status: strongly_supported confidence: high supporting_papers: - '2505.07916' - '2508.09767' - interspeech-2025-1344 - '2508.18006' - '2509.16010' - '2509.22727' contradicting_papers: [] refining_papers: [] caveats: - Lightweight updates may underfit highly expressive voices or new linguistic coverage. last_reviewed: '2026-07-29' - id: disentangling_identity_and_style_improves_control claim: Separating speaker identity from style, emotion, content, and prosody improves controllable personalization. status: strongly_supported confidence: high supporting_papers: - '2508.08399' - interspeech-2025-1440 - '2509.16010' contradicting_papers: [] refining_papers: [] caveats: - Identity and expressive behavior overlap, so hard separation can erase speaker-specific phrasing and timing. last_reviewed: '2026-07-29' - id: identity_and_expressiveness_trade_off_during_adaptation claim: Speaker identity, naturalness, intelligibility, and expressive-style transfer frequently trade off during adaptation. status: strongly_supported confidence: high supporting_papers: - '2406.04904' - '2406.05370' - '2502.18924' - '2506.18296' - 2025.acl-industry.42 - 2025.acl-long.87 - '2509.15969' - '2509.16010' contradicting_papers: [] refining_papers: - '2406.04904' - '2507.09282' - '2507.15272' - '2508.08399' - interspeech-2025-0596 - '2509.16010' caveats: - The apparent trade-off changes with listener task and automatic metric choice. last_reviewed: '2026-07-29' - id: cross_lingual_adaptation_remains_language_sensitive claim: Cross-lingual and multilingual adaptation can preserve speaker identity, but language coverage and duration modeling create persistent variability. status: strongly_supported confidence: high supporting_papers: - '2406.04904' - 2025.findings-naacl.279 - '2503.11026' - '2508.08715' - interspeech-2025-1344 - interspeech-2025-raju25_interspeech - '2509.22727' contradicting_papers: [] refining_papers: - '2406.04904' - 2025.findings-naacl.279 caveats: - Language, accent, phonetic coverage, and training-data scale are rarely controlled independently. last_reviewed: '2026-07-29' - id: continuous_representations_preserve_speaker_detail claim: Continuous acoustic representations often preserve speaker detail better than discrete content-oriented units in adaptation systems. status: strongly_supported confidence: high supporting_papers: - '2305.07243' - '2407.08551' - iclr-2025-cuFzE8Jlvb - 2025.findings-naacl.130 - '2509.04667' contradicting_papers: [] refining_papers: [] caveats: - Representation comparisons depend on token rate, decoder capacity, and whether semantic invariance is desirable. last_reviewed: '2026-07-29' - id: codec_prompting_supports_in_context_voice_cloning claim: Codec-token language modeling supports in-context voice cloning through acoustic prompts without explicit speaker-specific training. status: emerging confidence: medium supporting_papers: - '2301.02111' - '2406.05370' contradicting_papers: [] refining_papers: [] caveats: - Codec hierarchy and prompt placement affect how identity and acoustic detail are distributed. last_reviewed: '2026-07-29' - id: flow_and_diffusion_support_scalable_adaptation claim: Flow-matching and diffusion generators support high-quality speaker adaptation with parallel generation and effective distillation. status: strongly_supported confidence: high supporting_papers: - '2305.07243' - '2502.18924' - '2507.08319' - '2507.15272' - 2025.acl-long.790 - 2025.acl-long.87 - '2503.11026' - '2508.04996' - interspeech-2025-0854 - interspeech-2025-1434 - interspeech-2025-1747 - interspeech-2025-raju25_interspeech contradicting_papers: [] refining_papers: - '2507.15272' caveats: - Matched comparisons against autoregressive systems remain limited across data scale and inference budgets. last_reviewed: '2026-07-29' - id: data_scale_and_diversity_drive_unseen_speaker_generalization claim: Training-data scale and speaker diversity are major drivers of unseen-speaker similarity and robustness. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2305.07243' - 2025.naacl-short.65 - interspeech-2025-0596 - interspeech-2025-1726 - '2411.19770' - '2509.04702' - '2509.15969' contradicting_papers: [] refining_papers: - '2507.08319' - interspeech-2025-1536 caveats: - Scale effects are confounded by filtering, transcription quality, architecture, and proprietary data. last_reviewed: '2026-07-29' - id: automatic_similarity_metrics_do_not_replace_listener_judgments claim: Automatic speaker-similarity metrics can diverge from perceptual identity and adaptation suitability judgments. status: emerging confidence: medium supporting_papers: - '2406.18009' - '2509.09631' contradicting_papers: [] refining_papers: - '2507.15272' - interspeech-2025-1538 - interspeech-2025-1726 - '2412.16846' caveats: - Embedding metrics reward consistency in their own representation space and may miss identity, style, or assistive suitability. last_reviewed: '2026-07-29' - id: noise_and_domain_mismatch_degrade_reference_adaptation claim: Noise, channel, domain, and atypical-speech mismatch degrade reference-conditioned speaker adaptation unless explicitly addressed. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2508.04996' - '2508.09702' - interspeech-2025-0596 - interspeech-2025-0816 - interspeech-2025-1536 - interspeech-2025-1638 - interspeech-2025-1726 - '2411.19770' contradicting_papers: [] refining_papers: - interspeech-2025-1536 - '2411.19770' caveats: - Robustness interventions can alter timbre or normalize speaker-specific characteristics. last_reviewed: '2026-07-29' - id: speaker_adaptation_supports_personalized_assistive_voices claim: Speaker adaptation enables personalized assistive voices from limited or atypical reference speech. status: strongly_supported confidence: high supporting_papers: - '2507.09282' - interspeech-2025-0596 - interspeech-2025-1536 - interspeech-2025-1726 - interspeech-2025-2679 - interspeech-2025-2815 contradicting_papers: [] refining_papers: - interspeech-2025-1536 - interspeech-2025-1726 caveats: - Perceived communicative suitability may matter more than strict timbre matching in assistive settings. last_reviewed: '2026-07-29' - id: speaker_privacy_conflicts_with_identity_preservation claim: Speaker anonymization removes identity cues that adaptation systems otherwise seek to preserve, exposing a privacy–personalization tension. status: strongly_supported confidence: high supporting_papers: - '2506.18296' - '2508.15565' - '2509.04667' contradicting_papers: [] refining_papers: - '2507.20140' caveats: - Privacy gains can collapse under adaptive attacks and may not transfer across speaker extractors. last_reviewed: '2026-07-29' method_families: - id: autoregressive_prompt_conditioned_adaptation name: Autoregressive prompt-conditioned adaptation summary: Autoregressive speech language models adapt to unseen speakers through acoustic prompts, speaker embeddings, or in-context codec-token conditioning. papers: - '1609.03499' - '2301.02111' - '2305.07243' - '2406.04904' - '2406.05370' - '2406.07855' - '2407.08551' - '2409.03283' - iclr-2025-cuFzE8Jlvb - 2025.findings-naacl.279 - '2505.07916' - '2507.02380' - '2507.09282' - '2505.15670' - '2508.14049' - '2508.04141' - '2508.08715' - '2508.09767' - interspeech-2025-0047 - interspeech-2025-0787 - interspeech-2025-1538 - interspeech-2025-2679 - '2506.04077' - '2412.16846' - '2509.15462' - '2509.15969' open_questions: - How can prompt-conditioned autoregressive models preserve identity without inheriting prompt noise, content, or prosody? - id: flow_matching_speaker_adaptation name: Flow-matching speaker adaptation summary: Flow-matching systems condition parallel acoustic generation on speaker references, learned embeddings, or adapted latent representations. papers: - '2406.18009' - '2409.03283' - '2505.07916' - '2506.13053' - '2507.02380' - '2507.16875' - 2025.acl-long.790 - 2025.acl-long.87 - '2507.20140' - '2503.11026' - '2508.14049' - '2508.04996' - '2508.08715' - interspeech-2025-0854 - interspeech-2025-1344 - interspeech-2025-1536 - interspeech-2025-raju25_interspeech - '2509.09631' - '2509.15462' - '2509.22727' open_questions: - Which conditioning placement best preserves speaker identity at low inference cost across languages and recording domains? - id: gan_based_speaker_transfer name: GAN-based speaker transfer and adaptation summary: Adversarial generators and vocoders transfer target-speaker characteristics through speaker encoders, reference features, or speaker-specific fine-tuning. papers: - '2406.04904' - 2025.findings-naacl.130 - 2025.findings-naacl.279 - 2025.naacl-short.65 - 2025.americasnlp-1.1 - 2025.acl-industry.42 - '2508.07426' - '2508.08715' - interspeech-2025-0854 - interspeech-2025-1747 - '2508.15565' - '2508.17031' - '2508.18006' - '2509.04667' open_questions: - How can adversarial training suppress source-speaker leakage while maintaining stable reconstruction and naturalness? - id: transformer_encoder_decoder_adaptation name: Transformer encoder–decoder adaptation summary: Encoder–decoder systems use reference encoders, adapters, retrieval, or cross-attention to inject speaker information into acoustic generation. papers: - 2025.findings-naacl.130 - 2025.americasnlp-1.1 - '2507.00808' - '2507.08319' - 2025.acl-industry.42 - '2508.07426' - interspeech-2025-0596 - interspeech-2025-0854 - interspeech-2025-1034 - '2508.15931' - '2508.17031' - '2509.00675' - '2509.09631' open_questions: - When do local reference features outperform global speaker embeddings for identity and prosody preservation? - id: vae_latent_speaker_adaptation name: VAE latent speaker adaptation summary: VAE-based systems represent speaker identity in continuous latent spaces that support sampling, disentanglement, and lightweight adaptation. papers: - '2305.07243' - '2502.18924' - iclr-2025-cuFzE8Jlvb - '2505.07916' - '2507.09282' - '2508.07426' - '2508.08399' - interspeech-2025-1440 - '2509.07376' - '2412.16846' open_questions: - How should latent regularization balance reconstruction fidelity, speaker separability, and downstream modelability? - id: hybrid_semantic_acoustic_adaptation name: Hybrid semantic–acoustic adaptation summary: Hybrid systems combine semantic, acoustic, speaker, style, visual, or codec pathways to preserve complementary aspects of a target voice. papers: - '2409.03283' - 2025.findings-naacl.279 - 2025.naacl-short.65 - '2508.04141' - interspeech-2025-1478 - interspeech-2025-1638 - interspeech-2025-1726 - '2509.04667' - '2509.16010' - '2509.22718' open_questions: - Can identity, style, prosody, and content be separated without discarding speaker-specific timing and expression? - id: diffusion_based_speaker_adaptation name: Diffusion-based speaker adaptation summary: Diffusion systems condition iterative or distilled acoustic generation on reference speech, speaker embeddings, or adapted latent targets. papers: - '2305.07243' - '2502.18924' - '2507.09282' - '2507.15272' - 2025.ccl-1.77 - interspeech-2025-0816 - interspeech-2025-1434 - interspeech-2025-1747 - '2411.19770' open_questions: - Can distilled diffusion retain speaker fidelity and robustness under the inference budgets required for interactive use? reassessment_queue: - id: prompt_length_and_quality_control_cloning_quality type: claim_status reason: Prompt-duration findings vary across autoregressive architectures and clean versus noisy references. trigger: Matched multi-architecture studies isolate duration, content, and acoustic-quality effects. due: 2026-10 current_assessment: strongly_supported watch_for: - Prompt-duration sweeps - Clean/noisy reference controls - id: continuous_representations_preserve_speaker_detail type: claim_status reason: Continuous-versus-discrete comparisons also change rate, alignment, and decoder capacity. trigger: Rate- and decoder-matched studies reproduce the speaker-detail advantage. due: 2026-10 current_assessment: strongly_supported watch_for: - Rate-controlled representation comparisons - Perceptual identity tests - id: automatic_similarity_metrics_do_not_replace_listener_judgments type: benchmark_validity reason: Several systems expose disagreement between verification embeddings and listener judgments. trigger: Listener-calibrated identity metrics generalize across TTS, VC, languages, and atypical speech. due: 2026-10 current_assessment: emerging watch_for: - Cross-system metric calibration - Assistive-voice suitability benchmarks - id: cross_lingual_adaptation_remains_language_sensitive type: claim_status reason: Language coverage, data quality, and duration modeling are confounded in most comparisons. trigger: Matched multilingual studies separate language coverage from speaker-conditioning design. due: 2026-10 current_assessment: strongly_supported watch_for: - Matched-language training studies - Unseen-language speaker evaluations - id: speaker_privacy_conflicts_with_identity_preservation type: claim_status reason: Privacy evidence is heterogeneous and often evaluated against a limited set of speaker extractors. trigger: Adaptive, black-box, and perceptual privacy evaluations converge across anonymization methods. due: 2026-10 current_assessment: strongly_supported watch_for: - Adaptive privacy attacks - Perceptual identity leakage studies open_questions: - What reference duration, content, and acoustic quality are sufficient for robust adaptation across architectures and languages? - Which speaker representation best preserves perceptual identity without importing prompt noise, prosody, or linguistic content? - When should personalization use in-context prompting, parameter-efficient adaptation, or full speaker-specific fine-tuning? - How can identity and expressive style be separated without erasing speaker-specific phrasing, timing, and accent? - What listener-calibrated metric can compare speaker identity across TTS, voice conversion, atypical speech, and assistive use? - How should speaker adaptation systems balance personalization with consent, privacy, misuse resistance, and revocation? trend_notes: - Speaker adaptation has shifted from speaker-ID lookup and fixed embeddings toward reference-conditioned zero-shot and few-shot generation. - Codec language models, flow matching, and diffusion increasingly support adaptation without a dedicated per-speaker acoustic model. - Recent work increasingly combines global identity embeddings with local, semantic, prosodic, or style-specific conditioning. - Parameter-efficient personalization is expanding through adapters, LoRA, speaker vectors, and federated style modules. - Evaluation is moving beyond verification embeddings toward perceptual identity, robustness, expressive suitability, fairness, and privacy.