concept: emotion-synthesis last_updated: '2026-07-30' paper_count: 73 papers: - id: '2312.01479' published_date: "2023-12-03" entry_date: '2026-07-30' year: 2023 venue: arXiv task: - TTS architecture: - GAN - VAE relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - gan_based_expressive_transfer - vae_emotion_style_latents claims: - claim_id: decoupling_tone_color_transfer_from_style_and_language_generation_enables role: supports claim: Decoupling tone color transfer from style and language generation enables flexible post-cloning control over emotion, accent, and rhythm without requiring paired multi-attribute training data. source: §2.1, §2.2 evidence: Decoupling tone color transfer from style and language generation enables flexible post-cloning control over emotion, accent, and rhythm without requiring paired multi-attribute training data. confidence: high relevance: high - claim_id: using_ipa_as_a_universal_phoneme_representation_in_a_normalising role: supports claim: Using IPA as a universal phoneme representation in a normalising flow allows voice cloning to generalise to unseen languages without retraining on massive multilingual speaker data. source: §2.3, §3 "Importance of IPA" evidence: Using IPA as a universal phoneme representation in a normalising flow allows voice cloning to generalise to unseen languages without retraining on massive multilingual speaker data. confidence: high relevance: low - claim_id: feed_forward_voice_cloning_pipelines_encoder_flow_decoder_without_autoregressive role: supports claim: Feed-forward voice cloning pipelines (encoder-flow-decoder without autoregressive or diffusion components) achieve real-time or faster inference, making large-scale commercial deployment feasible. source: §3 "Fast Inference with Low Cost" evidence: Feed-forward voice cloning pipelines (encoder-flow-decoder without autoregressive or diffusion components) achieve real-time or faster inference, making large-scale commercial deployment feasible. confidence: high relevance: medium - claim_id: cross_study_quantitative_comparison_of_voice_cloning_systems_is_systematically role: supports claim: Cross-study quantitative comparison of voice cloning systems is systematically confounded by differences in test set composition, training data scale, and evaluation protocol. source: §3 evidence: Cross-study quantitative comparison of voice cloning systems is systematically confounded by differences in test set composition, training data scale, and evaluation protocol. confidence: high relevance: low limitations: - The paper reports no quantitative metrics anywhere. All evaluation is qualitative, with audio samples hosted on a demo website. This makes it impossible to independently verify the claimed advantages over VALL-E, XTTS, YourTTS, or Voicebox on any standard benchmark. - The base TTS model is trained on only four speakers across English, Chinese, and Japanese. Adding support for additional languages requires collecting at least one native single-speaker dataset, which while much cheaper than a full MSML system, still represents a non-trivial data acquisition effort. Style preservation through the tone color converter is imperfect for highly expressive inputs, with emotion occasionally being neutralised. The proprietary training data for both the base model and the MSML converter set cannot be replicated by the research community despite the open-source code and weights release. caveats: [] - id: '2402.05755' published_date: "2024-02-08" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models claims: - claim_id: continuous_pretraining_of_a_text_llm_on_interleaved_speech_and role: supports claim: Continuous pretraining of a text LLM on interleaved speech and text tokens transfers the text model's few-shot learning and semantic reasoning abilities to the speech modality. source: §4.2, §4.3, Table 4 evidence: Continuous pretraining of a text LLM on interleaved speech and text tokens transfers the text model's few-shot learning and semantic reasoning abilities to the speech modality. confidence: high relevance: medium - claim_id: word_level_interleaving_of_speech_and_text_during_training_is role: supports claim: Word-level interleaving of speech and text during training is more effective than parallel ASR/TTS training or speech-only fine-tuning for cross-modal semantic understanding. source: §4.2, Table 6 evidence: Word-level interleaving of speech and text during training is more effective than parallel ASR/TTS training or speech-only fine-tuning for cross-modal semantic understanding. confidence: high relevance: medium - claim_id: expressive_speech_properties_sentiment_pitch_style_can_be_modeled_in role: supports claim: Expressive speech properties (sentiment, pitch, style) can be modeled in a language model through discrete token streams that supplement phonetic tokens, enabling cross-modal sentiment preservation. source: §5, Table 3 evidence: Expressive speech properties (sentiment, pitch, style) can be modeled in a language model through discrete token streams that supplement phonetic tokens, enabling cross-modal sentiment preservation. confidence: high relevance: high - claim_id: adding_expressive_speech_tokens_to_a_speech_lm_improves_expressivity role: complicates claim: Adding expressive speech tokens to a speech LM improves expressivity at the cost of moderate degradation in lexical and grammatical speech understanding. source: §4.2, Table 4 evidence: Adding expressive speech tokens to a speech LM improves expressivity at the cost of moderate degradation in lexical and grammatical speech understanding. confidence: high relevance: high - claim_id: cascade_speech_pipelines_remain_substantially_stronger_than_end_to_end role: supports claim: Cascade speech pipelines remain substantially stronger than end-to-end unified models on task-specific metrics such as ASR WER and TTS intelligibility at equivalent model scale. source: §4.3, Table 5 evidence: Cascade speech pipelines remain substantially stronger than end-to-end unified models on task-specific metrics such as ASR WER and TTS intelligibility at equivalent model scale. confidence: high relevance: medium limitations: - Spirit LM's vocoder is conditioned on only 4 speaker voices from the Expresso dataset, which severely constrains the diversity and quality of synthesized speech; the model cannot generalize to arbitrary target speakers at inference without retraining the vocoder. - The STSP benchmark is evaluated using fine-tuned automatic classifiers rather than human listeners, which may not capture perceptual sentiment fidelity accurately. The model was evaluated only in English, leaving multilingual capabilities untested. The 7B scale represents a reasonable starting point, but the authors note that scaling beyond 7B could substantially improve both semantic and expressive performance. Toxicity analysis shows the model can add harmful speech content at levels comparable to cascades, with higher MUTOX scores in S→S, warranting safety work before deployment. caveats: [] - id: '2407.05407' published_date: "2024-07-07" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: inserting_a_vector_quantizer_into_a_supervised_asr_encoder_yields role: supports claim: Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. source: §2.1, §5.1, Table 7 evidence: Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. confidence: high relevance: medium - claim_id: in_autoregressive_codec_lm_tts_both_the_text_tokenizer_and role: supports claim: In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. source: §5.2, Table 7 evidence: In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. confidence: high relevance: low - claim_id: asr_re_ranking_is_an_effective_post_hoc_method_for role: complicates claim: ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. source: §5.3, Tables 8, 9 evidence: ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. confidence: high relevance: medium - claim_id: instruction_fine_tuning_on_a_modest_amount_of_labelled_data role: supports claim: Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. source: §2.4, §5.4, Table 10 evidence: Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. confidence: high relevance: high - claim_id: high_quality_tts_synthesized_speech_can_serve_as_effective_training role: supports claim: High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. source: §5.5, Table 11 evidence: High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. confidence: high relevance: medium limitations: - '- Only a single VQ codebook (4096 codes) is used; multi-level RVQ and its effect on quality vs. compression is left for future work. - The choice of VQ insertion layer (after layer 6 of 12) is not ablated — optimal placement is unresolved. - Cross-lingual cloning omits prompt prosody to prevent leakage, which may reduce naturalness in target language. - Instruction fine-tuning data amounts (556h) are modest; broader paralinguistic coverage remains open. - No subjective (MOS) evaluation in the main paper; relies entirely on objective WER/CER/SS metrics.' caveats: [] - id: '2409.00750' published_date: "2024-09-01" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - TTS - VC architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models claims: - claim_id: non_autoregressive_masked_generative_transformers_can_achieve_human_level_speaker role: supports claim: Non-autoregressive masked generative transformers can achieve human-level speaker similarity in zero-shot TTS without requiring explicit text-speech alignment or phone-level duration supervision. source: §4.2.1, Table 2 evidence: Non-autoregressive masked generative transformers can achieve human-level speaker similarity in zero-shot TTS without requiring explicit text-speech alignment or phone-level duration supervision. confidence: high relevance: low - claim_id: replacing_k_means_quantisation_of_ssl_features_with_vq_vae role: supports claim: Replacing k-means quantisation of SSL features with VQ-VAE vector quantisation reduces information loss in tonal languages and improves downstream acoustic token prediction. source: §3.2.1 evidence: Replacing k-means quantisation of SSL features with VQ-VAE vector quantisation reduces information loss in tonal languages and improves downstream acoustic token prediction. confidence: high relevance: medium - claim_id: masked_generative_tts_substantially_outperforms_autoregressive_tts_on_hard_text role: supports claim: Masked generative TTS substantially outperforms autoregressive TTS on hard-text robustness (tongue twisters, repeating phrases) while maintaining competitive naturalness on standard benchmarks. source: §4.2.2, Appendix J, Table 13 evidence: Masked generative TTS substantially outperforms autoregressive TTS on hard-text robustness (tongue twisters, repeating phrases) while maintaining competitive naturalness on standard benchmarks. confidence: high relevance: medium - claim_id: parallel_iterative_decoding_in_masked_generative_models_yields_constant_inference role: supports claim: Parallel iterative decoding in masked generative models yields constant inference cost regardless of output length, in contrast to autoregressive decoding whose cost scales linearly with utterance duration. source: §4.2.2 evidence: Parallel iterative decoding in masked generative models yields constant inference cost regardless of output length, in contrast to autoregressive decoding whose cost scales linearly with utterance duration. confidence: high relevance: medium - claim_id: zero_shot_style_cloning_via_in_context_learning_extends_to role: supports claim: Zero-shot style cloning via in-context learning extends to accent and emotion transfer without task-specific architectural changes. source: §4.3, Tables 4–5 evidence: Zero-shot style cloning via in-context learning extends to accent and emotion transfer without task-specific architectural changes. confidence: high relevance: high limitations: - Speech content editing is acknowledged as "not very robust" by the authors, who attribute this to a training objective mismatch (mask-and-predict vs. fill-in-mask). The editing capability is demonstrated qualitatively only, with no quantitative evaluation reported. - 'Training uses 100K hours of English and Chinese speech from Emilia, with multilingual extension at far smaller data budgets (2,500–8,200 hours per language). Multilingual performance is uneven: French and German show higher WER in cross-lingual dubbing, and the authors note limitations from insufficient retraining of all components on expanded data.' - Duration control requires either a ground-truth length or the flow-matching duration predictor; errors in predicted duration propagate to WER. The gap between predicted-length and ground-truth-length WER is measurable (e.g., 2.634 vs. 2.012 on LibriSpeech test-clean). - Inference steps of 25-50 for T2S plus the S2A step schedule add latency compared to single-pass systems, though the paper does not report real-time factor or wall-clock comparisons. - Emotion control requires post-training fine-tuning on labelled data; it is not available zero-shot from the base model alone. caveats: [] - id: '2501.06282' published_date: "2025-01-10" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: large_scale_multi_task_training_across_heterogeneous_speech_tasks_substantially role: supports claim: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. source: §3.4, §4.1–4.5 evidence: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. confidence: high relevance: medium - claim_id: aligned_multimodal_architectures_can_achieve_instruction_controlled_speech_style_emotion role: complicates claim: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. source: §3.2, §4.4, Table 18 evidence: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. confidence: high relevance: high - claim_id: full_duplex_spoken_dialogue_at_sub_second_latency_is_achievable role: supports claim: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. source: §3.1, §4.5, Table 21 evidence: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. confidence: high relevance: low - claim_id: an_autoregressive_streaming_voice_decoder_that_interleaves_text_hidden_states role: supports claim: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. source: §3.2, §4.4, Table 17 evidence: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. confidence: high relevance: low limitations: - The instruction-following voice generation evaluation uses a single in-house Chinese test set of 122 turns, making external validation of the 98.4% accuracy figure impossible. Code and model weights had not been released at time of publication. - LoRA-only updates to the text LLM during training limit the model's ability to follow diverse instructions; more comprehensive LLM updates with higher-quality text data remain unexplored. Long-tail pronunciation errors persist in end-to-end audio generation, partly due to special symbols that the decoder cannot reliably convert to speech. The full-duplex module still relies on external acoustic echo cancellation (AEC) and voice activity detection (VAD) modules, meaning a truly end-to-end duplex system has not been achieved. Performance on speech emotion recognition in low-resource languages shows mixed results (e.g., Polish at 55.9% F1), indicating that cross-lingual generalization is uneven despite the broad multilingual training. caveats: [] - id: 2025.findings-naacl.38 published_date: "2025-04-29" entry_date: '2026-07-30' year: 2025 venue: NAACL task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: selecting_and_acoustically_emphasizing_a_small_subset_of_emotion_salient role: supports claim: Selecting and acoustically emphasizing a small subset of emotion-salient words, rather than conditioning on a single holistic style embedding, improves emotion recognizability of synthesized dialogue speech. source: §5.1, Table 1 evidence: The proposed model improves Speech Emotion Accuracy by 12.98 points over an emotion-label-only baseline and 16.47 points over a dialogue-context-only baseline, with consistent MOS and ABX gains across all five emotion categories. confidence: high relevance: high - claim_id: large_language_models_can_identify_text_spans_that_carry_emotional role: supports claim: Large language models can identify text spans that carry emotional salience with moderate agreement to human judgment, without being trained on acoustic or prosodic data. source: §5.2, Table 2 evidence: GPT-3.5's word selections for emotion-bearing rationales overlap with a single human annotator's selections at a rate of 0.735, with comparable average word counts (2.5 vs. 3.2 words per utterance) across four non-neutral emotion categories. confidence: high relevance: high - claim_id: training_a_separate_modality_independent_representation_for_masked_acoustic_cues role: complicates claim: Training a separate, modality-independent representation for masked acoustic cues is necessary; forcing a single representation to jointly encode textual rationale and acoustic detail degrades performance. source: §5.3, Table 4 evidence: Removing the dedicated acoustic embedding and relying only on the phoneme-level binary rationale label drops SEA by 6.94 points relative to the full model; using full (unmasked) audio as the embedding target rather than selectively masked audio drops SEA by a further 9.59 points relative to the full model. confidence: high relevance: medium - claim_id: llm_based_emotion_tag_prediction_accuracy_varies_substantially_across_emotion role: complicates claim: LLM-based emotion tag prediction accuracy varies substantially across emotion categories and is markedly worse for emotions underrepresented in the training distribution. source: §5.2, Appendix A, Figure 4 evidence: Stage 1 prompting holistic emotion tag accuracy is 84% for happiness and 74% for surprise but only 57% for anger and sadness, the two emotion categories that together make up under 7% of the labelled training utterances. confidence: high relevance: high limitations: - 'The training and evaluation data are heavily emotion-imbalanced: only 20.4% of DailyTalk utterances carry a non-neutral emotion label, and within that subset happiness accounts for 85.8% while anger is only 2.55%. Results for the lowest-accuracy, lowest-frequency emotion categories (anger, sadness) should be treated with caution.' - The paper's own limitations section flags that the approach assumes GPT-3.5 can reliably correlate text-based word selection with the acoustic features a human speaker would naturally emphasize when reading the same text aloud, a capability the authors acknowledge is not independently verified beyond the single-annotator overlap study and the Llama-3 generalizability check. The evaluation uses only one dataset (DailyTalk) and one TTS backbone (FastSpeech2); whether the selective masking loss transfers to autoregressive or diffusion-based backbones, or to languages other than English, is untested. No code or demo availability is reported in the paper. caveats: [] - id: '2505.15772' published_date: "2025-05-21" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: automated_multimodal_annotation_can_match_human_accuracy_while_substantially_outperforming role: supports claim: Automated multimodal annotation can match human accuracy while substantially outperforming human annotation in inter-rater consistency for emotional speech labeling. source: §2.3, Table 1 evidence: MIKU-PAL achieves Fleiss kappa of 0.93-0.95 on IEMOCAP and MELD, compared to 0.40-0.43 for human annotators, with accuracy of 68.5% on MELD at approximately $0.50/hour versus $10/hour for manual labeling. confidence: high relevance: high - claim_id: expanding_emotional_speech_training_data_beyond_conventional_6_category_taxonomies role: supports claim: Expanding emotional speech training data beyond conventional 6-category taxonomies to psychologically grounded fine-grained categories improves downstream TTS emotion expressiveness. source: §3.2, Table 3 evidence: Fish-Speech fine-tuned on MIKU-EmoBench (26 categories) achieves MOS 4.12 and emotion similarity 0.92, outperforming fine-tuning on IEMOCAP and MELD (MOS 4.01/4.00, emotion similarity 0.89 each). confidence: high relevance: high - claim_id: standard_emotional_speech_benchmark_categories_introduce_annotation_ambiguity_that_inflates role: complicates claim: Standard emotional speech benchmark categories introduce annotation ambiguity that inflates apparent system error rates in automatic labeling. source: §2.3, Figure 4 evidence: MIKU-PAL accuracy on IEMOCAP rises from approximately 58.6% to approximately 75% when 'frustration' and 'neutral' categories (not recognized as distinct in established psychological classification) are excluded, suggesting benchmark taxonomy is itself a source of apparent error. confidence: high relevance: high - claim_id: visual_modality_information_contributes_substantially_to_speech_emotion_classification_accuracy role: supports claim: Visual modality information contributes substantially to speech emotion classification accuracy in multimodal annotation systems. source: §2.1 evidence: Ablation on MELD shows a 25.6% accuracy improvement when vision preprocessing (face detection and active speaker identification) is included relative to audio-only analysis. confidence: high relevance: high limitations: - MIKU-PAL's annotation quality depends on Gemini 2.0 Flash, a proprietary model whose availability, versioning, and cost structure may change; reproducibility over time is uncertain. - The 42% retention rate on raw YouTube video means more than half of collected audio is discarded, limiting efficiency for domain-specific collection. The dataset sources are predominantly English-language content from YouTube, introducing demographic and domain biases. Current SER models are not effective at distinguishing all 26 fine-grained categories, so MIKU-EmoBench's evaluation potential is limited until recognition models catch up. The downstream TTS validation uses only Fish-Speech with a simple special-token conditioning scheme; generalization to other architectures and emotion conditioning mechanisms is untested. caveats: [] - id: '2506.16381' published_date: "2025-06-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: fine_grained_paralinguistic_control_independent_manipulation_of_multiple_acoustic_attributes role: supports claim: Fine-grained paralinguistic control — independent manipulation of multiple acoustic attributes from natural-language instructions — remains an open challenge for current TTS systems, including commercial offerings. source: §4.3, Table 5 evidence: Fine-grained paralinguistic control — independent manipulation of multiple acoustic attributes from natural-language instructions — remains an open challenge for current TTS systems, including commercial offerings. confidence: high relevance: high - claim_id: a_substantial_performance_gap_exists_between_closed_source_commercial_tts role: supports claim: A substantial performance gap exists between closed-source commercial TTS systems and open-source alternatives on instruction-following benchmarks, widening as instruction abstraction increases. source: §4.3, Tables 5–6 evidence: A substantial performance gap exists between closed-source commercial TTS systems and open-source alternatives on instruction-following benchmarks, widening as instruction abstraction increases. confidence: high relevance: medium - claim_id: timbre_controllability_and_emotional_expressiveness_represent_orthogonal_capabilities_in_current role: complicates claim: Timbre controllability and emotional expressiveness represent orthogonal capabilities in current TTS systems, as models that succeed on one dimension frequently fail on the other. source: §4.4, Table 7 evidence: Timbre controllability and emotional expressiveness represent orthogonal capabilities in current TTS systems, as models that succeed on one dimension frequently fail on the other. confidence: high relevance: high - claim_id: automatic_llm_based_evaluation_of_instruction_following_in_speech_achieves role: supports claim: Automatic LLM-based evaluation of instruction-following in speech achieves approximately 79% agreement with human annotators, with lower reliability on subjective role-play tasks. source: §4.1, Table 4 evidence: Automatic LLM-based evaluation of instruction-following in speech achieves approximately 79% agreement with human annotators, with lower reliability on subjective role-play tasks. confidence: high relevance: low - claim_id: paralinguistic_vocal_events_such_as_laughter_sighs_and_screams_are role: supports claim: Paralinguistic vocal events such as laughter, sighs, and screams are not reliably reproducible by any current TTS system when specified in natural-language instructions. source: §4.4, Table 7 evidence: Paralinguistic vocal events such as laughter, sighs, and screams are not reliably reproducible by any current TTS system when specified in natural-language instructions. confidence: high relevance: high limitations: - Gemini is used both as the caption generator for benchmark construction and as the judge evaluating model outputs. Gemini TTS models score highest on the benchmark, and the authors acknowledge LLM self-evaluator bias (Panickssery et al., 2024) as a confound. Gemini scores should be treated with caution. - 'The RP task is inherently subjective: a scenario description like "nervous applicant in an interview" admits many valid acoustic realisations, and lower human-Gemini agreement (71% average) reflects this ambiguity rather than evaluator failure. The dataset construction pipeline draws from film and television sources, so emotional and stylistic distributions reflect those domains; certain categories (specific emotions, role archetypes) are under-represented by design. Continuous large-scale evaluation using Gemini-as-a-judge costs roughly $25 per full evaluation run, which may limit iterative use.' caveats: [] - id: '2507.00808' published_date: "2025-07-02" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: iterative_natural_language_feedback_can_progressively_refine_the_speaking_style role: supports claim: Iterative natural language feedback can progressively refine the speaking style of synthesized speech without accumulating naturalness degradation. source: §4.2, §4.4, Figure 4, Figure 6 evidence: Over three interaction sessions, Iterative (ours) significantly outperformed the Identical baseline on a 5-point style refinement MOS, and naturalness MOS showed no significant difference between Iterative and the oracle condition across all style groups. confidence: high relevance: high - claim_id: global_speech_embedding_based_conditioning_cannot_accurately_reflect_fine_grained role: complicates claim: Global speech embedding-based conditioning cannot accurately reflect fine-grained positional or linguistic instructions in expressive TTS. source: §5.1, §5.2, Table 4 evidence: Low-scoring examples in the test set contained directions targeting specific word positions ("at the beginning", "at the end", "for the part of...") or linguistic modifications ("hold your breath", "place just a slight pause between words"), which the speech embedding manipulation approach could not handle. confidence: high relevance: high - claim_id: semantic_similarity_of_style_direction_text_not_exact_wording_governs role: supports claim: Semantic similarity of style direction text, not exact wording, governs how well listeners perceive style refinement as aligned with the instruction. source: §4.3, Figure 5 evidence: In the style refinement accuracy evaluation, directions semantically similar to the one used for refinement (Random Similar) yielded scores comparable to the Matched condition, while semantically dissimilar directions scored significantly lower across all style groups. confidence: high relevance: medium - claim_id: holistic_subjective_evaluation_scales_may_not_adequately_capture_fine_grained role: complicates claim: Holistic subjective evaluation scales may not adequately capture fine-grained stylistic alignment in iterative TTS refinement tasks. source: §4.2, §5.3 evidence: Even the Actor-Guided oracle condition scored around 3 out of 5 on the iterative style refinement MOS, which the authors attribute to the evaluation task not fully discriminating subtle style differences; similar evaluation difficulties have been noted in text-to-image/video generation research. confidence: high relevance: low limitations: - All training and evaluation data is proprietary in-house Japanese speech from two voice actors. No public dataset is used, and no results are reported outside this setup. Reproducibility and generalization are untested. - 'The style refiner is speaker-dependent; the authors plan to extend to speaker-independent operation as future work. The current model refines only paralinguistic information (speaking style via global embeddings), not linguistic content, so instructions involving pauses, stress, or pitch accent placement cannot be followed. The directions cover only two of four practical categories from actual recording sessions (paralinguistic and text-expressible linguistic instructions), omitting demonstrative and gestural instructions entirely. The evaluation task design is also noted as an open problem: the relatively low absolute scores even under oracle conditions suggest that existing MOS paradigms do not cleanly measure this type of fine-grained iterative alignment, and more sensitive evaluation methods are needed.' caveats: [] - id: '2507.08012' published_date: "2025-07-05" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: latent_controllable_features_absent_from_a_tts_model_s_training role: supports claim: Latent controllable features absent from a TTS model's training annotations can be discovered by applying PCA to embeddings of fixed-input generated samples and enrolling the identified dimensions as new description-prompt labels. source: §4.3, Tables 2–3 evidence: For T3 (no emotion labels), PCA of 1,000 Wav2Vec2 embeddings revealed emotional intensity as the primary variance axis; iterative re-labelling and fine-tuning progressively separated neutral from emotive utterances, improving neutral cluster assignment from 59.3% to 89.3% across rounds. confidence: high relevance: medium - claim_id: variance_based_feature_discovery_in_prompt_based_tts_is_unreliable role: complicates claim: Variance-based feature discovery in prompt-based TTS is unreliable when the model's output distribution is highly constrained by existing conditioning labels. source: §4.4, Figure 6 evidence: T3-emotion, already trained with explicit emotion and intensity labels, generated highly consistent F0 contours under neutral-emotion prompts (Figure 6), leaving insufficient variance for meaningful feature discovery in the fixed-input analysis set. confidence: high relevance: medium - claim_id: embedding_based_clustering_of_tts_output_variance_can_surface_acoustic role: complicates claim: Embedding-based clustering of TTS output variance can surface acoustic artefacts of the training corpus (such as recording-condition variation) rather than prosodic features of interest. source: §4.4, Figure 8 evidence: For T3-emotion with a diverse analysis set, the principal component correlated strongly with GeMaps-v01b loudness features attributable to microphone distance differences in the Talromur-3 corpus, not to prosody. confidence: high relevance: medium - claim_id: self_supervised_speech_representations_suitable_for_prosody_analysis_are_inherently role: complicates claim: Self-supervised speech representations suitable for prosody analysis are inherently entangled with speaker identity and linguistic content, requiring fixed-input generation sets to isolate prosodic variation. source: §3.2, Figure 1 evidence: Wav2Vec2 summary embeddings cluster by target text and speaker identity regardless of which network layer is used (Figure 1), motivating the fixed-input analysis design where text, speaker, and prompt are held constant. confidence: high relevance: low limitations: - No subjective listening tests are reported. All quality and controllability assessments use automatic metrics (ASR-based WER, speaker embedding cosine similarity, diversity score). Whether the discovered features correspond to perceptually meaningful and user-controllable dimensions is not established. - The method is evaluated on a single Icelandic speaker (Ingrid) for the fine-tuning stages, with no cross-speaker or cross-language generalisation experiments. The paper does not address why T3-emotion fails and T3 succeeds beyond noting the former's consistent output distribution; whether adding a more diverse analysis set would help, or whether an alternative embedding choice (not Wav2Vec2) would be less sensitive to recording conditions, remains open. The use of Icelandic as the sole test language, while noted as language-agnostic in principle, is unverified on any other language. caveats: [] - id: '2507.04349' published_date: "2025-07-06" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_emotional_synthesis claims: - claim_id: frozen_backbone_adapter_training_can_add_fine_grained_time_varying role: supports claim: Frozen-backbone adapter training can add fine-grained, time-varying conditioning to pretrained flow-matching TTS with substantially less data than full fine-tuning. source: §1, §4.4, Table 5 evidence: ControlNet blocks trained on approximately 400 hours of public emotion data achieve higher Emo-SIM and Aro-Val SIM than EmoCtrl-TTS, which requires 87k hours of training including 27k hours of in-house emotion data, while preserving the backbone's zero-shot voice cloning capability. confidence: high relevance: medium - claim_id: stronger_emotion_conditioning_in_flow_matching_tts_introduces_a_trade role: complicates claim: Stronger emotion conditioning in flow-matching TTS introduces a trade-off with text intelligibility. source: §4.3.4, Table 4 evidence: Increasing control scale lambda from 0 to 1 consistently improves AutoPCP, Emo-SIM, and Aro-Val SIM but raises WER from 2.9% to 9.58% on the EMO-Change benchmark, showing that stronger emotion modulation introduces acoustic variations that degrade phoneme-level precision. confidence: high relevance: high - claim_id: transformer_blocks_in_dit_based_tts_models_contribute_unequally_to role: supports claim: Transformer blocks in DiT-based TTS models contribute unequally to speaker identity and text intelligibility, and block selection is important for conditional control. source: §4.3.1, Figure 2, Table 3 evidence: Layer-wise skip analysis on F5-TTS shows that removing specific blocks dramatically increases WER and reduces speaker similarity; excluding those critical blocks from ControlNet connections yields WER 0% and SIM-o 0.684 versus 8.9% WER and 0.630 SIM-o when all blocks are connected. confidence: high relevance: low - claim_id: emotion_in_a_flow_matching_trajectory_is_concentrated_at_early role: supports claim: Emotion in a flow-matching trajectory is concentrated at early denoising steps, and restricting conditioning to this interval improves both efficiency and intelligibility. source: §4.3.2, Table 1, Table 8 evidence: Training with flow step interval [0, 0.1] achieves Emo-SIM 0.565 and Aro-Val SIM 0.876 with 1.9% WER, whereas training on the full [0, 1] interval degrades to Emo-SIM 0.389 and Aro-Val SIM 0.674 with 0% WER; applying ControlNet only in early steps reduces per-sample inference time from 5.4s to 4.2s. confidence: high relevance: high - claim_id: frame_level_emotion_features_from_self_supervised_ser_models_require role: complicates claim: Frame-level emotion features from self-supervised SER models require temporal smoothing to serve as effective conditioning signals for TTS. source: §4.3.3, Table 2 evidence: Using emotion window size of 1 (no smoothing) produces Emo-SIM 0.500 and WER 4.7%; a window size of 30 achieves Emo-SIM 0.565 and WER 1.9%, demonstrating that token-level SER features without pooling lose emotional coherence. confidence: high relevance: high limitations: - The underlying SER model cannot reliably recognize non-verbal vocalizations (laughing, crying) since these are not well-captured by the arousal-valence-dominance regression framework trained on utterance-level labels. This restricts the expressiveness of the emotion conditioning relative to systems with dedicated non-verbal encoders (e.g., EmoCtrl-TTS with its NV encoder). - Most baseline comparisons use values reported in a prior paper (EmoCtrl-TTS) rather than independently reproduced, limiting the reliability of direct numerical comparisons for closed-source models. On the cross-lingual JVNV S2ST benchmark, TTS-CtrlNet's speaker similarity (0.464) remains behind several baselines trained on much larger data. The method inherits any failure modes of the backbone F5-TTS, including artifacts during high-pitch synthesis and language bias toward English and Chinese. caveats: [] - id: '2507.04598' published_date: "2025-07-07" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: hierarchical_dependency_aware_emotion_prediction_yields_better_emotional_naturalness_and role: supports claim: Hierarchical, dependency-aware emotion prediction yields better emotional naturalness and intelligibility than treating prosodic granularities independently. source: §5.1.1, Table 1 evidence: VA (Multi-Step) achieves MUSHRA naturalness 62.2 vs. 57.5 (single-step VA) with ground-truth ED, and WER 2.45 vs. 4.61 with predicted ED on LibriTTS-R. confidence: high relevance: high - claim_id: multi_level_emotion_intensity_control_when_implemented_at_the_phoneme role: supports claim: Multi-level emotion intensity control, when implemented at the phoneme and word levels, enables fine-grained and quantifiable manipulation of prosodic features consistent with perceptual expectations. source: §5.2, Table 4, Figure 6 evidence: BWS controllability tests show the proposed model more consistently associates low intensity with least-expressive and high intensity with most-expressive ratings across all five emotions compared to MsEmoTTS, and prosodic trend analysis (duration, pitch mean/std, energy) confirms expected acoustic correlates at each level. confidence: high relevance: high - claim_id: sequential_multi_step_prediction_across_hierarchical_levels_introduces_error_accumulation role: complicates claim: Sequential multi-step prediction across hierarchical levels introduces error accumulation that can degrade pitch and duration alignment even when overall naturalness improves. source: §5.1.2, Table 2 evidence: In the VA setting, multi-step prediction does not outperform single-step on pitch distortion or frame disturbance metrics, attributed to cascaded prediction error and increased sensitivity from joint training with ED difference loss. confidence: high relevance: medium - claim_id: the_gap_between_ground_truth_and_text_predicted_emotion_distributions role: refines claim: The gap between ground-truth and text-predicted emotion distributions at the word level is smaller than at the phoneme level, suggesting that phoneme-level emotion is harder to infer from text. source: §5.1.3, Table 3 evidence: Table 3 shows mean absolute ED differences are lower for words than phonemes in the multi-step (Predicted) condition, and the authors interpret this as the model prioritizing cross-segment dependencies over precise ED matching. confidence: high relevance: high limitations: - The system is evaluated exclusively on English data, and the authors identify extension to additional languages as future work. Both training datasets (LibriTTS-R, ESD) are relatively clean, and the method's robustness in noisy or spontaneous conditions is untested. The MsEmoTTS baseline is re-implemented within the FastSpeech2 framework rather than evaluated in its original form, which introduces a potential confound in the emotion editing comparison. The external module's claim of compatibility with arbitrary TTS systems is not verified against any system other than FastSpeech2. Model size and computational cost are not reported. caveats: [] - id: '2507.04817' published_date: "2025-07-07" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - VC architecture: - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - gan_based_expressive_transfer claims: - claim_id: explicit_interpretable_prosodic_conditioning_in_voice_conversion_can_match_the role: supports claim: Explicit, interpretable prosodic conditioning in voice conversion can match the intelligibility and speaker similarity of systems using implicit or disentangled representations, at a fraction of the parameter cost. source: §5.1.1, Table 2 evidence: Fast-VGAN (~3.2M params) achieves WER of 0.000% and cosine similarity 0.648, competitive with ControlVC (~20M params, WER 0.089%, similarity 0.652) and superior to HiFi-VC (~14M params, WER 2.857%) on seen-speaker VCTK conversion. confidence: high relevance: low - claim_id: adapting_target_speaker_specific_prosodic_parameters_pitch_ambitus_speech_rate role: complicates claim: Adapting target-speaker-specific prosodic parameters (pitch ambitus, speech rate) during voice conversion does not reliably improve perceived speaker identity, and combining multiple prosodic adaptations simultaneously degrades both naturalness and similarity. source: §5.2.1, Table 4 evidence: When ambitus and speech rate dilation are applied jointly, subjective MOS naturalness drops from 3.63 to 3.13 and speaker similarity from 3.47 to 2.99, compared to baseline Fast-VGAN conversion without prosodic parameter adaptation. confidence: high relevance: low - claim_id: neutral_to_expressive_speech_resynthesis_can_be_achieved_by_applying role: supports claim: Neutral-to-expressive speech resynthesis can be achieved by applying prosodic contours from expressive reference recordings to a model trained exclusively on neutral speech, without requiring expressive training data. source: §4.3, §5.1.3, Table 3 evidence: Fast-VGAN trained only on neutral Expresso speaker utterances achieves 0.867 ± 0.029 cosine speaker similarity on expressive resynthesis across confused, happy, and sad conditions, with WER (19.46%) comparable to real recordings (19.38%). confidence: high relevance: high - claim_id: extreme_prosodic_transformations_in_gan_based_voice_conversion_degrade_naturalness role: complicates claim: Extreme prosodic transformations in GAN-based voice conversion degrade naturalness and speaker similarity, with performance falling off on both sides of the neutral setting in a bell-shaped pattern. source: §5.2.2, Figure 3 evidence: MOS scores for naturalness and speaker similarity peak at unmodified (neutral) settings and decline for both compression and expansion of duration (up to 3x/0.33x), ambitus (up to ±1 octave), and F0 transposition (up to ±1 octave), as measured in subjective listening tests with approximately 20 participants per condition. confidence: high relevance: low limitations: - 'Speaker generalisation is restricted to seen speakers: speaker identity is encoded as a learned lookup embedding, so the model cannot convert to unseen target speakers without retraining. The evaluation uses only 8 VCTK speakers with 2 utterances per speaker pair -- a narrow test set that may not reflect performance across the full VCTK diversity or cross-corpus speakers.' - No demo or code is reported, limiting reproducibility. The speaker similarity metric uses Resemblyzer cosine embeddings, which may not correlate well with human speaker identity judgements; the subjective similarity results already show some misalignment (Fast-VGAN subjective similarity 3.47 vs ControlVC 2.82, while objective cosine similarity is nearly tied at 0.648 vs 0.652). The expressive synthesis experiment uses only 4 Expresso speakers (2M, 2F) and 3 emotions, leaving generalisation to other speakers and emotional categories open. The MBExWN vocoder introduces its own quality ceiling, and the combined system has not been evaluated on noisy or telephony-quality input speech. caveats: [] - id: '2507.21138' published_date: "2025-07-22" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models claims: - claim_id: rl_alignment_with_composite_perceptual_rewards_improves_multiple_speech_quality role: supports claim: RL alignment with composite perceptual rewards improves multiple speech quality dimensions simultaneously over SFT-only baselines in autoregressive codec TTS. source: §3.5, Table 8 evidence: GRPO with combined WER + speaker similarity + DNSMOS rewards achieves gains in all three metrics over the SFT baseline, while single-reward models improve only their target dimension; TTS-1 WER drops from 7.9% (SFT) to 6.3% (RL-aligned). confidence: high relevance: low - claim_id: audio_pre_training_on_large_scale_raw_speech_substantially_improves role: supports claim: Audio pre-training on large-scale raw speech substantially improves subsequent SFT quality in LLM-based TTS beyond what instruction-tuned LLM initialisation provides. source: §3.4, Figure 5 evidence: Initialising SFT from an audio pre-trained LLaMA-3.2-1B checkpoint yields lower SFT loss, approximately 15% lower WER, and approximately 3% higher speaker similarity than initialising from the base LLaMA-3.2-1B-Instruct checkpoint. confidence: high relevance: medium - claim_id: scaling_speechlm_parameter_count_in_autoregressive_codec_tts_consistently_improves role: supports claim: Scaling SpeechLM parameter count in autoregressive codec TTS consistently improves intelligibility and speaker fidelity across languages. source: §4, Figure 8, Table 8 evidence: TTS-1-Max (8.8B) achieves 5.1% overall WER and higher SIM across all 11 evaluated languages compared to TTS-1 (1.6B) at 6.3% WER, with performance correlating with pre-training loss differences. confidence: high relevance: low - claim_id: style_conditioning_via_discrete_text_tags_conflicts_with_speaker_identity role: complicates claim: Style conditioning via discrete text tags conflicts with speaker identity preservation in single-codebook codec TTS architectures. source: §3.6 evidence: Direct prepending of style markup tags during SFT produced no effect on synthesized speech because the single-codebook design entangles acoustic and semantic information; successful style control required constructing paired neutral/stylized utterances from the same speaker and applying LoRA fine-tuning. confidence: high relevance: low - claim_id: streaming_audio_delivery_in_autoregressive_tts_introduces_audible_artifacts_and role: complicates claim: Streaming audio delivery in autoregressive TTS introduces audible artifacts and volume inconsistencies at segment boundaries that require specific engineering mitigations independent of the generative model's quality. source: §5.1 evidence: Without concatenation restricted to non-voicing regions and context-aware decoder decoding with extended audio prompt context, segment boundaries introduce clicks and volume drops; these are engineering-layer problems independent of SpeechLM quality. confidence: high relevance: low limitations: - Model weights are not publicly released, making independent benchmarking and replication impossible. All evaluations use proprietary or internal test sets; the internal TTS arena covers only English and uses approximately 20 annotators with a modest vote count per pair. - The training data is drawn from a proprietary mixture of public and licensed sources whose exact composition is not disclosed, limiting reproducibility. The evaluation framework does not include standard public TTS benchmarks (e.g., LibriTTS or VCTK test sets), making direct numerical comparison with published systems that do report on these benchmarks difficult. - The paper notes that speaker similarity metrics fluctuate with emotionally expressive speech, suggesting that current automated evaluation protocols may not fully capture perceptual quality in dynamic scenarios. The audio markup system, while effective for English, shows "reduced fidelity" when generalizing style control to non-English languages. Prompt audio caching can cause prosodic bleed from the reference audio into generated speech, and longer sequences generated from short prompts may degrade in quality. caveats: [] - id: '2507.18119' published_date: "2025-07-25" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: modality_alignment_spoken_language_models_can_internalize_paralinguistic_cues_dialect role: supports claim: Modality-alignment spoken language models can internalize paralinguistic cues (dialect, age, emotion, non-speech vocalizations) through staged training without requiring explicit user instructions at inference time. source: §5.2, Table 7, Table 8 evidence: GOAT-SLM achieves 50.73% average dialect-following accuracy across five Chinese dialects without explicit prompting, versus 18.91% for the next-best model; it also reaches 40.91% on non-speech vocal response tasks where all other models score below 10%. confidence: high relevance: high - claim_id: initializing_a_speech_generation_head_from_a_pretrained_llm_text role: supports claim: Initializing a speech generation head from a pretrained LLM text head promotes parameter reuse and can preserve linguistic competence in the speech branch without full retraining. source: §3, §5.1, Table 6 evidence: The Speak module is initialized from Write module weights; the model retains multi-turn dialogue accuracy of 84% despite not training on multi-turn data, attributed to well-aligned input embeddings from this parameter sharing strategy. confidence: high relevance: medium - claim_id: incorporating_paralinguistic_awareness_into_a_spoken_language_model_introduces_a role: complicates claim: Incorporating paralinguistic awareness into a spoken language model introduces a trade-off with general semantic question-answering performance. source: §5.1, Table 4 evidence: GOAT-SLM's capability on general AQA (Table 4) declines compared to models without paralinguistic training, remaining at average level across eight QA datasets; the authors explicitly attribute this to the expanded training objectives. confidence: high relevance: high - claim_id: dialectal_speech_generation_quality_is_uneven_across_dialects_with_closely role: complicates claim: Dialectal speech generation quality is uneven across dialects, with closely related varieties more reliably acquired than typologically distant ones. source: §5.2, Table 10 evidence: In subjective evaluation of dialectal spoken response (Table 10), GOAT-SLM exceeds 90% consistency for Cantonese, Henan, Shanghainese, and Sichuanese, but achieves only 57.2% for northeastern Mandarin, which the authors note is lexically similar to standard Mandarin and may suffer from training ambiguity. confidence: high relevance: medium - claim_id: confidence_based_gradient_masking_during_speech_token_training_can_improve role: supports claim: Confidence-based gradient masking during speech token training can improve pronunciation stability by selectively updating tokens where the model's predictions are reliable. source: §4.3 evidence: GOAT-SLM applies confidence-based automatic gradient masking in Stage 3, masking low-confidence tokens during backpropagation; the technique is credited with significantly enhanced pronunciation stability and speech fidelity. confidence: high relevance: medium limitations: - The evaluation is restricted to Chinese-language dialects and Mandarin-English bilingual content; generalization to non-Chinese languages, especially under dialectal conditions, is untested. The TELEVAL benchmark is developed by the same research group, introducing potential evaluation-training alignment that may not reflect performance on independently constructed benchmarks. - The paper relies on GOAT-TTS to generate synthetic speech training targets in Stage 3-2, creating a dependency on a proprietary tool whose capabilities bound the upper limit of the attribute-aware training data. The paper does not ablate the impact of LLM-initialized speech branch weights versus random initialization in a controlled experiment, leaving the primary architectural design claim without direct causal evidence. - Northeastern Mandarin generates only 57.2% dialectal consistency in subjective evaluation, revealing that dialects with high lexical overlap with the training language (standard Mandarin) are harder to acquire distinctly, likely because the model defaults to standard Mandarin patterns. caveats: [] - id: 2025.findings-acl.101 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: decomposing_conversational_speech_synthesis_into_sequential_emotion_understanding_and_speech role: supports claim: Decomposing conversational speech synthesis into sequential emotion-understanding and speech-generation stages improves expressiveness over direct speech token prediction. source: §6.5, Table 2 evidence: Chain-Talker (DMOS-E 4.239) outperforms GPT-Talker's direct token prediction (3.913) and GPT-Talker_c, which adds emotion understanding without fully chaining it through caption prediction (4.102); the ablation w/o captions (4.084) confirms the caption conditioning is the key factor. confidence: high relevance: high - claim_id: natural_language_emotion_captions_are_more_effective_conditioning_signals_for role: supports claim: Natural language emotion captions are more effective conditioning signals for empathetic speech synthesis than discrete emotion category labels. source: §6.4, §6.5, Tables 1 and 2 evidence: Chain-Talker using empathetic captions (DMOS-E 4.239) outperforms Chain-Talker_e using emotion labels (4.127) and Chain-Talker_s using style labels (4.015); DMOS-C scores also show captions exceed ground-truth label quality (4.462 vs 4.327). confidence: high relevance: high - claim_id: supervised_asr_derived_semantic_speech_tokens_provide_more_interpretable_and role: supports claim: Supervised ASR-derived semantic speech tokens provide more interpretable and expressive conversational speech generation than unsupervised tokens that mix semantic and acoustic content. source: §2.3, §6.5, Table 2 evidence: The paper argues that HuBERT tokens in GPT-Talker contain entangled acoustic information that limits emotional comprehension; Chain-Talker's ASR-VQ tokens improve DDTW from 44.625 to 38.784 and ACC_m from 0.562 to 0.612. confidence: high relevance: high - claim_id: llm_driven_automatic_speech_emotion_captioning_with_multi_level_attribute role: supports claim: LLM-driven automatic speech emotion captioning with multi-level attribute extraction produces higher-quality annotations than single-modality or keyword-based description approaches. source: §6.4, Table 1 evidence: CSS-EmCap (DMOS-C 4.462, SIM_G 0.694, DIS-2 0.296) outperforms Qwen2-Audio (4.212, 0.534, 0.174) and SECap (4.268, 0.617, 0.186); ablations confirm that removing either sentence-level style factors or dialogue-level emotion significantly degrades both quality and diversity. confidence: high relevance: high - claim_id: autoregressive_chain_modeling_for_empathetic_conversational_speech_synthesis_introduces_latency role: complicates claim: Autoregressive chain modeling for empathetic conversational speech synthesis introduces latency that does not yet support real-time interaction requirements. source: §Limitations evidence: Chain-Talker generates average empathetic responses of 2.5 seconds duration on an RTX 4080 with 32 GB VRAM; authors describe this as a gap relative to real-time dialogue and identify streaming inference as a necessary future direction. confidence: high relevance: high limitations: - Chain-Talker is fine-tuned on only 384 hours of dialogue data drawn from daily conversational domains and predominantly young adult speakers. The authors explicitly note it may not capture the speaking styles of children or the elderly, and domain generalisation to non-conversational or non-English settings is untested. - 'Inference latency is a practical bottleneck: the three-stage autoregressive pipeline produces responses averaging 2.5 seconds, which is insufficient for low-latency real-time dialogue. Streaming inference is identified as future work but not yet implemented.' - The subjective evaluations rely on 30 university students rating 50 sentences, which is a small sample. Additionally, the empathetic caption generation pipeline depends on Gemini 1.5 Pro, which is closed-source, making the full CSS-EmCap pipeline difficult to replicate without API access. - The zero-shot voice cloning capability is noted to carry safety risks (voice spoofing), which the authors plan to address through open-source license restrictions. caveats: [] - id: 2025.findings-acl.687 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - singing - TTS architecture: - flow-matching - VAE relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_emotional_synthesis - vae_emotion_style_latents claims: - claim_id: boundary_masking_during_training_improves_zero_shot_singing_synthesis_naturalness role: supports claim: Boundary masking during training improves zero-shot singing synthesis naturalness by forcing the model to learn smooth phoneme and note transitions without relying on precise alignment annotations. source: §3.2, Table 4 evidence: BBC Encoder masks 8 tokens at each phoneme/note boundary; ablating this masking causes CMOS-Q to drop -0.36 in style transfer and -0.39 in style control, the largest single-component quality degradation. confidence: high relevance: medium - claim_id: contrastive_alignment_of_cross_modal_style_embeddings_singing_speech_natural role: supports claim: Contrastive alignment of cross-modal style embeddings (singing, speech, natural language) enables a single encoder to support style transfer, speech-to-singing, and instruction-conditioned synthesis without modality-specific architectures. source: §3.3, Table 4 evidence: Custom Audio Encoder trained with CLIP-style triplet contrastive loss achieves unified style space across modalities; removing it drops CMOS-S by -0.37 (style transfer) and CMOS-C by -0.41 (style control). confidence: high relevance: high - claim_id: language_conditioned_mixture_of_experts_routing_in_a_flow_matching role: supports claim: Language-conditioned mixture-of-experts routing in a flow-matching transformer improves multilingual singing synthesis quality by directing language-family-specific token processing to specialised experts. source: §3.4, Table 4, Table 7 evidence: Lingual-MOE ablation (replaced with standard FFN) causes CMOS-Q to drop -0.29 in style transfer; Stylistic-MOE ablation causes CMOS-S -0.26 and CMOS-C -0.33; four experts per group is the optimal configuration before diminishing returns. confidence: high relevance: low - claim_id: f0_supervision_applied_to_intermediate_transformer_representations_improves_both_synthesis role: supports claim: F0 supervision applied to intermediate transformer representations improves both synthesis quality and style fidelity in zero-shot singing voice synthesis. source: §3.4, Table 4 evidence: Removing F0 supervision from the first block's output causes CMOS-Q -0.33 and CMOS-S -0.24 in style transfer, and CMOS-Q -0.31 and CMOS-C -0.27 in style control. confidence: high relevance: medium - claim_id: multi_level_style_control_via_natural_language_prompts_in_singing role: complicates claim: Multi-level style control via natural language prompts in singing synthesis requires costly manual annotation, limiting dataset scale and introducing labelling errors. source: §6, Appendix B evidence: Style labels (emotion, singing method, vocal range, word-level techniques) annotated by music experts at $300/hour, then converted to natural language via GPT-4o; the paper cites labelling cost and annotation errors as primary limitations constraining generalisation. confidence: high relevance: low limitations: - 'The training and test sets share the same underlying singer pool: 30 "unseen" singers are held out from a pool drawn from the same data collection. This limits the assessment of generalisation to truly out-of-distribution singers or languages not present in training (all 9 test languages are also in training).' - Generation speed does not meet real-time industrial requirements at current inference settings, despite 25-step ODE inference; streaming inference is deferred to future work. - Multi-level style annotation requires music expert labour and is therefore not easily scalable; automatic labelling tools are noted as future work but not yet demonstrated. - The model's potential for misuse in unauthorised dubbing is acknowledged. Vocal watermarking is proposed as a mitigation but has not been implemented in this paper. caveats: [] - id: '2503.11026' published_date: "2025-07-30" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_emotional_synthesis claims: - claim_id: conditioning_a_flow_matching_mel_spectrogram_generator_on_rich_multimodal role: supports claim: Conditioning a flow matching mel-spectrogram generator on rich multimodal speaker representations produces more consistent speaker identity in zero-shot cross-lingual speech synthesis than injecting a single speaker embedding at the vocoder stage. source: §5.4, Table 1 evidence: MAVFlow achieves an average 36% improvement in speaker similarity (SS) over AV2AV across four language pairs on MuAViC, using OT-CFM conditioned on x-vector speaker embeddings plus facial emotion embeddings, while AV2AV uses d-vector conditioning in the vocoder only. confidence: high relevance: low - claim_id: higher_quality_intermediate_mel_spectrogram_synthesis_propagates_benefits_to_downstream role: supports claim: Higher-quality intermediate mel-spectrogram synthesis propagates benefits to downstream talking-face generation even when the face decoder itself is unchanged. source: §5.5, Table 5 evidence: MAVFlow improves LSE-C by +0.87, LSE-D by -0.49, and FID by -0.61 relative to AV2AV on LRS3 visual evaluation, despite using the same Wav2Lip face decoder, suggesting that the mel quality bottleneck affects face sync accuracy. confidence: high relevance: medium - claim_id: visual_emotion_conditioning_is_insufficient_on_its_own_to_improve role: complicates claim: Visual emotion conditioning is insufficient on its own to improve emotion expression in synthesized speech and requires concurrent audio speaker conditioning to be effective. source: §5.6, Table 8 evidence: Adding only visual guidance to the CFM model marginally maintains speaker similarity (SS 0.056 vs 0.057 without guidance) but reduces emotion accuracy from 28.66% to 26.83% on CREMA-D; the combination of audio and visual guidance is needed to reach 36.46%. confidence: high relevance: high - claim_id: paralinguistic_and_linguistic_generation_objectives_are_compatible_in_zero_shot role: supports claim: Paralinguistic and linguistic generation objectives are compatible in zero-shot cross-lingual speech synthesis; improving speaker fidelity does not require sacrificing translation accuracy. source: §5.4, Tables 1 and 3 evidence: MAVFlow maintains competitive ASR-BLEU scores (26.97 Es-En vs 26.57 for AV2AV and 28.66–30.55 for cascaded systems) while substantially improving speaker similarity, using the same unit translation module as AV2AV. confidence: high relevance: high - claim_id: emotion_recognition_accuracy_in_synthesized_cross_lingual_speech_remains_far role: complicates claim: Emotion recognition accuracy in synthesized cross-lingual speech remains far below ground-truth levels even with multimodal conditioning, indicating that paralinguistic preservation is an unsolved challenge. source: §5.4, Table 2; §5.6, Table 7 evidence: MAVFlow achieves 36.46% audio emotion accuracy vs a GT upper bound of 81.95% on CREMA-D, even with dual audio-visual guidance; additional training on an emotion-rich dataset (CREMA-D) raises this to 51.46% but still far below GT. confidence: high relevance: high limitations: - The system relies on emotional cues from facial video alone; audio-side paralinguistics such as prosody and timbre variation are not used as emotion conditioning signals, which may limit emotion transfer when visual input is unavailable or low quality. The unit extractor and unit-to-unit translation modules are adopted unchanged from AV2AV, so improvements in semantic translation quality require addressing those upstream components separately. The Duration Length Regulator interpolates to match source duration, which may introduce length-related artifacts when source and translated speech have substantially different natural lengths. Evaluation is limited to five European languages with English as the target; generalization to typologically distant language pairs (e.g., tonal languages, right-to-left scripts) is untested. caveats: [] - id: 2025.ccl-1.77 published_date: "2025-08-01" entry_date: '2026-07-30' year: 2025 venue: workshop task: - TTS architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_emotional_generation claims: - claim_id: cross_modal_facial_features_can_provide_sufficient_speaker_identity_signal role: supports claim: Cross-modal facial features can provide sufficient speaker identity signal for zero-shot voice generation without reference audio. source: §3.2, Table 1, Table 2 evidence: The cross-modal biometric unseen speaker modelling module maps a face image to a speaker embedding via ResNet50 and a trainable visual network; Id.Acc of 66.87/68.99 on LRS2/GRID exceeds all baselines that require reference audio at inference. confidence: high relevance: low - claim_id: diffusion_based_denoising_in_a_multimodal_tts_dubbing_pipeline_improves role: supports claim: Diffusion-based denoising in a multimodal TTS/dubbing pipeline improves prosodic diversity relative to deterministic autoregressive or attention-based baselines. source: §4.2.4, Figure 4 evidence: Running HFSD-V2C 10 times per speaker produces diverse F0 contours capturing individual accent patterns; Neural Dubber and HPMDubbing produce fixed prosodic distributions. confidence: high relevance: medium - claim_id: hierarchical_multimodal_prosody_modelling_video_audio_and_text_levels_jointly role: supports claim: Hierarchical multimodal prosody modelling (video, audio, and text levels jointly) improves emotion accuracy over methods relying on fewer modalities. source: §4.2.1, §4.2.5, Table 1, Table 2 evidence: Emo.Acc reaches 65.63 on LRS2 and 66.32 on GRID, against 61.46 and 63.66 for HPMDubbing; the ablation (w/o UP) drops Emo.Acc to 22.08/27.64, confirming the prosody module's role. confidence: high relevance: high - claim_id: zero_shot_visual_voice_cloning_accuracy_remains_significantly_below_ground role: complicates claim: Zero-shot visual voice cloning accuracy remains significantly below ground-truth speaker identity, indicating that cross-modal biometric embeddings do not fully replace reference audio. source: §4.2.1, Table 1 evidence: GT Id.Acc on LRS2 is 91.52 vs. HFSD-V2C's 66.87; GT MOS is 4.72 vs. 4.29 for HFSD-V2C, a gap of 0.43 that persists after hierarchical multimodal conditioning. confidence: high relevance: low limitations: - The evaluation is restricted to LRS2 and GRID, both of which contain constrained speaking styles (BBC broadcasts and phonetically structured lab speech), leaving generalisation to spontaneous conversational video undemonstrated. The subjective MOS is collected on only 20 clips rated by 20 evaluators, which is a limited sample for drawing robust conclusions. The model trains on both LRS2 and GRID but the zero-shot claim means unseen speakers at test time, not unseen datasets; the extent of genuine out-of-domain generalisation is not assessed. Code and demos are not released, limiting reproducibility. Finally, the method depends on visible, well-lit face images, which may not be robust in natural video production environments. caveats: [] - id: 2025.sigdial-1.27 published_date: "2025-08-01" entry_date: '2026-07-30' year: 2025 venue: workshop task: - SCA - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: integrating_emotion_conditioned_tts_into_a_spoken_dialogue_system_produces role: supports claim: Integrating emotion-conditioned TTS into a spoken dialogue system produces measurably more appropriate emotional speech than a neutral TTS baseline. source: §3.5, Table 2 evidence: Speech emotion appropriateness scores were 4.10 (proposed) vs. 1.70 (baseline) with a very large effect size (Cohen's d = 3.07, p < 0.001) across 20 human-judged conversations. confidence: high relevance: high - claim_id: sentiment_driven_emotional_speech_control_does_not_straightforwardly_translate_into role: complicates claim: Sentiment-driven emotional speech control does not straightforwardly translate into significantly higher user engagement in short task-oriented interactions. source: §3.5, Table 2 evidence: Engagement scores and number of SDS turns showed large effect sizes favouring the proposed system (d = 0.824, d = 0.831) but failed to reach statistical significance (p = 0.09, p = 0.07) with n = 20 conversations; the authors attribute this partly to abrupt discrete emotion shifts that ignore conversational history. confidence: high relevance: high - claim_id: discrete_context_free_emotion_label_assignment_produces_step_change_tonal role: complicates claim: Discrete, context-free emotion label assignment produces step-change tonal shifts that may undermine conversational naturalness even when individual utterance emotion is appropriate. source: §3.5 evidence: Mean engagement remained around 3.0 (on a 5-point scale), and the authors explicitly note that abrupt emotion transitions without considering prior context may have suppressed engagement gains. confidence: high relevance: high - claim_id: subjective_evaluation_of_emotional_spoken_dialogue_systems_requires_task_specific role: supports claim: Subjective evaluation of emotional spoken dialogue systems requires task-specific metrics beyond standard TTS naturalness; generic MOS-type ratings do not capture emotion appropriateness or conversational engagement. source: §3.3 evidence: The paper proposes a seven-item questionnaire covering RAG quality, task achievement, speech emotion appropriateness, and engagement, arguing that existing evaluation frameworks lack appropriate metrics for social goals in emotional SDSs. confidence: high relevance: high limitations: - The evaluation uses only 20 conversation samples (10 per system) conducted by the authors themselves, which is underpowered for the engagement and turn-count analyses and does not include independent users. Significance could not be established for engagement despite large effect sizes, and the results may not generalise beyond news summarisation. - 'The system uses discrete, context-free emotion tagging: sentiment is predicted independently for each LLM response without tracking emotional trajectory across turns. This produces abrupt shifts and likely suppresses engagement. Future work would benefit from emotion-aware dialogue state tracking and smoother emotion interpolation across conversational turns. Task achievement scores plateaued around 3.0 for both systems, suggesting that retrieval quality and response coherence are the binding constraints for this application, not TTS expressiveness.' caveats: [] - id: '2508.02038' published_date: "2025-08-04" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - VC architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: difference_in_means_emotion_embeddings_computed_from_paired_emotional_and role: supports claim: Difference-in-means emotion embeddings, computed from paired emotional and neutral speech, enable speaker-independent emotion representations that generalise across speakers. source: §2.2 evidence: Difference-in-means emotion embeddings, computed from paired emotional and neutral speech, enable speaker-independent emotion representations that generalise across speakers. confidence: high relevance: high - claim_id: enforcing_orthogonality_between_speaker_and_emotion_embedding_spaces_via_a role: supports claim: Enforcing orthogonality between speaker and emotion embedding spaces via a cross-dot-product loss enables independent control of voice identity and emotional style without degrading either. source: §2.3, Table 1 evidence: Enforcing orthogonality between speaker and emotion embedding spaces via a cross-dot-product loss enables independent control of voice identity and emotional style without degrading either. confidence: high relevance: high - claim_id: unified_joint_modeling_of_voice_cloning_and_emotion_control_produces role: supports claim: Unified joint modeling of voice cloning and emotion control produces higher speaker similarity and emotional naturalness than systems that address the two capabilities separately. source: §4.1, §4.2, Tables 1–2 evidence: Unified joint modeling of voice cloning and emotion control produces higher speaker similarity and emotional naturalness than systems that address the two capabilities separately. confidence: high relevance: high - claim_id: expressive_emotional_tts_systems_that_include_naturalistic_vocalized_fillers_exhibit role: supports claim: Expressive emotional TTS systems that include naturalistic vocalized fillers exhibit higher word error rates on clean-transcript benchmarks, even when perceived speech quality is high. source: §4.4, Table 4–5 evidence: Expressive emotional TTS systems that include naturalistic vocalized fillers exhibit higher word error rates on clean-transcript benchmarks, even when perceived speech quality is high. confidence: high relevance: high - claim_id: gender_imbalance_in_emotional_speech_training_data_produces_systematic_differences role: supports claim: Gender imbalance in emotional speech training data produces systematic differences in emotion recognition accuracy, with male speakers underperforming female speakers across most emotion categories. source: §4.4, Figure 6 evidence: Gender imbalance in emotional speech training data produces systematic differences in emotion recognition accuracy, with male speakers underperforming female speakers across most emotion categories. confidence: high relevance: high limitations: - '- Requires paired emotional/neutral speech data from the same speaker for emotion embedding extraction, which limits scalability and language coverage. - Higher WER on Mandarin AISHELL-3 relative to the CosyVoice1 baseline: the system inserts vocalized fillers (ah, um) that are natural in expressive speech but inflate WER against clean transcripts. - Model size not reported; training efficiency uncharacterized. - Evaluation is limited to 7 emotions — more naturalistic, multi-emotion or low-intensity affects are not tested. - No streaming or latency evaluation.' caveats: [] - id: '2508.03543' published_date: "2025-08-05" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - flow_matching_emotional_synthesis claims: - claim_id: training_free_inference_time_activation_steering_can_achieve_emotion_control role: supports claim: Training-free inference-time activation steering can achieve emotion control in flow-matching TTS models that is competitive with or superior to systems explicitly trained for emotion conditioning. source: §4.2, Table 1 evidence: Training-free inference-time activation steering can achieve emotion control in flow-matching TTS models that is competitive with or superior to systems explicitly trained for emotion conditioning. confidence: high relevance: high - claim_id: pre_trained_flow_matching_tts_models_encode_emotion_as_approximately role: supports claim: Pre-trained flow-matching TTS models encode emotion as approximately linear directions in DiT activation space, enabling vector arithmetic over multiple emotion categories. source: §3, §4.6 evidence: Pre-trained flow-matching TTS models encode emotion as approximately linear directions in DiT activation space, enabling vector arithmetic over multiple emotion categories. confidence: high relevance: high - claim_id: emotion_bearing_information_in_tts_dit_activations_is_concentrated_in role: supports claim: Emotion-bearing information in TTS DiT activations is concentrated in a subset of token positions identifiable via speech emotion recognition scores, and selective steering of these positions outperforms global activation modification. source: §3.3, §4.6 evidence: Emotion-bearing information in TTS DiT activations is concentrated in a subset of token positions identifiable via speech emotion recognition scores, and selective steering of these positions outperforms global activation modification. confidence: high relevance: high - claim_id: activation_steering_for_emotion_control_generalises_across_model_architectures_and role: supports claim: Activation steering for emotion control generalises across model architectures and scales without retuning, producing consistent results on F5-TTS (22 layers), E2-TTS (8 layers), and CosyVoice2 (56 layers). source: §4.2, §4.5 evidence: Activation steering for emotion control generalises across model architectures and scales without retuning, producing consistent results on F5-TTS (22 layers), E2-TTS (8 layers), and CosyVoice2 (56 layers). confidence: high relevance: high - claim_id: training_free_emotion_manipulation_via_activation_steering_incurs_a_moderate role: supports claim: Training-free emotion manipulation via activation steering incurs a moderate speaker similarity cost relative to the unsteered baseline, indicating residual speaker-emotion entanglement in DiT representations. source: §4.2, Table 1 evidence: Training-free emotion manipulation via activation steering incurs a moderate speaker similarity cost relative to the unsteered baseline, indicating residual speaker-emotion entanglement in DiT representations. confidence: high relevance: high limitations: - '- The steering vectors are derived from difference-in-means, which assumes linearity of emotional representations in DiT activations — a plausible but unverified assumption. - Speaker similarity drops slightly in EmoSteer+F5 (S-SIM 0.59) vs unsteered F5 (0.66), suggesting some speaker-emotion coupling in the activations. - The method requires access to DiT intermediate activations, ruling out fully black-box deployment. - The topk selection method depends on emotion2vec quality; errors in emotion token detection propagate to steering performance. - Evaluated on 6 basic emotions only; nuanced or blended emotions outside the training distribution are not assessed. - Laughter, whispering, and other non-categorical paralinguistic phenomena are not covered.' caveats: [] - id: '2508.04195' published_date: "2025-08-06" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: treating_paralinguistic_vocalizations_as_inline_vocabulary_tokens_in_asr_decoding role: supports claim: Treating paralinguistic vocalizations as inline vocabulary tokens in ASR decoding enables joint transcription of lexical and non-verbal content within a single unified sequence, without requiring a separate event detection module. source: §Paralinguistic Aware Speech Recognition evidence: Treating paralinguistic vocalizations as inline vocabulary tokens in ASR decoding enables joint transcription of lexical and non-verbal content within a single unified sequence, without requiring a separate event detection module. confidence: high relevance: high - claim_id: automatically_annotated_data_at_scale_can_match_or_exceed_the role: supports claim: Automatically annotated data at scale can match or exceed the training effectiveness of smaller human-annotated sets for paralinguistic-enhanced TTS, measured by CER and perceptual quality. source: §Main Results, Table 5 evidence: Automatically annotated data at scale can match or exceed the training effectiveness of smaller human-annotated sets for paralinguistic-enhanced TTS, measured by CER and perceptual quality. confidence: high relevance: high - claim_id: fine_tuning_zero_shot_tts_models_with_paralinguistic_vocabulary_extensions role: supports claim: Fine-tuning zero-shot TTS models with paralinguistic vocabulary extensions enables token-level control over non-verbal vocalization placement without material degradation of lexical quality or speaker similarity. source: §Paralinguistic-enhanced TTS Experiments, Table 5 evidence: Fine-tuning zero-shot TTS models with paralinguistic vocabulary extensions enables token-level control over non-verbal vocalization placement without material degradation of lexical quality or speaker similarity. confidence: high relevance: high - claim_id: listener_preference_for_para_enhanced_tts_is_strong_even_when role: supports claim: Listener preference for para-enhanced TTS is strong even when paralinguistic tag recall remains below 65%, suggesting that the mere presence of contextually appropriate non-verbal cues improves perceived naturalness. source: §Human Evaluation, Table 6 evidence: Listener preference for para-enhanced TTS is strong even when paralinguistic tag recall remains below 65%, suggesting that the mere presence of contextually appropriate non-verbal cues improves perceived naturalness. confidence: high relevance: high limitations: - The NVSpeech corpus is restricted to Chinese (Mandarin) and draws primarily from game-style voice acting rather than spontaneous conversational speech, which may limit generalization to natural dialogue scenarios. The auto-labeled data quality depends entirely on the SenseVoice ASR model's precision, and label errors accumulate at scale without a second annotation pass. - Paralinguistic tag recall in TTS reaches only 61.9%, leaving substantial room for improvement in ensuring that specified non-verbal cues are faithfully rendered. The paper evaluates TTS on its own in-domain and open-domain splits rather than a shared benchmark, making direct comparison with other paralinguistic TTS systems (such as EmoCtrl-TTS or ELaTE) difficult. Model size and inference cost are not reported. English experiments (Appendix D) use a small consolidated dataset and a single speaker source, limiting cross-lingual generalization claims. Potential misuse risks from highly naturalistic TTS with non-verbal behaviors (deepfakes, audio manipulation) are acknowledged but not technically mitigated. caveats: [] - id: '2508.04585' published_date: "2025-08-06" entry_date: '2026-07-30' year: 2025 venue: ACM MM task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: matching_the_token_rates_of_speech_and_facial_landmark_codecs role: supports claim: Matching the token rates of speech and facial landmark codecs enables frame-level synchronisation between synthesised speech and talking-face animations without post-hoc alignment. source: §4.1.3, §4.2 evidence: Matching the token rates of speech and facial landmark codecs enables frame-level synchronisation between synthesised speech and talking-face animations without post-hoc alignment. confidence: high relevance: medium - claim_id: llm_based_joint_prediction_of_interleaved_speech_and_visual_tokens role: supports claim: LLM-based joint prediction of interleaved speech and visual tokens in dialogue context outperforms cascaded speech-then-video generation on both emotional accuracy and lip synchronisation. source: §6.2, §6.3, Table 2, Table 3 evidence: LLM-based joint prediction of interleaved speech and visual tokens in dialogue context outperforms cascaded speech-then-video generation on both emotional accuracy and lip synchronisation. confidence: high relevance: high - claim_id: including_visual_dialogue_history_talking_face_animations_of_prior_turns role: supports claim: Including visual dialogue history (talking-face animations of prior turns) improves emotion accuracy and speaker consistency in conversational speech synthesis relative to audio-text-only context. source: §6.4, Table 4 evidence: Including visual dialogue history (talking-face animations of prior turns) improves emotion accuracy and speaker consistency in conversational speech synthesis relative to audio-text-only context. confidence: high relevance: high - claim_id: low_rate_discrete_tokenisation_of_facial_landmarks_1_token_per role: supports claim: Low-rate discrete tokenisation of facial landmarks (1 token per frame) is more effective for LLM contextual modelling than higher-rate representations, even at a cost in geometric reconstruction fidelity. source: §6.1, §6.2, Table 1, Table 2 evidence: Low-rate discrete tokenisation of facial landmarks (1 token per frame) is more effective for LLM contextual modelling than higher-rate representations, even at a cost in geometric reconstruction fidelity. confidence: high relevance: medium - claim_id: emotion_guided_conditioning_of_the_speech_renderer_including_predicted_emotion role: supports claim: Emotion-guided conditioning of the speech renderer, including predicted emotion labels as explicit conditioning, improves measured emotional expressiveness over systems that rely on implicit contextual inference alone. source: §6.4, Table 4 evidence: Emotion-guided conditioning of the speech renderer, including predicted emotion labels as explicit conditioning, improves measured emotional expressiveness over systems that rely on implicit contextual inference alone. confidence: high relevance: high limitations: - The Talking-face Animations Renderer (EchoMimic) is a pre-trained third-party module that receives no additional fine-tuning in this pipeline. Its outputs are constrained by the quality ceiling and biases of the EchoMimic base model, limiting the paper's ability to attribute animation quality gains to UniTalker specifically vs. the renderer. - 'Rendering latency is notable: speech generation takes approximately 2 seconds and animation rendering takes approximately 5 seconds per 25 frames on an RTX 4080 with 32 GB RAM, making the system unsuitable for real-time interaction in its current form. The paper acknowledges this and lists streaming optimisation as future work.' - The training data for dialogue context is limited to 113 hours of audio-only and 307 hours of visual dialogue, which is modest compared to the 170,000-hour pretraining corpus used for the speech renderer. Generalisation to diverse speaking styles and languages beyond the training distribution is untested. All subjective evaluations used 30 non-native English speakers recruited locally, raising questions about evaluation representativeness for naturalness judgements. - The CSVS task definition assumes that the target utterance's text is known at inference time. This is a constrained setting (closer to expressive TTS than to open-ended dialogue response generation) and does not address the harder problem of jointly generating text, speech, and visual responses. caveats: [] - id: '2507.20091' published_date: "2025-08-07" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models claims: - claim_id: codec_token_speech_lms_trained_on_tens_of_thousands_of role: supports claim: Codec-token speech LMs trained on tens of thousands of hours of data do not develop prosodic understanding or generation as emergent capabilities through pre-training alone. source: §1, §4.2–4.4 evidence: Codec-token speech LMs trained on tens of thousands of hours of data do not develop prosodic understanding or generation as emergent capabilities through pre-training alone. confidence: high relevance: low - claim_id: replacing_codec_tokens_with_explicit_interpretable_word_level_prosody_annotations role: supports claim: Replacing codec tokens with explicit, interpretable word-level prosody annotations enables a pre-trained text LLM to acquire prosody processing capabilities — including contrastive focus, emotion recognition, and style transfer — without task-specific fine-tuning. source: §3.1, §4.2–4.4 evidence: Replacing codec tokens with explicit, interpretable word-level prosody annotations enables a pre-trained text LLM to acquire prosody processing capabilities — including contrastive focus, emotion recognition, and style transfer — without task-specific fine-tuning. confidence: high relevance: high - claim_id: disentangling_prosodic_and_linguistic_content_in_the_token_sequence_trades role: supports claim: Disentangling prosodic and linguistic content in the token sequence trades voice quality fidelity for prosodic expressiveness and controllability. source: §3.1, §5 evidence: Disentangling prosodic and linguistic content in the token sequence trades voice quality fidelity for prosodic expressiveness and controllability. confidence: high relevance: medium - claim_id: prosody_to_content_dependencies_detecting_emphasis_or_emotion_from_speech role: supports claim: Prosody-to-content dependencies (detecting emphasis or emotion from speech) are more difficult for codec-token LMs than content-to-prosody generation. source: §4.3, Table 4 evidence: Prosody-to-content dependencies (detecting emphasis or emotion from speech) are more difficult for codec-token LMs than content-to-prosody generation. confidence: high relevance: high - claim_id: word_level_prosody_tokens_impose_only_marginal_degradation_on_text role: supports claim: Word-level prosody tokens impose only marginal degradation on text language modelling capability when interleaved with transcription tokens in a joint sequence. source: §4.5, Table 9 evidence: Word-level prosody tokens impose only marginal degradation on text language modelling capability when interleaved with transcription tokens in a joint sequence. confidence: high relevance: low limitations: - ProsodyLM is trained and evaluated exclusively on audiobooks (Librilight). Audiobook prosody is stylised and read-speech in character; generalisation to spontaneous conversation, emotional dialogue, or cross-domain settings is untested and likely requires substantial data re-collection or domain adaptation. - The decoder is frozen at StyleTTS2 pre-trained weights, which constrains output quality to that system's capability ceiling and limits the range of expressible voice characteristics. The five-dimensional prosody token cannot encode voice quality changes (breathiness, creakiness, vocal effort), which are important for fine-grained expressiveness. - The model is pretrained only; it has no instruction-following capability for arbitrary prosody control prompts at inference time. The global control token provides limited high-level steering but is a coarse mechanism. Whether this paradigm scales to longer contexts, diverse speaking styles, and multi-speaker dialogue settings beyond two-party audiobook narration remains open. caveats: [] - id: '2508.05385' published_date: "2025-08-07" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: automated_annotation_pipelines_for_non_verbal_vocalizations_can_match_or role: supports claim: Automated annotation pipelines for non-verbal vocalizations can match or exceed manually-annotated datasets in downstream NV generation and understanding tasks, while scaling at substantially lower cost. source: §5.1, Table 2; §5.2, Tables 4–5 evidence: Automated annotation pipelines for non-verbal vocalizations can match or exceed manually-annotated datasets in downstream NV generation and understanding tasks, while scaling at substantially lower cost. confidence: high relevance: medium - claim_id: positional_accuracy_of_non_verbal_tag_annotations_is_a_more role: supports claim: Positional accuracy of non-verbal tag annotations is a more important determinant of NV controllability in TTS than dataset size alone. source: §5.1, Table 2 evidence: Positional accuracy of non-verbal tag annotations is a more important determinant of NV controllability in TTS than dataset size alone. confidence: high relevance: medium - claim_id: frame_level_nv_detection_models_trained_exclusively_on_one_language role: supports claim: Frame-level NV detection models trained exclusively on one language can generalise to structurally different languages without retraining, suggesting that acoustic features of non-verbal sounds are largely language-agnostic. source: §2.2; §3 evidence: Frame-level NV detection models trained exclusively on one language can generalise to structurally different languages without retraining, suggesting that acoustic features of non-verbal sounds are largely language-agnostic. confidence: high relevance: low - claim_id: rule_based_nv_data_augmentation_produces_measurably_worse_nv_controllability role: supports claim: Rule-based NV data augmentation produces measurably worse NV controllability in fine-tuned TTS systems compared to models trained on naturally-occurring vocalizations. source: §5.1, Table 2 evidence: Rule-based NV data augmentation produces measurably worse NV controllability in fine-tuned TTS systems compared to models trained on naturally-occurring vocalizations. confidence: high relevance: medium limitations: - The dataset is heavily skewed toward Chinese (over two-thirds of samples), not because the pipeline is language-dependent but because the crawled source material is predominantly Chinese. This imbalance means English NV performance, while competitive, may underperform on broader English benchmarks, and the paper does not provide results on out-of-domain English test sets. - The detection model is trained and evaluated on simulated test data (augmented speech + NV clips), not on naturally-occurring NV events in the wild. Whether the ~91% F1 reflects real-world performance is untested. The evaluation covers only six NV categories despite the dataset spanning ten. NVS is also relatively small (38K samples, ~131 hours) compared to comparable speech datasets used for general TTS; the observed SSIM and WER gaps relative to the base F5-TTS suggest this is a binding constraint. caveats: [] - id: '2508.06890' published_date: "2025-08-09" entry_date: '2026-07-30' year: 2025 venue: ASRU task: - VC architecture: - GAN - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - gan_based_expressive_transfer - transformer_encoder_decoder_emotion_control claims: - claim_id: frame_level_emotion_representations_improve_speaker_emotion_classification_accuracy_and role: supports claim: Frame-level emotion representations improve speaker emotion classification accuracy and prosody transfer fidelity over utterance-level representations in voice conversion. source: §IV.C, Table I evidence: Frame-level emotion representations improve speaker emotion classification accuracy and prosody transfer fidelity over utterance-level representations in voice conversion. confidence: high relevance: high - claim_id: adversarial_disentanglement_via_gradient_reversal_layers_reduces_phonetic_leakage_into role: supports claim: Adversarial disentanglement via gradient reversal layers reduces phonetic leakage into emotion embeddings, improving intelligibility under cross-linguistic-content conversion. source: §IV.C, Table I evidence: Adversarial disentanglement via gradient reversal layers reduces phonetic leakage into emotion embeddings, improving intelligibility under cross-linguistic-content conversion. confidence: high relevance: high - claim_id: explicit_conditioning_on_extracted_prosodic_features_f0_and_energy_from role: supports claim: Explicit conditioning on extracted prosodic features (F0 and energy) from an emotion reference transfers temporal dynamics more faithfully than implicit prediction from latent codes. source: §IV.A, Table I evidence: Explicit conditioning on extracted prosodic features (F0 and energy) from an emotion reference transfers temporal dynamics more faithfully than implicit prediction from latent codes. confidence: high relevance: high - claim_id: training_time_prosody_augmentation_through_temporal_shifting_and_warping_improves role: supports claim: Training-time prosody augmentation through temporal shifting and warping improves robustness of prosody transfer under mismatched reference conditions without sacrificing naturalness. source: §IV.C, Table I evidence: Training-time prosody augmentation through temporal shifting and warping improves robustness of prosody transfer under mismatched reference conditions without sacrificing naturalness. confidence: high relevance: high limitations: - Training and primary evaluation use only the English ESD corpus — 350 parallel utterances across 10 speakers and 5 emotion categories. This is a narrow domain; generalisation to spontaneous, noisy, or multilingual emotional speech is entirely untested. - The small, parallel ESD corpus makes it difficult to assess whether the disentanglement holds under more naturalistic or non-parallel conditions. The ablation study evaluates the seen scenario only; it is not clear whether the ablated variants degrade similarly on unseen speakers and emotions. Speaker classification accuracy (SCA) is reported as a zero-shot metric for the seen-speaker scenario but becomes undefined for unseen speakers, so that dimension of the zero-shot evaluation lacks a corresponding metric. Model size and inference speed are not reported, which matters for the real-time dubbing applications the paper motivates. caveats: [] - id: '2508.07273' published_date: "2025-08-10" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: training_speech_llms_on_data_that_jointly_encodes_contextual_and role: supports claim: Training Speech-LLMs on data that jointly encodes contextual and paralinguistic reasoning substantially outperforms training on isolated paralinguistic QA templates, even when the underlying speech encoder is frozen. source: §V.B.1, Table III evidence: Training Speech-LLMs on data that jointly encodes contextual and paralinguistic reasoning substantially outperforms training on isolated paralinguistic QA templates, even when the underlying speech encoder is frozen. confidence: high relevance: high - claim_id: incorporating_dimensional_emotion_annotations_valence_arousal_dominance_alongside_categorical_labels role: supports claim: Incorporating dimensional emotion annotations (valence, arousal, dominance) alongside categorical labels into LLM-generated QA data diversifies training supervision and improves generalisation to complex emotional states. source: §II.D, §V.B.1 evidence: Incorporating dimensional emotion annotations (valence, arousal, dominance) alongside categorical labels into LLM-generated QA data diversifies training supervision and improves generalisation to complex emotional states. confidence: high relevance: high - claim_id: explicitly_injecting_emotion_metadata_into_inference_prompts_can_partially_compensate role: complicates claim: Explicitly injecting emotion metadata into inference prompts can partially compensate for a model's limited intrinsic paralinguistic understanding, but training on contextual-paralinguistic data yields more robust generalisation across question types. source: §V.B.3, Fig. 2 evidence: Explicitly injecting emotion metadata into inference prompts can partially compensate for a model's limited intrinsic paralinguistic understanding, but training on contextual-paralinguistic data yields more robust generalisation across question types. confidence: high relevance: high - claim_id: llm_judge_scores_for_open_ended_speech_language_model_evaluation role: supports claim: LLM judge scores for open-ended speech-language model evaluation correlate reliably with classification-based accuracy and F1 metrics on questions with deterministic answers, supporting their use as a proxy metric. source: §IV, §V.B.4, Table V evidence: LLM judge scores for open-ended speech-language model evaluation correlate reliably with classification-based accuracy and F1 metrics on questions with deterministic answers, supporting their use as a proxy metric. confidence: high relevance: low limitations: - The CPQA training data is derived from a proprietary in-house movie and TV dataset that is not publicly released, making exact replication of the training setup impossible for external researchers. - Emotion labels used both in training and inference prompts come from SER models rather than ground-truth annotations, introducing noise that may suppress performance on direct classification tasks while still benefiting contextual reasoning. The LLM-generated CPQA evaluation set contains evaluation confounds — direct emotion questions that benefit disproportionately from explicit metadata injection — which the authors flag but do not resolve in the current work, requiring stricter QA generation controls in follow-up. - 'The evaluation scope is narrow: CPQA and emotion-PQA benchmarks only. Performance on other SCA capabilities (dialogue coherence, turn-taking, ASR) is not assessed, so it is unclear whether the CPQA training data causes regression elsewhere. All experiments use a single base architecture (MERaLiON); generalisability to other Speech-LLM frameworks remains untested.' caveats: [] - id: '2508.07302' published_date: "2025-08-10" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: language_agnostic_emotional_embeddings_from_pre_trained_models_can_serve role: supports claim: Language-agnostic emotional embeddings from pre-trained models can serve as a reliable cross-lingual retrieval signal for zero-shot emotion transfer in TTS. source: §III.C, §IV.C evidence: Language-agnostic emotional embeddings from pre-trained models can serve as a reliable cross-lingual retrieval signal for zero-shot emotion transfer in TTS. confidence: high relevance: high - claim_id: retrieval_augmented_prompting_reduces_foreign_accent_artefacts_in_cross_lingual role: supports claim: Retrieval-augmented prompting reduces foreign-accent artefacts in cross-lingual emotional speech synthesis more effectively than direct prosody transfer between typologically distant languages. source: §III.C, §IV.E evidence: Retrieval-augmented prompting reduces foreign-accent artefacts in cross-lingual emotional speech synthesis more effectively than direct prosody transfer between typologically distant languages. confidence: high relevance: high - claim_id: flow_matching_alignment_between_discrete_codec_tokens_and_mel_spectrograms role: supports claim: Flow-matching alignment between discrete codec tokens and mel-spectrograms improves speaker identity preservation as well as prosodic naturalness in multilingual synthesis. source: §III.B, §IV.E evidence: Flow-matching alignment between discrete codec tokens and mel-spectrograms improves speaker identity preservation as well as prosodic naturalness in multilingual synthesis. confidence: high relevance: low - claim_id: clustering_based_retrieval_strategies_over_large_emotional_speech_pools_maintain role: supports claim: Clustering-based retrieval strategies over large emotional speech pools maintain higher accuracy and lower latency than exhaustive cosine similarity search as pool size grows. source: §IV.D, Table II evidence: Clustering-based retrieval strategies over large emotional speech pools maintain higher accuracy and lower latency than exhaustive cosine similarity search as pool size grows. confidence: high relevance: high - claim_id: two_stage_fine_tuning_first_on_phonetics_then_on_expressiveness role: supports claim: Two-stage fine-tuning — first on phonetics, then on expressiveness — enables effective emotion adaptation in low-resource target languages from a strong multilingual foundation model. source: §III.D, §IV.A evidence: Two-stage fine-tuning — first on phonetics, then on expressiveness — enables effective emotion adaptation in low-resource target languages from a strong multilingual foundation model. confidence: high relevance: high limitations: - All evaluation is conducted on internal, non-public datasets with a narrow test configuration (one Chinese speaker, proprietary Thai data). There is no standard benchmark, no released code, and no cross-lab reproducibility path. Claims about emotion transfer quality are difficult to verify independently. - The evaluation covers only the Chinese-to-Thai direction. The paper claims the framework is language-agnostic, but this is stated as future work rather than demonstrated. The listener panel is small (15 raters) and the Thai subset is particularly small (5 raters), raising questions about statistical reliability. The EMOS metric used here is a custom MOS variant not directly comparable to published results elsewhere. Comparisons with Typhoon2-Audio are limited to CER only, leaving the emotional quality comparison against a strong 8B-parameter baseline unanswered. caveats: [] - id: '2508.08095' published_date: "2025-08-11" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: disentangling_paralinguistic_and_linguistic_information_through_separate_adapter_architectures_improves role: supports claim: Disentangling paralinguistic and linguistic information through separate adapter architectures improves an SLM's ability to perceive both modalities independently without modifying the underlying encoder or LLM. source: §III.A, §V.A, Table I evidence: Disentangling paralinguistic and linguistic information through separate adapter architectures improves an SLM's ability to perceive both modalities independently without modifying the underlying encoder or LLM. confidence: high relevance: high - claim_id: training_time_randomisation_over_paralinguistic_embedding_sources_speech_text_caption role: supports claim: Training-time randomisation over paralinguistic embedding sources (speech, text caption, or absent) can prevent adapter collapse into task-specific vectors and preserve contextual generalisation. source: §III.C, §V.B evidence: Training-time randomisation over paralinguistic embedding sources (speech, text caption, or absent) can prevent adapter collapse into task-specific vectors and preserve contextual generalisation. confidence: high relevance: high - claim_id: parameter_efficient_adapter_only_training_on_a_frozen_llm_is role: supports claim: Parameter-efficient adapter-only training on a frozen LLM is competitive with full fine-tuning approaches for emotional dialogue tasks when the information encoding is structured by design. source: §V.A, Table II–III, §V.B, Table IV evidence: Parameter-efficient adapter-only training on a frozen LLM is competitive with full fine-tuning approaches for emotional dialogue tasks when the information encoding is structured by design. confidence: high relevance: high - claim_id: automatic_llm_as_judge_evaluation_of_emotional_conversation_quality_does role: supports claim: Automatic LLM-as-judge evaluation of emotional conversation quality does not substitute for human subjective evaluation; score magnitudes are judge-model-dependent. source: §IV.B, §V.B evidence: Automatic LLM-as-judge evaluation of emotional conversation quality does not substitute for human subjective evaluation; score magnitudes are judge-model-dependent. confidence: high relevance: high limitations: - The system generates text responses, not speech. Despite targeting emotional spoken conversation, the output modality is text only — the emotional response is encoded in linguistic content and style, not in synthesised speech prosody. This limits applicability to fully spoken dialogue pipelines. - The evaluation relies entirely on LLM-as-judge metrics (CS Score, EGS Score) with no human listening tests, making quality estimates harder to interpret and compare across papers. The StyleTalk test set is small, and the evaluation extension (reversing assistant/user roles) is non-standard. - The paper does not report model parameter counts for the adapter components, making efficiency claims difficult to quantify precisely. Code and pretrained models are not publicly released (at time of submission). - ERR assumes that each task type primarily relies on one information type, but in practice multi-dimensional emotional speech may engage both simultaneously; the degree to which the disentanglement generalises beyond the training task taxonomy is untested. caveats: [] - id: '2504.12867' published_date: "2025-08-13" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: fine_grained_natural_language_emotion_descriptions_provide_richer_control_over role: complicates claim: Fine-grained natural language emotion descriptions provide richer control over expressive speech synthesis than coarse categorical labels, at the cost of requiring emotion-specific training data. source: §2.1, §6.1 evidence: Fine-grained natural language emotion descriptions provide richer control over expressive speech synthesis than coarse categorical labels, at the cost of requiring emotion-specific training data. confidence: high relevance: high - claim_id: parallel_phoneme_token_prediction_as_a_secondary_output_head_reduces role: supports claim: Parallel phoneme token prediction as a secondary output head reduces intelligibility errors in LLM-based TTS, particularly on challenging inputs such as rare words and tongue twisters. source: §6.2.1, §6.2.2, Table 5, Table 6 evidence: Parallel phoneme token prediction as a secondary output head reduces intelligibility errors in LLM-based TTS, particularly on challenging inputs such as rare words and tongue twisters. confidence: high relevance: medium - claim_id: llm_pretraining_initialisation_meaningfully_benefits_emotion_controllable_tts_models_without role: supports claim: 'LLM pretraining initialisation meaningfully benefits emotion-controllable TTS: models without it show substantially higher word error rates and weaker emotion transfer.' source: §6.2.4, Table 8 evidence: 'LLM pretraining initialisation meaningfully benefits emotion-controllable TTS: models without it show substantially higher word error rates and weaker emotion transfer.' confidence: high relevance: high - claim_id: automatic_emotion_similarity_metrics_e_g_emotion2vec_cosine_similarity_correlate role: supports claim: Automatic emotion similarity metrics (e.g. emotion2vec cosine similarity) correlate reasonably at the system level but poorly at the utterance level with human perceptual judgments, limiting their utility for fine-grained model comparison. source: §7, Table 10 evidence: Automatic emotion similarity metrics (e.g. emotion2vec cosine similarity) correlate reasonably at the system level but poorly at the utterance level with human perceptual judgments, limiting their utility for fine-grained model comparison. confidence: high relevance: high - claim_id: multimodal_llms_are_not_yet_reliable_judges_of_emotional_speech role: supports claim: Multimodal LLMs are not yet reliable judges of emotional speech quality, exhibiting both low correlation with human ratings and inter-run instability. source: §7, Table 10 evidence: Multimodal LLMs are not yet reliable judges of emotional speech quality, exhibiting both low correlation with human ratings and inter-run instability. confidence: high relevance: high limitations: - 'The English model is trained and evaluated entirely on synthetic data generated by GPT-4o-audio. Both EmoVoice-DB (training) and the test set are GPT-4o-audio outputs, creating circularity: the model learns to mimic GPT-4o-audio''s synthesis style rather than natural human emotional speech. Generalisation to real human emotional recordings or to out-of-distribution TTS systems is undemonstrated.' - The emotion recall evaluation omits three of seven emotion categories (disgusted, fearful, surprised) due to low recognition accuracy from emotion2vec, which limits the scope of the emotional expressiveness claims. - 'EmoVoice does not support zero-shot speaker generalisation: speaker identity is controlled via a reference speech prompt passed to the CosyVoice flow-matching module, but the system is not evaluated in a cross-speaker zero-shot scenario.' - The phoneme-boost variant (EmoVoice-PP) requires phoneme sequences at training time, adding a preprocessing dependency (Phonemizer) and making extension to languages with complex or poorly supported phonemisers non-trivial. - No efficiency or latency analysis is provided; the streaming characteristics of the grouped decoding approach relative to real-time deployment requirements are not reported. caveats: [] - id: '2508.09600' published_date: "2025-08-13" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: explicit_chain_of_thought_reasoning_over_paralinguistic_cues_emotion_age role: supports claim: Explicit chain-of-thought reasoning over paralinguistic cues (emotion, age, gender, sound events) improves empathetic response generation in speech-to-speech dialogue systems. source: §OSUM-EChat "Training", Stage 3 Empathy; Table 2 evidence: Explicit chain-of-thought reasoning over paralinguistic cues (emotion, age, gender, sound events) improves empathetic response generation in speech-to-speech dialogue systems. confidence: high relevance: high - claim_id: pretraining_on_multitask_speech_understanding_before_speech_to_speech_dialogue role: supports claim: Pretraining on multitask speech understanding before speech-to-speech dialogue training reduces dependence on large-scale paired dialogue datasets while maintaining paralinguistic modelling quality. source: §OSUM-EChat "Training", Stage 1; Table 2 evidence: Pretraining on multitask speech understanding before speech-to-speech dialogue training reduces dependence on large-scale paired dialogue datasets while maintaining paralinguistic modelling quality. confidence: high relevance: high - claim_id: synthetic_speech_to_speech_data_derived_from_tts_systems_exhibits role: supports claim: Synthetic speech-to-speech data derived from TTS systems exhibits reduced emotional expressiveness compared to real human speech, creating a systematic domain gap that degrades empathetic dialogue evaluation. source: §EChat-200K Dataset evidence: Synthetic speech-to-speech data derived from TTS systems exhibits reduced emotional expressiveness compared to real human speech, creating a systematic domain gap that degrades empathetic dialogue evaluation. confidence: high relevance: high - claim_id: automatic_empathy_evaluation_pipelines_using_llm_scoring_and_automatic_emotion role: supports claim: Automatic empathy evaluation pipelines using LLM scoring and automatic emotion classifiers diverge measurably from human judgements, primarily due to emotion classifier errors and LLM hallucinations. source: §Main Results "Results of Empathetic Intelligence"; Table 3 evidence: Automatic empathy evaluation pipelines using LLM scoring and automatic emotion classifiers diverge measurably from human judgements, primarily due to emotion classifier errors and LLM hallucinations. confidence: high relevance: high - claim_id: native_multimodal_models_that_integrate_speech_token_prediction_directly_into role: supports claim: Native multimodal models that integrate speech token prediction directly into the LLM are better suited to capturing and generating paralinguistic nuance than modularly aligned architectures that treat speech decoding separately. source: §Introduction; §Related Work "End-to-End Spoken Dialogue System" evidence: Native multimodal models that integrate speech token prediction directly into the LLM are better suited to capturing and generating paralinguistic nuance than modularly aligned architectures that treat speech decoding separately. confidence: high relevance: high limitations: - 'The EChat-200K dataset is almost entirely synthetic: query audio is generated by CosyVoice2 and response audio likewise, with real recordings comprising only a minority of the data. The authors acknowledge that emotional expressiveness of synthesised audio lags real human speech, and the training corpus does not include dynamic paralinguistic scenarios (e.g. emotional transitions, multi-speaker interactions). Generalisability to natural in-the-wild speech remains unvalidated.' - 'The EChat-eval automatic scoring pipeline — combining GPT-4o and emotion2vec-Large — produces rankings consistent with human evaluation but absolute scores that diverge meaningfully. The benchmark is therefore more reliable for ranking systems than for measuring absolute empathy. Sound event capability is evaluated only within the categories present in EChat-200K, which may not reflect the diversity of real conversational contexts. General linguistic intelligence (UltraEval-Audio, Table 4) regresses compared to the Qwen2.5-3B base: GSM8K drops from 85 to 34 after the multi-stage training, indicating that the speech capability gain comes at a significant cost to LLM reasoning.' caveats: [] - id: '2508.11273' published_date: "2025-08-15" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: discretizing_self_supervised_speech_features_via_k_means_produces_more role: supports claim: Discretizing self-supervised speech features via k-means produces more stable prosodic control signals than retaining continuous SSL representations in encoder-decoder TTS. source: §5.6, Table 1 evidence: Discretizing self-supervised speech features via k-means produces more stable prosodic control signals than retaining continuous SSL representations in encoder-decoder TTS. confidence: high relevance: medium - claim_id: combining_continuous_spherical_emotion_vectors_with_discrete_ssl_prosody_tokens role: supports claim: Combining continuous spherical emotion vectors with discrete SSL prosody tokens improves both naturalness and intelligibility compared to spherical emotion vectors alone. source: §5.1, §5.3, Tables 1–2 evidence: Combining continuous spherical emotion vectors with discrete SSL prosody tokens improves both naturalness and intelligibility compared to spherical emotion vectors alone. confidence: high relevance: high - claim_id: speaker_independent_prosody_conditioning_via_ssl_tokens_can_generalize_across role: supports claim: Speaker-independent prosody conditioning via SSL tokens can generalize across reference speakers with minimal degradation, enabling robust cross-speaker emotion transfer. source: §5.6, Table 1 evidence: Speaker-independent prosody conditioning via SSL tokens can generalize across reference speakers with minimal degradation, enabling robust cross-speaker emotion transfer. confidence: high relevance: high - claim_id: semantic_text_encoders_contribute_to_emotional_and_prosodic_consistency_in role: supports claim: Semantic text encoders contribute to emotional and prosodic consistency in multilingual TTS even when they do not serve as direct decoder inputs. source: §5.6, Table 1 evidence: Semantic text encoders contribute to emotional and prosodic consistency in multilingual TTS even when they do not serve as direct decoder inputs. confidence: high relevance: high limitations: - '- Evaluations are limited to single-speaker datasets in two languages, making it unclear whether EmoSSLSphere generalises to multi-speaker, low-resource, or unseen-language scenarios. - Subjective listener panels are small, and emotional authenticity is assessed primarily via AVD RMSE as a proxy rather than direct perceptual emotion ratings. - Speaker similarity (SPK-SIM) is not evaluated, making it hard to quantify speaker fidelity claims. - Cross-speaker emotion transfer is described but not formally evaluated; inference always uses same-speaker reference audio. - Separate per-language encoder instances do not scale to many-language settings without significant parameter overhead. - Integration with semi-supervised training (EmoSphere++) and extension to zero-shot speaker scenarios are listed as future work.' caveats: [] - id: interspeech-2025-0166 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: training_a_speech_encoder_to_match_llm_responses_conditioned_on role: supports claim: Training a speech encoder to match LLM responses conditioned on emotion labels can transfer paralinguistic understanding to a frozen LLM without any LLM weight modification. source: §2.3, §4.1, §4.2, Table 1 evidence: SpeechEmotionLlama achieves an LLM-annotated emotion understanding score of 7.54 versus 5.83 for the best cascaded baseline, and 81.51% SER accuracy versus 80.81% for the standalone SER model, while keeping Llama 3 8B entirely frozen throughout training. confidence: high relevance: high - claim_id: cascading_a_high_accuracy_standalone_emotion_classifier_onto_a_speech role: complicates claim: Cascading a high-accuracy standalone emotion classifier onto a speech-LLM system provides only marginal improvement in emotional response quality. source: §3.2, §4.1, Table 1 evidence: Prepending SER model predictions (80.81% accuracy) to Baseline 2 raises the LLM-annotated emotion score from 5.59 to only 5.83, while encoder-level alignment raises it to 7.54, suggesting that discretized emotion tags do not capture the richness of paralinguistic information as effectively as continuous encoder embeddings. confidence: high relevance: high - claim_id: evaluating_paralinguistic_speech_llm_systems_is_inherently_difficult_when_training role: complicates claim: Evaluating paralinguistic speech-LLM systems is inherently difficult when training and evaluation data are fully proprietary. source: §3.1, §3.3 evidence: All datasets (pre-training, fine-tuning, and test sets) are proprietary, precluding external replication; the LLM-based evaluation metric (Llama 3 70B as judge) also introduces a dependency on the evaluator model's own behavior, which is not systematically validated against human listeners. confidence: high relevance: high - claim_id: self_supervised_speech_pre_training_on_large_scale_multilingual_data role: supports claim: Self-supervised speech pre-training on large-scale multilingual data provides a strong initialization for downstream paralinguistic understanding in speech encoders. source: §2.2, §3.2, §4.2 evidence: The Conformer encoder is pre-trained with BEST-RQ on approximately 15M hours before fine-tuning on emotion tasks; the system substantially outperforms baselines that use a speech encoder that is not further fine-tuned on emotion-related tasks, indicating the pre-training provides useful representations that emotion-specific fine-tuning can build upon. confidence: high relevance: high limitations: - All training and evaluation data are proprietary; results cannot be replicated externally and comparisons are restricted to in-house baselines. The role of data scale (1B-parameter encoder, 15M hours pre-training) versus the training paradigm itself is not ablated. - The system is evaluated only on English expressive speech from voice actors and a large but single-language corpus, leaving generalization to other languages, accents, and naturalistic (non-acted) expressive speech open. The test set is intentionally constructed to control for linguistic content by using utterances where the same sentence is spoken in multiple styles, which may not reflect realistic distribution. LLM-judged metrics (Llama 3 70B as scorer) have not been validated against human listener judgments for empathy or response quality in this setup. caveats: [] - id: interspeech-2025-0203 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - VC architecture: - flow-matching - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_emotional_synthesis - transformer_encoder_decoder_emotion_control claims: - claim_id: natural_language_prompts_can_control_emotional_voice_conversion_at_parity role: supports claim: Natural language prompts can control emotional voice conversion at parity with reference speech for the majority of listeners, reducing reliance on hard-to-source reference audio. source: §3.2.2 evidence: Natural language prompts can control emotional voice conversion at parity with reference speech for the majority of listeners, reducing reliance on hard-to-source reference audio. confidence: high relevance: high - claim_id: flow_matching_produces_noticeably_higher_speech_naturalness_and_audio_quality role: supports claim: Flow matching produces noticeably higher speech naturalness and audio quality in emotional voice conversion than GAN and autoencoder baselines. source: §3.2.1, Table 1 evidence: Flow matching produces noticeably higher speech naturalness and audio quality in emotional voice conversion than GAN and autoencoder baselines. confidence: high relevance: high - claim_id: combining_categorical_emotion_labels_with_free_form_prompt_labels_through role: supports claim: Combining categorical emotion labels with free-form prompt labels through soft-label contrastive training improves emotion embedding quality over prompt-only or label-only training. source: §3.3, Table 2 evidence: Combining categorical emotion labels with free-form prompt labels through soft-label contrastive training improves emotion embedding quality over prompt-only or label-only training. confidence: high relevance: high - claim_id: an_explicit_scalar_intensity_gate_applied_to_emotional_embeddings_before role: supports claim: An explicit scalar intensity gate applied to emotional embeddings before content-emotion fusion improves both naturalness and emotion similarity in converted speech. source: §3.3, Table 2 evidence: An explicit scalar intensity gate applied to emotional embeddings before content-emotion fusion improves both naturalness and emotion similarity in converted speech. confidence: high relevance: high limitations: - The system is trained and evaluated entirely on a proprietary internal Mandarin corpus. No open-source data or model weights are released, and no cross-lingual or multi-speaker generalisation is tested. - Comparisons are restricted to older GAN and autoencoder baselines (StarGAN-EVC, Seq2seq-EVC, MixEmo); no diffusion-based or recent flow-matching EVC systems are included, so the claimed state-of-the-art position cannot be verified against the most competitive contemporaries. The evaluation is any-to-one (fixed target speaker identity), leaving any-to-any EVC performance unaddressed. Emotion coverage is limited to seven categorical classes; whether the natural language conditioning generalises to subtler or blended emotional states is untested. caveats: [] - id: interspeech-2025-0408 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control - gan_based_expressive_transfer claims: - claim_id: combining_paralinguistic_intimacy_adaptation_with_linguistic_register_change_improves_user role: supports claim: Combining paralinguistic intimacy adaptation with linguistic register change improves user satisfaction more than linguistic-only adaptation in multi-session dialogue. source: §5.2, Table 2 evidence: Combining paralinguistic intimacy adaptation with linguistic register change improves user satisfaction more than linguistic-only adaptation in multi-session dialogue. confidence: high relevance: high - claim_id: intimacy_conditioned_tts_trained_on_spontaneous_conversational_data_can_reproduce role: supports claim: Intimacy-conditioned TTS trained on spontaneous conversational data can reproduce prosodic features (speech rate and F0) that distinguish high- and low-intimacy speech. source: §5.1 evidence: Intimacy-conditioned TTS trained on spontaneous conversational data can reproduce prosodic features (speech rate and F0) that distinguish high- and low-intimacy speech. confidence: high relevance: medium - claim_id: binary_intimacy_labels_are_sufficient_to_produce_measurable_prosodic_differences role: supports claim: Binary intimacy labels are sufficient to produce measurable prosodic differences in synthesised speech, suggesting that coarse-grained conditioning captures style variation in spontaneous dialogue. source: §5.1 evidence: Binary intimacy labels are sufficient to produce measurable prosodic differences in synthesised speech, suggesting that coarse-grained conditioning captures style variation in spontaneous dialogue. confidence: high relevance: low - claim_id: paralinguistic_intimacy_may_be_less_effectively_conveyed_through_speech_prosody role: supports claim: Paralinguistic intimacy may be less effectively conveyed through speech prosody alone than through linguistic register, as gains in friendliness ratings remain modest even when prosodic style shifts. source: §5.2 evidence: Paralinguistic intimacy may be less effectively conveyed through speech prosody alone than through linguistic register, as gains in friendliness ratings remain modest even when prosodic style shifts. confidence: high relevance: high limitations: - Only binary (high/low) intimacy labels are used; a continuous intimacy scale might enable finer-grained adaptation. The experiment spans only 3 days; longer interactions may reveal larger effects. The CG agent's gesture and emotion were frozen, which may have suppressed multimodal intimacy cues. The study is Japanese-language only, limiting generalizability. Future work intends to incorporate multimodal behaviors (gestures, facial expressions) and longer-term experiments. caveats: [] - id: interspeech-2025-0648 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: automated_multimodal_annotation_pipelines_can_achieve_annotation_consistency_significantly_exceeding role: supports claim: Automated multimodal annotation pipelines can achieve annotation consistency significantly exceeding that of human raters on emotional speech datasets, at substantially lower cost per hour. source: §2.3, Table 1 evidence: Automated multimodal annotation pipelines can achieve annotation consistency significantly exceeding that of human raters on emotional speech datasets, at substantially lower cost per hour. confidence: high relevance: high - claim_id: fine_grained_emotion_datasets_with_a_larger_category_space_and role: supports claim: Fine-grained emotion datasets with a larger category space and higher annotation consistency lead to measurable improvements in downstream emotional TTS quality compared to smaller manually annotated alternatives. source: §3.2, Table 3 evidence: Fine-grained emotion datasets with a larger category space and higher annotation consistency lead to measurable improvements in downstream emotional TTS quality compared to smaller manually annotated alternatives. confidence: high relevance: high - claim_id: visual_preprocessing_is_a_substantial_contributor_to_multimodal_emotion_recognition role: supports claim: Visual preprocessing is a substantial contributor to multimodal emotion recognition accuracy, as removing it causes a measurable drop in classification performance. source: §2.1 evidence: Visual preprocessing is a substantial contributor to multimodal emotion recognition accuracy, as removing it causes a measurable drop in classification performance. confidence: high relevance: high - claim_id: the_limited_emotion_category_sets_in_standard_ser_benchmarks_introduce role: complicates claim: The limited emotion category sets in standard SER benchmarks introduce systematic classification confusion for emotionally adjacent categories that are not psychologically distinct. source: §2.3 evidence: The limited emotion category sets in standard SER benchmarks introduce systematic classification confusion for emotionally adjacent categories that are not psychologically distinct. confidence: high relevance: high limitations: - MIKU-PAL accuracy (58.6–68.5%) remains below human accuracy on IEMOCAP (72.9%). Performance depends on the capabilities and potential biases of Gemini 2.0 Flash, which may change across model versions. YouTube-sourced data introduces demographic and cultural biases from the content distribution. Current SER models cannot reliably classify all 26 fine-grained emotion categories, so full evaluation of MIKU-EmoBench is limited. The pipeline retains only 42% of raw video, which is efficient but still discards a large fraction. Multilingual generalization beyond English is not demonstrated. caveats: [] - id: interspeech-2025-0704 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: computing_rewards_directly_on_codec_tokens_rather_than_synthesized_audio role: complicates claim: Computing rewards directly on codec tokens rather than synthesized audio reduces the computational cost of RLHF for codec LM-based TTS systems without sacrificing reward signal quality. source: §3.1 evidence: Computing rewards directly on codec tokens rather than synthesized audio reduces the computational cost of RLHF for codec LM-based TTS systems without sacrificing reward signal quality. confidence: high relevance: low - claim_id: differentiable_reward_optimization_via_gumbel_softmax_can_replace_ppo_style role: supports claim: Differentiable reward optimization via Gumbel-Softmax can replace PPO-style policy gradient training for codec LM fine-tuning, enabling direct backpropagation through the reward model. source: §3.1, §2.2 evidence: Differentiable reward optimization via Gumbel-Softmax can replace PPO-style policy gradient training for codec LM fine-tuning, enabling direct backpropagation through the reward model. confidence: high relevance: low - claim_id: a_multi_task_reward_model_trained_on_codec_tokens_can role: supports claim: A multi-task reward model trained on codec tokens can simultaneously improve pronunciation accuracy, emotion expressiveness, and audio quality in a single RL training phase. source: §3.2, §4.3 evidence: A multi-task reward model trained on codec tokens can simultaneously improve pronunciation accuracy, emotion expressiveness, and audio quality in a single RL training phase. confidence: high relevance: high - claim_id: rlhf_fine_tuning_on_mandarin_and_english_data_generalizes_to role: supports claim: RLHF fine-tuning on Mandarin and English data generalizes to unseen languages (Japanese, Korean) when the reward signal targets phonetic accuracy rather than language-specific patterns. source: §4.2, Table 2 evidence: RLHF fine-tuning on Mandarin and English data generalizes to unseen languages (Japanese, Korean) when the reward signal targets phonetic accuracy rather than language-specific patterns. confidence: high relevance: low - claim_id: quality_attributes_governed_primarily_by_the_flow_matching_and_vocoder role: supports claim: Quality attributes governed primarily by the flow-matching and vocoder stages (MOS, speaker characteristics) are largely resistant to optimization by LM-level RLHF alone. source: §4.3.2, Table 4 evidence: Quality attributes governed primarily by the flow-matching and vocoder stages (MOS, speaker characteristics) are largely resistant to optimization by LM-level RLHF alone. confidence: high relevance: low limitations: - DiffRO only optimizes the LM component; the FM and vocoder are frozen, which limits the impact of rewards that primarily affect post-LM processing (e.g., speaker age/gender, which are partly determined by the FM). The MTR reward model requires a large in-house multi-task labeled corpus (13,000+ hours) that is not publicly available, limiting reproducibility. The approach is validated only on Mandarin-focused data; generalization to languages with distinct phonological properties (tonal, agglutinative) needs further study. Constructing diverse RL training data remains nontrivial. caveats: [] - id: interspeech-2025-0706 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: automated_llm_based_qa_generation_can_produce_paralinguistic_evaluation_sets role: supports claim: Automated LLM-based QA generation can produce paralinguistic evaluation sets that correlate closely with human-authored ones for assessing speech LLMs. source: §4, Table 2 evidence: Qwen2-Audio-7B-Instruct achieves 60.28 (LLM QA) vs. 59.46 (Human QA) under the ChatGPT judge with Prompt 2, and 56.82 vs. 54.33 under the Llama-70B judge; both differences are small and directionally consistent. confidence: high relevance: high - claim_id: combining_categorical_and_dimensional_emotion_recognition_with_consistency_filtering_reduces role: supports claim: Combining categorical and dimensional emotion recognition with consistency filtering reduces annotation noise in in-the-wild speech data condensation. source: §3.1, Figure 3 evidence: The dual SER consistency condition (categorical sentiment class aligned with valence score) applied on SG TV/Movie data achieved the highest UWA of 33.65% at valence thresholds x=0.5, y=0.4, outperforming single-paradigm labelling and improving class balance. confidence: high relevance: high - claim_id: current_speech_llms_exhibit_weak_performance_on_contextual_empathetic_reasoning role: complicates claim: Current speech LLMs exhibit weak performance on contextual empathetic reasoning even when evaluated against well-formed paralinguistic QA. source: §4, §5 evidence: Evaluation of Qwen2-Audio-7B-Instruct on the CPQA set reveals limitations in handling empathetic reasoning tasks; the paper identifies this as a motivating gap requiring both better data and more robust models. confidence: high relevance: high - claim_id: scalable_automatic_qa_generation_from_speech_introduces_systematic_biases_that role: complicates claim: Scalable automatic QA generation from speech introduces systematic biases that require post-filtering to produce usable evaluation data. source: §3.2 evidence: ChatGPT-generated QA contained repetitive question variants (multiple paraphrases asking about reasons behind emotion) and irrelevant questions assuming a text transcript was available; keyword-based post-filtering was required to remove these. confidence: high relevance: low limitations: - The evaluation benchmark is small (480 samples, 6.5 hours) and drawn from a single domain (Singaporean YouTube channels) in a mostly English-Mandarin context, so generalisation to other languages and speaking styles is unknown. The entire framework is validated against a single speech LLM (Qwen2-Audio-7B-Instruct), and the LLM judge scores are relatively similar across generated and human QA, which leaves open the question of whether the difference would matter more with a weaker model or a harder task. Speaker diarization is absent, causing the LLM to under-generate questions about multi-speaker interactions when multiple speakers share a gender. The internal SG TV/Movie dataset used for parameter tuning is not publicly available, making exact replication of the condensation configuration difficult. caveats: [] - id: interspeech-2025-0754 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: expressive_emotional_prosody_can_suppress_or_distort_intended_word_level role: supports claim: Expressive emotional prosody can suppress or distort intended word-level emphasis without explicit attention-level intervention, degrading perceptual emphasis clarity particularly in high-arousal emotions. source: §1, §2.4 evidence: Expressive emotional prosody can suppress or distort intended word-level emphasis without explicit attention-level intervention, degrading perceptual emphasis clarity particularly in high-arousal emotions. confidence: high relevance: high - claim_id: variance_based_pitch_and_duration_features_derived_from_emphasis_pseudo role: supports claim: Variance-based pitch and duration features derived from emphasis pseudo-labels provide effective local prosody modulation signals for jointly supervised emphasis and emotion control in TTS. source: §2.3 evidence: Variance-based pitch and duration features derived from emphasis pseudo-labels provide effective local prosody modulation signals for jointly supervised emphasis and emotion control in TTS. confidence: high relevance: high - claim_id: an_attention_bias_mechanism_targeting_predefined_emphasis_positions_improves_listener role: supports claim: An attention-bias mechanism targeting predefined emphasis positions improves listener emphasis recognition accuracy across emotion categories compared to a system without such a mechanism. source: §3.2.1, Table 1 evidence: An attention-bias mechanism targeting predefined emphasis positions improves listener emphasis recognition accuracy across emotion categories compared to a system without such a mechanism. confidence: high relevance: high - claim_id: llm_predicted_emphasis_positions_can_substitute_for_manual_annotation_at role: supports claim: LLM-predicted emphasis positions can substitute for manual annotation at inference time in emphasis-controllable emotional TTS, enabling controllable emphasis without per-utterance human labeling. source: §1, §3.1 evidence: LLM-predicted emphasis positions can substitute for manual annotation at inference time in emphasis-controllable emotional TTS, enabling controllable emphasis without per-utterance human labeling. confidence: high relevance: high - claim_id: joint_emphasis_emotion_modeling_improves_subjective_emotion_recognition_accuracy_for role: supports claim: Joint emphasis-emotion modeling improves subjective emotion recognition accuracy for difficult emotions (angry, sad) while leaving high-arousal emotions (happy, surprise) less changed. source: §3.2.2, Table 3 evidence: Joint emphasis-emotion modeling improves subjective emotion recognition accuracy for difficult emotions (angry, sad) while leaving high-arousal emotions (happy, surprise) less changed. confidence: high relevance: high limitations: - '- Dataset is small (ESD, ~1.2 hours/speaker, 5 emotions) and single-language (English); broader emotional diversity and multilingual transfer are untested. - EPE `strength` is a fixed hyperparameter (0.2); adaptive or learned strength per emotion is not explored. - Inference-time reliance on GPT-4 for emphasis prediction adds latency and cost; a lightweight alternative would be needed for deployment. - Happy emotion shows limited improvement, suggesting the emotion-emphasis interaction is not uniform.' caveats: [] - id: interspeech-2025-0756 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: structuring_speech_adapter_routing_around_distinct_affective_dimensions_improves_emotion role: supports claim: Structuring speech adapter routing around distinct affective dimensions improves emotion prediction and response quality in spoken dialogue systems compared to single-path adapters. source: §3.8, Table 4 evidence: A-SMiLE's task-aware top-1/top-4 expert routing for VAD vs. response generation outperforms conv-based, Q-Former, and standard Transformer adapters across METEOR, ROUGE-L, and GPT-4o empathy scores on DailyTalk. confidence: high relevance: high - claim_id: multi_task_joint_training_of_emotion_regression_and_response_generation role: supports claim: Multi-task joint training of emotion regression and response generation improves both emotional alignment and text quality over response-only training in spoken dialogue. source: §3.6, Table 2 evidence: EAML fine-tuning (Stage 2) consistently raises CCC scores and METEOR/ROUGE-L for both 0.5B and 7B backbones over text-only and cascaded baselines that receive VAD labels as auxiliary text rather than learned representations. confidence: high relevance: high - claim_id: discrete_categorical_emotion_labels_are_insufficient_for_capturing_fine_grained role: complicates claim: Discrete categorical emotion labels are insufficient for capturing fine-grained paralinguistic states such as sarcasm and depression in spoken dialogue. source: §3.1, §3.5, Table 1 evidence: The hard-case benchmark (sourced from IEMOCAP and MUStARD) specifically targets emotionally complex states where existing systems trained on categorical emotion corpora fail to capture nuance; A-SMiLE's CCC-based VAD modeling shows substantial gains on exactly these cases. confidence: high relevance: high - claim_id: automatic_text_overlap_metrics_meteor_rouge_l_and_llm_based role: complicates claim: Automatic text-overlap metrics (METEOR, ROUGE-L) and LLM-based empathy scoring may not fully reflect human perceptual judgments of emotional appropriateness in dialogue responses. source: §3.2 evidence: The paper evaluates response quality exclusively with METEOR, ROUGE-L, and GPT-4o-1120 empathy scores; no human listening evaluation is reported, leaving open the question of whether score gains translate to perceived empathy. confidence: high relevance: high limitations: - The evaluation relies entirely on reference-text metrics (METEOR, ROUGE-L) and GPT-4o-based empathy scores, with no human perceptual study. The DailyTalk corpus is 20 hours of scripted dialogue between two fixed speakers and may not represent the diversity of real-world conversations. The hard-case evaluation set is small (0.8 hours). The system generates text responses rather than speech, leaving the downstream TTS step and its interaction with emotional conditioning unaddressed. Whether the cognitive specialisation motivation (routing to distinct brain-analogue experts) genuinely produces disentangled representations or is primarily a useful inductive bias remains untested analytically. caveats: [] - id: interspeech-2025-0815 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - VC architecture: - VAE - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - vae_emotion_style_latents - gan_based_expressive_transfer claims: - claim_id: discrete_speech_unit_representations_reduce_source_speaker_leakage_in_voice role: supports claim: Discrete speech unit representations reduce source speaker leakage in voice conversion but introduce pronunciation artefacts that degrade intelligibility compared to continuous feature counterparts. source: §4.2, Table 1 evidence: Discrete speech unit representations reduce source speaker leakage in voice conversion but introduce pronunciation artefacts that degrade intelligibility compared to continuous feature counterparts. confidence: high relevance: low - claim_id: mix_style_layer_normalisation_mitigates_the_train_inference_mismatch_caused role: supports claim: Mix-style layer normalisation mitigates the train-inference mismatch caused by content-style dependence in style encoders, improving zero-shot generalisation on unseen speakers. source: §4.3, Table 2 evidence: Mix-style layer normalisation mitigates the train-inference mismatch caused by content-style dependence in style encoders, improving zero-shot generalisation on unseen speakers. confidence: high relevance: medium - claim_id: enriching_global_style_embeddings_with_explicit_pitch_and_energy_features role: supports claim: Enriching global style embeddings with explicit pitch and energy features improves emotion transfer fidelity in expressive voice conversion beyond mel-spectrogram-only style encoding. source: §3.5, §4.3, Table 2 evidence: Enriching global style embeddings with explicit pitch and energy features improves emotion transfer fidelity in expressive voice conversion beyond mel-spectrogram-only style encoding. confidence: high relevance: high - claim_id: cross_attention_fusion_of_local_f0_contours_with_content_embeddings role: supports claim: Cross-attention fusion of local F0 contours with content embeddings produces stronger prosodic alignment to the target than additive F0 injection in non-autoregressive voice conversion. source: §3.1, §4.3, Table 2 evidence: Cross-attention fusion of local F0 contours with content embeddings produces stronger prosodic alignment to the target than additive F0 injection in non-autoregressive voice conversion. confidence: high relevance: medium - claim_id: zero_shot_cross_lingual_voice_conversion_is_achievable_with_a role: supports claim: Zero-shot cross-lingual voice conversion is achievable with a monolingual training corpus when content representations are extracted from a multilingual speech model, though intelligibility degrades for unseen source languages. source: §4.4, Table 4 evidence: Zero-shot cross-lingual voice conversion is achievable with a monolingual training corpus when content representations are extracted from a multilingual speech model, though intelligibility degrades for unseen source languages. confidence: high relevance: medium limitations: - 'The proposed system incurs a substantial WER penalty relative to baselines: 7.98% vs. 5.01% (ESD) and 8.84% vs. 3.48% (LibriTTS) for the full model, with discrete units identified as the cause. This intelligibility regression is acknowledged but not resolved; future work is deferred.' - The evaluation uses a small subjective panel (15 listeners, 10–15 samples per model), limiting the statistical power of MOS comparisons. The cross-lingual results are restricted to English and German; how performance degrades for more distant language pairs is untested. The model is trained on English-only data, and German-to-English conversion shows a 30.84% WER, suggesting significant cross-lingual generalisation limits. The model size and computational cost are not reported, making it difficult to assess deployment feasibility. caveats: [] - id: interspeech-2025-0948 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - VC architecture: - VAE - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - vae_emotion_style_latents - diffusion_based_emotional_generation claims: - claim_id: natural_language_prompts_enable_more_flexible_and_subjectively_accurate_emotion role: supports claim: Natural language prompts enable more flexible and subjectively accurate emotion control in voice conversion than numeric intensity values or reference audio selection. source: §1, §3.4 evidence: Natural language prompts enable more flexible and subjectively accurate emotion control in voice conversion than numeric intensity values or reference audio selection. confidence: high relevance: high - claim_id: a_diffusion_based_mapping_from_text_embeddings_to_speech_emotion role: supports claim: A diffusion-based mapping from text embeddings to speech emotion embeddings is sufficient to replace reference audio at inference time without significant quality loss. source: §2.1, §3.2, Table 1 evidence: A diffusion-based mapping from text embeddings to speech emotion embeddings is sufficient to replace reference audio at inference time without significant quality loss. confidence: high relevance: high - claim_id: joint_training_of_a_text_to_emotion_mapper_with_reference role: supports claim: Joint training of a text-to-emotion mapper with reference emotion embeddings improves prosody naturalness over direct prediction from text alone. source: §3.3, Table 1 evidence: Joint training of a text-to-emotion mapper with reference emotion embeddings improves prosody naturalness over direct prediction from text alone. confidence: high relevance: high - claim_id: preserving_speaker_identity_during_emotional_pitch_manipulation_requires_an_explicit role: supports claim: Preserving speaker identity during emotional pitch manipulation requires an explicit F0 constraint in the speaker encoder; adversarial training alone is insufficient. source: §2.3, §3.3, Table 1 evidence: Preserving speaker identity during emotional pitch manipulation requires an explicit F0 constraint in the speaker encoder; adversarial training alone is insufficient. confidence: high relevance: high - claim_id: mixed_emotion_synthesis_remains_harder_to_control_than_single_category role: supports claim: Mixed-emotion synthesis remains harder to control than single-category emotion intensity across both subjective and objective metrics. source: §3.4, Table 2, Table 3 evidence: Mixed-emotion synthesis remains harder to control than single-category emotion intensity across both subjective and objective metrics. confidence: high relevance: high limitations: - Training and evaluation are conducted entirely on TextrolSpeech, a single corpus with a limited speaker set. Generalisation to out-of-domain speakers, languages, or acoustic conditions is untested, and all reported numbers should be interpreted within that constraint. - The evaluation uses only 25 listeners for subjective MOS across 132 utterances — a borderline sample size that may limit statistical reliability. The mixed-emotion accuracy (61.3%) is notably lower than single-attribute control, and the system's handling of complex emotional blends (e.g., contempt with happiness) is not analysed in depth. The discrete HuBERT token approach for linguistic content may introduce quantisation artefacts not reported in the paper. Future real-time or streaming deployment, mentioned in the conclusion as a direction, is not addressed in the current architecture. caveats: [] - id: interspeech-2025-1101 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - VC architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - diffusion_based_emotional_generation claims: - claim_id: diffusion_based_voice_conversion_systems_can_achieve_strong_emotion_controllability role: supports claim: Diffusion-based voice conversion systems can achieve strong emotion controllability in zero-shot settings when combined with mutual-information disentanglement and inference-time guidance. source: §3.2, §3.3, Table 2 evidence: Diffusion-based voice conversion systems can achieve strong emotion controllability in zero-shot settings when combined with mutual-information disentanglement and inference-time guidance. confidence: high relevance: high - claim_id: disentangling_speaker_identity_and_emotion_via_mutual_information_minimisation_improves role: supports claim: Disentangling speaker identity and emotion via mutual information minimisation improves emotion controllability in voice conversion without requiring parallel or speaker-specific training data. source: §2.1.4, §3.3, Table 2 evidence: Disentangling speaker identity and emotion via mutual information minimisation improves emotion controllability in voice conversion without requiring parallel or speaker-specific training data. confidence: high relevance: high - claim_id: in_emotional_voice_conversion_autoencoder_based_methods_tend_to_achieve role: complicates claim: In emotional voice conversion, autoencoder-based methods tend to achieve higher emotion accuracy than GAN-based methods, but at the cost of substantially lower naturalness and higher speech distortion. source: §3.2, Table 1 evidence: In emotional voice conversion, autoencoder-based methods tend to achieve higher emotion accuracy than GAN-based methods, but at the cost of substantially lower naturalness and higher speech distortion. confidence: high relevance: high - claim_id: classifier_free_style_guidance_applied_to_emotion_representations_at_inference role: supports claim: Classifier-free-style guidance applied to emotion representations at inference time provides a direct lever for trading naturalness against emotion controllability in diffusion-based EVC. source: §2.1.3, §3.3, Table 2 evidence: Classifier-free-style guidance applied to emotion representations at inference time provides a direct lever for trading naturalness against emotion controllability in diffusion-based EVC. confidence: high relevance: high - claim_id: training_on_large_scale_in_the_wild_emotional_corpora_enables role: supports claim: Training on large-scale in-the-wild emotional corpora enables zero-shot generalisation to speakers absent from training, even when evaluation is conducted on acted-speech datasets with different recording conditions. source: §3.4, §4 evidence: Training on large-scale in-the-wild emotional corpora enables zero-shot generalisation to speakers absent from training, even when evaluation is conducted on acted-speech datasets with different recording conditions. confidence: high relevance: high limitations: - 'The comparison between ZSDEVC and EMOCONV-DIFF in Table 1 is not fully fair: EMOCONV-DIFF is evaluated in a seen-speaker scenario while ZSDEVC operates zero-shot. The naturalness gap may reflect this experimental asymmetry rather than a fundamental quality deficit.' - The model does not address intensity control within a target emotion category — prior work (Emovox) provides per-dimension arousal/valence control that ZSDEVC does not directly expose during inference. Evaluation covers only five emotion categories (angry, happy, sad, neutral, surprise) and excludes neutral-to-emotional conversion. Real-time or streaming use cases are not addressed. Results on out-of-domain acted speech (ESD) and in-the-wild speech (MSP-Podcast) show consistent trends, but the system's behaviour on highly expressive or non-English speech is untested. caveats: [] - id: interspeech-2025-1115 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - diffusion - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_emotional_generation - autoregressive_emotion_language_models claims: - claim_id: unified_multi_modal_emotion_encoders_that_align_different_prompt_modalities role: supports claim: Unified multi-modal emotion encoders that align different prompt modalities to a shared emotion representation space enable flexible emotion control in zero-shot TTS without sacrificing speaker similarity. source: §2.2, §3.4, Table 1 evidence: MPEE ablation shows removing the multi-modal encoder drops MOS from 3.73 to 3.20 and emotion accuracy from 48% to 35% in speech-prompt mode; SSMOS (3.73) is maintained even when emotion and timbre are drawn from different speakers. confidence: high relevance: high - claim_id: auxiliary_emotion_consistency_losses_applied_to_autoregressive_prosody_prediction_improve role: supports claim: Auxiliary emotion consistency losses applied to autoregressive prosody prediction improve emotion alignment in generated speech. source: §2.3, §3.4, Tables 1–3 evidence: ECL ablation reduces ESMOS from 4.05 to 3.83 and ACC from 48% to 45% in speech-prompt mode; the gain is larger for text and image prompts (ESMOS drops from 3.85 to 3.73 and from 3.90 to 3.75 respectively), where the emotion signal is less direct. confidence: high relevance: high - claim_id: emotion_accuracy_in_expressive_zero_shot_tts_remains_substantially_below role: complicates claim: Emotion accuracy in expressive zero-shot TTS remains substantially below ground truth even with explicit emotion conditioning and auxiliary training objectives. source: §3.4, Tables 1–3 evidence: Best system ACC is 48% vs 54% for ground-truth mel-spectrograms; the 6-point gap persists despite MPEE and ECL, indicating that fine-grained emotion control is not yet solved at this data scale. confidence: high relevance: high - claim_id: evaluating_multi_modal_tts_systems_against_prior_work_is_complicated role: complicates claim: Evaluating multi-modal TTS systems against prior work is complicated by the absence of official open-source implementations for key baselines. source: §3.4 evidence: MM-TTS has no official open-source release; all comparisons in this paper are against a reproduction based on the original paper's settings, which the authors acknowledge as a limitation of the experimental evaluation. confidence: high relevance: medium - claim_id: hierarchical_disentanglement_of_speech_attributes_at_different_granularity_levels_enables role: supports claim: Hierarchical disentanglement of speech attributes at different granularity levels enables fine-grained independent control over timbre and emotion in zero-shot TTS. source: §2.1, §3.4, Tables 1–3 evidence: The disentangling strategy separates coarse-grained features (timbre, emotion as global vectors) from fine-grained features (content, prosody at frame level), using distinct bottleneck designs; SSMOS remains stable (3.73–3.76) across all three prompt modalities despite using different emotion sources from the timbre reference. confidence: high relevance: high limitations: - The primary baseline comparison (MM-TTS) is based on the authors' own reproduction rather than an official implementation, as MM-TTS has no public open-source release. Observed performance differences cannot be independently verified without access to the original system. - 'Scalability is explicitly flagged as a limitation: MPE-TTS is fine-tuned on 36 hours of MEAD-TTS (48 actors, 8 emotions), and the authors note that larger emotional corpora and increased model scale would be needed to improve generalisation. The system evaluates only English speech, and multilingual emotional expressiveness is unexplored. At inference, only one emotion modality can be provided (no multi-modal fusion); exploring how to combine cues from multiple modalities simultaneously remains an open direction. WER performance is noticeably worse than ground-truth reconstructed mel (23.4% vs. 18.8%), suggesting the emotion-conditioning mechanism introduces some degradation in intelligibility.' caveats: [] - id: interspeech-2025-1192 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: adversarial_disentanglement_via_a_gradient_reversal_layer_can_separate_voice role: supports claim: Adversarial disentanglement via a gradient reversal layer can separate voice impression information from speaker identity in a zero-shot TTS speaker encoder, enabling independent modulation of perceived voice characteristics. source: §2.2, §4.1, Figure 5 evidence: The control module applies GRL + 0.8 dropout to the speaker embedding to remove impression signal, then reintroduces it via an 11-dim impression vector; cosine similarity to the target speaker remains above the inter-speaker distribution at all tested modulation levels. confidence: high relevance: low - claim_id: llms_can_generate_low_dimensional_speech_style_parameter_vectors_from role: supports claim: LLMs can generate low-dimensional speech style parameter vectors from free-form natural language descriptions, providing a usable zero-shot interface for voice characteristic control. source: §2.3, §4.3, Table 4 evidence: ChatGPT-4o prompted with dimension definitions and pre-modulation values produces impression vectors preferred over unmodulated speech in 94.6% of "sleepy" trials and 74.6% of "urgent, attention-grabbing" trials (n=436, crowdsourced). confidence: high relevance: low - claim_id: fine_grained_impression_control_in_tts_involves_a_trade_off role: complicates claim: 'Fine-grained impression control in TTS involves a trade-off: stronger modulation produces more perceptually distinct impressions but degrades naturalness, particularly when the target impression is socially incongruent with the source speaker.' source: §4.2, Table 3 evidence: MOS naturalness at maximum modulation (±3) falls to 2.71-2.88 for the Powerful-Weak and Dark-Bright dimensions, compared to 3.61-3.72 at zero modulation; low-scoring samples correspond to combinations such as "feminine male" or "strongly dark/bright." confidence: high relevance: low - claim_id: automatic_annotation_of_high_dimensional_perceptual_voice_attributes_at_training role: complicates claim: Automatic annotation of high-dimensional perceptual voice attributes at training scale requires indirect labeling pipelines that introduce estimation error, limiting the precision of supervision. source: §3.2 evidence: Crowdsourced subjective ratings were collected for only 1,154 of 20,270 speakers; an HuBERT-based estimator extrapolated labels to the remaining data with an RMSE of 0.338 on held-out utterances. confidence: high relevance: medium limitations: - All training and evaluation data is a proprietary in-house Japanese corpus; no public datasets are used, and the evaluation spans only two held-out speakers. The results are not directly reproducible, and generalization to other languages, speaking domains, or TTS architectures is untested. - Eleven impression dimensions exhibit substantial inter-correlation (Table 1 reports correlations up to 0.8, e.g., Thick-Thin vs. High-Low Pitched), which means independent perceptual dimensions are not fully captured by the vector design. The paper demonstrates stable simultaneous two-dimension modulation but does not test combinations across weakly correlated dimensions at extreme values. - No ablation is reported for the GRL component alone versus the dropout component; the relative contribution of each disentanglement mechanism to impression controllability is unclear. The evaluation uses the impression estimator (trained on the same data pipeline) as the primary objective metric, creating circularity between the data labeling and the evaluation. - Future directions noted by the authors include applying voice impression constraints to neural audio codecs, following the factorized-codec approach of NaturalSpeech 3. caveats: [] - id: interspeech-2025-1210 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - VC architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_emotional_generation claims: - claim_id: dual_granularity_emotion_feature_extraction_combining_utterance_level_and_frame role: supports claim: Dual-granularity emotion feature extraction (combining utterance-level and frame-level representations) improves emotion discriminability in voice conversion compared to single-scale approaches. source: §2.1.3, §3.3.3, Table 3 evidence: DiffEmotionVC's dual-granularity emotion encoder achieves 80% ECA and 0.78 Pearson Corr on the ESD dataset; ablation confirms removing the emotion encoder is the most damaging intervention, dropping Corr to 0.38. confidence: high relevance: high - claim_id: orthogonality_constraints_on_emotion_speaker_and_content_feature_spaces_provide role: supports claim: Orthogonality constraints on emotion, speaker, and content feature spaces provide a stable and effective disentanglement mechanism for emotional voice conversion. source: §2.2.2, §3.3.3, Table 3 evidence: Removing orthogonal loss reduces SECS from 0.73 to 0.70 and Corr from 0.78 to 0.72; the paper explicitly motivates orthogonal loss as a remedy for the training instability of the mutual information loss used in prior work. confidence: high relevance: high - claim_id: diffusion_based_evc_systems_achieve_strong_overall_emotion_accuracy_but role: complicates claim: Diffusion-based EVC systems achieve strong overall emotion accuracy but struggle to discriminate between high-arousal emotions sharing similar arousal-valence profiles. source: §3.3.1 evidence: DiffEmotionVC reaches 80% ECA overall but the paper notes difficulty distinguishing happy, surprised, and angry, attributing this to insufficient emotional diversity in the ESD training data rather than a fundamental model limitation. confidence: high relevance: high - claim_id: discretisation_of_continuous_speech_representations_degrades_emotion_voice_conversion_by role: complicates claim: Discretisation of continuous speech representations degrades emotion voice conversion by introducing content-emotion feature leakage. source: §3.3.2, Table 2 evidence: Replacing continuous ContentVec with VQ-ContentVec drops UTMOS from 4.04 to 2.54 and Corr from 0.78 to 0.50; SpeechTokenizer RVQ1 discrete features produce the worst performance (UTMOS 1.79), demonstrating that discrete tokens cause timbre and emotion entanglement. confidence: high relevance: high - claim_id: cross_attention_fusion_outperforms_additive_fusion_for_integrating_heterogeneous_speech role: supports claim: Cross-attention fusion outperforms additive fusion for integrating heterogeneous speech features in voice conversion systems. source: §3.3.3, Table 3 evidence: Ablation replacing gated cross-attention with simple additive fusion reduces UTMOS from 4.04 to 3.26, a 19% degradation in predicted audio quality. confidence: high relevance: medium limitations: - Evaluation is limited to the ESD dataset (five emotions, primarily Mandarin Chinese; the ablation table specifically targets the zh-Angry subset), restricting generalisability to other languages and broader emotion categories. The model size is unreported, making deployment trade-off analysis impossible. Distinguishing between high-arousal emotions (happy, surprised, angry) remains unresolved; the paper identifies more naturalistic emotional data as the likely remedy but leaves this to future work. No code is released, and there is no cross-lingual evaluation. caveats: [] - id: interspeech-2025-1394 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: self_supervised_distillation_with_emotion_specific_inductive_biases_can_learn role: supports claim: Self-supervised distillation with emotion-specific inductive biases can learn speaker-independent emotion embeddings more effectively than GRL-based or VQ-based disentanglement. source: §3.4, Table 1 evidence: DiEmo-TTS surpasses Trans-GRL, Trans-VQ, and Trans-Ort on eMOS across all four emotion categories in subjective evaluation, without requiring explicit speaker labels during emotion encoder training. confidence: high relevance: high - claim_id: formant_based_speaker_perturbation_is_more_effective_for_disentangling_speaker role: supports claim: Formant-based speaker perturbation is more effective for disentangling speaker identity from emotion than non-targeted noise augmentation in cross-speaker emotion transfer. source: §3.6, Table 2 evidence: Replacing formant perturbation with MUSAN/RIR noise augmentation in the ablation increases WER and degrades SECS, whereas formant perturbation exploits the timbre-formant correlation to distort identity while preserving emotional expression. confidence: high relevance: high - claim_id: multi_factor_conditioning_via_shared_attention_mechanisms_produces_better_balance role: supports claim: Multi-factor conditioning via shared attention mechanisms produces better balance between speaker fidelity and emotional expressiveness than concatenation conditioning in transformer-based TTS. source: §3.6, Table 2 evidence: Adding the DCT block improves eMOS from 3.89 to 4.07 while keeping sMOS comparable in the ablation; the block applies weight-sharing multi-head attention per style signal and fuses outputs via MLP. confidence: high relevance: high - claim_id: achieving_speaker_independent_emotion_representations_via_self_supervised_methods_still role: complicates claim: Achieving speaker-independent emotion representations via self-supervised methods still depends on labeled auxiliary data for emotion space definition. source: §4, §3.2 evidence: The emotion clustering step requires a WavLM-based emotional attribute predictor fine-tuned on the MSP-Podcast labeled corpus; the authors identify dependence on this predictor as a limitation for generalising to unseen speakers. confidence: high relevance: high limitations: - Evaluation uses only 2 target speakers (one male, one female from ESD), both trained exclusively on neutral utterances. Generalisation to speakers trained on mixed emotional data or to out-of-domain voices is not assessed. - WER of 16.16% indicates non-trivial intelligibility degradation compared to clean TTS; the source of this degradation (emotion conditioning, model architecture, or dataset characteristics) is not explicitly discussed. The emotional attribute predictor is fine-tuned on MSP-Podcast, a separate labeled corpus, introducing a labeled-data dependency that the authors acknowledge as a target for future unsupervised replacement. Scalability to languages beyond English and to larger multi-emotion multi-speaker datasets remains undemonstrated. caveats: [] - id: interspeech-2025-1638 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: treating_emotion_as_a_disentangled_parallel_synthesis_branch_improves_emotional role: supports claim: Treating emotion as a disentangled parallel synthesis branch improves emotional expressiveness over global style conditioning in zero-shot TTS. source: §3.6, Table 2 evidence: Ablation removing emotion decoupling reduces EMOS from 3.96 to 3.45 on LibriTTS test, with Emotion Discrepancy rising from 0.571 to 0.939, confirming that explicit emotion isolation drives the expressiveness gains. confidence: high relevance: high - claim_id: llm_based_emotion_transformation_conditioned_on_target_text_semantics_reduces role: supports claim: LLM-based emotion transformation conditioned on target text semantics reduces emotion-content mismatch in zero-shot speech synthesis. source: §3.6, Table 2 evidence: Removing the emotion-aware LLM raises ED from 0.571 to 1.056 and drops EMOS from 3.96 to 3.73 on LibriTTS test, with both components contributing independently to emotional consistency. confidence: high relevance: high - claim_id: direct_reference_emotion_transfer_in_zero_shot_tts_produces_emotionally role: complicates claim: Direct reference emotion transfer in zero-shot TTS produces emotionally inconsistent speech when the reference and target texts differ in emotional valence. source: §3.3, §3.5, Table 1 evidence: All five comparison zero-shot TTS baselines (YourTTS, TransferTTS, VALL-E, E2-TTS, CosyVoice) show Emotion Discrepancy scores of 0.802 to 1.399 versus EATS-Speech at 0.571, suggesting that without explicit emotion adaptation, standard conditioning mechanisms misalign emotion to content. confidence: high relevance: high - claim_id: evaluation_of_emotion_expressiveness_with_metrics_derived_from_the_same role: complicates claim: Evaluation of emotion expressiveness with metrics derived from the same model used in training introduces circularity that limits the reliability of reported Emotion Discrepancy scores. source: §2.1, §3.5 evidence: ED is computed using utterance-level emotion2vec embeddings, the same pre-trained model whose frame-level features the Emotion Tokenizer builds on; this means the metric and the system share the same representational basis, potentially inflating reported gains. confidence: high relevance: high limitations: - The Emotion Discrepancy metric is computed using emotion2vec embeddings, the same model used as the feature extractor inside EATS-Speech's Emotion Tokenizer. This circularity means the objective metric may overstate the system's emotion alignment advantage relative to baselines that do not use emotion2vec internally. - Evaluation is limited to LibriTTS, an audiobook corpus with naturally constrained emotional variability. How well the framework transfers to genuinely expressive speech (acted emotion corpora, spontaneous speech) is not assessed. The paper does not report how reference and target samples are matched for emotional contrast in the test set, making it difficult to assess task difficulty. - The model size is not reported; the system combines a flow transformer, an emotion tokenizer, and a GPT-2-based LLM across three sequential training stages, suggesting non-trivial resource requirements that are not characterized. Only English is evaluated; extension to multilingual settings with different prosodic and emotional conventions is left open. caveats: [] - id: interspeech-2025-2075 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - codec architecture: [] relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: [] claims: - claim_id: multi_granularity_quantization_codebooks_encode_paralinguistic_and_prosodic_features_more role: supports claim: Multi-granularity quantization codebooks encode paralinguistic and prosodic features more efficiently than single-resolution codebooks at equivalent or higher bitrates. source: §6.2, Table 3 evidence: SVCs (k=500 per codebook, ~544 bits/s on Expresso) outperform frame-level k=2000 baselines (~548 bits/s) on all emotion sub-categories and prominence classification, with angry speech F1 rising from 0.298 to 0.614. confidence: high relevance: high - claim_id: pooling_continuous_speech_representations_before_discretization_retains_more_paralinguistic_and role: supports claim: Pooling continuous speech representations before discretization retains more paralinguistic and prosodic information than pooling after discretization. source: §6.1, Table 2 evidence: Pre-pooling consistently outperforms post-pooling in both utterance-level SER (0.5074 vs 0.2834 accuracy) and word-level prominence classification (0.3423 vs 0.1210 F-score) across single-level and multi-level codebook conditions. confidence: high relevance: high - claim_id: dsu_based_expressive_speech_resynthesis_retains_a_large_gap_in role: complicates claim: DSU-based expressive speech resynthesis retains a large gap in style fidelity relative to continuous-feature resynthesis, even with improved codebook designs. source: §6.2, §6.3, Table 4 evidence: SVCs achieve 41.22% style classification accuracy on Expresso versus 74.72% for continuous features and 88.42% for ground truth, a 33-point gap despite SVCs outperforming all other DSU baselines tested. confidence: high relevance: high - claim_id: forced_alignment_requirements_constrain_multi_granularity_dsu_methods_to_text role: complicates claim: Forced-alignment requirements constrain multi-granularity DSU methods to text-paired speech settings, limiting their applicability to spontaneous or unlabelled data. source: §4.1, §7 evidence: Phone, word, and utterance boundaries for SVC segmentation are derived from forced alignments using the Montreal Forced Aligner and HTK; no unsupervised segmentation is evaluated, and the authors list automatic segmentation as future work. confidence: high relevance: medium limitations: - The method depends on forced alignments at three linguistic levels (phone, word, utterance), which requires text transcriptions and limits use in low-resource or spontaneous speech domains. The authors acknowledge this and identify unsupervised or automatic segmentation as the most promising avenue for removing this constraint. - Evaluation uses automated quality proxies only (UTMOS, style classifier accuracy, WER). The authors note that human listening tests and qualitative error analysis are needed and explicitly defer them to future work. - The merged frame-level DSU representation discards part of the factorized structure SVCs provide. Whether architectures that natively consume multi-stream DSUs would yield further gains is unanswered. - Finally, all codebook vocabulary sizes are fixed at k=500. The paper suggests that per-level vocabulary tuning could improve task adaptation, but this is not explored empirically. caveats: [] - id: interspeech-2025-2586 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_emotion_control - gan_based_expressive_transfer claims: - claim_id: restricting_style_quantization_to_acoustically_informative_voiced_speech_regions_improves role: supports claim: Restricting style quantization to acoustically informative (voiced) speech regions improves both style expressiveness and prosodic accuracy in reference-based TTS. source: §4.4.1, Table 4.3 evidence: On ESD, removing voiced extraction raises RMSE_F0 from 8.27 to 11.48 Hz and WER from 12.64 to 14.06; the full model achieves the best style similarity MOS (3.84) over all baselines. confidence: high relevance: medium - claim_id: pairing_style_disentanglement_with_a_complementary_prosody_preserving_loss_stabilises role: supports claim: Pairing style disentanglement with a complementary prosody-preserving loss stabilises training and prevents prosody degradation from aggressive content removal. source: §4.4.2, Table 4.3 evidence: Removing only the SP loss raises pitch error to 9.74 Hz; removing both SD and SP degrades nMOS to 3.66 and WER to 15.38, with the SP-only-removed condition showing worse prosody than the case with no disentanglement losses at all. confidence: high relevance: medium - claim_id: asymmetric_biased_self_attention_in_unvoiced_region_filling_allowing_information role: supports claim: Asymmetric (biased) self-attention in unvoiced region filling, allowing information flow from voiced to unvoiced positions but not the reverse, outperforms symmetric or fully blocked alternatives for prosodic continuity. source: §4.4.3, Table 4.4 evidence: Standard self-attention in the UF module degrades pitch error to 16.38 Hz and F1 v/uv to 0.6668; binary masking partially recovers at 13.19 Hz versus 8.27 Hz for biased attention. confidence: high relevance: medium - claim_id: style_transfer_quality_gains_from_region_specific_quantization_have_been role: complicates claim: Style transfer quality gains from region-specific quantization have been established only within a controlled emotional corpus, leaving open the question of whether the approach generalises to broader speaking styles or modern generative architectures. source: §4.1, Table 4.1 evidence: All baselines are FastSpeech 2 variants evaluated solely on ESD (10 speakers, 5 discrete emotions); no comparison with flow-matching, diffusion, or large-scale systems is provided. confidence: high relevance: high limitations: - All evaluations are conducted on a single English emotional speech corpus (ESD, 10 speakers, 5 emotions). Baselines are restricted to FastSpeech 2-based systems; no comparison with flow-matching or diffusion TTS is included, and there is no evidence of generalisation to multilingual or out-of-domain speech. - Model size is not reported. The voiced/unvoiced segmentation depends on pre-extracted V/UV flags, introducing a dependency on external pitch tracking that may degrade under noisy or spontaneous speech conditions. Non-parallel style transfer performance, while positive, consistently trails the parallel setting, indicating residual sensitivity to content mismatch even after disentanglement. The rotation trick hyperparameters and UF module depth are fixed without sensitivity analysis. caveats: [] - id: '2508.13028' published_date: "2025-08-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: minor method_family: - transformer_encoder_decoder_emotion_control claims: - claim_id: feedback_loss_from_a_sarcasm_classifier_can_be_integrated_into role: supports claim: Feedback loss from a sarcasm classifier can be integrated into TTS training to bias synthesized speech toward detector-recognizable sarcastic prosody. source: §2.2, §4.2, Table 2 evidence: Feedback loss from a sarcasm classifier can be integrated into TTS training to bias synthesized speech toward detector-recognizable sarcastic prosody. confidence: high relevance: medium - claim_id: bi_modal_sarcasm_detection_that_combines_acoustic_features_and_text role: supports claim: Bi-modal sarcasm detection that combines acoustic features and text substantially outperforms audio-only detection, suggesting that sarcasm in speech is often semantically encoded and not recoverable from prosody alone. source: §4.1, Table 1 evidence: Bi-modal sarcasm detection that combines acoustic features and text substantially outperforms audio-only detection, suggesting that sarcasm in speech is often semantically encoded and not recoverable from prosody alone. confidence: high relevance: medium - claim_id: two_stage_fine_tuning_from_neutral_read_speech_to_conversational role: supports claim: Two-stage fine-tuning — from neutral read speech to conversational speech and then to target style — provides a viable data strategy for low-resource expressive speech synthesis. source: §2.3, §3.1 evidence: Two-stage fine-tuning — from neutral read speech to conversational speech and then to target style — provides a viable data strategy for low-resource expressive speech synthesis. confidence: high relevance: high - claim_id: subjective_sarcasm_perception_in_listening_tests_is_difficult_to_isolate role: supports claim: Subjective sarcasm perception in listening tests is difficult to isolate from the sarcastic content of the text, creating ambiguity in whether listeners respond to prosody or semantics. source: §5, §4.3 evidence: Subjective sarcasm perception in listening tests is difficult to isolate from the sarcastic content of the text, creating ambiguity in whether listeners respond to prosody or semantics. confidence: high relevance: medium limitations: - 'The evaluation design has a critical circularity: the same detector architecture trained on the same data distribution is used both as the training feedback signal and as the primary objective metric. Improvements in detection score on synthesized outputs are therefore expected by construction and cannot be treated as independent evidence of sarcasm-aware synthesis.' - The paper does not include an ablation that isolates the contribution of the feedback loss from the two-stage fine-tuning; it is therefore unclear which component drives the reported gains. The subjective study is small (13 listeners) and does not report MOS confidence intervals, making statistical significance uncertain. All evaluation stimuli are drawn from a sarcasm corpus, which may prime listener responses. The system has not been compared against more expressive modern TTS models (e.g., flow-matching or diffusion-based systems with style conditioning). Finally, the system is English-only and trained on North American sitcom speech, limiting generalizability to other languages, speakers, or sarcasm conventions. caveats: [] - id: '2508.16188' published_date: "2025-08-22" entry_date: '2026-07-30' year: 2025 venue: EMNLP task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models claims: - claim_id: full_face_visual_features_encode_complementary_emotional_information_not_captured role: supports claim: Full-face visual features encode complementary emotional information not captured by audio alone, and combining both modalities yields meaningfully higher emotion recognition accuracy than either modality individually. source: §3, Table 1 evidence: Full-face visual features encode complementary emotional information not captured by audio alone, and combining both modalities yields meaningfully higher emotion recognition accuracy than either modality individually. confidence: high relevance: high - claim_id: prefix_based_visual_fusion_using_compressed_q_former_query_latents role: supports claim: Prefix-based visual fusion using compressed Q-Former query latents integrates more effectively into autoregressive speech LMs than direct feature concatenation, which collapses perplexity. source: §5.1, Table 3 evidence: Prefix-based visual fusion using compressed Q-Former query latents integrates more effectively into autoregressive speech LMs than direct feature concatenation, which collapses perplexity. confidence: high relevance: medium - claim_id: visual_guidance_during_training_improves_not_only_classification_accuracy_but role: supports claim: Visual guidance during training improves not only classification accuracy but also the emotional alignment of generated speech in conversational settings. source: §5.3, Table 7 evidence: Visual guidance during training improves not only classification accuracy but also the emotional alignment of generated speech in conversational settings. confidence: high relevance: high - claim_id: emotion_controllability_through_prompt_only_label_manipulation_is_insufficient_when role: supports claim: Emotion controllability through prompt-only label manipulation is insufficient when a model is trained on data where input and response emotions are correlated; in-context demonstrations are necessary to override this bias. source: §5.3, Figure 5 evidence: Emotion controllability through prompt-only label manipulation is insufficient when a model is trained on data where input and response emotions are correlated; in-context demonstrations are necessary to override this bias. confidence: high relevance: high limitations: - The expressive generation evaluation relies on a third-party model (Qwen2-Audio) to classify emotion in synthesised speech — there is no human perceptual evaluation of naturalness, similarity, or overall quality. The fine-tuning dataset is small (4,859 synthetic pairs from IEMOCAP), and the system is evaluated on a closed four-class emotion taxonomy, limiting generalisability to naturalistic conversation. - The decoding strategy enforces structural constraints on SpiritLM's interleaved token types (style → pitch → semantic), which adds inference complexity and may suppress the model's capacity for stylistic diversity beyond the four emotion classes. The AVSR results (3.5% WER clean) lag specialist systems by more than 2.5 percentage points, a gap the paper attributes to design choices rather than fundamental limitations — though this distinction is not tested directly. Cross-language and multi-speaker generalisation beyond IEMOCAP's two-speaker dyadic setup are not evaluated. caveats: [] - id: '2508.17623' published_date: "2025-08-25" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: current_spoken_dialogue_systems_reliably_fail_to_sustain_emotionally_coherent role: complicates claim: Current spoken dialogue systems reliably fail to sustain emotionally coherent responses across multi-turn conversations, with degradation compounding across turns. source: §V, Table V evidence: Current spoken dialogue systems reliably fail to sustain emotionally coherent responses across multi-turn conversations, with degradation compounding across turns. confidence: high relevance: low - claim_id: automatic_continuous_and_categorical_emotion_metrics_correlate_moderately_with_human role: supports claim: Automatic continuous and categorical emotion metrics correlate moderately with human perceptual judgements on most SDMs, but overestimate emotional coherence on systems with atypical output characteristics. source: §V, Figure 3, Figure 4 evidence: Automatic continuous and categorical emotion metrics correlate moderately with human perceptual judgements on most SDMs, but overestimate emotional coherence on systems with atypical output characteristics. confidence: high relevance: high - claim_id: optimising_for_categorical_emotion_appropriateness_and_optimising_for_continuous_affect role: supports claim: Optimising for categorical emotion appropriateness and optimising for continuous affect trajectory alignment represent distinct objectives that current SDMs trade off against each other rather than jointly satisfying. source: §V, Table IV evidence: Optimising for categorical emotion appropriateness and optimising for continuous affect trajectory alignment represent distinct objectives that current SDMs trade off against each other rather than jointly satisfying. confidence: high relevance: high - claim_id: synthetic_emotional_speech_generated_by_controllable_tts_systems_provides_sufficient role: supports claim: Synthetic emotional speech generated by controllable TTS systems provides sufficient variety and quality to serve as evaluation stimulus material when real annotated data is scarce. source: §IV-A, Table III evidence: Synthetic emotional speech generated by controllable TTS systems provides sufficient variety and quality to serve as evaluation stimulus material when real annotated data is scarce. confidence: high relevance: high limitations: - Human perceptual evaluation covers only 20 randomly sampled dialogues per system evaluated by 20 annotators; at this sample size, inter-rater agreement statistics are not reported, limiting confidence in the perceptual ERS rankings. The authors acknowledge this and commit to releasing agreement analyses upon acceptance. - The benchmark currently covers English only, constraining applicability to multilingual SDMs. The categorical metric relies on four high-reliability emotion classes (neutral, happy, angry, sad); more nuanced affective states (contempt, fear, surprise) are captured only indirectly through continuous VAD, leaving a gap between the benchmark's coverage and the full affective spectrum. Metric thresholds are derived from MSP-Conversation statistics; their transferability to other languages or domains is untested, though the authors show robustness to small percentile shifts. The CSER model underperforms the baseline on arousal (CCC 0.554 vs. 0.594) while outperforming on valence and dominance; whether the arousal shortfall materially affects benchmark rankings is not examined. caveats: [] - id: '2509.02020' published_date: "2025-09-02" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models claims: - claim_id: reducing_speech_tokenizer_frame_rate_to_12_5hz_with_explicit role: supports claim: Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection. source: §2.1, §4.1, Table 1 evidence: Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection. confidence: high relevance: low - claim_id: a_dual_transformer_architecture_for_multi_layer_rvq_prediction_achieves role: supports claim: A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns. source: §2.2 evidence: A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns. confidence: high relevance: low - claim_id: autoregressive_tts_systems_trained_on_multi_speaker_dialogue_data_with role: supports claim: Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels. source: §3.2, §4.3, Table 3 evidence: Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels. confidence: high relevance: high - claim_id: sentence_by_sentence_multi_speaker_dialogue_tts_systems_produce_more role: supports claim: Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track. source: §4.4, Table 4 evidence: Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track. confidence: high relevance: low - claim_id: fine_tuning_a_post_trained_dialogue_tts_model_on_as role: supports claim: Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials. source: §4.4, Figure 4 evidence: Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials. confidence: high relevance: low limitations: - '- Currently limited to 3-minute dialogues with up to 4 speakers; scaling requires extending training corpus. - English speaker similarity (SIM 0.665) lags Mandarin (0.736), attributed to limited English voice diversity in training data — a data rather than architectural limitation. - Trails Mimi on PESQ metrics, likely because Mimi was trained on a massive English-only corpus closely matching LibriSpeech. - Emotion fine-tuning is demonstrated for a single distinctive female voice; generalisation to arbitrary voices and more nuanced emotional transitions is not evaluated. - No ablation of the semantic supervision contribution vs. the lower frame rate independently.' caveats: [] - id: '2506.21619' published_date: "2025-09-03" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - gan_based_expressive_transfer - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: tying_the_duration_target_embedding_to_the_semantic_positional_embedding role: supports claim: Tying the duration-target embedding to the semantic positional embedding table enables near-perfect token-count precision in autoregressive TTS without sequence-level supervision. source: § Proposed Method — Duration Control, Table 4 evidence: Tying the duration-target embedding to the semantic positional embedding table enables near-perfect token-count precision in autoregressive TTS without sequence-level supervision. confidence: high relevance: medium - claim_id: gradient_reversal_layer_training_effectively_disentangles_emotional_prosody_from_speaker role: supports claim: Gradient Reversal Layer training effectively disentangles emotional prosody from speaker timbre in zero-shot TTS, enabling independent control of expression and identity from separate reference signals. source: § Proposed Method — Emotional Control, Table 2 evidence: Gradient Reversal Layer training effectively disentangles emotional prosody from speaker timbre in zero-shot TTS, enabling independent control of expression and identity from separate reference signals. confidence: high relevance: high - claim_id: gpt_latent_fusion_between_the_autoregressive_semantic_module_and_the role: supports claim: GPT-latent fusion between the autoregressive semantic module and the flow-matching acoustic module improves phonetic clarity under high emotional expressiveness, as measured by WER on the emotional test set. source: § Semantic-to-Mel Module, Table 2 evidence: GPT-latent fusion between the autoregressive semantic module and the flow-matching acoustic module improves phonetic clarity under high emotional expressiveness, as measured by WER on the emotional test set. confidence: high relevance: high - claim_id: a_three_stage_curriculum_that_separates_base_capability_training_from role: supports claim: A three-stage curriculum that separates base capability training from emotion-conditioner fine-tuning is essential for emotional expressiveness; removing it causes EMOS to collapse from 4.22 to 2.82. source: § Experiment Results — Emotional Performance, Table 2 evidence: A three-stage curriculum that separates base capability training from emotion-conditioner fine-tuning is essential for emotional expressiveness; removing it causes EMOS to collapse from 4.22 to 2.82. confidence: high relevance: high - claim_id: knowledge_distillation_from_a_large_reasoning_llm_into_a_small role: supports claim: Knowledge distillation from a large reasoning LLM into a small fine-tuned model provides a usable soft emotion routing interface for TTS that outperforms instruction-tuned baselines on naturalness and prosody. source: § Text-to-Emotion, Table 3/5 evidence: Knowledge distillation from a large reasoning LLM into a small fine-tuned model provides a usable soft emotion routing interface for TTS that outperforms instruction-tuned baselines on naturalness and prosody. confidence: high relevance: high limitations: - The emotional training data (135 hours across 361 speakers) is modest and partly commercial, raising questions about reproducibility and demographic coverage. The GRL-based disentanglement works for the 7 basic emotions defined, but the emotion space is discrete and limited — continuous or compositional emotion representations are not addressed. The paper notes WER degrades slightly on AISHELL-1 relative to IndexTTS, suggesting the emotional training may slightly hurt clean-speech intelligibility. Duration control is evaluated only at up to 1.25× scaling; extreme compression or expansion is untested. The code release is stated as planned but not confirmed at submission. caveats: [] - id: '2509.03940' published_date: "2025-09-04" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - evaluation_caution - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: acoustic_quality_paralinguistic_appropriateness_is_the_most_consistently_underperformed_capability role: supports claim: Acoustic quality (paralinguistic appropriateness) is the most consistently underperformed capability across current spoken dialogue models, even for proprietary systems, lagging well behind text generation and reasoning dimensions. source: '§Experiments: LLM-based Result, Table 3' evidence: Acoustic quality (paralinguistic appropriateness) is the most consistently underperformed capability across current spoken dialogue models, even for proprietary systems, lagging well behind text generation and reasoning dimensions. confidence: high relevance: high - claim_id: model_parameter_count_does_not_reliably_predict_speech_synthesis_quality role: supports claim: 'Model parameter count does not reliably predict speech synthesis quality in spoken dialogue systems: smaller, better-trained models can match or exceed much larger counterparts on naturalness metrics.' source: '§Experiments: Metric-based Results, Table 2' evidence: 'Model parameter count does not reliably predict speech synthesis quality in spoken dialogue systems: smaller, better-trained models can match or exceed much larger counterparts on naturalness metrics.' confidence: high relevance: low - claim_id: llm_based_evaluation_augmented_with_discrete_acoustic_features_achieves_strong role: supports claim: LLM-based evaluation augmented with discrete acoustic features achieves strong alignment with human judgment on multi-dimensional spoken role-playing tasks. source: '§Evaluation Framework, §Experiments: Subjective experimental results' evidence: LLM-based evaluation augmented with discrete acoustic features achieves strong alignment with human judgment on multi-dimensional spoken role-playing tasks. confidence: high relevance: low - claim_id: there_is_an_optimal_context_window_length_for_spoken_role role: supports claim: There is an optimal context window length for spoken role-playing models beyond which additional conversational history degrades rather than improves performance. source: '§Experiments: Ablation study on context length, Table 4' evidence: There is an optimal context window length for spoken role-playing models beyond which additional conversational history degrades rather than improves performance. confidence: high relevance: medium limitations: - The benchmark is sourced exclusively from movie dialogue, which is scripted and stylistically distinct from spontaneous conversational speech. Generalisation of the evaluation findings to natural speech interaction scenarios is not established. - Character quality validation covers only 20 characters rated by 5 annotators — the 93% satisfactory/acceptable rate is encouraging but the sample is small. The human evaluation for judge validation similarly covers only 20 dialogue instances. The benchmark is currently restricted to two-speaker conversations, limiting applicability to multi-party or group dialogue settings. Future work noted by the authors includes expanding the dataset and fine-tuning models on the role-playing task, but neither training data nor fine-tuning baselines are provided in this paper. caveats: [] - id: '2509.04072' published_date: "2025-09-04" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: narrative_aware_segmentation_of_audiobook_data_into_character_quotation_and role: supports claim: Narrative-aware segmentation of audiobook data into character quotation and narration subsets yields training material with measurably higher emotional diversity than standard sentence-level audiobook splits. source: §3.2, Table 5 evidence: Narrative-aware segmentation of audiobook data into character quotation and narration subsets yields training material with measurably higher emotional diversity than standard sentence-level audiobook splits. confidence: high relevance: high - claim_id: flow_matching_tts_models_show_larger_expressivity_gains_from_fine role: supports claim: Flow-matching TTS models show larger expressivity gains from fine-tuning on targeted expressive speech data than autoregressive models with equivalent training setups. source: §4.3, Table 2 evidence: Flow-matching TTS models show larger expressivity gains from fine-tuning on targeted expressive speech data than autoregressive models with equivalent training setups. confidence: high relevance: high - claim_id: conditioning_tts_synthesis_on_surrounding_narrative_context_rather_than_only role: complicates claim: Conditioning TTS synthesis on surrounding narrative context rather than only the target utterance text improves contextual appropriateness of synthesized speech at the cost of modest intelligibility degradation. source: §4.3, Table 2 evidence: Conditioning TTS synthesis on surrounding narrative context rather than only the target utterance text improves contextual appropriateness of synthesized speech at the cost of modest intelligibility degradation. confidence: high relevance: medium - claim_id: current_open_source_tts_systems_are_substantially_less_expressive_than role: supports claim: Current open-source TTS systems are substantially less expressive than human audiobook narrators on contextual benchmarks, even when naturalness scores (MOS) are comparable. source: §5.2, Table 4 evidence: Current open-source TTS systems are substantially less expressive than human audiobook narrators on contextual benchmarks, even when naturalness scores (MOS) are comparable. confidence: high relevance: high - claim_id: llm_extracted_speech_delivery_pseudo_labels_verbs_and_adverbs_from role: supports claim: LLM-extracted speech-delivery pseudo-labels (verbs and adverbs) from narrative prose are reliable enough to serve as training signals for expressive TTS, achieving high precision when confidence-filtered. source: §3.3, Figure 2 evidence: LLM-extracted speech-delivery pseudo-labels (verbs and adverbs) from narrative prose are reliable enough to serve as training signals for expressive TTS, achieving high precision when confidence-filtered. confidence: high relevance: high limitations: - The LibriQuotetest ground-truth is recorded by amateur LibriVox volunteers, who themselves show insufficient expressivity for many quotations. ContextMOS scores for ground-truth (3.55 average) are only marginally above the best TTS systems, raising questions about whether the benchmark captures professional audiobook narration standards or amateur reading behaviour. - The training set is not WER-filtered, meaning a small proportion of transcription errors may remain. The gender distribution of LibriVox speakers is unknown, introducing potential bias. Experiments use only English fiction; cross-lingual and non-fiction applicability is untested. The contextual conditioning results are encouraging but use only one-paragraph context windows — longer narrative context might yield larger gains. LibriQuote's utility for neural audio codec training (noted as future work) remains to be demonstrated. caveats: [] - id: '2509.06502' published_date: "2025-09-08" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: speaker_conditioned_vad_substantially_reduces_false_barge_in_rates_in role: supports claim: Speaker-conditioned VAD substantially reduces false barge-in rates in full-duplex spoken conversational agents compared to standard VAD, at the cost of slightly higher target-speaker response latency. source: §3.1, Table 2 evidence: pVAD achieves a 10.2% false barge-in rate versus 33.4% (LiveKit SileroVAD) and 78.1% (Ten VAD), with T90 of 170 ms versus 140 ms and 90 ms respectively. confidence: high relevance: low - claim_id: compact_task_specific_classifiers_for_semantic_end_of_turn_detection role: supports claim: Compact task-specific classifiers for semantic end-of-turn detection can match parameter-heavy general-purpose language models while imposing negligible latency overhead. source: §3.2, Table 3 evidence: A 170M-parameter BERT-based EoT model achieves 96.0%/94.9% average accuracy on Chinese/English, on par with a 7B-parameter model (95.8%/94.4%), while using far fewer parameters and thus adding minimal latency to the pipeline. confidence: high relevance: low - claim_id: semi_cascaded_architectures_that_replace_the_asr_llm_cascade_with role: supports claim: Semi-cascaded architectures that replace the ASR+LLM cascade with a single AudioLLM offer paralinguistic awareness and reduced error propagation while maintaining competitive end-to-end latency. source: §2.3.2 evidence: The semi-cascaded variant conditioned on user speech emotion achieves a simpler pipeline with potential latency benefits, mitigates ASR error propagation, and enables paralinguistic-consistent TTS synthesis, without sacrificing the modularity needed for independent component optimisation. confidence: high relevance: high - claim_id: end_to_end_speech_to_speech_models_face_training_and role: complicates claim: End-to-end speech-to-speech models face training and architectural challenges that limit their use as a foundation for full-duplex interaction systems. source: §1 evidence: The authors identify catastrophic forgetting in LLM backbones and token length mismatch between text and speech sequences as key drawbacks of end-to-end approaches, motivating the pluggable modular controller design. confidence: high relevance: medium - claim_id: system_level_evaluation_of_full_duplex_voice_interaction_requires_metrics role: supports claim: System-level evaluation of full-duplex voice interaction requires metrics beyond component-level quality scores, covering barge-in accuracy, semantic end-of-turn detection, and end-to-end latency jointly. source: §3 evidence: FireRedChat introduces three interaction-level metrics (barge-in T90, false barge-in rate, and end-to-first-response P50/P95) that capture naturalness, control accuracy, and efficiency independently of individual ASR, LLM, or TTS scores. confidence: high relevance: low limitations: - All evaluation uses proprietary test data for barge-in and latency experiments. The barge-in test set is 1,000 utterances per language constructed by the authors, and latency is measured on 25 synthetic utterances. External reproducibility of these results is not possible. - Latency remains above industrial-grade systems (DouBao) by 0.3 s (P50) and 0.6 s (P95). The authors identify three concrete engineering improvements, streaming ASR output, streaming TTS input, and reduced VAD chunk size, as a path to closing this gap, but these are not demonstrated. The semi-cascaded pipeline's paralinguistic advantages are described qualitatively; no objective metric (MOS, emotion accuracy) quantifies the improvement in response coherence or emotional consistency. The AudioLLM and FireRedTTS-2 components are in-house proprietary models not released, limiting direct reproducibility of the semi-cascaded results. caveats: [] - id: '2509.09716' published_date: "2025-09-09" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: large_audio_language_model_judges_can_approximate_human_evaluation_of role: supports claim: Large audio-language model judges can approximate human evaluation of speaking style adherence with near-human-level reliability. source: §3.3, Table 2 evidence: LALM-as-a-Judge (Gemini-2.5-pro) achieves Spearman correlations of 77.01% (English) and 73.03% (Chinese) with consensus human scores, comparable to inter-human agreement of 78.58% and 70.54%. confidence: high relevance: high - claim_id: current_open_source_spoken_language_models_exhibit_substantial_deficits_in role: complicates claim: Current open-source spoken language models exhibit substantial deficits in controllable expressive speech generation compared to commercial systems. source: §3.2, Table 1 evidence: Open-source models (Step-Audio, Kimi-Audio, Qwen2.5-Omni, Baichuan-Audio) score 2.46-3.11 overall on VStyle, while commercial systems (GPT-4o, Doubao) reach 3.74-4.10, with the gap attributed to weak acoustic feature modeling and lower instruction-following robustness. confidence: high relevance: high - claim_id: speaking_style_control_does_not_transfer_evenly_across_languages_even role: complicates claim: Speaking style control does not transfer evenly across languages even within the same system, indicating language-specific rather than language-general style representations in current models. source: §3.2 evidence: 'Doubao and Kimi-Audio score substantially higher in Chinese than English (e.g., Doubao: 4.10 vs. 3.88), while the GPT-4o series shows the reverse, attributed to imbalanced training data distributions and differences in pronunciation habits.' confidence: high relevance: high - claim_id: staged_hierarchical_evaluation_frameworks_capture_finer_distinctions_in_speech_generation role: supports claim: Staged hierarchical evaluation frameworks capture finer distinctions in speech generation quality than flat or single-dimension scoring approaches. source: §2.3, Figure 2, §3.3 evidence: VStyle's LALM-as-a-Judge separates content correctness, style adherence, and naturalness into sequential assessment gates, enabling discrimination between systems that achieve content but fail on style versus those achieving both; this structure is validated against human evaluation at near-human-level correlation. confidence: high relevance: low limitations: - The instruction dataset is built from manually designed seeds expanded by LLMs, meaning its distribution reflects annotator preferences and model-driven patterns that may diverge from real user interaction needs. Despite strong aggregate correlation between LALM scoring and human judgment, large audio-language model evaluators remain prone to hallucinations; the paper mitigates this with step-by-step prompts but does not eliminate the risk. The benchmark synthesizes audio instructions using commercial voice cloning, meaning style signals in the input prompts may exhibit a narrower range of speaking styles than naturally produced speech. caveats: [] - id: '2504.20581' published_date: "2025-09-17" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: automated_speaker_similarity_metrics_can_serve_as_architecture_agnostic_proxies role: supports claim: Automated speaker-similarity metrics can serve as architecture-agnostic proxies for voice cloning quality, enabling reproducible evaluation without human raters. source: §3, §4 evidence: ClonEval evaluates five open-source models as black boxes using WavLM cosine similarity, requiring no human listeners or model internals access, while revealing systematic performance differences across systems and conditions. confidence: high relevance: low - claim_id: benchmark_evaluations_using_only_neutral_read_speech_overestimate_the_practical role: complicates claim: Benchmark evaluations using only neutral read-speech overestimate the practical voice cloning performance of current TTS systems. source: §7.1, Table 1 evidence: All five models achieved WavLM cosine similarity in the 0.88–0.91 range on LibriSpeech test-clean, but scores dropped substantially on emotional datasets; OuteTTS fell to 0.54 on SAVEE, revealing that neutral-data performance does not generalize to expressive conditions. confidence: high relevance: medium - claim_id: emotion_transfer_fidelity_varies_systematically_by_emotion_category_with_high role: complicates claim: Emotion transfer fidelity varies systematically by emotion category, with high-arousal negative emotions consistently harder to clone than neutral or positive-valence speech. source: §7.3, Table 3 evidence: 'Across all five evaluated models in Table 3, fear produced the lowest or near-lowest per-emotion scores (e.g., XTTS-v2: 0.7929 for fear vs. 0.848 for neutral), and disgust and anger also ranked consistently below neutral for most models.' confidence: high relevance: high limitations: - The benchmark is English-only; generalization of the protocol and leaderboard to multilingual voice cloning is deferred to future work. Speaker similarity via WavLM cosine similarity does not capture naturalness, intelligibility, or audio artefacts, so a system with high speaker similarity scores may still produce perceptually poor speech. The paper acknowledges the absence of human evaluation and plans to add it. Because the leaderboard relies on vendors self-reporting generated samples, it cannot rule out benchmark-specific optimizations. WavLM representations may not generalize equally across all speaker demographics and acoustic environments, which could introduce bias in rankings for out-of-distribution reference voices. caveats: [] - id: '2509.13989' published_date: "2025-09-17" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: fine_grained_prosodic_control_in_instruction_guided_tts_does_not role: complicates claim: Fine-grained prosodic control in instruction-guided TTS does not improve monotonically with overall system capability. source: §4.3, §4.4, Table 3 evidence: GPT-4o-mini-TTS led all five systems on adverbs-of-degree and emotion-intensity tasks yet achieved only 0.265 on word-level emphasis accuracy (barely above the 0.143 random baseline), and all systems showed near-zero F1 on child-voice generation regardless of system tier. confidence: high relevance: medium - claim_id: human_perceptual_evaluation_reveals_itts_alignment_gaps_that_automated_classifier role: supports claim: Human perceptual evaluation reveals ITTS alignment gaps that automated classifier-based metrics are likely to underestimate. source: §2.2, §3.3, §4.3 evidence: The E-VOC corpus collected 60,000+ ratings from 165+ raters on Prolific; large-scale human annotation found all five systems defaulted to adult-sounding voices regardless of explicit child or elderly instructions, a systematic bias that per-class classifiers tied to their training distribution would likely miss. confidence: high relevance: low - claim_id: current_itts_systems_exhibit_a_systematic_adult_voice_bias_that role: complicates claim: Current ITTS systems exhibit a systematic adult-voice bias that natural-language age instructions alone cannot overcome. source: §4.3, Table 3, Table 4 evidence: GPT-4o confusion matrix (Table 4) shows utterances labeled Child, Teenager, and Elderly were predominantly perceived as Adult by human listeners; the best child-voice F1 across all five systems was 0.113 (Parler-TTS-large), with GPT-4o reaching only 0.074. confidence: high relevance: low - claim_id: commercial_itts_systems_can_achieve_perceptually_ordered_degree_scaled_speech role: supports claim: Commercial ITTS systems can achieve perceptually ordered degree-scaled speech control that open-source alternatives fail to replicate at comparable levels. source: §4.1 evidence: GPT-4o-mini-TTS was the only system producing consistent objective acoustic progressions (LUFS, F0, words per second) and corresponding perceptual gradations under degree modifiers; Parler-TTS variants, PromptTTS++, and UniAudio showed minimal, irregular, or reversed variation. confidence: high relevance: medium limitations: - The study generates audio from only eight conversational transcripts paired with fixed style prompts, which limits the diversity of linguistic contexts covered. Generalisation of findings about speaker age and word-level emphasis to a wider range of sentence types, speaking styles, or TTS voices not included in the study is untested. - Annotation covers only native English speakers from the United States, so judgments of perceived age and prosodic emphasis may reflect dialect-specific or cultural norms. Inter-rater agreement (Cohen's kappa) was low for the adverbs-of-degree task (0.17) and emotion-intensity task (0.226), indicating that even human raters find degree-scaled emotion perception genuinely ambiguous, which complicates interpretation of model scores on those dimensions. The study measures the instruction-perception gap but does not diagnose whether failures arise from model architecture, training data distribution, or prompt formulation, leaving the design question open for future work. caveats: [] - id: '2509.14946' published_date: "2025-09-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis claims: - claim_id: automated_mixing_of_isolated_paralinguistic_audio_events_with_clean_conversational role: supports claim: Automated mixing of isolated paralinguistic audio events with clean conversational speech, combined with voice conversion for timbre alignment, can produce effective fine-tuning data for paralinguistic TTS. source: §4.2.1, Table 3 evidence: CosyVoice2 SFT on SynParaSpeech achieves PMOS 3.31 versus 2.35 for NVS SFT and 1.88 baseline; F5-TTS SFT on SynParaSpeech similarly achieves PMOS 3.10 versus 1.49 for NVS SFT. confidence: high relevance: high - claim_id: dpo_with_naturally_paired_utterances_with_vs_without_paralinguistic_events role: supports claim: DPO with naturally paired utterances (with vs. without paralinguistic events) can improve paralinguistic speech quality beyond SFT alone. source: §4.2.1, Table 3 evidence: CosyVoice2 DPO-Joint achieves PMOS 3.46 and QMOS 4.12 versus SFT-only PMOS 3.31 and QMOS 4.01; joint SFT+DPO outperforms staged DPO across all subjective dimensions. confidence: high relevance: high - claim_id: standard_automatic_speech_quality_metrics_are_unreliable_for_evaluating_paralinguistic role: complicates claim: Standard automatic speech quality metrics are unreliable for evaluating paralinguistic TTS systems. source: §4.2.1, Table 3 evidence: All models fine-tuned on paralinguistic data show degraded CER, SECS, and UTMOSv2 relative to baselines, despite achieving higher subjective NMOS and QMOS; laughter vocalizations raise CER because ASR transcribes them as "ha ha". confidence: high relevance: high - claim_id: few_shot_in_context_prompting_with_paralinguistic_examples_improves_paralinguistic role: supports claim: Few-shot in-context prompting with paralinguistic examples improves paralinguistic event detection in multimodal LLMs. source: §4.2.2, Table 4 evidence: SynParaSpeech prompt tuning raises accuracy and macro F1 for both Kimi Audio and Qwen 2.5 Omni over no-context baselines, with optimal performance at 5-shot context for both models. confidence: high relevance: high - claim_id: increasing_prompt_context_length_for_in_context_learning_in_paralinguistic role: complicates claim: Increasing prompt context length for in-context learning in paralinguistic event detection yields diminishing returns and eventually degrades performance. source: §4.2.2, Table 4 evidence: Kimi Audio accuracy peaks at 5-shot (0.382) and declines at 7-shot (0.371 for Kimi Audio); Qwen 2.5 Omni also peaks at 5-shot (0.473) and declines at 7-shot (0.423), reflecting input overload beyond an optimal context window. confidence: high relevance: high limitations: - The corpus is Chinese-only and covers only 6 paralinguistic categories, limiting direct applicability to other languages or richer paralinguistic inventories. The paralinguistic events are inserted synthetically rather than captured in natural speech, which may introduce distribution mismatch relative to genuinely spontaneous vocalizations. - Standard objective metrics (CER, UTMOSv2, SECS) degrade after paralinguistic fine-tuning and do not reflect the true quality improvement observed in subjective evaluation. This leaves the field without reliable automatic metrics for paralinguistic speech, requiring human listener studies for valid comparisons. The paper does not evaluate zero-shot generalization to unseen paralinguistic categories or cross-lingual transfer. caveats: [] - id: '2509.15253' published_date: "2025-09-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: minor method_family: [] claims: - claim_id: end_to_end_speech_generation_pipelines_that_chain_multiple_vision role: complicates claim: End-to-end speech generation pipelines that chain multiple vision and language components suffer from compounding errors that substantially limit overall system accuracy. source: §3.5 evidence: Speaker attribution accuracy in Setting C is 64.8% and emotion classification micro-F1 is 42.9%, but joint speaker-and-emotion accuracy is only 20.4%, illustrating how independent module errors multiply at the pipeline level. confidence: high relevance: low - claim_id: llm_based_dialogue_attribution_for_tts_struggles_in_spatially_ambiguous role: complicates claim: LLM-based dialogue attribution for TTS struggles in spatially ambiguous layouts where dialogue text is not co-located with the speaking character. source: §3.3, Table 1 evidence: Setting C achieves 79.2% speaker accuracy on easy cases but only 20.5% on hard cases (spatially separated dialogue), showing that the LLM falls back to proximity heuristics that fail when layout breaks co-location assumptions. confidence: high relevance: low - claim_id: imbalanced_emotion_datasets_make_fine_grained_emotion_conditioning_for_tts role: complicates claim: Imbalanced emotion datasets make fine-grained emotion conditioning for TTS unreliable, particularly for the neutral category which dominates real-world speech. source: §2.4, §3.4, Table 3 evidence: 'KangaiSet''s extreme class imbalance (neutral: 35%, disgust: 0.47%) required binary reformulation of 7-class emotion classification; the neutral class still achieves only 34.6% recall in 5-way LLM-based emotion prediction.' confidence: high relevance: high - claim_id: standard_accuracy_metrics_for_emotion_classification_may_underestimate_the_perceptual role: complicates claim: Standard accuracy metrics for emotion classification may underestimate the perceptual quality of emotion-conditioned speech when ground-truth emotion labels are inherently ambiguous. source: §3.4 evidence: Manual inspection of Setting C mispredictions reveals that many misclassified emotions still produce perceptually appropriate voice outputs, and the authors attribute this to inherent visual and contextual ambiguity in comic-style emotional expression. confidence: high relevance: high limitations: - No perceptual evaluation of synthesised speech is conducted. The pipeline's audio output quality, naturalness, and emotional appropriateness are not assessed with listening tests or any objective metric (MOS, UTMOS, WER). The speech synthesis module cannot be independently compared to other TTS systems. - The TTS component's architecture and training setup are not described, making the synthesis stage unreproducible. The pipeline is evaluated exclusively on Japanese manga (Manga109/KangaiSet), leaving generalisation to Western comics, webtoons, or other visual styles untested. Character identification at 62.9% accuracy with only 40 reference images per character introduces substantial upstream error that propagates through speaker attribution. Emotion classification accuracy remains low even with LLM-based reasoning, and the 20.4% joint accuracy suggests significant headroom before deployment quality is reached. The evaluation covers only the first 10 pages per comic, which may not capture the full distribution of visual complexity encountered in complete volumes. caveats: [] - id: '2509.15626' published_date: "2025-09-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - VAE relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - vae_emotion_style_latents claims: - claim_id: impression_leakage_in_controllable_tts_arises_structurally_when_a_single role: supports claim: Impression leakage in controllable TTS arises structurally when a single reference utterance is used for both speaker identity and style conditioning during training. source: §2, §5.2, Table 2 evidence: VIC-base achieves ∆V = 0.22 (significantly different from zero), confirming that the reference audio biases synthesized VI toward the reference's inherent voice impression even when GRL and high-rate dropout are applied. confidence: high relevance: low - claim_id: decoupled_training_using_distinct_utterances_for_speaker_and_style_conditioning role: supports claim: Decoupled training using distinct utterances for speaker and style conditioning reduces style leakage without changing the underlying TTS architecture. source: §4.1, §5.2, Table 2 evidence: VIC-dis uses a separate utterance from the same speaker for reference conditioning during training, reducing ∆V from 0.22 to 0.14 with a statistically significant improvement in RVI-MSE (0.61 to 0.51). confidence: high relevance: low - claim_id: eliminating_the_speaker_reference_and_conditioning_solely_on_a_style role: supports claim: Eliminating the speaker reference and conditioning solely on a style vector achieves the strongest leakage reduction, at a moderate cost in speaker fidelity. source: §4.2, §5.2, Table 2 evidence: VIC-srf reduces ∆V to 0.05 (not significantly different from zero), while SECS drops to 0.72, which remains above the cross-speaker bound of 0.63 but below the same-speaker bound of 0.81. confidence: high relevance: low - claim_id: llm_based_tts_systems_conditioned_on_natural_language_prompts_are role: complicates claim: LLM-based TTS systems conditioned on natural language prompts are inadequate for fine-grained numerical voice impression control. source: §5.1, §5.2, Table 2, Table 3 evidence: Qwen3-TTS (zero-shot) achieves VI-MSE of 0.82 vs. 0.39 for the VITS-based baseline; fine-tuning worsens controllability further (VI-MSE 0.87), and text-VI entanglement is confirmed empirically by punctuation biasing predicted impressions. confidence: high relevance: low - claim_id: perceptual_voice_impression_annotation_can_achieve_inter_annotator_agreement_comparable role: supports claim: Perceptual voice impression annotation can achieve inter-annotator agreement comparable to other subjective speech tasks, supporting the validity of VI corpora. source: §3, Table 1 evidence: Krippendorff's alpha averages 0.470 across 10 VI dimensions, exceeding reported agreement for speech emotion recognition (0.442) and singing voice preference (0.153). confidence: high relevance: medium limitations: - The LibriTTS-VI corpus contains only 130 manually annotated utterances from 130 speakers; the remaining LibriTTS-R annotations are estimated by the VIE trained with data augmentation. Several VI dimensions fall below the reliable inter-annotator agreement threshold (alpha below 0.667), particularly J) Cold-Warm (0.197) and D) Calm-Restless (0.295), limiting annotation reliability for these dimensions. - Evaluation is confined to the LibriTTS-R audiobook domain, which is relatively clean and controlled. Generalisation of both the VIE and the VIC systems to spontaneous or noisy speech is untested. The subjective MOS evaluation uses only two speakers and the authors note potential interface bias due to rating-scale anchoring. The comparison with Qwen3-TTS relies on a single external system evaluated in a zero-shot configuration for which the model was not explicitly designed. caveats: [] - id: '2509.15845' published_date: "2025-09-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - flow_matching_emotional_synthesis - autoregressive_emotion_language_models claims: - claim_id: conditioning_a_flow_matching_model_to_generate_speaker_embeddings_from role: supports claim: Conditioning a flow matching model to generate speaker embeddings from natural language descriptions enables automatic voice assignment for characters without requiring reference audio. source: §2.2, §3.3, Table 2 evidence: The TTT model, trained with OT-CFM to map text descriptions (age, gender, personality, identity) to speaker embeddings, achieves CMS of 2.866 and sex accuracy above 96% for adult demographics; Qwen3-Embedding-0.6B outperforms T5-Large and RoBERTa-Large on all character attribute metrics. confidence: high relevance: low - claim_id: providing_llm_derived_emotion_scene_instructions_as_an_additional_conditioning role: supports claim: Providing LLM-derived emotion-scene instructions as an additional conditioning signal to TTS improves emotional expressiveness without substantially degrading intelligibility. source: §3.3, Table 3 evidence: CA-Instruct-TTS achieves MOS-E of 4.15 vs. 3.67 and MOS-N of 3.33 vs. 3.10 compared to the instruction-free CA-TTS baseline, with WER increasing only from 2.39% to 2.54%. confidence: high relevance: high - claim_id: embedding_based_speaker_generation_from_text_degrades_for_child_voices role: complicates claim: Embedding-based speaker generation from text degrades for child voices due to acoustic similarity and training data artifacts. source: §3.3, Figure 2 evidence: The TTT model shows notably lower age accuracy for the child category (74.38% with TTT-Qwen3-0.6B) compared to adult categories (90%+). The paper attributes this to pre-pubertal voice similarity and to adult speakers imitating child voices in available training data. confidence: high relevance: low - claim_id: larger_semantically_richer_text_encoders_improve_text_conditioned_speaker_embedding role: supports claim: Larger, semantically richer text encoders improve text-conditioned speaker embedding generation across demographic categories. source: §3.3, Table 2 evidence: TTT-Qwen3-0.6B achieves substantially higher CMS (2.866) and age accuracy across most categories compared to TTT-T5-Large (CMS 2.375) and TTT-RoBERTa-Large (CMS 2.359), suggesting that language understanding capacity from pre-trained LLMs transfers to voice attribute generation. confidence: high relevance: low - claim_id: the_flow_matching_framework_extends_naturally_from_speech_waveform_generation role: refines claim: The flow matching framework extends naturally from speech waveform generation to speaker embedding synthesis. source: §2.2 evidence: The OT-CFM framework is applied directly to the generation of continuous speaker embedding vectors from noise, using the same DiT backbone and conditioning architecture as the speech-side flow matching decoder. The speaker embedding space is treated as the target data distribution. confidence: high relevance: low limitations: - All evaluations use proprietary internal data. No comparisons to published expressive audiobook TTS systems are included, limiting the ability to independently verify the reported gains in emotional expressiveness. - The TTT model struggles with child voice generation, an acknowledged limitation tied to data scarcity and label noise in child speech corpora. Future work is noted toward collecting authentic child speech. - The CA-Instruct-TTS relies on a closed-source base model (QinYu), which prevents reproducibility of the LM component despite the open demo and dataset release. - The system has not been evaluated in multilingual settings, and the emotion-scene instruction templates were designed for Chinese-language audiobooks, raising open questions about cross-lingual or cross-cultural generalisation. caveats: [] - id: '2509.16010' published_date: "2025-09-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - VC architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: disentangling_speaker_identity_from_expressive_style_into_separate_adapter_modules role: supports claim: Disentangling speaker identity from expressive style into separate adapter modules improves both speaker similarity and style expressivity in personalised voice cloning. source: §3.3, Table 2 evidence: Ablations on GPT-SOVITS-V4 show removing the private ID-LoRA drops SS from 0.645 to 0.507, while removing the Style-LoRA drops SE from 0.704 to 0.588; a single unified adapter cannot optimise both objectives without mutual degradation. confidence: high relevance: high - claim_id: collaborative_style_learning_across_clients_in_a_federated_setting_can role: supports claim: Collaborative style learning across clients in a federated setting can overcome per-client data scarcity and produce higher speaker similarity than purely local fine-tuning. source: §3.2, Table 2 evidence: Fed-PISA exceeds Local FT (LoRA) in both SS (0.645 vs 0.529) and nMOS (4.08 vs 3.36), demonstrating that sharing expressive style information across clients compensates for limited per-client data. confidence: high relevance: low - claim_id: personalized_aggregation_weighted_by_stylistic_similarity_between_clients_outperforms_uniform role: supports claim: Personalized aggregation weighted by stylistic similarity between clients outperforms uniform averaging for federated style learning. source: §3.3, Table 2 evidence: Under identical GPT-SOVITS-V4 backbone and LoRA configuration, attention-based personalized aggregation achieves SE 0.704 and SS 0.645 versus FedAvg's SE 0.476 and SS 0.523, confirming that style-averaged updates homogenise client models and harm expressivity. confidence: high relevance: medium - claim_id: in_speaker_adaptation_stylisation_steps_beyond_a_small_fraction_of role: complicates claim: In speaker adaptation, stylisation steps beyond a small fraction of total training degrade speaker identity and peak naturalness, establishing a ceiling on achievable style transfer without identity loss. source: §3.4, Figure 2 evidence: Holding total steps at 100, speaker similarity decreases monotonically as stylisation steps increase, while naturalness peaks at 20% stylisation (m=20); excessive stylisation corrupts identity and degrades nMOS. confidence: high relevance: high - claim_id: fair_cross_framework_comparison_in_federated_tts_is_obstructed_when role: complicates claim: Fair cross-framework comparison in federated TTS is obstructed when competing methods are architecturally coupled to legacy backbones and cannot be reimplemented on modern foundations. source: §3.1.2 evidence: FedSpeech and Federated Dynamic Transformer use FastSpeech2-XL and Transformer-TTS respectively and are incompatible with LoRA; the evaluation must compare them on their original configurations, making it impossible to isolate the contribution of the federation strategy from the backbone quality. confidence: high relevance: medium limitations: - The federated baselines (FedSpeech, Federated Dynamic Transformer) run on architecturally distinct and weaker backbones; the performance gap reflects backbone modernization in addition to the proposed federation strategy. No ablation on GPT-SOVITS-V4 with a standard FedAvg baseline using the same LoRA parameter budget is reported to cleanly isolate the aggregation contribution. - Evaluation uses 60 speakers treated as 60 independent clients from four relatively small emotion datasets, which may not represent the statistical diversity or scale of real-world federated deployments. Naturalness is evaluated by 22 in-lab researchers rather than crowd-sourced listeners, which may limit generalisability of the nMOS scores. The paper does not address communication round latency or asynchronous client availability, both of which matter in practical on-device deployment. How the framework scales to many more clients or to cross-lingual speaker sets remains untested. caveats: [] - id: '2509.20378' published_date: "2025-09-20" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models claims: - claim_id: sub_sentence_emotion_conditioning_produces_more_accurate_emotional_dynamics_in role: supports claim: Sub-sentence emotion conditioning produces more accurate emotional dynamics in TTS than sentence-level conditioning. source: §4.1, Table 1 evidence: Emo-FiLM outperforms CosyVoice2 by 9.1% on DTW and 12.7% on ESD DTW, with higher EMOS on both test sets, when comparing word-level emotion labels against global prompt conditioning. confidence: high relevance: high - claim_id: feature_wise_linear_modulation_is_an_effective_mechanism_for_injecting role: supports claim: Feature-wise Linear Modulation is an effective mechanism for injecting word-level conditioning signals into the text representations of autoregressive LLM-TTS systems. source: §4.2, Table 2 evidence: Ablation replacing the FiLM layer with simple addition increases FEDD DTW from 49.62 to 70.5, demonstrating that the non-linear affine modulation is essential beyond mere feature fusion. confidence: high relevance: medium - claim_id: self_supervised_speech_emotion_models_provide_useful_word_aligned_supervision role: supports claim: Self-supervised speech emotion models provide useful word-aligned supervision for fine-grained emotional TTS when combined with forced alignment. source: §2.1, §4.2, Table 2 evidence: emotion2vec frame-level features aligned via MFA and pooled to word boundaries form the annotation backbone; removing word-level data tuning causes the most severe degradation (FEDD DTW 49.62 to 133.97). confidence: high relevance: high - claim_id: frame_averaged_emotion_similarity_metrics_are_insufficient_for_evaluating_intra role: complicates claim: Frame-averaged emotion similarity metrics are insufficient for evaluating intra-utterance emotional dynamics in TTS output. source: §3.4 evidence: The authors note that Emo SIM averages frame-level emotion vectors, potentially obscuring dynamic information, and introduce DTW as a complementary metric that more sensitively distinguishes the models' ability to track emotional transitions. confidence: high relevance: high - claim_id: explicit_emotion_classification_as_an_auxiliary_training_objective_improves_fine role: supports claim: Explicit emotion classification as an auxiliary training objective improves fine-grained emotion control in multi-task TTS training. source: §4.2, Table 2 evidence: Removing the emotion classification loss increases FEDD DTW from 49.62 to 73.96, a degradation comparable to removing the FiLM layer entirely. confidence: high relevance: high limitations: - The FEDD dataset is small (1,000 utterances, 5 speakers) and constructed partly via concatenation of emotionally distinct segments, which may not capture naturally occurring emotional transitions. The method is not evaluated for spontaneous or conversational speech, where emotion boundaries are less well-defined. The backbone (CosyVoice2) is frozen, so the approach inherits any limitations of that system in speaker diversity or prosodic range. Model size and speaker generalization are not reported. caveats: [] - id: '2509.17516' published_date: "2025-09-22" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - gan_based_expressive_transfer claims: - claim_id: decoupling_the_training_pairing_between_the_speech_prompt_and_the role: supports claim: Decoupling the training pairing between the speech prompt and the synthesis target (rather than using identical prompt/target audio) improves text-prosody alignment in style-controllable TTS, at the cost of an inflated speaker-similarity score under the coupled setup. source: §3.3, Table 2 evidence: Switching from non-decoupled to decoupled prompt/target training raises S-MOS from 3.45±0.09 to 3.82-3.93±0.06-0.07 while reducing speaker-similarity (SS) from 0.87 (interpreted as over-similar timbre/prosody) to 0.69-0.8. confidence: high relevance: low - claim_id: providing_an_autoregressive_speech_language_model_with_explicit_surrounding_sentence role: supports claim: Providing an autoregressive speech language model with explicit surrounding-sentence text context improves narrative coherence in long-form speech generation beyond what sentence-isolated synthesis achieves. source: §2.1, §3.3, Table 1 evidence: Adding pre/post-context text sequences raises Chapter M-MOS from 3.88±0.07 (CosyVoice2 baseline, sentence-level synthesis) to 4.13±0.09 (Infer-ctx), and a qualitative example shows the model generating contextually appropriate laughter for a sentence with no explicit emotional markers when given the preceding sentence's context. confidence: high relevance: low - claim_id: the_strength_of_the_prompt_target_decoupling_threshold_used_during role: complicates claim: The strength of the prompt/target decoupling threshold used during training trades off speaker-identity stability against naturalness, rather than improving both simultaneously. source: §3.3, Table 2 evidence: Sweeping the voiceprint-similarity clustering threshold shows a lower threshold reduces speaker-similarity (with occasional timbre discontinuities) but yields slightly higher S-MOS, while a higher threshold approaches the non-decoupled model's inflated similarity at the cost of lower S-MOS. confidence: high relevance: low - claim_id: self_distillation_from_a_pretrained_emotional_tts_model_filtered_by role: supports claim: Self-distillation from a pretrained emotional TTS model, filtered by objective quality metrics, can mitigate the scarcity of high-intensity emotional training data and improve controllable emotional expressiveness. source: §2.2, §3.3, Table 3, Table 4 evidence: On the CV3-Eval text-unrelated test set, the model trained with 500 hours of emotional data plus 5,300 hours of self-distilled augmentation achieves larger high- vs. low-intensity F1 discriminability than a CosyVoice2-instruct baseline (e.g. angry ΔF1 0.31 vs. 0.07) and higher mixed-emotion S-MOS (3.87±0.09 vs. 3.35±0.07). confidence: high relevance: high limitations: - All training data (1M-hour audiobook corpus, 100K-hour context-aware corpus, 500-hour emotional corpus, 5K-hour augmentation set) and all three test sets (Test-NAR, Test-DIA, Test-CHAP) are proprietary and internally constructed; no code, model weights, or evaluation data are released, which limits independent verification of the reported gains. - The evaluation is confined to Chinese-language audiobook content with a single subjective rater pool (50 native Chinese speakers); generalization to other languages or narration styles is untested. The decoupling-threshold ablation reports trends across tested values without identifying a clearly optimal setting, and the paper notes the risk of persona instability at high thresholds is hypothesized rather than directly measured across the full range. The authors flag future work on scaling chapter-level context data and applying reinforcement learning for further quality improvement. caveats: [] - id: '2501.04561' published_date: "2025-09-23" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: progressive_text_pivoted_alignment_across_modality_pairs_can_substitute_for role: supports claim: Progressive, text-pivoted alignment across modality pairs can substitute for paired tri-modal training data without sacrificing downstream omnimodal task performance. source: §4.2, Table 1 evidence: OpenOmni trains only on speech-text and image-text pairs (no image-speech-text triples) yet outperforms VITA, which is trained on 5M tri-modal samples, by 4 points on OmniBench while using a 7B rather than 7×8B language model and roughly 5x fewer training samples. confidence: high relevance: medium - claim_id: non_autoregressive_discrete_unit_speech_decoding_trades_generation_quality_for role: complicates claim: Non-autoregressive discrete-unit speech decoding trades generation quality for latency relative to autoregressive decoding. source: §4.2, §D "AR mode" evidence: The paper reports that its AR mode (NTP loss, 16K-unit vocabulary) yields higher speech generation quality but slower streaming, while the NAR mode (CTC loss, 6K-unit vocabulary) achieves under-1-second latency for up to 30 seconds of speech (5x faster) at the cost of "slightly worse" generation quality. confidence: high relevance: low - claim_id: direct_preference_optimization_can_be_adapted_to_discrete_unit_ctc role: supports claim: Direct preference optimization can be adapted to discrete-unit, CTC-trained speech generators to improve emotional coherence without an auxiliary emotion-control module. source: §4.2, Table 4 evidence: CTC-DPO training on the 9K-pair EO2S-9K preference dataset (Plutchik-based emotion categories, CosyVoice-synthesized positive/negative pairs) raises Emotion2Vec-classified accuracy from 57.9% to 70.4% on Chinese and 62.6% to 65.4% on English test speech. confidence: high relevance: high - claim_id: mixture_of_experts_capacity_is_necessary_not_merely_beneficial_for role: complicates claim: Mixture-of-experts capacity is necessary, not merely beneficial, for stabilizing CTC-loss training of multilingual non-autoregressive speech decoders. source: §3.4, §C, Table 6 evidence: Ablations show a single feed-forward decoder layer (1 expert) fails to converge on bilingual WeNetSpeech/LibriSpeech data (CER/WER of 113.6/129.7/87.8/96.5), while increasing to a 4-expert MoE layer brings these down to single digits (8.5/8.4/4.2/4.7); the paper states that "without this layer, the speech decoder fails to train effectively." confidence: high relevance: medium limitations: - The system is trained and validated only on Chinese and English; the authors explicitly note that multilingual speech data beyond these two languages was not used due to resource constraints, leaving the generalization of the alignment and speech-generation strategy to other languages untested (§Limitation). Separately, the paper acknowledges that because the speech decoder conditions on the LLM's internal hidden states rather than its final decoded text, occasional mismatches between the LLM's actual text answer and the conditioning features used for speech generation can still occur despite the text-guided fusion module designed to mitigate this (§Appendix D). Evaluation of emotional and omnimodal quality relies on automated classifiers (Emotion2Vec, Whisper-based WER) and the authors' own benchmarks rather than independent human listening tests, so subjective perceptual quality of the generated speech is not directly reported. caveats: [] - id: '2505.10599' published_date: "2025-09-25" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - gan_based_expressive_transfer - hybrid_semantic_acoustic_emotion_systems claims: - claim_id: combining_discrete_emotion_labels_and_continuous_dimensional_emotion_representations_within role: supports claim: Combining discrete emotion labels and continuous dimensional emotion representations within one conditioning scheme, rather than choosing one, expands the practically controllable range of emotional TTS beyond what either representation supports alone. source: §4.4, §4.6, Figure 4 evidence: Semi-supervised training across label-only, ADV-only, and jointly-annotated datasets raises ADV-space controllable coverage from 70% (training on jointly-annotated data alone) to 89.35% (full semi-supervised training), including synthesis of previously unseen ADV combinations. confidence: high relevance: high - claim_id: the_reliability_of_continuous_emotion_control_depends_heavily_on_how role: complicates claim: The reliability of continuous emotion control depends heavily on how the continuous space is quantized into controllable units, not just on the underlying representation. source: §4.4, Table 2, §4.6 evidence: Replacing the clustering-based nonlinear ADV quantizer with linear binning drops Spearman's Rank Correlation for perceived linear emotion control across all three ADV dimensions (e.g., valence SRC falls from 0.92 to 0.57), showing that imbalanced raw annotation distributions bias the model toward overrepresented regions of the space unless the quantizer compensates. confidence: high relevance: high - claim_id: predicting_an_emotion_conditioning_signal_directly_from_input_text_allows role: supports claim: Predicting an emotion conditioning signal directly from input text allows emotional TTS to operate without an explicit emotion label or reference at inference time. source: §4.5, §4.6, Table 3 evidence: An ADV predictor infers pseudo-ADV tokens from text alone; removing it biases synthesis toward neutral emotion and drops pairwise preference against baselines from 67.33%/58.60% to 46.88%/28.50% in end-to-end preference tests. confidence: high relevance: high - claim_id: fine_grained_control_over_continuous_emotion_representations_remains_bounded_by role: complicates claim: Fine-grained control over continuous emotion representations remains bounded by the consistency of the human annotations used to train the control mechanism. source: §Limitations evidence: The authors note that subjective inter-annotator variation in ADV labels degrades linear control accuracy, and that texts with ambiguous emotional attributes cause the ADV predictor to infer inappropriate values, since the same text can express different emotions in different contexts. confidence: high relevance: high limitations: - Training and evaluation are restricted to English-language datasets; the paper does not report whether the semi-supervised ADV/label fusion strategy transfers to multilingual emotional speech. The ADV predictor's text-only inference mode is explicitly noted to fail on texts with ambiguous emotional attributes, since a single sentence can carry different emotions depending on context that the model cannot observe. The framework also depends on the quality and consistency of ADV annotations, which the authors identify as inherently noisy due to subjective variation across human annotators; they suggest more data and more consistent annotation as the primary mitigation rather than a modeling fix. Speaker conditioning uses a fixed embedding averaged from a speaker's neutral-emotion utterances rather than zero-shot cloning from an arbitrary reference clip, which is a narrower speaker-control setting than several of the LLM-based baselines it is compared against. caveats: [] - id: '2509.25416' published_date: "2025-09-29" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - diffusion_based_emotional_generation claims: - claim_id: applying_preference_supervision_at_each_denoising_step_of_a_diffusion role: supports claim: Applying preference supervision at each denoising step of a diffusion model, rather than only at the final generated output, improves alignment with a fine-grained target attribute compared to endpoint-propagated preference optimization. source: §3.4, Table 8 evidence: Under a matched Grad-TTS backbone, the stepwise method outperforms three prior diffusion-DPO variants (DDPO, D3PO, Diffusion-DPO) on emotion similarity, prosody similarity, WER, and UTMOS. confidence: high relevance: medium - claim_id: the_assumption_underlying_trajectory_level_preference_propagation_in_diffusion_models role: complicates claim: The assumption underlying trajectory-level preference propagation in diffusion models, that all intermediate states on a path toward a preferred final output are themselves preferred, does not reliably hold. source: §1, §2.2 evidence: The paper motivates its stepwise design specifically by rejecting this assumption, and shows local win/lose selection at a shared latent outperforms endpoint-propagated variants. confidence: high relevance: medium - claim_id: a_contrastive_audio_language_scoring_model_can_be_adapted_to role: supports claim: A contrastive audio-language scoring model can be adapted to evaluate noisy intermediate diffusion states, enabling automatic preference-pair construction without categorical emotion classifiers. source: §2.1, Table 3 evidence: Adding a time-aware normalization layer to a CLAP-based encoder (CLEP) and training it with diffusion-perturbed audio pairs lets it score emotional-prompt consistency directly on noisy mel-spectrograms; removing this timestep conditioning degrades Emo SIM, Prosody SIM, WER, and UTMOS. confidence: high relevance: high - claim_id: in_rlhf_style_fine_tuning_of_diffusion_generation_continuing_rollouts role: complicates claim: In RLHF-style fine-tuning of diffusion generation, continuing rollouts from the highest-reward candidate at each step, rather than a randomly selected candidate, biases training toward degenerate regions and reduces overall quality. source: §2.1, §3.4, Table 4 evidence: The paper reports that random selection of the next rollout state from the candidate pool consistently improves performance relative to continuing from the top-ranked (winning) candidate. confidence: high relevance: medium limitations: - 'Evaluation is confined to a single diffusion TTS backbone (Grad-TTS) and a single, small English emotion corpus (ESD: 5 emotions × 10 speakers, 350 utterances/emotion). Generalization to other diffusion or flow-based TTS architectures, other languages, or continuous/non-categorical emotional targets is untested.' - The preference model (EASPM) is trained on categorical emotion labels plus a fixed set of acoustic descriptors (pitch, loudness, jitter, shimmer) from MSP-Podcast, so its notion of "emotional preference" is still anchored to discrete emotion categories rather than free-form style description. The paper does not report parameter counts for the fine-tuned decoder or wall-clock/inference overhead introduced by the k-candidate sampling and reward scoring at each training step, which matters for RLHF-style diffusion fine-tuning cost. Code availability is not stated in the paper; only a demo page is mentioned. caveats: [] - id: '2509.26514' published_date: "2025-09-30" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_emotion_language_models - flow_matching_emotional_synthesis - gan_based_expressive_transfer claims: - claim_id: decoupling_instruction_interpretation_from_speech_rendering_via_an_explicit_intermediate role: supports claim: Decoupling instruction interpretation from speech rendering, via an explicit intermediate textual feature representation, can match or exceed instruction-annotated controllable TTS systems without requiring any manually labeled instruction-speech data. source: §3.2, Table 1 evidence: BatonVoice-1.7B reaches 57.6% emotion accuracy using 0 hours of manual instruction annotation, exceeding CosyVoice (43.8%, 556 hours) and CosyVoice2 (37.8%, 1,500 hours) on the same Seed-TTS-derived English emotion benchmark. confidence: high relevance: medium - claim_id: when_conditioning_signals_are_represented_as_explicit_text_rather_than role: supports claim: When conditioning signals are represented as explicit text rather than learned embeddings, a downstream generation model can gain synthesis quality from a stronger upstream language model without any retraining of the generation model itself. source: §3.5, Figure 3b evidence: Holding BatonTTS fixed and only changing the external "conductor" LLM that produces the vocal plan, emotion accuracy rises monotonically from 29.8% (Qwen3-1.7B) through 39.8% (Qwen3-80B) and 47.8% (Qwen3-Max) to 57.6% (Gemini-2.5-Pro). confidence: high relevance: low - claim_id: automatic_emotion_classification_accuracy_and_human_judged_naturalness_can_diverge role: complicates claim: Automatic emotion-classification accuracy and human-judged naturalness can diverge, so strong performance on an LLM-judged style-control metric does not guarantee a favorable human preference outcome against commercial systems. source: §3.3, Table 2 evidence: On a free-form instruction-following test set judged by trained human annotators, BatonVoice wins only 56% of comparisons against CosyVoice and just 30% against Minimax-2.5-HD, with annotators specifically citing weaker fluency and naturalness. confidence: high relevance: high - claim_id: numerical_structured_representations_of_prosodic_control_targets_transfer_more_precisely role: refines claim: Numerical, structured representations of prosodic control targets transfer more precisely to a conditioned TTS decoder than free-text qualitative descriptions of the same target style. source: §B.2, Table 5 evidence: On a RAVDESS reconstruction task, the structured numerical vocal-plan format achieves 1.54 Mel-Cepstral Distortion versus 2.62 for an equivalent caption-based qualitative description, and ablating any single numerical feature (pitch, energy, or spectral centroid) increases MCD. confidence: high relevance: medium limitations: - The paper's headline emotion-control results across both English and Chinese benchmarks rely entirely on an LLM (Gemini-2.5-Pro) as an automatic judge rather than human raters; the one benchmark that does use human evaluation shows BatonVoice losing to the top closed-source baseline on naturalness and fluency (30% win rate against Minimax-2.5-HD), a gap not visible in the automated emotion-accuracy metric. - The vocal-plan representation is limited to pitch, energy, and spectral centroid; the authors note that finer-grained paralinguistic features such as emphatic stress and non-verbal vocalizations are not captured and are left to future work. Best reported results depend on using a large, capable, and likely costly external LLM (Gemini 2.5 Pro) as the conductor at inference time; using the in-house model of the same size as the orchestra as its own conductor (Qwen3-1.7B, 29.8% accuracy) performs far worse, so the practical quality of the system is bottlenecked by conductor access and cost, not by BatonTTS alone. Cross-lingual generalization is demonstrated on one unseen language (Chinese) with instructions and text machine-translated by the same LLM family used for judging, which leaves open how the approach performs on languages more typologically distant from English or evaluated independently of the conductor/judge model. caveats: [] claim_clusters: - id: explicit_emotion_conditioning_improves_expressiveness claim: Explicit emotion conditioning improves the recognizability and perceived expressiveness of synthesized speech. status: strongly_supported confidence: high supporting_papers: - 2025.findings-naacl.38 - '2507.04349' - '2507.04598' - '2508.02038' - '2508.04585' - '2508.06890' - '2508.07273' - '2504.12867' - interspeech-2025-0203 - interspeech-2025-0948 - '2506.21619' - '2509.15845' - '2509.20378' - '2509.17516' contradicting_papers: [] refining_papers: - '2503.11026' - 2025.sigdial-1.27 - '2504.12867' caveats: - Results depend on emotion taxonomy, corpus balance, and the strength of the neutral baseline. last_reviewed: '2026-07-30' - id: natural_language_instructions_support_rich_style_control claim: Natural-language descriptions and instructions support richer expressive control than fixed categorical labels alone. status: strongly_supported confidence: high supporting_papers: - '2407.05407' - 2025.findings-acl.101 - 2025.findings-acl.687 - interspeech-2025-0948 contradicting_papers: [] refining_papers: - '2504.12867' caveats: - Instruction understanding and acoustic realization remain separable sources of failure. last_reviewed: '2026-07-30' - id: fine_grained_temporal_conditioning_improves_emotional_dynamics claim: Word-, phoneme-, frame-, and sub-sentence conditioning improves fine-grained emotional dynamics over utterance-level control. status: strongly_supported confidence: high supporting_papers: - '2505.15772' - '2506.16381' - '2507.04349' - '2507.04598' - 2025.ccl-1.77 - '2507.20091' - '2508.06890' - interspeech-2025-0648 - interspeech-2025-0754 - interspeech-2025-1115 - interspeech-2025-1210 - '2509.20378' contradicting_papers: [] refining_papers: - '2507.04349' - '2507.04598' - '2504.12867' - '2509.13989' - '2509.15253' - '2505.10599' caveats: - Finer control requires reliable alignment and can accumulate prediction errors. last_reviewed: '2026-07-30' - id: speaker_emotion_disentanglement_enables_independent_control claim: Disentangling speaker identity from emotion and style enables more independent control of voice and expression. status: strongly_supported confidence: high supporting_papers: - '2508.02038' - interspeech-2025-1101 - interspeech-2025-1115 - interspeech-2025-1210 - interspeech-2025-1394 - '2506.21619' - '2509.16010' contradicting_papers: [] refining_papers: [] caveats: - Identity and habitual expression overlap perceptually, so complete separation may be undesirable. last_reviewed: '2026-07-30' - id: emotion_strength_trades_off_with_intelligibility_and_identity claim: Stronger emotion and style control frequently trades off against intelligibility, naturalness, or speaker identity. status: strongly_supported confidence: high supporting_papers: - '2507.00808' - '2507.04349' - '2507.04598' - 2025.ccl-1.77 - 2025.sigdial-1.27 - '2508.02038' - '2508.03543' - '2508.04195' - '2508.06890' - '2508.07302' - '2508.09600' - '2508.11273' - interspeech-2025-0203 - interspeech-2025-0648 - interspeech-2025-0754 - interspeech-2025-0948 - interspeech-2025-1101 - interspeech-2025-1115 - interspeech-2025-2586 - '2508.17623' - '2506.21619' - '2509.04072' - '2509.15845' - '2509.16010' contradicting_papers: [] refining_papers: - '2402.05755' - '2507.04349' - '2507.18119' - 2025.sigdial-1.27 - '2504.12867' - interspeech-2025-1101 - '2509.14946' - '2509.26514' caveats: - Trade-offs vary with architecture, control strength, and whether vocal events count as transcription errors. last_reviewed: '2026-07-30' - id: multimodal_context_improves_affect_modeling claim: Visual, textual, and conversational context provides complementary affect information that improves emotional speech modeling. status: emerging confidence: medium supporting_papers: - 2025.ccl-1.77 - '2508.16188' contradicting_papers: [] refining_papers: - '2503.11026' caveats: - Multimodal pipelines can compound upstream recognition and attribution errors. last_reviewed: '2026-07-30' - id: dialogue_context_supports_appropriate_emotional_responses claim: Dialogue and narrative context improve the emotional appropriateness and cross-turn coherence of generated speech. status: strongly_supported confidence: high supporting_papers: - '2409.00750' - 2025.findings-acl.101 - 2025.sigdial-1.27 - '2508.04195' - '2508.04585' - interspeech-2025-0408 - '2508.13028' - '2509.02020' - '2509.04072' contradicting_papers: [] refining_papers: - 2025.sigdial-1.27 - '2509.04072' caveats: - Short laboratory interactions do not establish sustained engagement or long-horizon coherence. last_reviewed: '2026-07-30' - id: emotion_representations_benefit_from_combining_categorical_and_continuous_axes claim: Combining categorical emotions with continuous affect dimensions expands controllability beyond either representation alone. status: strongly_supported confidence: high supporting_papers: - interspeech-2025-0948 - '2508.17623' - '2505.10599' contradicting_papers: [] refining_papers: [] caveats: - Continuous controls inherit ambiguity and sparsity from human annotations and quantization choices. last_reviewed: '2026-07-30' - id: data_balance_and_taxonomy_limit_emotion_generalization claim: Emotion taxonomy design, class imbalance, and annotation ambiguity materially limit generalization and control quality. status: emerging confidence: medium supporting_papers: - '2505.15772' - '2508.02038' contradicting_papers: [] refining_papers: - 2025.findings-naacl.38 - '2509.15253' caveats: - Category frequency, speaker demographics, and recording conditions are often confounded. last_reviewed: '2026-07-30' - id: synthetic_and_automatic_annotations_scale_emotional_data claim: Automatic annotation and synthetic augmentation can expand emotional-speech training data when curated labels are scarce. status: strongly_supported confidence: high supporting_papers: - '2505.15772' - 2025.findings-acl.101 - '2508.07273' - interspeech-2025-0166 - interspeech-2025-0648 - interspeech-2025-0754 - '2508.17623' - '2509.14946' - '2509.17516' contradicting_papers: [] refining_papers: - 2025.findings-naacl.38 caveats: - Automatically generated labels can reproduce classifier and prompt biases. last_reviewed: '2026-07-30' - id: reference_and_style_transfer_enable_expressive_resynthesis claim: Reference-based style and prosody transfer enables expressive resynthesis without requiring matched expressive training for every target voice. status: strongly_supported confidence: high supporting_papers: - '2507.04817' - '2508.11273' - interspeech-2025-0815 - interspeech-2025-0948 - '2501.04561' - '2509.25416' contradicting_papers: [] refining_papers: - interspeech-2025-1638 - interspeech-2025-2075 - interspeech-2025-2586 caveats: - Extreme transformations and mismatched references can degrade naturalness and identity. last_reviewed: '2026-07-30' - id: flow_and_diffusion_support_flexible_emotion_control claim: Flow-matching and diffusion generators support flexible emotional conditioning and preference-based refinement. status: strongly_supported confidence: high supporting_papers: - '2507.04349' - '2508.03543' - interspeech-2025-0203 - interspeech-2025-0948 - interspeech-2025-1101 - '2506.21619' - '2509.04072' - '2509.25416' contradicting_papers: [] refining_papers: - interspeech-2025-1210 caveats: - Few studies compare flow, diffusion, and autoregressive systems under matched data and inference budgets. last_reviewed: '2026-07-30' - id: paralinguistic_events_require_dedicated_modeling claim: Laughter, sighs, fillers, emphasis, and other paralinguistic events require dedicated data and objectives beyond ordinary TTS training. status: strongly_supported confidence: high supporting_papers: - '2407.05407' - '2506.16381' - '2507.18119' - '2503.11026' - '2508.02038' - '2508.04195' - '2508.07273' - '2508.09600' - '2509.03940' - '2509.14946' contradicting_papers: [] refining_papers: - '2507.18119' - '2508.07273' - interspeech-2025-0166 caveats: - Lexical metrics penalize intentional non-verbal events and therefore understate perceptual quality. last_reviewed: '2026-07-30' - id: preference_optimization_can_improve_emotional_alignment claim: Preference optimization can improve emotional coherence and expressive alignment beyond supervised fine-tuning alone. status: strongly_supported confidence: high supporting_papers: - interspeech-2025-0704 - '2509.14946' - '2501.04561' contradicting_papers: [] refining_papers: - '2509.25416' caveats: - Reward models may favor classifier-recognizable cues or degenerate trajectories rather than natural expression. last_reviewed: '2026-07-30' - id: automatic_emotion_metrics_do_not_replace_human_evaluation claim: Automatic emotion and style metrics can diverge from human judgments and do not replace perceptual evaluation. status: strongly_supported confidence: high supporting_papers: - 2025.sigdial-1.27 - '2508.04195' - '2508.08095' - '2504.12867' - '2508.09600' - interspeech-2025-0706 - interspeech-2025-0754 - interspeech-2025-1101 - '2508.17623' - '2506.21619' - '2509.04072' - '2509.09716' contradicting_papers: [] refining_papers: - interspeech-2025-0756 - interspeech-2025-1192 - interspeech-2025-1638 - '2509.14946' - '2509.15253' - '2509.26514' caveats: - Agreement varies by emotion, language, system family, and whether judgments target category or trajectory. last_reviewed: '2026-07-30' - id: cross_lingual_emotion_control_remains_uneven claim: Emotion and speaking-style control transfer unevenly across languages, accents, and speaker groups. status: strongly_supported confidence: high supporting_papers: - '2409.00750' - 2025.findings-acl.687 - '2508.02038' - '2508.07302' contradicting_papers: [] refining_papers: [] caveats: - Language coverage, demographic balance, and emotion frequency are rarely isolated. last_reviewed: '2026-07-30' - id: current_systems_lack_robust_emotional_coherence claim: Current speech generators still struggle with robust, sustained emotional coherence under realistic instructions and dialogue conditions. status: emerging confidence: medium supporting_papers: [] contradicting_papers: [] refining_papers: - '2509.09716' caveats: - Capability varies sharply across closed and open systems, languages, and evaluation protocols. last_reviewed: '2026-07-30' method_families: - id: autoregressive_emotion_language_models name: Autoregressive emotion-aware speech language models summary: Autoregressive speech language models generate expressive speech from emotion labels, natural-language instructions, dialogue context, or learned paralinguistic tokens. papers: - '2402.05755' - '2407.05407' - '2409.00750' - '2501.06282' - '2507.21138' - '2507.18119' - 2025.findings-acl.101 - '2508.02038' - '2508.04195' - '2508.04585' - '2507.20091' - '2508.07302' - '2504.12867' - '2508.09600' - interspeech-2025-0704 - interspeech-2025-1115 - '2508.16188' - '2509.02020' - '2506.21619' - '2509.04072' - '2509.14946' - '2509.15845' - '2509.20378' - '2509.17516' - '2501.04561' - '2505.10599' - '2509.26514' open_questions: - How can autoregressive systems retain fine-grained emotional dynamics without sacrificing latency, lexical accuracy, or conversational coherence? - id: flow_matching_emotional_synthesis name: Flow-matching emotional synthesis summary: Flow-matching systems inject global, local, reference-derived, or instruction-derived affect controls into parallel acoustic generation. papers: - '2407.05407' - '2501.06282' - '2507.04349' - '2507.18119' - 2025.findings-acl.101 - 2025.findings-acl.687 - '2503.11026' - '2508.02038' - '2508.03543' - '2508.04195' - '2508.04585' - '2508.07302' - '2504.12867' - '2508.09600' - interspeech-2025-0203 - interspeech-2025-0704 - '2506.21619' - '2509.04072' - '2509.14946' - '2509.15845' - '2509.17516' - '2505.10599' - '2509.26514' open_questions: - Where in the flow trajectory should affect conditioning be applied to maximize control while preserving intelligibility? - id: transformer_encoder_decoder_emotion_control name: Transformer encoder–decoder emotion control summary: Encoder–decoder systems predict or inject emotion and style at sentence, word, phoneme, or frame level through attention and hierarchical conditioning. papers: - 2025.findings-naacl.38 - '2507.00808' - '2507.08012' - '2507.04598' - 2025.sigdial-1.27 - '2508.05385' - '2508.06890' - '2508.07273' - '2508.08095' - '2508.11273' - interspeech-2025-0166 - interspeech-2025-0203 - interspeech-2025-0408 - interspeech-2025-0754 - interspeech-2025-1192 - interspeech-2025-1394 - interspeech-2025-2586 - '2508.13028' open_questions: - Which temporal granularity best balances stable synthesis with perceptually accurate within-utterance emotional change? - id: gan_based_expressive_transfer name: GAN-based expressive transfer summary: Adversarial systems transfer emotional prosody or disentangle it from speaker identity through reference encoders, explicit contours, and auxiliary objectives. papers: - '2312.01479' - '2507.04817' - '2508.06890' - interspeech-2025-0408 - interspeech-2025-0815 - interspeech-2025-2586 - '2506.21619' - '2509.17516' - '2505.10599' - '2509.26514' open_questions: - How far can adversarial prosody transfer move from neutral speech before naturalness and identity deteriorate? - id: hybrid_semantic_acoustic_emotion_systems name: Hybrid semantic–acoustic emotion systems summary: Hybrid systems separate semantic interpretation, affect planning, acoustic rendering, and sometimes speaker or dialogue pathways to retain complementary signals. papers: - '2407.05407' - '2501.06282' - '2508.02038' - interspeech-2025-0756 - interspeech-2025-1638 - '2506.21619' - '2509.06502' - '2509.16010' - '2501.04561' - '2505.10599' open_questions: - Can modular affect planning improve controllability without compounding errors or adding conversational latency? - id: diffusion_based_emotional_generation name: Diffusion-based emotional generation summary: Diffusion systems model expressive acoustic variation through iterative denoising, multimodal conditioning, or trajectory-level preference optimization. papers: - 2025.ccl-1.77 - interspeech-2025-0948 - interspeech-2025-1101 - interspeech-2025-1115 - interspeech-2025-1210 - '2509.25416' open_questions: - Can diffusion preference alignment remain stable when rewards are evaluated on noisy intermediate states? - id: vae_emotion_style_latents name: VAE emotion and style latents summary: VAE-based systems encode emotion, style, and speaker factors in continuous latent spaces for transfer, interpolation, and disentangled control. papers: - '2312.01479' - 2025.findings-acl.687 - interspeech-2025-0815 - interspeech-2025-0948 - '2509.15626' open_questions: - How should latent spaces be regularized so that emotion remains controllable without leaking speaker identity or corpus artefacts? reassessment_queue: - id: emotion_strength_trades_off_with_intelligibility_and_identity type: claim_status reason: Reported trade-offs vary with control strength, architecture, and transcription treatment of vocal events. trigger: Matched control-strength sweeps compare intelligibility, identity, naturalness, and emotion perception. due: 2026-10 current_assessment: strongly_supported watch_for: - Matched intensity sweeps - Listener and lexical evaluation together - id: automatic_emotion_metrics_do_not_replace_human_evaluation type: benchmark_validity reason: Metric–listener agreement varies across categories, languages, and atypical system outputs. trigger: Cross-system, multilingual studies calibrate automatic scores against fine-grained listener judgments. due: 2026-10 current_assessment: strongly_supported watch_for: - Cross-language calibration - Trajectory-sensitive perceptual metrics - id: speaker_emotion_disentanglement_enables_independent_control type: claim_status reason: Identity and habitual expression are correlated, making independence difficult to verify. trigger: Factorial listener studies isolate identity, emotion, accent, and habitual prosody across unseen speakers. due: 2026-10 current_assessment: strongly_supported watch_for: - Factorial identity–emotion tests - Unseen-speaker leakage studies - id: cross_lingual_emotion_control_remains_uneven type: claim_status reason: Language effects remain confounded with training scale, speaker balance, and annotation coverage. trigger: Matched multilingual evaluations reproduce language-specific emotion-control gaps. due: 2026-10 current_assessment: strongly_supported watch_for: - Matched-language evaluations - Demographically balanced emotion corpora - id: preference_optimization_can_improve_emotional_alignment type: claim_status reason: Evidence spans incompatible reward models and includes trajectory-level failure modes. trigger: Independent studies compare preference objectives with matched human evaluation and safety checks. due: 2026-10 current_assessment: strongly_supported watch_for: - Human preference replication - Reward-hacking and degeneration audits open_questions: - Which combination of categorical labels, continuous affect dimensions, and natural-language descriptions gives the most reliable control? - What temporal granularity is sufficient for natural within-utterance emotional dynamics without fragile alignment or error accumulation? - How can systems separate speaker identity from emotion while preserving speaker-specific expressive habits? - Which perceptual protocol can jointly evaluate emotional appropriateness, trajectory, naturalness, intelligibility, and identity? - How can emotional coherence be maintained across long, multi-turn dialogue rather than isolated utterances? - How should consent, cultural variation, demographic balance, and misuse risk be handled in emotional voice generation? trend_notes: - Emotion synthesis is moving from utterance-level category labels toward natural-language, continuous, and temporally localized controls. - Flow matching and speech language models increasingly combine semantic emotion planning with separate acoustic rendering. - Multimodal and conversational context is becoming a primary source of affect rather than an optional auxiliary feature. - Disentanglement objectives increasingly target independent control of identity, emotion, prosody, and paralinguistic events. - Evaluation is broadening from emotion-classification accuracy toward listener judgments of appropriateness, trajectory, coherence, and trade-offs.