concept: instruction-conditioned-tts last_updated: '2026-07-25' paper_count: 44 papers: - id: '2305.11000' published_date: "2023-05-18" entry_date: '2026-07-25' year: 2023 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: - direct_prompt_conditioned_speech_lm - multimodal_spoken_instruction_model - instruction_data_and_annotation_pipeline claims: - claim_id: expanding_an_llm_s_token_vocabulary_with_discretised role: supports claim: Expanding an LLM's token vocabulary with discretised speech units enables a single model to perform both speech comprehension and speech generation without a cascade pipeline. source: §4.1 evidence: Stage 1 (modality-adaptation pre-training) trains LLaMA-13B on LibriLight (60K hours) with a next-token prediction objective over discrete unit sequences, teaching the model to handle the speech modality at scale (96 A100 GPUs, 900 steps). confidence: high relevance: high - claim_id: a_multi_stage_training_curriculum_separating_modality_adaptation role: supports claim: A multi-stage training curriculum, separating modality adaptation, cross-modal instruction tuning, and chain-of-modality alignment, is necessary to acquire reliable cross-modal instruction-following from an LLM backbone. source: §4.2 evidence: Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability. confidence: high relevance: medium - claim_id: the_chain_of_modality_pattern_generating_a_text role: supports claim: The chain-of-modality pattern, generating a text intermediate before the speech response, is a practical mechanism for transferring LLM reasoning capability to speech output. source: §3.2, §4.2 evidence: This chain-of-modality pattern serialises cross-modal reasoning as an explicit text intermediate, mitigating the need for direct speech-to-speech reasoning. confidence: high relevance: medium - claim_id: large_scale_instruction_dataset_construction_via_gpt_4 role: supports claim: Large-scale instruction dataset construction via GPT-4-generated task descriptions applied to existing ASR corpora is a scalable approach to bootstrapping cross-modal training data. source: §3.1 evidence: Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability. confidence: high relevance: medium limitations: - 'The paper provides no quantitative evaluation: no MOS, WER, or speaker similarity scores are reported, and no comparison to cascade baselines is made. All results are case studies. Claims about spoken dialogue quality and instruction-following capability cannot be independently verified from the paper alone.' - 'Additional limitations acknowledged by the authors:' caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2310.00704' published_date: "2023-10-01" entry_date: '2026-07-25' year: 2023 venue: arXiv task: - TTS - VC - singing architecture: - autoregressive-LM - hybrid relevance: low evidence_role: - historical_context current_role: influential method_family: [] claims: - claim_id: training_a_single_audio_language_model_across_diverse role: supports claim: Training a single audio language model across diverse generation tasks (TTS, voice conversion, sound synthesis, music, singing) produces consistent performance improvements over task-specific models trained on the same data. source: §3.4.1, Appendix C.1, Table 17 evidence: 'UniAudio surpasses LM-VC and Make-A-Voice on both speaker similarity and naturalness. (Table 10, Appendix B.1) - Singing voice synthesis (M4Singer): MOS 4.08, comparable to Make-A-Voice (3.96) and DiffSinger (3.94). (Table 12, Appendix B.3) - Speech enhancement (VCTK): Best DNSMOS (3.66) and MOS (3.68) among compared systems, but lower PESQ (2.63) than discriminative baselines, consistent with a known limitation of generative methods on signal-level metrics. (Table 11, Appendix B.2) The ablation on the multi-scale Transformer (Table 4) shows it matches flat-sequence prediction in TTS quality (MOS 3.77 vs. 3.80) at substantially lower memory cost (19.4 GB vs. 36.7 GB) and training time (0.73s vs. 1.63s per iteration), while enabling nq=8 which yields MOS 3.84.' confidence: high relevance: high - claim_id: the_autoregressive_property_is_critical_for_audio_generation role: supports claim: 'The autoregressive property is critical for audio generation quality: parallel and delay-based codec prediction approaches yield measurably lower naturalness than fully autoregressive methods when codec quantization levels are held constant.' source: §3.4.2, Tables 4–5 evidence: 'UniAudio surpasses LM-VC and Make-A-Voice on both speaker similarity and naturalness. (Table 10, Appendix B.1) - Singing voice synthesis (M4Singer): MOS 4.08, comparable to Make-A-Voice (3.96) and DiffSinger (3.94). (Table 12, Appendix B.3) - Speech enhancement (VCTK): Best DNSMOS (3.66) and MOS (3.68) among compared systems, but lower PESQ (2.63) than discriminative baselines, consistent with a known limitation of generative methods on signal-level metrics. (Table 11, Appendix B.2) The ablation on the multi-scale Transformer (Table 4) shows it matches flat-sequence prediction in TTS quality (MOS 3.77 vs. 3.80) at substantially lower memory cost (19.4 GB vs. 36.7 GB) and training time (0.73s vs. 1.63s per iteration), while enabling nq=8 which yields MOS 3.84.' confidence: high relevance: high - claim_id: hierarchical_factorisation_of_rvq_codec_token_sequences_into role: supports claim: Hierarchical factorisation of RVQ codec token sequences into inter-frame and intra-frame modeling substantially reduces training memory and time relative to flat-sequence autoregressive prediction, with comparable generation quality. source: §2.3, §3.4.2, Table 4 evidence: 'UniAudio surpasses LM-VC and Make-A-Voice on both speaker similarity and naturalness. (Table 10, Appendix B.1) - Singing voice synthesis (M4Singer): MOS 4.08, comparable to Make-A-Voice (3.96) and DiffSinger (3.94). (Table 12, Appendix B.3) - Speech enhancement (VCTK): Best DNSMOS (3.66) and MOS (3.68) among compared systems, but lower PESQ (2.63) than discriminative baselines, consistent with a known limitation of generative methods on signal-level metrics. (Table 11, Appendix B.2) The ablation on the multi-scale Transformer (Table 4) shows it matches flat-sequence prediction in TTS quality (MOS 3.77 vs. 3.80) at substantially lower memory cost (19.4 GB vs. 36.7 GB) and training time (0.73s vs. 1.63s per iteration), while enabling nq=8 which yields MOS 3.84.' confidence: high relevance: high - claim_id: pre_training_on_a_broad_multi_task_audio role: supports claim: Pre-training on a broad multi-task audio corpus enables strong adaptation to unseen audio generation tasks via fine-tuning on small datasets, outperforming task-specific models trained from scratch on those tasks. source: §3.3, Appendix B.5–B.8, Table 17 evidence: 'Training proceeds in two stages: joint pretraining on 7 tasks (165K hours), followed by fine-tuning to add 4 new tasks (audio editing, speech dereverberation, instructed TTS, speech editing).' confidence: high relevance: low - claim_id: signal_level_metrics_such_as_pesq_are_poorly role: supports claim: 'Signal-level metrics such as PESQ are poorly suited for evaluating generative audio models: systems achieving higher perceptual MOS scores routinely score lower on PESQ than discriminative baselines.' source: §3.2, §3.4.2, Table 11 evidence: 'UniAudio surpasses LM-VC and Make-A-Voice on both speaker similarity and naturalness. (Table 10, Appendix B.1) - Singing voice synthesis (M4Singer): MOS 4.08, comparable to Make-A-Voice (3.96) and DiffSinger (3.94). (Table 12, Appendix B.3) - Speech enhancement (VCTK): Best DNSMOS (3.66) and MOS (3.68) among compared systems, but lower PESQ (2.63) than discriminative baselines, consistent with a known limitation of generative methods on signal-level metrics. (Table 11, Appendix B.2) The ablation on the multi-scale Transformer (Table 4) shows it matches flat-sequence prediction in TTS quality (MOS 3.77 vs. 3.80) at substantially lower memory cost (19.4 GB vs. 36.7 GB) and training time (0.73s vs. 1.63s per iteration), while enabling nq=8 which yields MOS 3.84.' confidence: high relevance: low limitations: - Model checkpoints are not released due to misuse concerns, limiting reproducibility. Only code and demos are public. Researchers cannot directly reproduce the full 165K-hour training run or perform ablations at scale. - 'UniAudio does not handle all known audio tasks: noise removal, noisy speech editing, and speech-to-speech translation are explicitly excluded. New modalities cannot be introduced during fine-tuning, only new combinations of modalities already seen at training time. The system relies entirely on labeled data; self-supervised or weakly supervised pre-training from unlabeled audio, which could substantially increase coverage, is left as future work.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2312.15821' published_date: "2023-12-25" entry_date: '2026-07-25' year: 2023 venue: arXiv task: - TTS architecture: - flow-matching relevance: low evidence_role: - historical_context current_role: influential method_family: - description_embedding_style_control claims: - claim_id: a_single_generative_model_trained_across_speech_sound role: supports claim: A single generative model trained across speech, sound, and music modalities can match or surpass modality-specific models on dedicated benchmarks. source: §5.4, §6.4, §7.5, Tables 1, 5, 13, 14 evidence: The unified Audiobox model, handling both speech and sound, achieves FAD of 1.10 on AudioCaps, which places it above all unified baselines and above TANGO-full-FT among sound-only models, only trailing the dedicated Audiobox Sound model (0.77). confidence: high relevance: low - claim_id: self_supervised_pre_training_on_large_scale_unlabeled role: supports claim: Self-supervised pre-training on large-scale unlabeled audio substantially improves multi-domain style generalisation in subsequent supervised fine-tuning, with gains most pronounced on out-of-domain test sets. source: §5.5, Table 3 evidence: 'More strikingly, cross-domain generalisation is large: Audiobox Speech improves over Voicebox by 0.096 to 0.156 absolute on CommonVoice, Switchboard, Expresso, and internal Accent datasets.' confidence: high relevance: low - claim_id: general_audio_language_embedding_models_trained_primarily_on role: complicates claim: General audio-language embedding models trained primarily on sound events fail to capture fine-grained speech attributes, rendering them unreliable for evaluating description-conditioned speech generation. source: §7.3.1, Table 8 evidence: 'Second, the pre-trained model is fine-tuned for specific tasks: Audiobox Speech fine-tunes with transcript-guided speech infilling on 100K hours of transcribed multi-domain speech; Audiobox Sound fine-tunes for text-to-sound generation using a two-stage curriculum of noisy tag-based data followed by high-quality captioned data.' confidence: high relevance: high - claim_id: flow_matching_models_admit_post_training_inference_optimisation role: supports claim: Flow-matching models admit post-training inference optimisation via learned ODE reparameterisation (Bespoke Solvers) that reduces function evaluations by 25x without measurable quality degradation. source: §8, Table 15 evidence: For inference acceleration, the paper integrates Bespoke Solvers (Shaul et al., 2023), which learn a reparameterisation of the ODE trajectory to allow accurate integration in as few as 4 steps (8 NFE), compared to 32 steps for the standard midpoint solver and roughly 280 for the adaptive dopri5 solver, without measurable degradation in speaker similarity or WER. confidence: high relevance: low - claim_id: removing_trailing_silence_from_audio_context_prompts_substantially role: supports claim: Removing trailing silence from audio context prompts substantially improves speaker similarity in zero-shot TTS, particularly on datasets with long prompt silences. source: §5.5, Table 3 evidence: 'Third, unified Audiobox combines both tasks in a single model with four conditioning streams: a frame-aligned character transcript, a T5-based text description embedding, a voice prompt encoder (a 3-layer Transformer over EnCodec features), and a masked audio context.' confidence: high relevance: low limitations: - The speech description data is primarily English, and the description-based conditioning pipeline depends on LLM-generated captions derived from categorical attribute labels. The attribute vocabulary (age, gender, pitch, speaking rate, accent, emotion, environment) has limited granularity; fine-grained attributes like specific regional accents or breed-level sound events cannot be reliably generated if no paired training examples exist for those distinctions. - The gap between dedicated Audiobox Sound (FAD 0.77) and unified Audiobox (FAD 1.10) on the AudioCaps benchmark suggests that task-specific models still have an edge in their domain, even at this training scale. The description-based TTS evaluation relies on Joint-CLAP, which is introduced by the same paper and has not been independently validated as a general benchmark metric. The model and training data are not released publicly (no code repository linked), limiting reproducibility. The voice restylization capability, which requires disentangling vocal identity from environment and emotion, is evaluated subjectively on only two internal test sets, and the degree to which the voice prompt de-correlation training objective succeeds across diverse speakers remains unclear. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2402.01912' published_date: "2024-02-02" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - direct_prompt_conditioned_speech_lm - instruction_data_and_annotation_pipeline claims: - claim_id: automatic_acoustic_labeling_can_substitute_for_human_annotations role: supports claim: Automatic acoustic labeling can substitute for human annotations in training large-scale instruction-conditioned speech language models without a loss in attribute control accuracy relative to human-labeled systems. source: §3.1, §3.2, §4.1 evidence: Objective attribute control is validated using the same automatic classifiers that labeled the training data. confidence: high relevance: high - claim_id: including_a_small_proportion_of_high_fidelity_audio role: supports claim: Including a small proportion of high-fidelity audio (approximately 1%) in a predominantly noisy training corpus, combined with explicit recording-quality labels, enables a speech LM to generate professional-sounding speech on demand from text prompts alone. source: §3.1.2, §4.2, Table 1 evidence: 'High audio fidelity is achieved through a combination of: (1) using DAC rather than EnCodec, which the authors report provides subjectively and objectively superior audio quality, and (2) including LibriTTS-R in training, which provides approximately 500 hours (roughly 1%) of clean, professionally enhanced speech alongside the crowd-sourced data.' confidence: high relevance: high - claim_id: the_choice_of_neural_audio_codec_has_a role: supports claim: The choice of neural audio codec has a measurable effect on perceptual audio quality in autoregressive TTS; higher-fidelity codecs translate directly to higher MOS and objective quality scores. source: §3.3, §4.2, Table 1–2 evidence: 'Objective audio quality metrics (PESQ, STOI, SI-SDR) evaluated via TorchAudio-SQUIM on 20 samples with "excellent recording quality" descriptions show the proposed model substantially outperforms Audiobox and approaches ground-truth values: PESQ 3.84 vs. 3.46 (Audiobox), STOI 0.996 vs. 0.988, SI-SDR 26.53 dB vs. 21.84 dB (Table 1).' confidence: high relevance: high - claim_id: natural_language_conditioning_on_accent_can_be_achieved role: supports claim: Natural language conditioning on accent can be achieved in a single TTS model covering dozens of accents, though classifier accuracy reflects the noise and imbalance inherent in automatic accent labeling of crowd-sourced data. source: §3.1.1, §4.1 evidence: A large language model (Stable Beluga 2) then converts keyword tuples into natural language sentences, eg. "a woman with a deep voice speaking slowly and somewhat monotonously with a Hungarian accent in an echoey room with background noise." This approach is applied to the 45k-hour English Multilingual LibriSpeech corpus and the smaller 585-hour LibriTTS-R corpus (included for its higher audio fidelity from the Miipher speech-enhancement model). confidence: high relevance: low limitations: - The evaluation compares only against Audiobox. No standard TTS baselines (reference-based zero-shot systems, encoder-decoder models) are included, making it impossible to assess whether the MOS gains arise from the conditioning approach, the codec choice, or the training data mix. - The system is evaluated only on English audiobook speech; generalization to conversational, spontaneous, or non-English speech is stated as future work but untested. The accent accuracy of 68% indicates that discrete accent labels in crowd-sourced data are noisy, and the model's C50 control was found to be unreliable even after training. Model size and total compute are not reported. The automatic labeling pipeline requires training multiple specialized classifiers (accent, gender), each of which introduces its own noise floor. The demo website is the only verification source; no code or model weights are released. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: '2407.05407' published_date: "2024-07-07" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: influential method_family: - direct_prompt_conditioned_speech_lm claims: - claim_id: inserting_a_vector_quantizer_into_a_supervised_asr role: supports claim: Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. source: §2.1, §5.1, Table 7 evidence: 'Token comparison (Table 7, LibriTTS test-clean): - VALL-E (Encodec tokens): WER 18.7%, SS 53.19 - HuBERT-based (Exp-1): WER 7.41%, SS 67.85 - S3 tokens, BPE text (Exp-3): WER 3.93%, SS 67.85 — best content consistency with same speaker similarity - S3 tokens, large-scale multilingual data (Exp-4-Large): WER 3.17%, SS 69.49 — human parity (original: 3.01%, SS 69.67) CosyVoice vs. ChatTTS (Tables 8, 9): - EN: CosyVoice WER 2.89% vs. ChatTTS 8.32%; SS 74.30 (no ChatTTS SS reported) - ZH: CosyVoice CER 3.82% vs. ChatTTS 3.87%, but ChatTTS has far more ins/del errors (111 vs. 24.4), indicating speaker leakage - With 5x ASR re-ranking: EN WER 1.51%, ZH CER 1.84% Emotion control (Table 10, CosyVoice-instruct): - Happy: 1.00→1.00 (base→instruct), Sad: 0.45→0.98, Angry: 0.59→0.83, Surprised: 0.26→0.64 ASR data generation: ASR model trained on CosyVoice-synthesized LibriSpeech achieves WER 3.0%/6.59% (test-clean/other) vs. 2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement.' confidence: high relevance: high - claim_id: in_autoregressive_codec_lm_tts_both_the_text role: supports claim: In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. source: §5.2, Table 7 evidence: 'Token comparison (Table 7, LibriTTS test-clean): - VALL-E (Encodec tokens): WER 18.7%, SS 53.19 - HuBERT-based (Exp-1): WER 7.41%, SS 67.85 - S3 tokens, BPE text (Exp-3): WER 3.93%, SS 67.85 — best content consistency with same speaker similarity - S3 tokens, large-scale multilingual data (Exp-4-Large): WER 3.17%, SS 69.49 — human parity (original: 3.01%, SS 69.67) CosyVoice vs. ChatTTS (Tables 8, 9): - EN: CosyVoice WER 2.89% vs. ChatTTS 8.32%; SS 74.30 (no ChatTTS SS reported) - ZH: CosyVoice CER 3.82% vs. ChatTTS 3.87%, but ChatTTS has far more ins/del errors (111 vs. 24.4), indicating speaker leakage - With 5x ASR re-ranking: EN WER 1.51%, ZH CER 1.84% Emotion control (Table 10, CosyVoice-instruct): - Happy: 1.00→1.00 (base→instruct), Sad: 0.45→0.98, Angry: 0.59→0.83, Surprised: 0.26→0.64 ASR data generation: ASR model trained on CosyVoice-synthesized LibriSpeech achieves WER 3.0%/6.59% (test-clean/other) vs. 2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement.' confidence: high relevance: high - claim_id: asr_re_ranking_is_an_effective_post_hoc role: complicates claim: ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. source: §5.3, Tables 8, 9 evidence: 'Token comparison (Table 7, LibriTTS test-clean): - VALL-E (Encodec tokens): WER 18.7%, SS 53.19 - HuBERT-based (Exp-1): WER 7.41%, SS 67.85 - S3 tokens, BPE text (Exp-3): WER 3.93%, SS 67.85 — best content consistency with same speaker similarity - S3 tokens, large-scale multilingual data (Exp-4-Large): WER 3.17%, SS 69.49 — human parity (original: 3.01%, SS 69.67) CosyVoice vs. ChatTTS (Tables 8, 9): - EN: CosyVoice WER 2.89% vs. ChatTTS 8.32%; SS 74.30 (no ChatTTS SS reported) - ZH: CosyVoice CER 3.82% vs. ChatTTS 3.87%, but ChatTTS has far more ins/del errors (111 vs. 24.4), indicating speaker leakage - With 5x ASR re-ranking: EN WER 1.51%, ZH CER 1.84% Emotion control (Table 10, CosyVoice-instruct): - Happy: 1.00→1.00 (base→instruct), Sad: 0.45→0.98, Angry: 0.59→0.83, Surprised: 0.26→0.64 ASR data generation: ASR model trained on CosyVoice-synthesized LibriSpeech achieves WER 3.0%/6.59% (test-clean/other) vs. 2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement.' confidence: high relevance: high - claim_id: instruction_fine_tuning_on_a_modest_amount_of role: supports claim: Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. source: §2.4, §5.4, Table 10 evidence: 'CosyVoice-instruct: Built on CosyVoice-base with instruction fine-tuning on 556h of curated data covering speaker identity descriptions, speaking style (emotion, gender, rate, pitch), and fine-grained paralinguistics (laughter, breath, emphasis).' confidence: high relevance: low - claim_id: high_quality_tts_synthesized_speech_can_serve_as role: supports claim: High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. source: §5.5, Table 11 evidence: 'Token comparison (Table 7, LibriTTS test-clean): - VALL-E (Encodec tokens): WER 18.7%, SS 53.19 - HuBERT-based (Exp-1): WER 7.41%, SS 67.85 - S3 tokens, BPE text (Exp-3): WER 3.93%, SS 67.85 — best content consistency with same speaker similarity - S3 tokens, large-scale multilingual data (Exp-4-Large): WER 3.17%, SS 69.49 — human parity (original: 3.01%, SS 69.67) CosyVoice vs. ChatTTS (Tables 8, 9): - EN: CosyVoice WER 2.89% vs. ChatTTS 8.32%; SS 74.30 (no ChatTTS SS reported) - ZH: CosyVoice CER 3.82% vs. ChatTTS 3.87%, but ChatTTS has far more ins/del errors (111 vs. 24.4), indicating speaker leakage - With 5x ASR re-ranking: EN WER 1.51%, ZH CER 1.84% Emotion control (Table 10, CosyVoice-instruct): - Happy: 1.00→1.00 (base→instruct), Sad: 0.45→0.98, Angry: 0.59→0.83, Surprised: 0.26→0.64 ASR data generation: ASR model trained on CosyVoice-synthesized LibriSpeech achieves WER 3.0%/6.59% (test-clean/other) vs. 2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement.' confidence: high relevance: low limitations: - '- Only a single VQ codebook (4096 codes) is used; multi-level RVQ and its effect on quality vs. compression is left for future work. - The choice of VQ insertion layer (after layer 6 of 12) is not ablated — optimal placement is unresolved. - Cross-lingual cloning omits prompt prosody to prevent leakage, which may reduce naturalness in target language. - Instruction fine-tuning data amounts (556h) are modest; broader paralinguistic coverage remains open. - No subjective (MOS) evaluation in the main paper; relies entirely on objective WER/CER/SS metrics.' caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: '2408.16725' published_date: "2024-08-29" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: low evidence_role: - historical_context current_role: influential method_family: - multimodal_spoken_instruction_model claims: - claim_id: simultaneous_text_and_audio_generation_conditioned_on_text role: supports claim: Simultaneous text and audio generation, conditioned on text tokens generated in parallel, enables streaming speech output without the latency penalty of sequential text-then-audio decoding. source: §3.2 evidence: This avoids the latency penalty of sequential text-then-audio generation while retaining the reasoning structure of the text modality. confidence: high relevance: high - claim_id: audio_reasoning_quality_in_end_to_end_speech role: supports claim: Audio reasoning quality in end-to-end speech LMs lags behind text reasoning quality when trained on similar data volumes, and batch inference strategies can partially bridge this gap. source: §3.2, §4.4 evidence: A second inference strategy, batch parallel decoding, addresses the observed gap between text and audio reasoning quality. confidence: high relevance: high - claim_id: a_three_stage_adapter_based_training_curriculum_can role: supports claim: A three-stage adapter-based training curriculum can integrate speech input and output into a frozen language model backbone with minimal degradation to text capabilities. source: §3.3 evidence: 'Mini-Omni builds on Qwen2-0.5B as its backbone language model and extends it with two lightweight adapters: a Whisper-small encoder adapter for speech input and a TTS adapter consisting of six additional transformer blocks for speech output.' confidence: high relevance: high - claim_id: multi_codebook_audio_codecs_with_high_token_rates role: supports claim: Multi-codebook audio codecs with high token rates require parallel decoding schemes to maintain practical streaming throughput in autoregressive speech LMs. source: §3.1, §3.2 evidence: The high codebook depth makes flattened token sequences impractically long, so the model instead uses parallel decoding across all eight heads (one text head plus seven SNAC codebook heads) with a one-step delay between adjacent layers. confidence: high relevance: high limitations: - The paper reports no MOS or naturalness metrics for speech output, making it impossible to quantitatively compare audio quality against TTS or SCA baselines. The claim that quality is "on par with common TTS systems" is unsupported. - Evaluation is restricted to ASR performance on LibriSpeech; there is no evaluation of conversational quality, response coherence, or latency. The 0.5B model size limits reasoning depth, and the paper acknowledges that audio reasoning remains weaker than text reasoning. The VoiceAssistant-400K dataset is entirely synthesized by GPT-4o, which may introduce systematic biases in prosody and topic coverage. The model supports only English. The paper is described as a work-in-progress technical report, with some experiments deferred to a future version. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2409.03283' published_date: "2024-09-05" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - direct_prompt_conditioned_speech_lm claims: - claim_id: separating_the_waveform_generation_stage_into_a_low role: supports claim: Separating the waveform generation stage into a low-sampling-rate Mel decoder and a super-resolution vocoder allows a system trained predominantly on low-sampling-rate data to produce high-fidelity 48kHz output. source: §3.3 evidence: In both cases, a BigVGAN-V2-based super-resolution vocoder upsamples the 16kHz Mel spectrogram to 48kHz waveform, trained on a curated 294-hour high-sampling-rate subset. confidence: high relevance: low - claim_id: few_shot_fine_tuning_of_a_large_foundation role: supports claim: Few-shot fine-tuning of a large foundation TTS model substantially outperforms zero-shot in-context learning for highly expressive, distinctive target voices, even with only one hour of data. source: §5.2.1, Table 5 evidence: 'Two downstream applications are demonstrated: (1) zero-shot and few-shot voice cloning for video dubbing, where UGC scenarios use in-context learning and PUGC scenarios use 1-hour supervised fine-tuning of the LM and flow-matching decoder; (2) instruction-tuned human-like speech generation adding four emotion categories via a dedicated embedding layer and 13 paralinguistic behaviour types via token insertion or embedding injection.' confidence: high relevance: low - claim_id: prompt_audio_enhancement_improves_voice_cloning_quality_for role: supports claim: Prompt audio enhancement improves voice cloning quality for noisy prompts but can slightly degrade performance when prompts are already clean. source: §5.2.2, Table 6 evidence: Prompt enhancement before voice cloning is beneficial at SNR 0dB and 10dB but can slightly degrade performance at SNR 20dB (Table 6), recommending selective application based on noise level. confidence: high relevance: low - claim_id: instruction_tuning_with_a_small_domain_specific_dataset role: supports claim: Instruction tuning with a small domain-specific dataset dramatically improves emotion controllability in a pre-trained TTS language model, raising accuracy from near-chance to near-ceiling. source: §5.3, Table 7 evidence: Instruction tuning lifts emotion classification accuracy from 45–87% (pre-trained) to 97–100% across four emotion categories (Table 7), and paralinguistic behaviour synthesis achieves 45% listener preference vs. 26% without paralinguistics (Table 8). confidence: high relevance: high - claim_id: autoregressive_tts_systems_trained_on_predominantly_one_language role: supports claim: Autoregressive TTS systems trained on predominantly one language show markedly higher pronunciation error rates on under-represented languages, even at large data scales. source: §5.1.2, Table 3 evidence: On stability (pronunciation error rate) across 2,000 utterances, FireRedTTS reduces overall sentence-level error rate to 2.09% vs. CosyVoice's 5.68% in Chinese, but both systems show substantially higher rates in English and code-switch conditions (12% and 8.5% for FireRedTTS), attributed to the limited English proportion in training data (Table 3). confidence: high relevance: high limitations: - All evaluations are conducted on proprietary internal test sets with no publicly released benchmarks, data, or model weights. This makes direct comparison with other systems difficult to reproduce and limits the generalisability of the reported numbers. - The streamable decoder incurs a measurable quality penalty (0.07 CoMOS) and the paper notes that Mel codec quality is a bottleneck, which the authors flag for future work. The English and code-switch pronunciation error rates remain high (12% and 8.5%), driven by limited language diversity in training data. The paralinguistic behaviour framework currently supports 13 types targeting primarily Chinese conversational speech; coverage of other languages and more complex prosodic phenomena is not addressed. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: '2409.06666' published_date: "2024-09-10" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - SCA architecture: - hybrid relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: - multimodal_spoken_instruction_model claims: - claim_id: end_to_end_speech_llms_with_parallel_text role: supports claim: End-to-end speech LLMs with parallel text and speech generation can achieve lower response latency than cascaded ASR-LLM-TTS pipelines without sacrificing prosody coherence under low-latency streaming conditions. source: §4.5 evidence: Critically, unlike cascaded systems where reducing the word chunk size degrades speech rate and naturalness, LLaMA-Omni maintains stable prosody across latency settings because streaming unit generation is end-to-end — only the final vocoder step is cascaded. confidence: high relevance: high - claim_id: aligning_llm_output_to_speech_interaction_conventions_through role: supports claim: Aligning LLM output to speech interaction conventions through targeted instruction data rewriting substantially improves response style suitability, independently of model architecture. source: §3, §4.4 evidence: 'To provide suitable training data, the authors construct InstructS2S-200K: 200K speech instruction-response pairs derived from Alpaca and UltraChat text data by rewriting instructions for spoken style (via Llama-3-70B-Instruct) and synthesising responses to speech using VITS trained on LJSpeech.' confidence: high relevance: medium - claim_id: non_autoregressive_ctc_decoding_from_llm_hidden_states role: supports claim: Non-autoregressive CTC decoding from LLM hidden states enables streaming speech synthesis whose speech rate and naturalness are robust to chunk size variation, unlike word-level streaming TTS cascades. source: §4.5, Table 4 evidence: Critically, unlike cascaded systems where reducing the word chunk size degrades speech rate and naturalness, LLaMA-Omni maintains stable prosody across latency settings because streaming unit generation is end-to-end — only the final vocoder step is cascaded. confidence: high relevance: high - claim_id: training_an_end_to_end_speech_interaction_model role: supports claim: Training an end-to-end speech interaction model on a small, carefully curated speech instruction dataset is sufficient to significantly close the gap with models trained on orders of magnitude more data, provided the LLM backbone is sufficiently capable. source: §4.4, §5 evidence: 'To provide suitable training data, the authors construct InstructS2S-200K: 200K speech instruction-response pairs derived from Alpaca and UltraChat text data by rewriting instructions for spoken style (via Llama-3-70B-Instruct) and synthesising responses to speech using VITS trained on LJSpeech.' confidence: high relevance: medium limitations: - The ASR-WER of 10.82% is notably higher than cascaded baselines (3.78% for SALMONN+Orca), reflecting that the speech decoder is trained on only approximately 1K hours of response speech — far below industrial TTS scale. Intelligibility limitations restrict applicability in domains requiring precise spoken content. - The evaluation benchmark (InstructS2S-Eval) is derived from AlpacaEval with math and code questions removed, which skews toward conversational helpfulness and may not represent more demanding speech interaction tasks. The speech encoder relies on Whisper, which is optimised for ASR rather than general speech understanding, potentially limiting response to prosodic or para-linguistic cues in the user's speech. The current architecture does not support full-duplex interaction (interruption, turn-taking) — speech responses are generated after the full instruction is received. The training data is synthesised from text corpora, which may not capture the naturalness and variability of real spoken dialogue. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2410.17196' published_date: "2024-10-22" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - evaluation - SCA architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: - multimodal_spoken_instruction_model claims: - claim_id: pipeline_based_voice_assistants_asr_plus_llm_substantially role: supports claim: Pipeline-based voice assistants (ASR plus LLM) substantially outperform end-to-end audio LLMs on spoken instruction understanding, with the gap persisting even for state-of-the-art proprietary end-to-end systems. source: §3.3, Table 3 evidence: The Naive-4o pipeline (GPT-4o backend) reaches 87.2, and GPT-4o-Audio achieves 86.4 as a proprietary end-to-end system, closing most but not all of the gap. confidence: high relevance: medium - claim_id: end_to_end_audio_llm_training_can_degrade role: supports claim: End-to-end audio LLM training can degrade the underlying LLM's text-instruction capabilities, including safety alignment, compared to using the same base model in a pipeline configuration. source: §3.3 evidence: The naive Whisper + LLaMA-3.1-8B pipeline achieves 79.1 on the composite VoiceBench score (speech form), while the best open-source end-to-end model, DiVA, scores 55.7 (Table 3). confidence: high relevance: medium - claim_id: mispronunciation_causes_significantly_greater_performance_degradation_in_voice role: supports claim: Mispronunciation causes significantly greater performance degradation in voice assistants than disfluencies or grammatical errors, because it increases effective word error rate rather than merely adding irrelevant context. source: §6.2, Table 5 evidence: Mispronunciation causes the largest content degradation, with an average score reduction of 20.3% across models; repair disfluencies cause a 12.6% reduction (Table 5). confidence: high relevance: medium - claim_id: evaluations_using_clean_synthetic_speech_substantially_overestimate_voice role: supports claim: Evaluations using clean synthetic speech substantially overestimate voice assistant robustness relative to real recorded speech, particularly for models with non-standard speech encoders. source: §3.3, Table 4 evidence: Real accent data (SD-QA) is substantially harder than synthetic accent data, particularly for low-resource accents such as Philippine and Indian English. confidence: high relevance: medium - claim_id: the_choice_of_speech_encoder_determines_a_voice role: supports claim: The choice of speech encoder determines a voice assistant's robustness to accent and speaker variation more than the choice of base LLM. source: §4.2 evidence: VITA shows the greatest accent sensitivity, hypothesised to stem from its non-Whisper CNN+Transformer speech encoder. confidence: high relevance: medium limitations: - VoiceBench evaluates only the text content of responses, not speech output quality. Moshi, the only tested model that natively outputs speech, is therefore evaluated under the same protocol as text-output models, which means the benchmark does not measure naturalness, prosody, or latency in the response speech stream. - The benchmark covers evaluation in English only, and generalisation claims about accent robustness are limited by the diversity of the accent TTS systems used. Content perturbations are all generated synthetically via GPT-4o rewriting, which may not fully capture the distribution of natural human disfluencies. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2412.02612' published_date: "2024-12-03" entry_date: '2026-07-25' year: 2024 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: low evidence_role: - historical_context current_role: influential method_family: - direct_prompt_conditioned_speech_lm - multimodal_spoken_instruction_model claims: - claim_id: speech_text_interleaved_pre_training_at_trillion_token role: supports claim: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. source: §4.1, Table 4 evidence: 'The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607).' confidence: high relevance: high - claim_id: single_codebook_supervised_speech_tokenizers_derived_from_asr role: supports claim: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. source: §3.1, Table 1 evidence: 'The design deliberately sacrifices full acoustic fidelity for semantic compactness: a single codebook at 12.5Hz avoids the multi-codebook complexity of RVQ-based codecs while preserving enough semantic content for accurate ASR (2.10 WER on LibriSpeech clean) and high-quality reconstruction (MOSNet 3.39).' confidence: high relevance: high - claim_id: streaming_interleaved_generation_templates_alternating_text_and_speech role: supports claim: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. source: §3.3 evidence: 'The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607).' confidence: high relevance: high - claim_id: end_to_end_speech_language_models_that_include role: supports claim: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. source: §5.2, Table 6 evidence: Training uses a mix of supervised ASR data (LibriSpeech, GigaSpeech, CommonVoice, AISHELL-1, MLS-Eng, Wenet) and 700k hours of pseudo-labelled unsupervised speech. confidence: high relevance: high - claim_id: decoupling_the_text_and_speech_output_subtasks_during role: supports claim: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. source: §4.2.2 evidence: Separate loss-masking passes train the text-output and speech-output subtasks at different rates (4 vs. 20 epochs) to account for their different learning speeds. confidence: high relevance: low limitations: - The paper reports no subjective listening test (MOS/MUSHRA) on the chat model output; UTMOS is used as a proxy for speech naturalness, and the chat evaluation relies on GPT-4o scoring of ASR transcriptions, introducing cascaded error from both the vocoder quality and the Whisper transcription step. - 'The 175bps tokenizer trades acoustic fidelity for compactness: VisQOL at 12.5Hz (2.52) is lower than SpeechTokenizer variants and the 50Hz variant of the same system. This may limit voice cloning quality and the fidelity of paralinguistic feature reproduction (accent, fine-grained emotion), though the paper does not directly evaluate these.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2501.06282' published_date: "2025-01-10" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: - direct_prompt_conditioned_speech_lm - multimodal_spoken_instruction_model claims: - claim_id: large_scale_multi_task_training_across_heterogeneous_speech role: supports claim: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. source: §3.4, §4.1–4.5 evidence: MinMo adopts an aligned multimodal architecture that adapts a pretrained text LLM by adding audio input and output pathways without fundamentally replacing the LLM's operating regime. confidence: high relevance: high - claim_id: aligned_multimodal_architectures_can_achieve_instruction_controlled_speech role: supports claim: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. source: §3.2, §4.4, Table 18 evidence: End-to-end instruction control (emotion, dialect, speaking rate, voice identity) is achieved by conditioning on hidden embeddings from the LLM, which carry the user instruction context through to the audio output. confidence: high relevance: medium - claim_id: full_duplex_spoken_dialogue_at_sub_second_latency role: supports claim: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. source: §3.1, §4.5, Table 21 evidence: The end-to-end full-duplex response latency is ~600ms in theory (250ms duplex decision + 150ms text generation + 70ms speech token generation + 130ms token-to-wave), and ~800ms in practice on L20 GPUs. confidence: high relevance: high - claim_id: an_autoregressive_streaming_voice_decoder_that_interleaves_text role: supports claim: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. source: §3.2, §4.4, Table 17 evidence: 'The decoder operates on interleaved sequences of semantic vectors and speech tokens in a fixed 5:15 ratio: every five text tokens produce five semantic vectors via a linear output projector, and these drive fifteen autoregressive speech tokens from the Voice Token LM.' confidence: high relevance: high limitations: - The instruction-following voice generation evaluation uses a single in-house Chinese test set of 122 turns, making external validation of the 98.4% accuracy figure impossible. Code and model weights had not been released at time of publication. - LoRA-only updates to the text LLM during training limit the model's ability to follow diverse instructions; more comprehensive LLM updates with higher-quality text data remain unexplored. Long-tail pronunciation errors persist in end-to-end audio generation, partly due to special symbols that the decoder cannot reliably convert to speech. The full-duplex module still relies on external acoustic echo cancellation (AEC) and voice activity detection (VAD) modules, meaning a truly end-to-end duplex system has not been achieved. Performance on speech emotion recognition in low-resource languages shows mixed results (e.g., Polish at 55.9% F1), indicating that cross-lingual generalization is uneven despite the broad multilingual training. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2502.11946' published_date: "2025-02-17" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: low evidence_role: - historical_context current_role: influential method_family: [] claims: - claim_id: a_dual_codebook_interleaved_tokenizer_that_combines_linguistic role: supports claim: A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality. source: §4.4, §6.2.1 evidence: 'This interleaving is the core tokenizer innovation: semantic tokens maintain semantic coherence in generated continuations, while linguistic tokens preserve the acoustic quality needed for intelligible reconstruction.' confidence: high relevance: high - claim_id: scaling_autoregressive_llm_backbone_size_from_3b_to role: supports claim: Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive. source: §6.2.2, Table 3 evidence: On the SEED-TTS benchmark for TTS quality, Step-Audio-TTS-3B achieves 1.31% CER (Chinese) and 2.31% WER (English), outperforming CosyVoice, MaskGCT, and FireRedTTS on intelligibility while remaining competitive on speaker similarity. confidence: high relevance: high - claim_id: rlhf_reward_models_trained_on_speech_interaction_data role: supports claim: RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training. source: §5.2.6 evidence: The AQTA system is post-trained with PPO-based RLHF using a two-stage reward model, where a known failure mode ("deaf hacking") was identified and mitigated by constructing explicit counter-examples in the reward training data. confidence: high relevance: low - claim_id: speculative_response_generation_triggered_by_voice_activity_detection role: supports claim: Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems. source: §3.4 evidence: Real-time inference uses speculative response generation triggered by voice activity detection pauses. confidence: high relevance: low - claim_id: synthetic_speech_data_generated_by_a_large_multi role: supports claim: Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles. source: §5.1.1 evidence: The same decoder architecture underlies the distilled standalone Step-Audio-TTS-3B model, which is trained on synthetic data produced by the 130B model rather than manually curated recordings. confidence: high relevance: low limitations: - The StepEval-Audio-360 benchmark is proprietary and created by the same team; human evaluation results on it cannot be independently reproduced. Open-source benchmark comparisons mix locally re-run models with results taken from original publications, complicating direct numerical comparison. - 'Speaker similarity scores for the distilled Step-Audio-TTS-3B are noticeably lower than CosyVoice 2 on both Chinese and English SEED-TTS tests, suggesting that the dual-codebook approach trades some acoustic identity preservation for intelligibility gains. The AQTA+TTS design still relies on a cascade: errors in ASR transcription of history or in text generation propagate to the TTS stage. The paper''s future work section acknowledges that purely end-to-end audio-in/audio-out (AQAA) remains unsolved. Evaluation for singing, RAP, and dialect control is limited to instruction following scores without reference audio; absolute quality in these dimensions is difficult to assess from the reported numbers alone.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: 2025.findings-naacl.298 published_date: "2025-04-29" entry_date: '2026-07-25' year: 2025 venue: NAACL task: - TTS - evaluation architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: instruction_conditioned_tts_models_reflect_and_amplify_occupational role: supports claim: Instruction-conditioned TTS models reflect and amplify occupational gender stereotypes present in training data when interpreting ambiguous style prompts. source: §4.3, Table 1 evidence: Across four Parler-TTS variants and 109 occupations, chi-square analysis shows statistically significant overrepresentation of male voices for male-coded occupations and female voices for female-coded occupations, with standardized residuals up to 27.2 for individual occupation-model pairs. confidence: high relevance: medium - claim_id: prompt_based_fairness_interventions_can_introduce_compensatory_bias role: complicates claim: Prompt-based fairness interventions can introduce compensatory bias rather than achieving gender neutrality in instruction-conditioned TTS. source: §4.4, Tables 2–4 evidence: Methods 2 and 3 (adding explicit fairness language to style prompts) frequently reverse the bias direction for traditionally masculine occupations across all four models, shifting overrepresentation from male to female rather than reducing bias magnitude. confidence: high relevance: medium - claim_id: gender_bias_magnitude_in_instruction_guided_tts_varies role: complicates claim: Gender bias magnitude in instruction-guided TTS varies substantially across model variants trained on the same data distribution, limiting the generalizability of any single model's bias profile. source: §4.3 evidence: Large v1 and Mini v1 share training data but display different bias patterns by occupation; Mini v0.1 and Mini Expresso (fine-tuned variant) likewise differ, suggesting model scale and fine-tuning interact with bias in ways not fully explained by data alone. confidence: high relevance: medium - claim_id: evaluating_gender_bias_in_tts_requires_controlling_for role: supports claim: Evaluating gender bias in TTS requires controlling for the model's inherent gendered tendency in baseline generation, separate from the effect of specific style prompts. source: §3.3, §4.2 evidence: Even with an empty style prompt or neutral non-occupational prompts, Parler-TTS models generate voices skewed toward a particular gender, necessitating three distinct control groups to isolate the effect of occupation-related prompting. confidence: high relevance: medium limitations: - The analysis is restricted to Parler-TTS, the only publicly available open-source model family that accepts free-form style prompts without requiring a reference audio clip at the time of publication. All quantitative findings are specific to this model family and may not generalise. - The study uses a binary gender classification framework (male/female) constrained by current automatic gender recognition models. Occupations are drawn from a fixed list of 109 that, while broad, cannot represent the full diversity of professions. Emotion and speaking rate are measured but no occupation-related effects are found for either attribute, leaving open the question of whether other speech attributes (accent, perceived age, confidence) also carry occupational bias. No training-time debiasing approaches are evaluated. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: 2025.findings-naacl.38 published_date: "2025-04-29" entry_date: '2026-07-25' year: 2025 venue: NAACL task: - TTS architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - structured_intermediate_control claims: - claim_id: selecting_and_acoustically_emphasizing_a_small_subset_of role: supports claim: Selecting and acoustically emphasizing a small subset of emotion-salient words, rather than conditioning on a single holistic style embedding, improves emotion recognizability of synthesized dialogue speech. source: §5.1, Table 1 evidence: The proposed model improves Speech Emotion Accuracy by 12.98 points over an emotion-label-only baseline and 16.47 points over a dialogue-context-only baseline, with consistent MOS and ABX gains across all five emotion categories. confidence: high relevance: low - claim_id: large_language_models_can_identify_text_spans_that role: supports claim: Large language models can identify text spans that carry emotional salience with moderate agreement to human judgment, without being trained on acoustic or prosodic data. source: §5.2, Table 2 evidence: GPT-3.5's word selections for emotion-bearing rationales overlap with a single human annotator's selections at a rate of 0.735, with comparable average word counts (2.5 vs. 3.2 words per utterance) across four non-neutral emotion categories. confidence: high relevance: high - claim_id: training_a_separate_modality_independent_representation_for_masked role: complicates claim: Training a separate, modality-independent representation for masked acoustic cues is necessary; forcing a single representation to jointly encode textual rationale and acoustic detail degrades performance. source: §5.3, Table 4 evidence: Removing the dedicated acoustic embedding and relying only on the phoneme-level binary rationale label drops SEA by 6.94 points relative to the full model; using full (unmasked) audio as the embedding target rather than selectively masked audio drops SEA by a further 9.59 points relative to the full model. confidence: high relevance: low - claim_id: llm_based_emotion_tag_prediction_accuracy_varies_substantially role: complicates claim: LLM-based emotion tag prediction accuracy varies substantially across emotion categories and is markedly worse for emotions underrepresented in the training distribution. source: §5.2, Appendix A, Figure 4 evidence: Stage 1 prompting holistic emotion tag accuracy is 84% for happiness and 74% for surprise but only 57% for anger and sadness, the two emotion categories that together make up under 7% of the labelled training utterances. confidence: high relevance: low limitations: - 'The training and evaluation data are heavily emotion-imbalanced: only 20.4% of DailyTalk utterances carry a non-neutral emotion label, and within that subset happiness accounts for 85.8% while anger is only 2.55%. Results for the lowest-accuracy, lowest-frequency emotion categories (anger, sadness) should be treated with caution.' - The paper's own limitations section flags that the approach assumes GPT-3.5 can reliably correlate text-based word selection with the acoustic features a human speaker would naturally emphasize when reading the same text aloud, a capability the authors acknowledge is not independently verified beyond the single-annotator overlap study and the Llama-3 generalizability check. The evaluation uses only one dataset (DailyTalk) and one TTS backbone (FastSpeech2); whether the selective masking loss transfers to autoregressive or diffusion-based backbones, or to languages other than English, is untested. No code or demo availability is reported in the paper. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2506.16381' published_date: "2025-06-19" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: fine_grained_paralinguistic_control_independent_manipulation_of_multiple role: complicates claim: Fine-grained paralinguistic control — independent manipulation of multiple acoustic attributes from natural-language instructions — remains an open challenge for current TTS systems, including commercial offerings. source: §4.3, Table 5 evidence: 'Three task types are defined at increasing levels of abstraction: - Acoustic-Parameter Specification (APS): structured, fine-grained instructions covering all 12 features, generated by prompting Gemini to caption each reference clip. - Descriptive-Style Directive (DSD): free-form prose descriptions rewritten from APS captions by GPT-4o, with random attribute dropout to simulate underspecified prompts. - Role-Play (RP): high-level role or scenario descriptions (eg., "a teacher scolding a student") that require the TTS system to infer appropriate acoustic properties from world knowledge, using chain-of-thought GPT-4o generation.' confidence: high relevance: medium - claim_id: a_substantial_performance_gap_exists_between_closed_source role: supports claim: A substantial performance gap exists between closed-source commercial TTS systems and open-source alternatives on instruction-following benchmarks, widening as instruction abstraction increases. source: §4.3, Tables 5–6 evidence: Among open-source systems, VoxInstruct leads at 50.4%, with Parler-TTS-mini (46.9%), Parler-TTS-large (45.7%), and PromptTTS/PromptStyle below 48%. confidence: high relevance: medium - claim_id: timbre_controllability_and_emotional_expressiveness_represent_orthogonal_capabil role: complicates claim: Timbre controllability and emotional expressiveness represent orthogonal capabilities in current TTS systems, as models that succeed on one dimension frequently fail on the other. source: §4.4, Table 7 evidence: 'Case studies reveal three systematic failure modes across all evaluated systems: reproduction of paralinguistic sound events (sighs, laughter, screams), extreme emotional intensity and rapid affective transitions, and singing or melodic vocal production.' confidence: high relevance: medium - claim_id: automatic_llm_based_evaluation_of_instruction_following_in role: supports claim: Automatic LLM-based evaluation of instruction-following in speech achieves approximately 79% agreement with human annotators, with lower reliability on subjective role-play tasks. source: §4.1, Table 4 evidence: Human-Gemini consistency is validated on 50 reference samples per task and language, achieving 79% average agreement across tasks and languages (87% on APS, 79% on DSD, 71% on RP). confidence: high relevance: medium - claim_id: paralinguistic_vocal_events_such_as_laughter_sighs_and role: supports claim: Paralinguistic vocal events such as laughter, sighs, and screams are not reliably reproducible by any current TTS system when specified in natural-language instructions. source: §4.4, Table 7 evidence: 'Case studies reveal three systematic failure modes across all evaluated systems: reproduction of paralinguistic sound events (sighs, laughter, screams), extreme emotional intensity and rapid affective transitions, and singing or melodic vocal production.' confidence: high relevance: medium limitations: - Gemini is used both as the caption generator for benchmark construction and as the judge evaluating model outputs. Gemini TTS models score highest on the benchmark, and the authors acknowledge LLM self-evaluator bias (Panickssery et al., 2024) as a confound. Gemini scores should be treated with caution. - 'The RP task is inherently subjective: a scenario description like "nervous applicant in an interview" admits many valid acoustic realisations, and lower human-Gemini agreement (71% average) reflects this ambiguity rather than evaluator failure. The dataset construction pipeline draws from film and television sources, so emotional and stylistic distributions reflect those domains; certain categories (specific emotions, role archetypes) are under-represented by design. Continuous large-scale evaluation using Gemini-as-a-judge costs roughly $25 per full evaluation run, which may limit iterative use.' caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2507.00808' published_date: "2025-07-02" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - description_embedding_style_control claims: - claim_id: iterative_natural_language_feedback_can_progressively_refine_the role: supports claim: Iterative natural language feedback can progressively refine the speaking style of synthesized speech without accumulating naturalness degradation. source: §4.2, §4.4, Figure 4, Figure 6 evidence: Over three interaction sessions, Iterative (ours) significantly outperformed the Identical baseline on a 5-point style refinement MOS, and naturalness MOS showed no significant difference between Iterative and the oracle condition across all style groups. confidence: high relevance: medium - claim_id: global_speech_embedding_based_conditioning_cannot_accurately_reflect role: complicates claim: Global speech embedding-based conditioning cannot accurately reflect fine-grained positional or linguistic instructions in expressive TTS. source: §5.1, §5.2, Table 4 evidence: Low-scoring examples in the test set contained directions targeting specific word positions ("at the beginning", "at the end", "for the part of...") or linguistic modifications ("hold your breath", "place just a slight pause between words"), which the speech embedding manipulation approach could not handle. confidence: high relevance: medium - claim_id: semantic_similarity_of_style_direction_text_not_exact role: supports claim: Semantic similarity of style direction text, not exact wording, governs how well listeners perceive style refinement as aligned with the instruction. source: §4.3, Figure 5 evidence: In the style refinement accuracy evaluation, directions semantically similar to the one used for refinement (Random Similar) yielded scores comparable to the Matched condition, while semantically dissimilar directions scored significantly lower across all style groups. confidence: high relevance: medium - claim_id: holistic_subjective_evaluation_scales_may_not_adequately_capture role: complicates claim: Holistic subjective evaluation scales may not adequately capture fine-grained stylistic alignment in iterative TTS refinement tasks. source: §4.2, §5.3 evidence: Even the Actor-Guided oracle condition scored around 3 out of 5 on the iterative style refinement MOS, which the authors attribute to the evaluation task not fully discriminating subtle style differences; similar evaluation difficulties have been noted in text-to-image/video generation research. confidence: high relevance: medium limitations: - All training and evaluation data is proprietary in-house Japanese speech from two voice actors. No public dataset is used, and no results are reported outside this setup. Reproducibility and generalization are untested. - 'The style refiner is speaker-dependent; the authors plan to extend to speaker-independent operation as future work. The current model refines only paralinguistic information (speaking style via global embeddings), not linguistic content, so instructions involving pauses, stress, or pitch accent placement cannot be followed. The directions cover only two of four practical categories from actual recording sessions (paralinguistic and text-expressible linguistic instructions), omitting demonstrative and gestural instructions entirely. The evaluation task design is also noted as an open problem: the relatively low absolute scores even under oracle conditions suggest that existing MOS paradigms do not cleanly measure this type of fine-grained iterative alignment, and more sensitive evaluation methods are needed.' caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2507.08012' published_date: "2025-07-05" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: minor method_family: - description_embedding_style_control claims: - claim_id: latent_controllable_features_absent_from_a_tts_model role: supports claim: Latent controllable features absent from a TTS model's training annotations can be discovered by applying PCA to embeddings of fixed-input generated samples and enrolling the identified dimensions as new description-prompt labels. source: §4.3, Tables 2–3 evidence: For T3 (no emotion labels), PCA of 1,000 Wav2Vec2 embeddings revealed emotional intensity as the primary variance axis; iterative re-labelling and fine-tuning progressively separated neutral from emotive utterances, improving neutral cluster assignment from 59.3% to 89.3% across rounds. confidence: high relevance: low - claim_id: variance_based_feature_discovery_in_prompt_based_tts role: complicates claim: Variance-based feature discovery in prompt-based TTS is unreliable when the model's output distribution is highly constrained by existing conditioning labels. source: §4.4, Figure 6 evidence: T3-emotion, already trained with explicit emotion and intensity labels, generated highly consistent F0 contours under neutral-emotion prompts (Figure 6), leaving insufficient variance for meaningful feature discovery in the fixed-input analysis set. confidence: high relevance: low - claim_id: embedding_based_clustering_of_tts_output_variance_can role: complicates claim: Embedding-based clustering of TTS output variance can surface acoustic artefacts of the training corpus (such as recording-condition variation) rather than prosodic features of interest. source: §4.4, Figure 8 evidence: For T3-emotion with a diverse analysis set, the principal component correlated strongly with GeMaps-v01b loudness features attributable to microphone distance differences in the Talromur-3 corpus, not to prosody. confidence: high relevance: low - claim_id: self_supervised_speech_representations_suitable_for_prosody_analysis role: complicates claim: Self-supervised speech representations suitable for prosody analysis are inherently entangled with speaker identity and linguistic content, requiring fixed-input generation sets to isolate prosodic variation. source: §3.2, Figure 1 evidence: Wav2Vec2 summary embeddings cluster by target text and speaker identity regardless of which network layer is used (Figure 1), motivating the fixed-input analysis design where text, speaker, and prompt are held constant. confidence: high relevance: low limitations: - No subjective listening tests are reported. All quality and controllability assessments use automatic metrics (ASR-based WER, speaker embedding cosine similarity, diversity score). Whether the discovered features correspond to perceptually meaningful and user-controllable dimensions is not established. - The method is evaluated on a single Icelandic speaker (Ingrid) for the fine-tuning stages, with no cross-speaker or cross-language generalisation experiments. The paper does not address why T3-emotion fails and T3 succeeds beyond noting the former's consistent output distribution; whether adding a more diverse analysis set would help, or whether an alternative embedding choice (not Wav2Vec2) would be less sensitive to recording conditions, remains open. The use of Icelandic as the sole test language, while noted as language-agnostic in principle, is unverified on any other language. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2507.07799' published_date: "2025-07-10" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: minor method_family: - prompt_conditioned_identity_and_privacy claims: - claim_id: instruction_conditioned_tts_enables_speaker_anonymization_without_relying role: supports claim: Instruction-conditioned TTS enables speaker anonymization without relying on source speaker audio or speaker embeddings. source: §3.3, §3.4.1, Table 1 evidence: Parler-TTS with natural language speaker descriptions achieves FAR=0% on SLUE-VoxPopuli, matching zero-shot TTS baselines (VALL-E, XTTS-v2) that require reference audio, while offering explicit control over gender, pitch, accent, speaking rate, and channel conditions. confidence: high relevance: medium - claim_id: false_acceptance_rate_is_an_unreliable_discriminator_of role: complicates claim: False acceptance rate is an unreliable discriminator of speaker anonymization quality when evaluation design guarantees disjoint training and test speakers. source: §3.4.1 evidence: FAR=0% is achieved by every TTS-based system regardless of architecture or conditioning, because TTS models are trained on speakers absent from the evaluation set; the metric cannot distinguish the proposed method from any TTS-based pipeline in this setting. confidence: high relevance: medium - claim_id: asr_ner_pipeline_based_content_privacy_is_limited role: complicates claim: ASR + NER pipeline-based content privacy is limited by recognition and detection errors that allow a meaningful share of sensitive content to pass through. source: §3.4.1, §4 evidence: The ASR component achieves 19.00% WER on original speech and DeBERTa-L NER achieves only 71.80% F1 on predicted transcriptions, meaning many named entities are missed at the detection stage despite 99.95% replacement accuracy among detected entities. confidence: high relevance: medium - claim_id: generative_variability_in_prompt_based_tts_limits_consistent role: complicates claim: Generative variability in prompt-based TTS limits consistent voice identity reproduction across utterances sharing the same speaker description. source: §4 evidence: Parler-TTS outputs may sound like different persons across generations even with an identical speaker description, undermining multi-utterance anonymization coherence and making deterministic identity control difficult. confidence: high relevance: medium - claim_id: speaker_attribute_choice_in_instruction_conditioned_tts_introduces role: supports claim: Speaker attribute choice in instruction-conditioned TTS introduces measurable variation in downstream speech intelligibility that practitioners must account for. source: §3.4.2, Tables 2 and 3 evidence: WER varies from 12.07% (Slovak accent) to 23.76% (Italian accent) across 38 accent descriptions, and from 12.39% (normal rate) to 18.56% (very fast) across speaking rate settings, while speaker privacy (FAR=0%) remains constant throughout. confidence: high relevance: medium limitations: - FAR=0% is observed for all TTS-based systems including both baselines, because the evaluation speakers are disjoint from TTS training speakers. The privacy metric cannot distinguish the proposed approach from any TTS pipeline in this experimental design, which materially limits the privacy claims. - The NER component (71.80% F1) misses a substantial share of named entities, so content privacy is incomplete even when replacement accuracy is near-perfect among detected entities. WER-based content privacy evaluation does not capture semantic re-identification risks from context, paraphrase, or implicit references. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: 2025.acl-long.346 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - TTS architecture: - transformer-enc-dec - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - description_embedding_style_control - prompt_conditioned_identity_and_privacy claims: - claim_id: disentangling_speaker_timbre_and_speaking_style_into_separate role: supports claim: Disentangling speaker timbre and speaking style into separate codec representations is a necessary condition for simultaneous zero-shot speaker cloning and style control; without explicit decoupling, the two conditioning signals interfere and controllability collapses. source: §4.3, Table 4 evidence: 'The codec decoupling ablation (test set A) is striking: without decoupling, controllability collapses (pitch 0.492, speed 0.517, emotion 0.237), confirming that independently disentangling the speech prompt and style prompt is necessary to prevent cross-channel interference.' confidence: high relevance: high - claim_id: natural_language_style_descriptions_have_an_inherent_many role: supports claim: Natural language style descriptions have an inherent many-to-many relationship with audio that cannot be resolved by timbre conditioning alone, requiring a probabilistic model of style variation such as a mixture density network. source: §3.3 evidence: On test set C (out-of-domain style descriptions), ControlSpeech degrades much less than baselines across all metrics, indicating substantially better generalization from the SMSD's mixture density mechanism. confidence: high relevance: low - claim_id: zero_shot_speaker_cloning_capability_in_style_controllable role: supports claim: Zero-shot speaker cloning capability in style-controllable TTS can be achieved by building on a large-scale pre-trained disentangled codec without sacrificing audio quality relative to dedicated zero-shot TTS systems. source: §4.2, Table 2 evidence: ControlSpeech achieves the best speaker similarity (Spk-sv 0.89), WER (2.9%), and MOS-Q (3.91 ± 0.09), outperforming all baselines on audio quality while uniquely providing zero-shot timbre cloning (all style-only baselines lack this capability). confidence: high relevance: high - claim_id: probabilistic_sampling_from_a_mixture_density_model_of role: supports claim: Probabilistic sampling from a mixture density model of style representations improves both style diversity and generalization to out-of-domain style descriptions compared to deterministic style encoding. source: §4.2, §4.3, Table 3 evidence: On test set C (out-of-domain style descriptions), ControlSpeech degrades much less than baselines across all metrics, indicating substantially better generalization from the SMSD's mixture density mechanism. confidence: high relevance: low - claim_id: pitch_control_is_measurably_harder_to_preserve_when role: supports claim: Pitch control is measurably harder to preserve when timbre and style are controlled simultaneously, suggesting that pitch conditioning interacts with speaker identity in ways that speed, volume, and emotion do not. source: §4.2, Table 1 evidence: On VccmDataset test set A (style controllability, using GT timbre as prompt), ControlSpeech achieves the best speed accuracy (0.829 vs. 0.786 for PromptStyle), volume accuracy (0.894 vs. 0.831 for Salle), and emotion accuracy (0.557 vs. 0.412 for InstructTTS). confidence: high relevance: low limitations: - 'The paper explicitly notes two limitations: (1) The training dataset is still limited in scale for style-controllable TTS; significantly larger datasets (tens of thousands of hours with style annotations) may be needed to achieve more advanced controllability. (2) The exploration of generative model architectures is narrow — only non-autoregressive parallel decoding is tried. Diffusion or flow-matching generators operating in the disentangled codec space might offer better quality or diversity.' - 'Beyond the paper''s self-assessment: pitch accuracy is the only metric where ControlSpeech underperforms style-only baselines on both in-domain and out-of-domain test sets. The authors attribute this to simultaneous timbre-style control, but the mechanism is unexplained and unresolved. The paper does not evaluate on standard TTS benchmarks (LibriSpeech, VCTK), relying entirely on VccmDataset, which makes external comparison difficult. The demo availability is not confirmed in the paper. Ethical risks from zero-shot voice cloning are acknowledged but only partially addressed (watermarking is proposed as future work).' caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: 2025.acl-long.681 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - direct_prompt_conditioned_speech_lm - instruction_data_and_annotation_pipeline claims: - claim_id: large_scale_instruction_datasets_that_span_diverse_acoustic role: supports claim: Large-scale instruction datasets that span diverse acoustic and paralinguistic dimensions improve speech-text LLM generalization on instruction-following benchmarks beyond what smaller task-specific corpora achieve. source: §5.2, Table 3 evidence: SIFT-LLM outperforms O-ASQA-LLM (same backbone, fine-tuned on 2.7M-example OpenASQA) across all benchmarks, including 46.1% vs. 22.9% on EvalSIFT closed-ended and 57.4% vs. 45.9% on Dynamic-Superb. confidence: high relevance: medium - claim_id: instruction_fine_tuning_on_diverse_speech_tasks_reduces role: complicates claim: Instruction fine-tuning on diverse speech tasks reduces specialized performance on foundational tasks relative to the pre-trained model checkpoint. source: §5.3, Table 4 evidence: SIFT-LLM WER on LibriSpeech test-clean rises from 2.5% (50K pre-training checkpoint) to 3.5% after instruction fine-tuning, consistent with behavior observed in Qwen2-Audio and its instruction fine-tuned variant. confidence: high relevance: medium - claim_id: codec_representations_that_jointly_encode_acoustic_and_semantic role: supports claim: Codec representations that jointly encode acoustic and semantic content enable more accurate controllable speech generation than semantic-only representations. source: §5.4, Table 6 evidence: X-codec2 (fused semantic and acoustic codes) achieves QWK of 0.69 for pitch variation versus 0.15 for HuBERT codes (semantic-only) on SIFT-LLM GEN controllable generation evaluations. confidence: high relevance: high - claim_id: semantic_only_discrete_speech_representations_are_insufficient_for role: complicates claim: Semantic-only discrete speech representations are insufficient for reliable speaker-dependent controllable generation, as they primarily encode content rather than prosodic and acoustic identity. source: §5.4, §F.3 evidence: The HuBERT-code vocoder setup achieves approximately 50% gender accuracy against instruction-specified gender, compared to 95.8% for X-codec2; pitch variation QWK from original audio degrades substantially in HuBERT re-synthesis (0.11 QWK, Table 18 Appendix F.3). confidence: high relevance: medium limitations: - SIFT-LLM does not achieve state-of-the-art results on foundational speech tasks; the instruction fine-tuning trades off specialized ASR and translation capability, limiting its use as a drop-in replacement for task-specialized models. - Controllable generation evaluation relies on automatically extracted acoustic features compared against instruction-specified targets, which indirectly assesses controllability but cannot measure perceptual quality or naturalness. The QWK and MAE metrics measure categorical agreement rather than human-perceived fidelity. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: 2025.acl-long.911 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - TTS - evaluation architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - context_conditioned_situated_tts claims: - claim_id: incorporating_conversational_and_environmental_context_into_tts_prompts role: supports claim: Incorporating conversational and environmental context into TTS prompts produces measurably more contextually appropriate speech than attribute-label prompts, even when the model architecture is otherwise unchanged. source: §4.3, Table 2 evidence: 'It is a FastSpeech2-style (transformer-enc-dec) architecture with five modules: a BERT-based Phoneme Encoder, a BERT-based Context Encoder (with auxiliary classification tasks for emotion, pitch, energy, and speed), a Style Fusion module (cross-attention for aligning context with phonemes), a Variance Adaptor (duration, pitch, loudness prediction inspired by FastSpeech2), and a pre-trained HiFi-GAN vocoder for waveform generation.' confidence: high relevance: low - claim_id: codec_based_tts_systems_despite_not_being_designed role: supports claim: Codec-based TTS systems, despite not being designed for contextual conditioning, outperform purpose-built prompt-based baselines on most metrics when evaluated on a situated TTS benchmark. source: §4.4, Table 2 evidence: 'Key findings: - Codec TTS models (VALL-E, NaturalSpeech2, VoiceCraft) achieve the best objective scores overall (PESQ up to 4.38, WER down to 6.03 for VoiceCraft). - VoiceCraft leads on MOS-D (3.88) and MOS-S (3.89), marginally above DNA-TTS (3.85 / 3.83). - DNA-TTS achieves the best PESQ (4.10) and WER (6.36) among non-codec models, and edges out prompt-based baselines on MOS-E (3.96 vs. 3.94 for InstructTTS/VoiceLDM). - None-prompt TTS methods score consistently lower on MOS-D and MOS-S (3.65–3.78), confirming that contextual prompts carry measurable benefit. - Data quality is validated on general TTS: models trained on DNASpeech achieve MOS comparable to LJSpeech and DailyTalk (Table 1; FastSpeech2 on DNASpeech: 4.01 ± 0.07 vs. 3.97 ± 0.08 on LJSpeech).' confidence: high relevance: high - claim_id: the_benefit_of_dialogue_context_in_speech_generation role: complicates claim: 'The benefit of dialogue context in speech generation is bounded: performance peaks at 4–6 dialogue turns and degrades significantly beyond 8 turns, suggesting that long-context integration remains an open problem.' source: Appendix F, Table 3 evidence: Analysis of dialogue turn count shows performance peaks at 4–6 turns for both DNA-TTS and VALL-E, then degrades as context grows beyond 8 turns. confidence: high relevance: high - claim_id: movie_scripts_provide_a_viable_automated_source_for role: supports claim: Movie scripts provide a viable automated source for constructing contextually rich TTS datasets, with speech quality comparable to purpose-built corpora on standard naturalness metrics. source: §3.4, Table 1 evidence: 'Key findings: - Codec TTS models (VALL-E, NaturalSpeech2, VoiceCraft) achieve the best objective scores overall (PESQ up to 4.38, WER down to 6.03 for VoiceCraft). - VoiceCraft leads on MOS-D (3.88) and MOS-S (3.89), marginally above DNA-TTS (3.85 / 3.83). - DNA-TTS achieves the best PESQ (4.10) and WER (6.36) among non-codec models, and edges out prompt-based baselines on MOS-E (3.96 vs. 3.94 for InstructTTS/VoiceLDM). - None-prompt TTS methods score consistently lower on MOS-D and MOS-S (3.65–3.78), confirming that contextual prompts carry measurable benefit. - Data quality is validated on general TTS: models trained on DNASpeech achieve MOS comparable to LJSpeech and DailyTalk (Table 1; FastSpeech2 on DNASpeech: 4.01 ± 0.07 vs. 3.97 ± 0.08 on LJSpeech).' confidence: high relevance: low limitations: - '- Movie-sourced speech may not generalize to real-world acoustic conditions or spontaneous conversational speech. The authors note plans to diversify to real-world contexts. - DNA prompt types, while more comprehensive than prior work, do not cover all possible contextual cues (e.g., prosodic history, listener affect, cultural context). - Performance degrades with more than 8 dialogue turns; long-context integration remains an open problem. - The dataset is 18 hours with English speech only, limiting multilingual and scale applications. - Human evaluator pool is small (8 individuals, Kendall''s W ≥ 0.5 threshold), which may reduce reliability of MOS estimates. - No zero-shot generalization experiments are reported; it is unclear whether CS-TTS benefits transfer to unseen speaker-scene combinations.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: 2025.findings-acl.101 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - structured_intermediate_control - context_conditioned_situated_tts - instruction_data_and_annotation_pipeline claims: - claim_id: decomposing_conversational_speech_synthesis_into_sequential_emotion_understandin role: supports claim: Decomposing conversational speech synthesis into sequential emotion-understanding and speech-generation stages improves expressiveness over direct speech token prediction. source: §6.5, Table 2 evidence: Chain-Talker (DMOS-E 4.239) outperforms GPT-Talker's direct token prediction (3.913) and GPT-Talker_c, which adds emotion understanding without fully chaining it through caption prediction (4.102); the ablation w/o captions (4.084) confirms the caption conditioning is the key factor. confidence: high relevance: high - claim_id: natural_language_emotion_captions_are_more_effective_conditioning role: supports claim: Natural language emotion captions are more effective conditioning signals for empathetic speech synthesis than discrete emotion category labels. source: §6.4, §6.5, Tables 1 and 2 evidence: Chain-Talker using empathetic captions (DMOS-E 4.239) outperforms Chain-Talker_e using emotion labels (4.127) and Chain-Talker_s using style labels (4.015); DMOS-C scores also show captions exceed ground-truth label quality (4.462 vs 4.327). confidence: high relevance: low - claim_id: supervised_asr_derived_semantic_speech_tokens_provide_more role: supports claim: Supervised ASR-derived semantic speech tokens provide more interpretable and expressive conversational speech generation than unsupervised tokens that mix semantic and acoustic content. source: §2.3, §6.5, Table 2 evidence: The paper argues that HuBERT tokens in GPT-Talker contain entangled acoustic information that limits emotional comprehension; Chain-Talker's ASR-VQ tokens improve DDTW from 44.625 to 38.784 and ACC_m from 0.562 to 0.612. confidence: high relevance: high - claim_id: llm_driven_automatic_speech_emotion_captioning_with_multi role: supports claim: LLM-driven automatic speech emotion captioning with multi-level attribute extraction produces higher-quality annotations than single-modality or keyword-based description approaches. source: §6.4, Table 1 evidence: CSS-EmCap (DMOS-C 4.462, SIM_G 0.694, DIS-2 0.296) outperforms Qwen2-Audio (4.212, 0.534, 0.174) and SECap (4.268, 0.617, 0.186); ablations confirm that removing either sentence-level style factors or dialogue-level emotion significantly degrades both quality and diversity. confidence: high relevance: low - claim_id: autoregressive_chain_modeling_for_empathetic_conversational_speech_synthesis role: complicates claim: Autoregressive chain modeling for empathetic conversational speech synthesis introduces latency that does not yet support real-time interaction requirements. source: §Limitations evidence: Chain-Talker generates average empathetic responses of 2.5 seconds duration on an RTX 4080 with 32 GB VRAM; authors describe this as a gap relative to real-time dialogue and identify streaming inference as a necessary future direction. confidence: high relevance: high limitations: - Chain-Talker is fine-tuned on only 384 hours of dialogue data drawn from daily conversational domains and predominantly young adult speakers. The authors explicitly note it may not capture the speaking styles of children or the elderly, and domain generalisation to non-conversational or non-English settings is untested. - 'Inference latency is a practical bottleneck: the three-stage autoregressive pipeline produces responses averaging 2.5 seconds, which is insufficient for low-latency real-time dialogue. Streaming inference is identified as future work but not yet implemented.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: 2025.findings-acl.1226 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - structured_intermediate_control - context_conditioned_situated_tts - instruction_data_and_annotation_pipeline claims: - claim_id: parallel_multi_agent_llm_architectures_can_produce_more role: supports claim: Parallel multi-agent LLM architectures can produce more informative and engaging conversational scripts than single-prompt generation for long-form spoken content. source: §5.1, §5.3, Table 1, Table 2 evidence: Host-Guest-Writer system outperforms direct GPT-4 baseline on all five quantitative metrics and all five LLM-as-a-Judge dimensions across four topic categories; the single-agent ablation also shows substantially lower scores on all metrics. confidence: high relevance: low - claim_id: llm_predicted_speaking_style_instructions_improve_speech_expressiveness role: supports claim: LLM-predicted speaking style instructions improve speech expressiveness in zero-shot TTS when used for long-form conversational audio. source: §5.2, Figure 7 evidence: All CMOS scores (0.2-0.9) are positive and preference tests favour LLM-instructed synthesis over uninstructed synthesis across all four topic categories in a 9-evaluator study. confidence: high relevance: low - claim_id: automatic_voice_role_matching_from_a_speaker_pool role: complicates claim: Automatic voice-role matching from a speaker pool cannot guarantee perceptual alignment between voice characteristics and speaker roles in conversational audio generation. source: §5.2, §7 evidence: 87.4% average pass rate implies roughly 1 in 8 voice assignments is judged inappropriate by participants; the voice pool is also limited to read-speech data (LibriTTS-R), and the paper notes the need for more conversational-style voices. confidence: high relevance: low - claim_id: increasing_the_number_of_participants_in_multi_agent role: supports claim: Increasing the number of participants in multi-agent dialogue generation does not monotonically improve content quality, and smaller groups with more focused interactions outperform larger ensembles. source: §5.3, Table 2 evidence: Ablation across 1-5 guest configurations shows that 2-guest setups score highest on Distinct-N, MATTR, and Semantic-Div; quality degrades progressively from 3 to 5 guests. confidence: high relevance: low limitations: - The voice pool is constructed entirely from read-speech corpora (LibriTTS-R, AISHELL-3). Conversational podcasts require naturalistic, spontaneous-sounding voices; the paper acknowledges that current voice quality limitations arise directly from this mismatch, and that long-form generation with CosyVoice2 still produces quality inconsistencies. - Evaluation relies exclusively on GPT-4 for LLM tasks and CosyVoice2 for synthesis, so results reflect the capabilities of those specific systems rather than the framework in general. The content evaluation uses LLM-as-a-Judge (GPT-4 evaluating GPT-4 outputs), which introduces well-documented position and self-preference biases, though the paper mitigates this by averaging forward and reverse comparison orders. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: 2025.findings-acl.534 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - SCA architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - instruction_data_and_annotation_pipeline claims: - claim_id: semantic_similarity_in_embedding_space_can_serve_as role: supports claim: Semantic similarity in embedding space can serve as a reliable proxy for linguistic equivalence when filtering TTS-synthesized speech instruction data, avoiding the over-rejection caused by WER-based criteria. source: §5.2, §5.3, Table 1 evidence: The SIM metric achieves over 98% consistency with WER when strict linguistic equivalence holds, and above 92.5% across all conditions and three TTS models on seven QA datasets. confidence: high relevance: medium - claim_id: training_large_speech_language_models_on_llm_continued role: supports claim: Training large speech language models on LLM-continued alignment targets produces systematically weaker instruction-following capability than training on human-annotated responses, regardless of the speech synthesis quality. source: §5.3, Table 3 evidence: Models trained with LLM-continued answers achieve 52.54 average ROUGE-L vs. 60.5 with human-annotated golden answers at identical data quality thresholds; the gap persists across all ablation settings. confidence: high relevance: high - claim_id: parallel_query_rewriting_with_multiple_structurally_diverse_llms role: supports claim: Parallel query rewriting with multiple structurally diverse LLMs improves TTS data usability compared to any single-LLM rewriting approach, due to complementary performance across text types. source: §5.4, Table 2 evidence: Using Llama-3, Phi-3, and Qwen2 together raises average Pass rate by 2–3 percentage points over the best single-LLM baseline across MeloTTS and ParlerTTS-Large in the single-speaker setting. confidence: high relevance: medium - claim_id: wer_based_quality_filtering_of_tts_synthesized_speech role: complicates claim: WER-based quality filtering of TTS-synthesized speech instruction data causes inappropriate data rejection due to incidental ASR errors, distorting the resulting dataset distribution. source: §3, §5.3, Table 13 evidence: Strict WER filtering rejects samples where ASR recognition errors occur despite semantic equivalence; SIM-based filtering recovers these samples while maintaining downstream task performance, as confirmed by case analysis in Table 13. confidence: high relevance: medium limitations: - The framework is validated only on short questions (under 100 words); the authors note that constructing spoken instructions from longer texts through rewriting remains an open problem. LSLM evaluation covers only four reading comprehension tasks measured with ROUGE-L, which may not reflect instruction-following quality in conversational or multi-turn settings. All experiments are English-only. The full multi-speaker pipeline requires approximately 558 GPU-hours on a single NVIDIA A40, which is non-trivial for reproduction at scale (Table 8). caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: 2025.findings-acl.687 published_date: "2025-07-27" entry_date: '2026-07-25' year: 2025 venue: ACL task: - singing - TTS architecture: - flow-matching - VAE relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - description_embedding_style_control claims: - claim_id: boundary_masking_during_training_improves_zero_shot_singing role: supports claim: Boundary masking during training improves zero-shot singing synthesis naturalness by forcing the model to learn smooth phoneme and note transitions without relying on precise alignment annotations. source: §3.2, Table 4 evidence: BBC Encoder masks 8 tokens at each phoneme/note boundary; ablating this masking causes CMOS-Q to drop -0.36 in style transfer and -0.39 in style control, the largest single-component quality degradation. confidence: high relevance: medium - claim_id: contrastive_alignment_of_cross_modal_style_embeddings_singing role: supports claim: Contrastive alignment of cross-modal style embeddings (singing, speech, natural language) enables a single encoder to support style transfer, speech-to-singing, and instruction-conditioned synthesis without modality-specific architectures. source: §3.3, Table 4 evidence: Custom Audio Encoder trained with CLIP-style triplet contrastive loss achieves unified style space across modalities; removing it drops CMOS-S by -0.37 (style transfer) and CMOS-C by -0.41 (style control). confidence: high relevance: medium - claim_id: language_conditioned_mixture_of_experts_routing_in_a role: supports claim: Language-conditioned mixture-of-experts routing in a flow-matching transformer improves multilingual singing synthesis quality by directing language-family-specific token processing to specialised experts. source: §3.4, Table 4, Table 7 evidence: Lingual-MOE ablation (replaced with standard FFN) causes CMOS-Q to drop -0.29 in style transfer; Stylistic-MOE ablation causes CMOS-S -0.26 and CMOS-C -0.33; four experts per group is the optimal configuration before diminishing returns. confidence: high relevance: high - claim_id: f0_supervision_applied_to_intermediate_transformer_representations_improves role: supports claim: F0 supervision applied to intermediate transformer representations improves both synthesis quality and style fidelity in zero-shot singing voice synthesis. source: §3.4, Table 4 evidence: Removing F0 supervision from the first block's output causes CMOS-Q -0.33 and CMOS-S -0.24 in style transfer, and CMOS-Q -0.31 and CMOS-C -0.27 in style control. confidence: high relevance: medium - claim_id: multi_level_style_control_via_natural_language_prompts role: complicates claim: Multi-level style control via natural language prompts in singing synthesis requires costly manual annotation, limiting dataset scale and introducing labelling errors. source: §6, Appendix B evidence: Style labels (emotion, singing method, vocal range, word-level techniques) annotated by music experts at $300/hour, then converted to natural language via GPT-4o; the paper cites labelling cost and annotation errors as primary limitations constraining generalisation. confidence: high relevance: medium limitations: - 'The training and test sets share the same underlying singer pool: 30 "unseen" singers are held out from a pool drawn from the same data collection. This limits the assessment of generalisation to truly out-of-distribution singers or languages not present in training (all 9 test languages are also in training).' - Generation speed does not meet real-time industrial requirements at current inference settings, despite 25-step ODE inference; streaming inference is deferred to future work. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2508.07273' published_date: "2025-08-10" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: training_speech_llms_on_data_that_jointly_encodes role: supports claim: Training Speech-LLMs on data that jointly encodes contextual and paralinguistic reasoning substantially outperforms training on isolated paralinguistic QA templates, even when the underlying speech encoder is frozen. source: §V.B.1, Table III evidence: The implicit method generates a large-scale contextual-paralinguistic question-answering (CPQA) training corpus by prompting GPT-4o with word-level transcripts alongside both categorical emotion labels (Emotion2Vec) and dimensional scores (valence, arousal, dominance from a continuous emotion recognition model), producing 32,960 QA pairs from 4,740 emotion-rich speech clips. confidence: high relevance: low - claim_id: incorporating_dimensional_emotion_annotations_valence_arousal_dominance_alongsid role: supports claim: Incorporating dimensional emotion annotations (valence, arousal, dominance) alongside categorical labels into LLM-generated QA data diversifies training supervision and improves generalisation to complex emotional states. source: §II.D, §V.B.1 evidence: The implicit method generates a large-scale contextual-paralinguistic question-answering (CPQA) training corpus by prompting GPT-4o with word-level transcripts alongside both categorical emotion labels (Emotion2Vec) and dimensional scores (valence, arousal, dominance from a continuous emotion recognition model), producing 32,960 QA pairs from 4,740 emotion-rich speech clips. confidence: high relevance: low - claim_id: explicitly_injecting_emotion_metadata_into_inference_prompts_can role: supports claim: Explicitly injecting emotion metadata into inference prompts can partially compensate for a model's limited intrinsic paralinguistic understanding, but training on contextual-paralinguistic data yields more robust generalisation across question types. source: §V.B.3, Fig. 2 evidence: The explicit method augments question prompts at both training and inference time with time-stamped emotion labels derived from a pre-processing speech emotion recognition (SER) module, giving the model direct access to affective context. confidence: high relevance: low - claim_id: llm_judge_scores_for_open_ended_speech_language role: supports claim: LLM judge scores for open-ended speech-language model evaluation correlate reliably with classification-based accuracy and F1 metrics on questions with deterministic answers, supporting their use as a proxy metric. source: §IV, §V.B.4, Table V evidence: Performance is measured using GPT-4o-as-judge scores (scaled 0–100) on human-annotated and LLM-generated CPQA evaluation sets, and with LLM-judged scores plus estimated weighted accuracy and F1 on emotion-PQA benchmarks. confidence: high relevance: high limitations: - The CPQA training data is derived from a proprietary in-house movie and TV dataset that is not publicly released, making exact replication of the training setup impossible for external researchers. - Emotion labels used both in training and inference prompts come from SER models rather than ground-truth annotations, introducing noise that may suppress performance on direct classification tasks while still benefiting contextual reasoning. The LLM-generated CPQA evaluation set contains evaluation confounds — direct emotion questions that benefit disproportionately from explicit metadata injection — which the authors flag but do not resolve in the current work, requiring stricter QA generation controls in follow-up. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2508.08095' published_date: "2025-08-11" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: disentangling_paralinguistic_and_linguistic_information_through_separate_adapter role: supports claim: Disentangling paralinguistic and linguistic information through separate adapter architectures improves an SLM's ability to perceive both modalities independently without modifying the underlying encoder or LLM. source: §III.A, §V.A, Table I evidence: The paralinguistic adapter applies a compact transformer block (single layer, 8 heads) to the speech encoder output, then uses adaptive pooling to reduce the sequence to a fixed length of 10 embeddings. confidence: high relevance: low - claim_id: training_time_randomisation_over_paralinguistic_embedding_sources_speech role: supports claim: Training-time randomisation over paralinguistic embedding sources (speech, text caption, or absent) can prevent adapter collapse into task-specific vectors and preserve contextual generalisation. source: §III.C, §V.B evidence: 'Stage 2 applies Equivalence Replacement Regularization (ERR): when training the linguistic adapter on linguistic tasks, the paralinguistic adapter is frozen and linguistic embeddings are randomly combined with paralinguistic embeddings drawn from text captions, speech, or nothing at equal probability.' confidence: high relevance: low - claim_id: parameter_efficient_adapter_only_training_on_a_frozen role: supports claim: Parameter-efficient adapter-only training on a frozen LLM is competitive with full fine-tuning approaches for emotional dialogue tasks when the information encoding is structured by design. source: §V.A, Table II–III, §V.B, Table IV evidence: 'Stage 2 applies Equivalence Replacement Regularization (ERR): when training the linguistic adapter on linguistic tasks, the paralinguistic adapter is frozen and linguistic embeddings are randomly combined with paralinguistic embeddings drawn from text captions, speech, or nothing at equal probability.' confidence: high relevance: low - claim_id: automatic_llm_as_judge_evaluation_of_emotional_conversation role: supports claim: Automatic LLM-as-judge evaluation of emotional conversation quality does not substitute for human subjective evaluation; score magnitudes are judge-model-dependent. source: §IV.B, §V.B evidence: The emotional conversation evaluation uses LLM-as-judge scoring on a relatively small extended StyleTalk test set, and the metric definitions (CS Score, EGS Score) are non-standard and paper-specific. confidence: high relevance: low limitations: - The system generates text responses, not speech. Despite targeting emotional spoken conversation, the output modality is text only — the emotional response is encoded in linguistic content and style, not in synthesised speech prosody. This limits applicability to fully spoken dialogue pipelines. - The evaluation relies entirely on LLM-as-judge metrics (CS Score, EGS Score) with no human listening tests, making quality estimates harder to interpret and compare across papers. The StyleTalk test set is small, and the evaluation extension (reversing assistant/user roles) is non-standard. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2504.12867' published_date: "2025-08-13" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: fine_grained_natural_language_emotion_descriptions_provide_richer role: complicates claim: Fine-grained natural language emotion descriptions provide richer control over expressive speech synthesis than coarse categorical labels, at the cost of requiring emotion-specific training data. source: §2.1, §6.1 evidence: 'Training is two-stage: a standard TTS pretraining phase on large synthetic neutral speech (VoiceAssistant, 3,234h English; Belle, 6,418h Chinese), followed by emotional fine-tuning on EmoVoice-DB, LAION Got Talent, and in-house Chinese data.' confidence: high relevance: low - claim_id: parallel_phoneme_token_prediction_as_a_secondary_output role: supports claim: Parallel phoneme token prediction as a secondary output head reduces intelligibility errors in LLM-based TTS, particularly on challenging inputs such as rare words and tongue twisters. source: §6.2.1, §6.2.2, Table 5, Table 6 evidence: EmoVoice-PP (Phoneme Boost variant) adds a parallel prediction head that simultaneously outputs phoneme tokens (~11Hz) alongside audio tokens (~17Hz). confidence: high relevance: high - claim_id: llm_pretraining_initialisation_meaningfully_benefits_emotion_controllable_tts role: supports claim: 'LLM pretraining initialisation meaningfully benefits emotion-controllable TTS: models without it show substantially higher word error rates and weaker emotion transfer.' source: §6.2.4, Table 8 evidence: On the Chinese Secap test set, EmoVoice-PP achieves the best WER (7.6), emotion similarity (0.7939), and recall (0.434) among evaluated models, outperforming CosyVoice2 and GPT-series models, which struggle with non-English prosody and timbre. confidence: high relevance: low - claim_id: automatic_emotion_similarity_metrics_e_g_emotion2vec_cosine role: supports claim: Automatic emotion similarity metrics (e.g. emotion2vec cosine similarity) correlate reasonably at the system level but poorly at the utterance level with human perceptual judgments, limiting their utility for fine-grained model comparison. source: §7, Table 10 evidence: On the English EmoVoice-DB test set (objective), EmoVoice(1.5B) achieves the highest emotion similarity (0.9118) and recall rate (0.424) among open systems, with WER of 2.62 and UTMOS of 4.35. confidence: high relevance: low - claim_id: multimodal_llms_are_not_yet_reliable_judges_of role: supports claim: Multimodal LLMs are not yet reliable judges of emotional speech quality, exhibiting both low correlation with human ratings and inter-run instability. source: §7, Table 10 evidence: 'The ablation on LLM initialization shows it is essential: removing pretrained Qwen2.5 weights raises WER from 2.73 to 6.16 for the base model, confirming that LLM language understanding transfers meaningfully to emotional TTS quality.' confidence: high relevance: low limitations: - 'The English model is trained and evaluated entirely on synthetic data generated by GPT-4o-audio. Both EmoVoice-DB (training) and the test set are GPT-4o-audio outputs, creating circularity: the model learns to mimic GPT-4o-audio''s synthesis style rather than natural human emotional speech. Generalisation to real human emotional recordings or to out-of-distribution TTS systems is undemonstrated.' - The emotion recall evaluation omits three of seven emotion categories (disgusted, fearful, surprised) due to low recognition accuracy from emotion2vec, which limits the scope of the emotional expressiveness claims. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2508.09600' published_date: "2025-08-13" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - multimodal_spoken_instruction_model claims: - claim_id: explicit_chain_of_thought_reasoning_over_paralinguistic_cues role: supports claim: Explicit chain-of-thought reasoning over paralinguistic cues (emotion, age, gender, sound events) improves empathetic response generation in speech-to-speech dialogue systems. source: §OSUM-EChat "Training", Stage 3 Empathy; Table 2 evidence: 'Stage 1 (understanding) jointly trains ASR and paralinguistic prediction tasks (ASR+P: emotion, age, gender, sound events), keeping the encoder and adapter trainable; pseudo-labelling expands single-task data to all paralinguistic dimensions.' confidence: high relevance: low - claim_id: pretraining_on_multitask_speech_understanding_before_speech_to role: supports claim: Pretraining on multitask speech understanding before speech-to-speech dialogue training reduces dependence on large-scale paired dialogue datasets while maintaining paralinguistic modelling quality. source: §OSUM-EChat "Training", Stage 1; Table 2 evidence: Training follows a three-stage understanding-driven spoken dialogue curriculum. confidence: high relevance: low - claim_id: synthetic_speech_to_speech_data_derived_from_tts role: supports claim: Synthetic speech-to-speech data derived from TTS systems exhibits reduced emotional expressiveness compared to real human speech, creating a systematic domain gap that degrades empathetic dialogue evaluation. source: §EChat-200K Dataset evidence: Stage 2 (generation) first fine-tunes the LLM on TTS data alone, then jointly trains on speech-to-speech dialogue in both non-streaming (text tokens then speech tokens) and streaming (interleaved 6:18 text-to-speech token ratio) modes. confidence: high relevance: low - claim_id: automatic_empathy_evaluation_pipelines_using_llm_scoring_and role: supports claim: Automatic empathy evaluation pipelines using LLM scoring and automatic emotion classifiers diverge measurably from human judgements, primarily due to emotion classifier errors and LLM hallucinations. source: §Main Results "Results of Empathetic Intelligence"; Table 3 evidence: 'There is a documented divergence between automated and human scores: emotion2vec-Large labelling errors and GPT-4o hallucinations inflate discrepancies, though model rankings remain consistent between the two evaluation modes.' confidence: high relevance: low - claim_id: native_multimodal_models_that_integrate_speech_token_prediction role: supports claim: Native multimodal models that integrate speech token prediction directly into the LLM are better suited to capturing and generating paralinguistic nuance than modularly aligned architectures that treat speech decoding separately. source: §Introduction; §Related Work "End-to-End Spoken Dialogue System" evidence: On general dialogue quality (UltraEval-Audio, Table 4), OSUM-EChat's S2S mode scores comparably to other open-source native multimodal models (eg. DeepTalk, GLM-4-Voice) but below modularly aligned systems such as MinMo and the commercial ChatGPT-4o. confidence: high relevance: high limitations: - 'The EChat-200K dataset is almost entirely synthetic: query audio is generated by CosyVoice2 and response audio likewise, with real recordings comprising only a minority of the data. The authors acknowledge that emotional expressiveness of synthesised audio lags real human speech, and the training corpus does not include dynamic paralinguistic scenarios (e.g. emotional transitions, multi-speaker interactions). Generalisability to natural in-the-wild speech remains unvalidated.' - 'The EChat-eval automatic scoring pipeline — combining GPT-4o and emotion2vec-Large — produces rankings consistent with human evaluation but absolute scores that diverge meaningfully. The benchmark is therefore more reliable for ranking systems than for measuring absolute empathy. Sound event capability is evaluated only within the categories present in EChat-200K, which may not reflect the diversity of real conversational contexts. General linguistic intelligence (UltraEval-Audio, Table 4) regresses compared to the Qwen2.5-3B base: GSM8K drops from 85 to 34 after the multi-stage training, indicating that the speech capability gain comes at a significant cost to LLM reasoning.' caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2508.11326' published_date: "2025-08-15" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - direct_prompt_conditioned_speech_lm - instruction_data_and_annotation_pipeline claims: - claim_id: freezing_the_backbone_llm_and_routing_modality_specific role: supports claim: Freezing the backbone LLM and routing modality-specific tokens to separate expert sets can preserve pre-trained text understanding capabilities during speech generation fine-tuning. source: §3.2 evidence: A deterministic modality-based routing strategy assigns text tokens exclusively to the original (frozen) text experts and speech tokens exclusively to the newly trained speech experts. confidence: high relevance: high - claim_id: instruction_conditioned_tts_systems_trained_on_tag_derived role: supports claim: Instruction-conditioned TTS systems trained on tag-derived description datasets exhibit significant performance degradation when faced with figurative or metaphorical natural language at inference time. source: §1, §4.1 evidence: 'On the alignment dimensions — which directly measure description-following — MoE-TTS outperforms both commercial systems (OA: 3.61 vs 3.26 ElevenLabs, 3.46 MiniMax on in-domain).' confidence: high relevance: low - claim_id: commercial_speech_synthesis_products_are_not_immune_to role: supports claim: Commercial speech synthesis products are not immune to the out-of-domain description challenge, suggesting that this generalisation gap is not solved by scale alone. source: §4.2, Table 2 evidence: On in-domain descriptions, MoE-TTS is competitive with the commercial systems on basic speech quality dimensions but does not lead; MiniMax achieves the highest OS (3.83 vs 3.82 for MoE-TTS). confidence: high relevance: low - claim_id: modality_separation_techniques_from_multimodal_vision_language_research role: supports claim: Modality separation techniques from multimodal vision-language research transfer meaningfully to the speech domain, reducing catastrophic forgetting without requiring multi-modal data mixing during pre-training. source: §2.2, §3.2 evidence: During training, only the speech-modality expert weights are updated; the LLM backbone remains entirely frozen throughout. confidence: high relevance: low limitations: - The evaluation is based on 20 in-domain and 40 out-of-domain test samples, annotated by 21 evaluators. These are very small test sets for drawing strong comparative conclusions. Both the in-domain and out-of-domain test sets were constructed by the MoE-TTS authors, introducing potential design bias toward cases where the proposed approach excels. - The system currently supports only English text descriptions, due to the limited scope of available open-source description-based TTS datasets. The LLM architecture sensitivity is unexplored — all experiments use Qwen3-4B, and the impact of model scale (smaller or larger LLM backbones) on the MoE approach is left for future work. The diffusion and VAEGAN components are adapted from Stable Audio without fine-tuning on description-based data, and their contribution to description alignment is not ablated. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: interspeech-2025-0047 published_date: "2025-08-17" entry_date: '2026-07-25' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - prompt_conditioned_identity_and_privacy claims: - claim_id: mixing_high_quality_audio_only_data_with_audio role: supports claim: Mixing high-quality audio-only data with audio-visual corpora via contrastive modality alignment substantially improves speech naturalness in face-driven TTS systems. source: §2.3, Table 1 evidence: Ablation confirms that mixing high-quality LibriTTS-R audio data contributes the largest MOS gain (+0.50), followed by style augmentation (+0.26 in MOS), with contrastive learning primarily benefiting face-voice association (SIM metric) rather than naturalness. confidence: high relevance: low - claim_id: neural_style_transfer_augmentation_of_training_face_images role: supports claim: Neural style transfer augmentation of training face images reduces the domain gap between photorealistic faces and artistic portraits, improving both naturalness and face-matching scores. source: §2.2, Table 1 evidence: 'Style augmentation: During training, input face images are randomly passed through CAST neural style transfer and augmented with gray-scaling and blurring (50% chance).' confidence: high relevance: low - claim_id: in_context_prompting_with_a_user_selected_speech role: supports claim: In-context prompting with a user-selected speech sample enables consistent voice generation across multiple utterances from the same face, addressing the inherent one-to-many ambiguity of face-to-voice mapping. source: §2.4 evidence: For diverse but consistent voice generation, the model uses sampling-based decoding to produce multiple candidate voices from the same face, then accepts a user-selected sample as an in-context prompt to enforce voice consistency across subsequent utterances. confidence: high relevance: low - claim_id: contrastive_pre_training_of_face_and_audio_encoders role: supports claim: Contrastive pre-training of face and audio encoders primarily improves face-voice association rather than overall speech naturalness. source: §3.4.1, Table 1 evidence: Ablation confirms that mixing high-quality LibriTTS-R audio data contributes the largest MOS gain (+0.50), followed by style augmentation (+0.26 in MOS), with contrastive learning primarily benefiting face-voice association (SIM metric) rather than naturalness. confidence: high relevance: low - claim_id: objective_acoustic_metrics_speaking_rate_si_sdr_c50 role: supports claim: Objective acoustic metrics (speaking rate, SI-SDR, C50, pitch standard deviation) shift predictably with natural language descriptors, confirming that text-driven style control transfers to measurable acoustic properties. source: §3.4.3, Table 3 evidence: Controllability is verified using objective metrics (speaking rate, SI-SDR, C50, pitch std) that shift predictably with descriptive text changes. confidence: high relevance: low limitations: - '- Voice consistency score (VCS) for RV-TTS (3.96) is below the audio-driven YourTTS (4.42), indicating residual inconsistency in pure face-conditioned generation. - The artistic portrait test uses only 20 copyright-free images; broader and more controlled evaluation is needed. - Descriptive text label generation for LRS3/VoxCeleb2 relies on Data-Speech auto-annotation rather than human labels. - Face encoder is ResNet50; more powerful vision-language models may improve face-voice matching. - No multilingual evaluation.' caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: interspeech-2025-0203 published_date: "2025-08-17" entry_date: '2026-07-25' year: 2025 venue: Interspeech task: - VC architecture: - flow-matching - transformer-enc-dec relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - description_embedding_style_control claims: - claim_id: natural_language_prompts_can_control_emotional_voice_conversion role: supports claim: Natural language prompts can control emotional voice conversion at parity with reference speech for the majority of listeners, reducing reliance on hard-to-source reference audio. source: §3.2.2 evidence: An ABX preference test comparing reference-speech-driven versus prompt-driven conversion found that 57.4% of 47 participants expressed no preference between the two modes, with 19.1% favouring the prompt-driven output, suggesting the natural language conditioning is effective without a meaningful quality penalty. confidence: high relevance: low - claim_id: flow_matching_produces_noticeably_higher_speech_naturalness_and role: supports claim: Flow matching produces noticeably higher speech naturalness and audio quality in emotional voice conversion than GAN and autoencoder baselines. source: §3.2.1, Table 1 evidence: Against three baselines (StarGAN-EVC, Seq2seq-EVC, MixEmo) on an internal Mandarin test set, ClapFM-EVC substantially outperforms all comparators across both speech quality and emotion similarity dimensions (Table 1). confidence: high relevance: low - claim_id: combining_categorical_emotion_labels_with_free_form_prompt role: supports claim: Combining categorical emotion labels with free-form prompt labels through soft-label contrastive training improves emotion embedding quality over prompt-only or label-only training. source: §3.3, Table 2 evidence: 'Ablation results confirm that soft-label training (categorical labels + symKL-loss) and the AIG module are each individually necessary: removing categorical labels causes EECS to drop from 0.79 to 0.66; replacing symKL with KL causes a 10.1% EECS decline; removing AIG degrades both nMOS (4.01 to 3.62) and EECS (0.79 to 0.74).' confidence: high relevance: low - claim_id: an_explicit_scalar_intensity_gate_applied_to_emotional role: supports claim: An explicit scalar intensity gate applied to emotional embeddings before content-emotion fusion improves both naturalness and emotion similarity in converted speech. source: §3.3, Table 2 evidence: Central to FuEncoder is an adaptive intensity gate (AIG), a learnable scalar applied to the emotional embedding before fusion, giving the user direct control over expression strength. confidence: high relevance: low limitations: - The system is trained and evaluated entirely on a proprietary internal Mandarin corpus. No open-source data or model weights are released, and no cross-lingual or multi-speaker generalisation is tested. - Comparisons are restricted to older GAN and autoencoder baselines (StarGAN-EVC, Seq2seq-EVC, MixEmo); no diffusion-based or recent flow-matching EVC systems are included, so the claimed state-of-the-art position cannot be verified against the most competitive contemporaries. The evaluation is any-to-one (fixed target speaker identity), leaving any-to-any EVC performance unaddressed. Emotion coverage is limited to seven categorical classes; whether the natural language conditioning generalises to subtler or blended emotional states is untested. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: interspeech-2025-0948 published_date: "2025-08-17" entry_date: '2026-07-25' year: 2025 venue: Interspeech task: - VC architecture: - VAE - diffusion relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - description_embedding_style_control claims: - claim_id: natural_language_prompts_enable_more_flexible_and_subjectively role: supports claim: Natural language prompts enable more flexible and subjectively accurate emotion control in voice conversion than numeric intensity values or reference audio selection. source: §1, §3.4 evidence: An emotion descriptor uses a pre-trained RoBERTa model followed by a linear projection to convert a natural-language description into a coarse emotion embedding. confidence: high relevance: low - claim_id: a_diffusion_based_mapping_from_text_embeddings_to role: supports claim: A diffusion-based mapping from text embeddings to speech emotion embeddings is sufficient to replace reference audio at inference time without significant quality loss. source: §2.1, §3.2, Table 1 evidence: This embedding is then refined by a prompt mapper — a score-based diffusion model implemented as stacked Transformer encoder layers — that is trained to predict the fine-grained emotion representation from the coarser text embedding, conditioned on reference embeddings extracted by the Emotion2Vec model (a large-scale SSL model for speech emotion). confidence: high relevance: low - claim_id: joint_training_of_a_text_to_emotion_mapper role: supports claim: Joint training of a text-to-emotion mapper with reference emotion embeddings improves prosody naturalness over direct prediction from text alone. source: §3.3, Table 1 evidence: This embedding is then refined by a prompt mapper — a score-based diffusion model implemented as stacked Transformer encoder layers — that is trained to predict the fine-grained emotion representation from the coarser text embedding, conditioned on reference embeddings extracted by the Emotion2Vec model (a large-scale SSL model for speech emotion). confidence: high relevance: low - claim_id: preserving_speaker_identity_during_emotional_pitch_manipulation_requires role: supports claim: Preserving speaker identity during emotional pitch manipulation requires an explicit F0 constraint in the speaker encoder; adversarial training alone is insufficient. source: §2.3, §3.3, Table 1 evidence: Speaker identity is preserved by an augmented speaker verification model trained with an F0 regression constraint (log-F0 L2 loss), discouraging identity drift when pitch is manipulated. confidence: high relevance: low - claim_id: mixed_emotion_synthesis_remains_harder_to_control_than role: complicates claim: Mixed-emotion synthesis remains harder to control than single-category emotion intensity across both subjective and objective metrics. source: §3.4, Table 2, Table 3 evidence: 'Objective metrics follow the same pattern: MCD 4.70, CER 4.09%, and log-F0 RMSE 42.58, all best among the five systems tested (Table 1).' confidence: high relevance: low limitations: - Training and evaluation are conducted entirely on TextrolSpeech, a single corpus with a limited speaker set. Generalisation to out-of-domain speakers, languages, or acoustic conditions is untested, and all reported numbers should be interpreted within that constraint. - The evaluation uses only 25 listeners for subjective MOS across 132 utterances — a borderline sample size that may limit statistical reliability. The mixed-emotion accuracy (61.3%) is notably lower than single-attribute control, and the system's handling of complex emotional blends (e.g., contempt with happiness) is not analysed in depth. The discrete HuBERT token approach for linguistic content may introduce quantisation artefacts not reported in the paper. Future real-time or streaming deployment, mentioned in the conclusion as a direction, is not addressed in the current architecture. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: interspeech-2025-1192 published_date: "2025-08-17" entry_date: '2026-07-25' year: 2025 venue: Interspeech task: - TTS architecture: - transformer-enc-dec relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - description_embedding_style_control claims: - claim_id: adversarial_disentanglement_via_a_gradient_reversal_layer_can role: supports claim: Adversarial disentanglement via a gradient reversal layer can separate voice impression information from speaker identity in a zero-shot TTS speaker encoder, enabling independent modulation of perceived voice characteristics. source: §2.2, §4.1, Figure 5 evidence: The control module applies GRL + 0.8 dropout to the speaker embedding to remove impression signal, then reintroduces it via an 11-dim impression vector; cosine similarity to the target speaker remains above the inter-speaker distribution at all tested modulation levels. confidence: high relevance: low - claim_id: llms_can_generate_low_dimensional_speech_style_parameter role: supports claim: LLMs can generate low-dimensional speech style parameter vectors from free-form natural language descriptions, providing a usable zero-shot interface for voice characteristic control. source: §2.3, §4.3, Table 4 evidence: ChatGPT-4o prompted with dimension definitions and pre-modulation values produces impression vectors preferred over unmodulated speech in 94.6% of "sleepy" trials and 74.6% of "urgent, attention-grabbing" trials (n=436, crowdsourced). confidence: high relevance: low - claim_id: fine_grained_impression_control_in_tts_involves_a role: complicates claim: 'Fine-grained impression control in TTS involves a trade-off: stronger modulation produces more perceptually distinct impressions but degrades naturalness, particularly when the target impression is socially incongruent with the source speaker.' source: §4.2, Table 3 evidence: MOS naturalness at maximum modulation (±3) falls to 2.71-2.88 for the Powerful-Weak and Dark-Bright dimensions, compared to 3.61-3.72 at zero modulation; low-scoring samples correspond to combinations such as "feminine male" or "strongly dark/bright." confidence: high relevance: low - claim_id: automatic_annotation_of_high_dimensional_perceptual_voice_attributes role: complicates claim: Automatic annotation of high-dimensional perceptual voice attributes at training scale requires indirect labeling pipelines that introduce estimation error, limiting the precision of supervision. source: §3.2 evidence: Crowdsourced subjective ratings were collected for only 1,154 of 20,270 speakers; an HuBERT-based estimator extrapolated labels to the remaining data with an RMSE of 0.338 on held-out utterances. confidence: high relevance: low limitations: - All training and evaluation data is a proprietary in-house Japanese corpus; no public datasets are used, and the evaluation spans only two held-out speakers. The results are not directly reproducible, and generalization to other languages, speaking domains, or TTS architectures is untested. - Eleven impression dimensions exhibit substantial inter-correlation (Table 1 reports correlations up to 0.8, e.g., Thick-Thin vs. High-Low Pitched), which means independent perceptual dimensions are not fully captured by the vector design. The paper demonstrates stable simultaneous two-dimension modulation but does not test combinations across weakly correlated dimensions at extreme values. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: interspeech-2025-2679 published_date: "2025-08-17" entry_date: '2026-07-25' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - direct_prompt_conditioned_speech_lm claims: - claim_id: natural_language_prompt_conditioning_provides_insufficient_fine_grained role: complicates claim: Natural language prompt conditioning provides insufficient fine-grained control for voice reconstruction tasks requiring consistent intelligibility and speaker identity. source: §5.2, §5.3, §6 evidence: WER for synthesised speech remains approximately 20-25% across all prompted intelligibility levels (from "extremely poor" to "extremely good") and across all six test speakers; speaker similarity scores are similarly insensitive to prompt variation; expert listening confirms stochastic sampling across traits despite fixed prompts. confidence: high relevance: medium - claim_id: large_instruction_conditioned_tts_models_can_be_adapted role: supports claim: Large instruction-conditioned TTS models can be adapted to dysarthric speech through fine-tuning on small curated datasets. source: §4.1, §5.1 evidence: Parler TTS Mini fine-tuned on ~23 hours of combined SAP dysarthric and MLS healthy speech learns to generate from the combined distribution; WER for fine-tuning data correlates with intelligibility ratings, confirming the model internalises the distribution. confidence: high relevance: medium - claim_id: automatic_evaluation_metrics_calibrated_on_healthy_synthetic_speech role: complicates claim: Automatic evaluation metrics calibrated on healthy synthetic speech do not generalise reliably to dysarthric speech and require domain validation before use. source: §5.1 evidence: UTMOS scores fall consistently below 3 for all SAP dysarthric speakers, even those with mild impairment (rating 1-2), indicating a domain mismatch that makes UTMOS unusable in this context; Resemblyzer speaker similarity shows comparable ranges across healthy and dysarthric conditions, demonstrating metric-level robustness varies substantially. confidence: high relevance: medium limitations: - UTMOS is found unusable for evaluating dysarthric speech synthesis in this work. Papers that report UTMOS for atypical speech without a domain validation step should be interpreted with caution. - The dataset is limited to 22 speakers across two etiologies (ALS and Cerebral Palsy), with no speakers at the most severe intelligibility levels (ratings 6-7). Generalisation to other dysarthric conditions or more severe impairment is undemonstrated. Subjective evaluation was restricted to expert listening rather than a formal MOS study, which makes comparison to other voice reconstruction and personalised TTS systems difficult. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2508.17494' published_date: "2025-08-24" entry_date: '2026-07-25' year: 2025 venue: workshop task: - TTS architecture: - hybrid relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - structured_intermediate_control claims: - claim_id: cascaded_task_decomposition_separating_structural_tag_prediction_from role: supports claim: Cascaded task decomposition — separating structural tag prediction from numerical parameter regression — substantially outperforms joint LLM generation for SSML-based prosody control. source: §4.4, §5.4, Table 4, Table 5 evidence: Few-shot prompting improves numerical accuracy but can cause unexpected structural collapses (Llama 3's prosody tagging nearly disappears in few-shot mode). confidence: high relevance: low - claim_id: prompt_only_llms_zero_shot_and_few_shot role: supports claim: Prompt-only LLMs (zero-shot and few-shot) systematically under-generate prosodic markup tags relative to gold annotations, and this failure persists across architectures and scales. source: §5.3, Figure 3 evidence: The LLM benchmarking (§5.3, Table 3) finds that all zero-shot and few-shot prompted models consistently under-generate break and prosody tags relative to the gold standard. confidence: high relevance: low - claim_id: prosody_enhancement_via_ssml_yields_substantial_perceptual_gains role: supports claim: Prosody enhancement via SSML yields substantial perceptual gains over neutral commercial TTS voices, even when the underlying synthesiser is not retrained. source: §5.1 evidence: Perceptual evaluation (18 listeners, 30 one-minute pairs each) shows MOS rising from 3.20 (Azure Henri baseline) to 3.87 with SSML enhancement (p < 0.005), a 20% improvement. 15 of 18 listeners preferred the enhanced version in over half of comparisons; 7 preferred it in more than 75% of comparisons (§5.1). confidence: high relevance: low - claim_id: french_tts_prosody_normalised_relative_to_a_synthetic role: supports claim: French TTS prosody normalised relative to a synthetic baseline captures linguistically meaningful patterns — phrase-final pitch rises, deliberate pacing — without requiring manual annotation. source: §3, Appendix A evidence: Each syntagm is annotated with four prosodic features — median pitch (expressed as a semitone offset converted to percentage), volume (LUFS-derived gain), speaking rate (words per second delta), and inter-syntagm break duration — normalised relative to a Microsoft Azure Henri baseline voice to produce relative delta values suitable for SSML encoding. confidence: high relevance: low limitations: - The entire pipeline is calibrated and evaluated against a single commercial TTS voice (Azure Henri, fr-FR). SSML tag semantics — the acoustic realisation of percentage pitch and rate adjustments — are implementation-dependent and voice-dependent. Transfer to any other engine or voice requires voice-specific recalibration, limiting the method's out-of-the-box generalisability. - The dataset is 14 hours of proprietary French podcasts; generalisation to other French domains, other speech styles (spontaneous, informal, unpunctuated text), or other languages is unvalidated. The paper's pipeline assumes that punctuation and syntactic cues reliably predict prosodic boundaries — an assumption that breaks down for social media text or transcribed spontaneous speech. Fine-tuning each Qwen 2.5-7B stage requires approximately 15 GB of GPU memory at 4-bit quantisation, which constrains deployment in low-resource settings. The perceptual test involved 18 listeners, a sample size sufficient for statistical significance but small for robust effect-size estimation across listener backgrounds. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: 2025.clicit-1.27 published_date: "2025-09-01" entry_date: '2026-07-25' year: 2025 venue: workshop task: - TTS - evaluation architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: minor method_family: - direct_prompt_conditioned_speech_lm claims: - claim_id: orthographic_pre_processing_of_input_text_accent_marking role: supports claim: Orthographic pre-processing of input text (accent marking, grapheme substitution, elision resolution) can reduce segmental substitution errors in LLM-based TTS applied to low-resource or historical languages without any model retraining. source: §3.2, §5.2 evidence: Replacing rare digraphs and adapting graphemes (〈c〉 to 〈k〉, 〈ae〉 to 〈ai〉, 〈qu〉 to 〈kw〉) reduced segmental substitutions by approximately one third; no retraining of GPT-4o-mini-tts was performed. confidence: high relevance: medium - claim_id: natural_language_prompt_instructions_can_steer_general_purpose role: supports claim: Natural language prompt instructions can steer general-purpose LLM-based TTS toward prosodically-correct speech in low-resource languages without fine-tuning. source: §3.3, §3.4 evidence: A concise system prompt specifying slow pace, explicit stress, and syllable articulation, combined with orthographically pre-processed input, produced at least one expert-validated recording for all 216 Latin verse lines. confidence: high relevance: medium - claim_id: llm_based_tts_systems_trained_on_modern_multilingual role: complicates claim: LLM-based TTS systems trained on modern multilingual data produce systematic cross-lingual phonological interference when applied to historically or orthographically distinct languages. source: §5.1 evidence: GPT-4o-mini-tts imposed Italian, English, or Spanish prosodic templates on Latin text depending on vocabulary similarity; the phenomenon confirms that Latin-specific data is essentially absent from the model's training corpus. confidence: high relevance: medium - claim_id: prompt_based_prosody_control_is_more_effective_for role: complicates claim: Prompt-based prosody control is more effective for stress placement than for enforcing absolute phonemic length distinctions. source: §5.6, §6 evidence: The workflow approximated Latin vowel quantity through slower pacing on ictic syllables but could not enforce a fixed heavy-to-light duration ratio, the standard for quantitative metre in phonetic work. confidence: high relevance: medium limitations: - The corpus depends on GPT-4o-mini-tts, a proprietary model with inaccessible weights. Should the API access policy change, exact reproduction of the dataset becomes impossible. - 'The corpus is small (216 lines, approximately 24 minutes), falling well short of the material typically required for full end-to-end TTS fine-tuning. Stylistic range is narrow: both texts belong to the Augustan literary period and formal register; colloquial Latin, post-Classical spelling, and late-antique forms are untested. A single synthetic voice is used throughout, precluding multi-speaker or gender-diverse analysis. Only the classical reconstructed pronunciation is represented; ecclesiastical pronunciation is excluded. Expert validation, while rigorous, introduces selection bias from a single institution''s phonological tradition. Quantity is approximated through pacing rather than explicit duration control; no fixed heavy-to-light ratio is enforced, leaving a gap relative to the phonetic gold standard.' caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2509.09716' published_date: "2025-09-09" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: large_audio_language_model_judges_can_approximate_human role: supports claim: Large audio-language model judges can approximate human evaluation of speaking style adherence with near-human-level reliability. source: §3.3, Table 2 evidence: LALM-as-a-Judge (Gemini-2.5-pro) achieves Spearman correlations of 77.01% (English) and 73.03% (Chinese) with consensus human scores, comparable to inter-human agreement of 78.58% and 70.54%. confidence: high relevance: high - claim_id: current_open_source_spoken_language_models_exhibit_substantial role: complicates claim: Current open-source spoken language models exhibit substantial deficits in controllable expressive speech generation compared to commercial systems. source: §3.2, Table 1 evidence: Open-source models (Step-Audio, Kimi-Audio, Qwen2.5-Omni, Baichuan-Audio) score 2.46-3.11 overall on VStyle, while commercial systems (GPT-4o, Doubao) reach 3.74-4.10, with the gap attributed to weak acoustic feature modeling and lower instruction-following robustness. confidence: high relevance: high - claim_id: speaking_style_control_does_not_transfer_evenly_across role: complicates claim: Speaking style control does not transfer evenly across languages even within the same system, indicating language-specific rather than language-general style representations in current models. source: §3.2 evidence: 'Doubao and Kimi-Audio score substantially higher in Chinese than English (e.g., Doubao: 4.10 vs. 3.88), while the GPT-4o series shows the reverse, attributed to imbalanced training data distributions and differences in pronunciation habits.' confidence: high relevance: low - claim_id: staged_hierarchical_evaluation_frameworks_capture_finer_distinctions_in role: supports claim: Staged hierarchical evaluation frameworks capture finer distinctions in speech generation quality than flat or single-dimension scoring approaches. source: §2.3, Figure 2, §3.3 evidence: VStyle's LALM-as-a-Judge separates content correctness, style adherence, and naturalness into sequential assessment gates, enabling discrimination between systems that achieve content but fail on style versus those achieving both; this structure is validated against human evaluation at near-human-level correlation. confidence: high relevance: high limitations: - The instruction dataset is built from manually designed seeds expanded by LLMs, meaning its distribution reflects annotator preferences and model-driven patterns that may diverge from real user interaction needs. Despite strong aggregate correlation between LALM scoring and human judgment, large audio-language model evaluators remain prone to hallucinations; the paper mitigates this with step-by-step prompts but does not eliminate the risk. The benchmark synthesizes audio instructions using commercial voice cloning, meaning style signals in the input prompts may exhibit a narrower range of speaking styles than naturally produced speech. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: '2509.13989' published_date: "2025-09-17" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - instruction_following_benchmark_and_audit claims: - claim_id: fine_grained_prosodic_control_in_instruction_guided_tts role: complicates claim: Fine-grained prosodic control in instruction-guided TTS does not improve monotonically with overall system capability. source: §4.3, §4.4, Table 3 evidence: GPT-4o-mini-TTS led all five systems on adverbs-of-degree and emotion-intensity tasks yet achieved only 0.265 on word-level emphasis accuracy (barely above the 0.143 random baseline), and all systems showed near-zero F1 on child-voice generation regardless of system tier. confidence: high relevance: medium - claim_id: human_perceptual_evaluation_reveals_itts_alignment_gaps_that role: supports claim: Human perceptual evaluation reveals ITTS alignment gaps that automated classifier-based metrics are likely to underestimate. source: §2.2, §3.3, §4.3 evidence: The E-VOC corpus collected 60,000+ ratings from 165+ raters on Prolific; large-scale human annotation found all five systems defaulted to adult-sounding voices regardless of explicit child or elderly instructions, a systematic bias that per-class classifiers tied to their training distribution would likely miss. confidence: high relevance: medium - claim_id: current_itts_systems_exhibit_a_systematic_adult_voice role: complicates claim: Current ITTS systems exhibit a systematic adult-voice bias that natural-language age instructions alone cannot overcome. source: §4.3, Table 3, Table 4 evidence: GPT-4o confusion matrix (Table 4) shows utterances labeled Child, Teenager, and Elderly were predominantly perceived as Adult by human listeners; the best child-voice F1 across all five systems was 0.113 (Parler-TTS-large), with GPT-4o reaching only 0.074. confidence: high relevance: medium - claim_id: commercial_itts_systems_can_achieve_perceptually_ordered_degree role: supports claim: Commercial ITTS systems can achieve perceptually ordered degree-scaled speech control that open-source alternatives fail to replicate at comparable levels. source: §4.1 evidence: GPT-4o-mini-TTS was the only system producing consistent objective acoustic progressions (LUFS, F0, words per second) and corresponding perceptual gradations under degree modifiers; Parler-TTS variants, PromptTTS++, and UniAudio showed minimal, irregular, or reversed variation. confidence: high relevance: medium limitations: - The study generates audio from only eight conversational transcripts paired with fixed style prompts, which limits the diversity of linguistic contexts covered. Generalisation of findings about speaker age and word-level emphasis to a wider range of sentence types, speaking styles, or TTS voices not included in the study is untested. - Annotation covers only native English speakers from the United States, so judgments of perceived age and prosodic emphasis may reflect dialect-specific or cultural norms. Inter-rater agreement (Cohen's kappa) was low for the adverbs-of-degree task (0.17) and emotion-intensity task (0.226), indicating that even human raters find degree-scaled emotion perception genuinely ambiguous, which complicates interpretation of model scores on those dimensions. The study measures the instruction-perception gap but does not diagnose whether failures arise from model architecture, training data distribution, or prompt formulation, leaving the design question open for future work. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2505.17093' published_date: "2025-09-19" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: [] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - structured_intermediate_control - instruction_following_benchmark_and_audit claims: - claim_id: an_llm_based_intermediate_conversion_layer_can_reduce role: supports claim: An LLM-based intermediate conversion layer can reduce the expertise burden on users of instruction-conditioned TTS without requiring model retraining. source: §4.3, Table 2 evidence: P2VA-C converts persona descriptions to structured prompts via GPT-4o-mini zero-shot prompting, improving human MOS from 3.09 to 3.42 and reducing WER from 22% to 17% relative to an unguided baseline on 1,000 LJSpeech-transcript persona pairs. confidence: high relevance: medium - claim_id: natural_language_voice_attribute_prompts_generated_by_llms role: complicates claim: Natural language voice attribute prompts generated by LLMs should not be assumed to be demographically neutral. source: §5.1, Table 3 evidence: When personas lack explicit gender cues, LLM-generated voice attributes assign male gender 64% of the time (vs. 10% in source descriptions), and 90% of accent assignments default to North American or British variants. confidence: high relevance: medium - claim_id: the_trade_off_between_structured_and_free_form role: refines claim: The trade-off between structured and free-form style conditioning in instruction-conditioned TTS extends to intermediate conversion layers, not only the TTS interface itself. source: §4.3, Table 2 evidence: P2VA-C (structured, predefined attributes) achieves lower WER and higher human MOS than P2VA-O (free-form), while P2VA-O preserves expressivity for attributes outside the predefined vocabulary. confidence: high relevance: medium - claim_id: fairness_in_voice_generation_systems_requires_addressing_bias role: complicates claim: Fairness in voice generation systems requires addressing bias in LLM-mediated style inference, not only in training data or TTS model design. source: §5.1, §5.2, Tables 4-6 evidence: Stereotyped gender-tone and gender-pitch co-occurrences (e.g., male assigned "Cognitive and Analytical" tone 41%, female assigned "Warm and Supportive" 44% and high pitch 62%) emerge from the LLM conversion layer itself, independent of the downstream TTS model's behavior. confidence: high relevance: medium limitations: - Evaluation is confined to English speech using a single-speaker dataset (LJSpeech), and the bias analysis covers only the five predefined attributes in P2VA-C. The scope of demographic bias in open-ended P2VA-O outputs is not characterized. - The user study does not report participant demographics, task instructions, or statistical significance, limiting the generalizability of the 90% alignment claim. The framework's dependence on Parler-TTS means results may not transfer to TTS systems with different conditioning interfaces. Bias mitigation methods are identified as future work; this paper provides quantification but no remediation. The attribute preset (P2VA-C) was derived from Parler-TTS training data labels, coupling the framework's controllability vocabulary to one model's training choices. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2509.15845' published_date: "2025-09-19" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - description_embedding_style_control - prompt_conditioned_identity_and_privacy claims: - claim_id: conditioning_a_flow_matching_model_to_generate_speaker role: supports claim: Conditioning a flow matching model to generate speaker embeddings from natural language descriptions enables automatic voice assignment for characters without requiring reference audio. source: §2.2, §3.3, Table 2 evidence: The TTT model, trained with OT-CFM to map text descriptions (age, gender, personality, identity) to speaker embeddings, achieves CMS of 2.866 and sex accuracy above 96% for adult demographics; Qwen3-Embedding-0.6B outperforms T5-Large and RoBERTa-Large on all character attribute metrics. confidence: high relevance: low - claim_id: providing_llm_derived_emotion_scene_instructions_as_an role: supports claim: Providing LLM-derived emotion-scene instructions as an additional conditioning signal to TTS improves emotional expressiveness without substantially degrading intelligibility. source: §3.3, Table 3 evidence: CA-Instruct-TTS achieves MOS-E of 4.15 vs. 3.67 and MOS-N of 3.33 vs. 3.10 compared to the instruction-free CA-TTS baseline, with WER increasing only from 2.39% to 2.54%. confidence: high relevance: low - claim_id: embedding_based_speaker_generation_from_text_degrades_for role: complicates claim: Embedding-based speaker generation from text degrades for child voices due to acoustic similarity and training data artifacts. source: §3.3, Figure 2 evidence: The TTT model shows notably lower age accuracy for the child category (74.38% with TTT-Qwen3-0.6B) compared to adult categories (90%+). The paper attributes this to pre-pubertal voice similarity and to adult speakers imitating child voices in available training data. confidence: high relevance: low - claim_id: larger_semantically_richer_text_encoders_improve_text_conditioned role: supports claim: Larger, semantically richer text encoders improve text-conditioned speaker embedding generation across demographic categories. source: §3.3, Table 2 evidence: TTT-Qwen3-0.6B achieves substantially higher CMS (2.866) and age accuracy across most categories compared to TTT-T5-Large (CMS 2.375) and TTT-RoBERTa-Large (CMS 2.359), suggesting that language understanding capacity from pre-trained LLMs transfers to voice attribute generation. confidence: high relevance: low - claim_id: the_flow_matching_framework_extends_naturally_from_speech role: refines claim: The flow matching framework extends naturally from speech waveform generation to speaker embedding synthesis. source: §2.2 evidence: The OT-CFM framework is applied directly to the generation of continuous speaker embedding vectors from noise, using the same DiT backbone and conditioning architecture as the speech-side flow matching decoder. The speaker embedding space is treated as the target data distribution. confidence: high relevance: low limitations: - All evaluations use proprietary internal data. No comparisons to published expressive audiobook TTS systems are included, limiting the ability to independently verify the reported gains in emotional expressiveness. - The TTT model struggles with child voice generation, an acknowledged limitation tied to data scarcity and label noise in child speech corpora. Future work is noted toward collecting authentic child speech. caveats: - Instruction conditioning is an evaluated capability but not the paper's sole architectural contribution. - id: '2509.17516' published_date: "2025-09-22" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: low evidence_role: - historical_context current_role: active_evidence method_family: - context_conditioned_situated_tts claims: - claim_id: decoupling_the_training_pairing_between_the_speech_prompt role: supports claim: Decoupling the training pairing between the speech prompt and the synthesis target (rather than using identical prompt/target audio) improves text-prosody alignment in style-controllable TTS, at the cost of an inflated speaker-similarity score under the coupled setup. source: §3.3, Table 2 evidence: Switching from non-decoupled to decoupled prompt/target training raises S-MOS from 3.45±0.09 to 3.82-3.93±0.06-0.07 while reducing speaker-similarity (SS) from 0.87 (interpreted as over-similar timbre/prosody) to 0.69-0.8. confidence: high relevance: low - claim_id: providing_an_autoregressive_speech_language_model_with_explicit role: supports claim: Providing an autoregressive speech language model with explicit surrounding-sentence text context improves narrative coherence in long-form speech generation beyond what sentence-isolated synthesis achieves. source: §2.1, §3.3, Table 1 evidence: Adding pre/post-context text sequences raises Chapter M-MOS from 3.88±0.07 (CosyVoice2 baseline, sentence-level synthesis) to 4.13±0.09 (Infer-ctx), and a qualitative example shows the model generating contextually appropriate laughter for a sentence with no explicit emotional markers when given the preceding sentence's context. confidence: high relevance: high - claim_id: the_strength_of_the_prompt_target_decoupling_threshold role: complicates claim: The strength of the prompt/target decoupling threshold used during training trades off speaker-identity stability against naturalness, rather than improving both simultaneously. source: §3.3, Table 2 evidence: Sweeping the voiceprint-similarity clustering threshold shows a lower threshold reduces speaker-similarity (with occasional timbre discontinuities) but yields slightly higher S-MOS, while a higher threshold approaches the non-decoupled model's inflated similarity at the cost of lower S-MOS. confidence: high relevance: low - claim_id: self_distillation_from_a_pretrained_emotional_tts_model role: supports claim: Self-distillation from a pretrained emotional TTS model, filtered by objective quality metrics, can mitigate the scarcity of high-intensity emotional training data and improve controllable emotional expressiveness. source: §2.2, §3.3, Table 3, Table 4 evidence: On the CV3-Eval text-unrelated test set, the model trained with 500 hours of emotional data plus 5,300 hours of self-distilled augmentation achieves larger high- vs. low-intensity F1 discriminability than a CosyVoice2-instruct baseline (e.g. angry ΔF1 0.31 vs. 0.07) and higher mixed-emotion S-MOS (3.87±0.09 vs. 3.35±0.07). confidence: high relevance: low limitations: - All training data (1M-hour audiobook corpus, 100K-hour context-aware corpus, 500-hour emotional corpus, 5K-hour augmentation set) and all three test sets (Test-NAR, Test-DIA, Test-CHAP) are proprietary and internally constructed; no code, model weights, or evaluation data are released, which limits independent verification of the reported gains. - The evaluation is confined to Chinese-language audiobook content with a single subjective rater pool (50 native Chinese speakers); generalization to other languages or narration styles is untested. The decoupling-threshold ablation reports trends across tested values without identifying a clearly optimal setting, and the paper notes the risk of persona instability at high thresholds is hypothesized rather than directly measured across the full range. The authors flag future work on scaling chapter-level context data and applying reinforcement learning for further quality improvement. caveats: - The connection to instruction-conditioned speech generation is contextual or benchmark-oriented rather than a direct conditioning contribution. - id: '2509.24570' published_date: "2025-09-29" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS - VC - evaluation architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: - direct_prompt_conditioned_speech_lm - instruction_data_and_annotation_pipeline claims: - claim_id: automated_pipelines_combining_expressive_tts_voice_conversion_and role: supports claim: Automated pipelines combining expressive TTS, voice conversion, and LLM-based instruction generation can produce large-scale paired speech style editing data without manual recording or annotation, provided a multi-criterion filtering step is applied. source: §2.1, §2.2, Fig. 3 evidence: The three-stage pipeline (EmoCapTTS synthesis + Chatterbox voice conversion + Qwen3-8B instruction generation) yields 382 hours and ~100,000 pairs from EARS and Expresso source material, filtered to WER < 10, style similarity > 0.5, and speaker similarity > 0.5 *(§2.1, §2.2, Fig. 3)*. confidence: medium relevance: medium - claim_id: fine_grained_diverse_natural_language_instructions_improve_both role: supports claim: Fine-grained, diverse natural-language instructions improve both in-domain accuracy and cross-domain generalization of instruction-guided speech style editing models relative to coarse, templated instruction sets. source: §4.2, Table 2 evidence: LlasaEdit trained on ISSE outperforms the same architecture trained on ESD across WER, style similarity, speaker similarity, and UTMOS in-domain (8.06 vs. 10.07 WER; 0.68 vs. 0.64 style-sim), and the ISSE-trained model's cross-domain performance on ESD exceeds the ESD-trained model's in-domain performance on several metrics *(§4.2, Table 2)*. confidence: medium relevance: medium - claim_id: instruction_guided_style_editing_models_trained_on_narrow role: complicates claim: Instruction-guided style editing models trained on narrow, templated-instruction datasets fail catastrophically when evaluated on more diverse, fine-grained instruction distributions. source: §4.2, Table 2 evidence: The ESD-trained LlasaEdit model, when evaluated on the ISSE test set, produces a WER of 68.17, compared to 10.07 on its own in-domain ESD test set, indicating the model does not generalize beyond the coarse single-attribute instructions it was trained on *(§4.2, Table 2)*. confidence: medium relevance: medium - claim_id: isolating_style_variation_from_speaker_identity_in_synthetically role: complicates claim: Isolating style variation from speaker identity in synthetically generated paired training data requires an explicit voice-conversion correction step, since expressive TTS models conditioned on style descriptions alone conflate style and timbre changes. source: §2.1 evidence: EmoCapTTS-generated stylized speech differs from the anchor speech in timbre because the model lacks explicit speaker control; a separate voice conversion stage (Chatterbox) is needed to re-align target timbre to the anchor speaker before the pair can be used to define a style-only edit *(§2.1)*. confidence: medium relevance: medium limitations: - The dataset and benchmark are limited to English, which the authors explicitly flag as constraining applicability to multilingual editing scenarios. The generated portion of ISSE (292 of 382 hours) is itself the product of a TTS+VC synthesis pipeline rather than real recordings, so any systematic biases or artifacts introduced by EmoCapTTS or Chatterbox could propagate into models trained on it; the quality-filtering thresholds (WER < 10, similarity > 0.5) are relatively loose and their effect on downstream editing fidelity is not separately ablated. The benchmark comparison is against a single alternative dataset (ESD) and a single model architecture (LlasaEdit); no comparison is made against other instruction-guided editing systems such as InstructSpeech, and no ablation isolates the individual contribution of instruction diversity versus raw data scale. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. - id: '2509.26514' published_date: "2025-09-30" entry_date: '2026-07-25' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - GAN relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: - structured_intermediate_control claims: - claim_id: decoupling_instruction_interpretation_from_speech_rendering_via_an role: supports claim: Decoupling instruction interpretation from speech rendering, via an explicit intermediate textual feature representation, can match or exceed instruction-annotated controllable TTS systems without requiring any manually labeled instruction-speech data. source: §3.2, Table 1 evidence: BatonVoice-1.7B reaches 57.6% emotion accuracy using 0 hours of manual instruction annotation, exceeding CosyVoice (43.8%, 556 hours) and CosyVoice2 (37.8%, 1,500 hours) on the same Seed-TTS-derived English emotion benchmark. confidence: high relevance: medium - claim_id: when_conditioning_signals_are_represented_as_explicit_text role: supports claim: When conditioning signals are represented as explicit text rather than learned embeddings, a downstream generation model can gain synthesis quality from a stronger upstream language model without any retraining of the generation model itself. source: §3.5, Figure 3b evidence: Holding BatonTTS fixed and only changing the external "conductor" LLM that produces the vocal plan, emotion accuracy rises monotonically from 29.8% (Qwen3-1.7B) through 39.8% (Qwen3-80B) and 47.8% (Qwen3-Max) to 57.6% (Gemini-2.5-Pro). confidence: high relevance: high - claim_id: automatic_emotion_classification_accuracy_and_human_judged_naturalness role: complicates claim: Automatic emotion-classification accuracy and human-judged naturalness can diverge, so strong performance on an LLM-judged style-control metric does not guarantee a favorable human preference outcome against commercial systems. source: §3.3, Table 2 evidence: On a free-form instruction-following test set judged by trained human annotators, BatonVoice wins only 56% of comparisons against CosyVoice and just 30% against Minimax-2.5-HD, with annotators specifically citing weaker fluency and naturalness. confidence: high relevance: medium - claim_id: numerical_structured_representations_of_prosodic_control_targets_transfer role: refines claim: Numerical, structured representations of prosodic control targets transfer more precisely to a conditioned TTS decoder than free-text qualitative descriptions of the same target style. source: §B.2, Table 5 evidence: On a RAVDESS reconstruction task, the structured numerical vocal-plan format achieves 1.54 Mel-Cepstral Distortion versus 2.62 for an equivalent caption-based qualitative description, and ablating any single numerical feature (pitch, energy, or spectral centroid) increases MCD. confidence: high relevance: medium limitations: - The paper's headline emotion-control results across both English and Chinese benchmarks rely entirely on an LLM (Gemini-2.5-Pro) as an automatic judge rather than human raters; the one benchmark that does use human evaluation shows BatonVoice losing to the top closed-source baseline on naturalness and fluency (30% win rate against Minimax-2.5-HD), a gap not visible in the automated emotion-accuracy metric. - The vocal-plan representation is limited to pitch, energy, and spectral centroid; the authors note that finer-grained paralinguistic features such as emphatic stress and non-verbal vocalizations are not captured and are left to future work. Best reported results depend on using a large, capable, and likely costly external LLM (Gemini 2.5 Pro) as the conductor at inference time; using the in-house model of the same size as the orchestra as its own conductor (Qwen3-1.7B, 29.8% accuracy) performs far worse, so the practical quality of the system is bottlenecked by conductor access and cost, not by BatonTTS alone. Cross-lingual generalization is demonstrated on one unseen language (Chinese) with instructions and text machine-translated by the same LLM family used for judging, which leaves open how the approach performs on languages more typologically distant from English or evaluated independently of the conductor/judge model. caveats: - Instruction or description conditioning is central to the paper's speech-generation contribution. claim_clusters: - id: natural_language_prompts_enable_multidimensional_style_control claim: Natural-language instructions can control multiple speech attributes within one synthesis system, including emotion, accent, speaking rate, prosody, and vocal identity. status: strongly_supported confidence: high supporting_papers: - '2402.01912' - '2407.05407' - '2409.03283' - '2501.06282' - 2025.findings-acl.1226 - interspeech-2025-0203 - interspeech-2025-0948 - 2025.clicit-1.27 contradicting_papers: [] refining_papers: - '2506.16381' - '2509.13989' caveats: - Control accuracy varies sharply by attribute and instruction abstraction; success on emotion or rate does not imply reliable control of age, timbre, non-verbal events, or local emphasis. last_reviewed: '2026-07-25' - id: small_instruction_finetuning_adapts_pretrained_tts claim: Instruction fine-tuning can add useful controllability to a pretrained speech generator with substantially less data than full pretraining. status: strongly_supported confidence: high supporting_papers: - '2407.05407' - '2409.03283' - '2501.06282' - interspeech-2025-2679 contradicting_papers: [] refining_papers: - 2025.acl-long.681 caveats: - Small-data adaptation is demonstrated for bounded domains or attribute sets; diverse instruction tuning can reduce performance on foundational tasks. last_reviewed: '2026-07-25' - id: instruction_diversity_improves_generalization claim: Diverse, fine-grained instruction supervision generalizes better than narrow templates or single-task instruction corpora. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - 2025.acl-long.681 - 2025.findings-acl.534 - '2509.24570' contradicting_papers: [] refining_papers: - '2409.06666' caveats: - Instruction diversity is often confounded with dataset scale, source-domain diversity, and filtering quality. last_reviewed: '2026-07-25' - id: automatic_instruction_data_construction_scales_supervision claim: Automatic labeling, LLM captioning or rewriting, and synthetic-speech pipelines can scale instruction-speech supervision beyond manual annotation. status: strongly_supported confidence: medium supporting_papers: - '2305.11000' - '2402.01912' - 2025.findings-acl.101 - 2025.findings-acl.1226 - 2025.findings-acl.534 - '2509.24570' contradicting_papers: [] refining_papers: - 2025.findings-acl.687 caveats: - Automatically generated labels inherit classifier, LLM, TTS, and voice-conversion errors; several studies do not isolate scale from annotation quality. last_reviewed: '2026-07-25' - id: structured_intermediates_decouple_interpretation_and_rendering claim: Explicit intermediate representations can decouple instruction interpretation from speech rendering and make controllability more modular. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - 2025.findings-acl.101 - 2025.findings-acl.1226 - '2505.17093' - '2509.26514' contradicting_papers: [] refining_papers: - '2508.17494' caveats: - 'The best representation is unsettled: free-form descriptions are accessible, while categorical, SSML, and numerical plans are usually more precise.' last_reviewed: '2026-07-25' - id: global_embeddings_limit_local_and_compositional_control claim: Global style embeddings are insufficient for instructions that specify local timing, linguistic position, or independently composable acoustic attributes. status: emerging confidence: medium supporting_papers: - '2507.00808' - '2509.26514' contradicting_papers: [] refining_papers: - 2025.acl-long.346 - '2509.24570' caveats: - Evidence comes from different tasks and representation families, so the boundary between embedding limitations and training-data limitations is not yet isolated. last_reviewed: '2026-07-25' - id: style_and_timbre_control_remain_entangled claim: Instruction-driven style changes can unintentionally alter speaker identity unless style and timbre are explicitly separated or corrected. status: emerging confidence: medium supporting_papers: - 2025.acl-long.346 - '2509.24570' contradicting_papers: [] refining_papers: - '2506.16381' - '2509.15845' caveats: - Explicit separation improves controllability, but its necessity has not been established across all prompt-conditioned architectures. last_reviewed: '2026-07-25' - id: open_source_systems_lag_on_abstract_instruction_following claim: Open-source instruction-conditioned speech systems lag closed commercial systems most clearly on abstract, compositional, and degree-scaled control. status: strongly_supported confidence: medium supporting_papers: - '2506.16381' - '2509.09716' - '2509.13989' contradicting_papers: [] refining_papers: [] caveats: - Commercial systems are opaque and change over time, while evaluations cover different languages and prompt taxonomies. last_reviewed: '2026-07-25' - id: fine_grained_control_is_attribute_dependent claim: 'Fine-grained instruction following remains attribute-dependent: models that control emotion or speaking rate may fail on age, timbre, phonemic length, emphasis, or non-verbal vocal events.' status: emerging confidence: high supporting_papers: - 2025.clicit-1.27 contradicting_papers: [] refining_papers: - '2506.16381' - interspeech-2025-2679 - '2509.09716' - '2509.13989' caveats: - Most evidence is diagnostic or complicating rather than a controlled comparison over a shared attribute inventory. last_reviewed: '2026-07-25' - id: automatic_judges_do_not_fully_replace_human_evaluation claim: Automatic judges can approximate human instruction-adherence ratings, but they do not reliably capture all perceptual alignment, naturalness, or domain-specific failure modes. status: contested confidence: high supporting_papers: - '2506.16381' - '2508.07273' - '2509.09716' contradicting_papers: - '2504.12867' - '2509.13989' - '2509.26514' refining_papers: - '2507.00808' - interspeech-2025-2679 caveats: - Agreement depends on task subjectivity, judge model, prompt design, and whether the target domain resembles the judge's calibration data. last_reviewed: '2026-07-25' - id: prompt_interpretation_can_amplify_demographic_bias claim: Natural-language style and identity prompts can reproduce or amplify demographic stereotypes through both TTS training data and LLM-mediated prompt interpretation. status: strongly_supported confidence: medium supporting_papers: - 2025.findings-naacl.298 - '2508.11326' - '2505.17093' contradicting_papers: [] refining_papers: - '2509.13989' caveats: - Bias profiles vary across models trained on similar data, and naive fairness prompting can introduce compensatory rather than neutral behavior. last_reviewed: '2026-07-25' - id: situated_context_improves_expressive_speech_with_bounded_returns claim: Dialogue, scene, and narrative context improve expressive or coherent speech generation, but gains diminish or reverse when context becomes too long. status: strongly_supported confidence: medium supporting_papers: - 2025.acl-long.911 - 2025.findings-acl.101 - 2025.findings-acl.1226 - '2509.17516' contradicting_papers: [] refining_papers: - 2025.acl-long.911 caveats: - The optimal context window is benchmark-dependent, and some evidence uses automatically derived emotion or scene labels. last_reviewed: '2026-07-25' - id: instruction_control_transfers_to_low_resource_and_clinical_domains claim: General instruction-conditioned TTS can be adapted or prompted for low-resource, historical-language, and atypical-speech domains with limited task-specific data. status: emerging confidence: medium supporting_papers: - interspeech-2025-2679 - 2025.clicit-1.27 contradicting_papers: [] refining_papers: [] caveats: - Evidence covers isolated languages and one clinical speech domain; phonological interference and metric validity remain unresolved. last_reviewed: '2026-07-25' - id: instruction_conditioning_enables_audio_free_anonymization claim: Textual speaker descriptions can support speech anonymisation without passing source-speaker audio or embeddings to the generator. status: emerging confidence: medium supporting_papers: - '2507.07799' contradicting_papers: [] refining_papers: [] caveats: - Prompt-based identities vary across utterances, and privacy conclusions depend strongly on attacker and evaluation design. last_reviewed: '2026-07-25' - id: stronger_interpreters_improve_fixed_renderers claim: A stronger upstream language model can improve instruction-conditioned synthesis through an explicit control plan without retraining the downstream renderer. status: emerging confidence: medium supporting_papers: - '2509.26514' - '2505.17093' contradicting_papers: [] refining_papers: [] caveats: - The strongest result depends on a proprietary external LLM, so cost, reproducibility, and portability remain open. last_reviewed: '2026-07-25' method_families: - id: direct_prompt_conditioned_speech_lm name: Direct prompt-conditioned speech language models summary: Autoregressive or hybrid speech language models consume natural-language style or interaction instructions alongside text and generate speech tokens directly. papers: - '2305.11000' - '2402.01912' - '2407.05407' - '2409.03283' - '2412.02612' - '2501.06282' - 2025.acl-long.681 - '2508.11326' - interspeech-2025-2679 - 2025.clicit-1.27 - '2509.24570' open_questions: - Which instruction representations retain fine-grained acoustic control without weakening content accuracy or base-model capabilities? - id: multimodal_spoken_instruction_model name: Multimodal spoken-instruction models summary: Speech-capable LLMs learn instruction following across comprehension, dialogue, and generation, often using interleaved or parallel text and speech outputs. papers: - '2305.11000' - '2408.16725' - '2409.06666' - '2410.17196' - '2412.02612' - '2501.06282' - '2508.09600' open_questions: - Can native spoken-instruction models match text-pipeline reasoning and safety while preserving low response latency? - id: description_embedding_style_control name: Description-embedding style control summary: Text descriptions are embedded into a continuous conditioning space that controls emotion, prosody, speaker attributes, or singing style in encoder-decoder, flow-matching, diffusion, and hybrid generators. papers: - '2312.15821' - '2507.00808' - '2507.08012' - 2025.acl-long.346 - 2025.findings-acl.687 - interspeech-2025-0203 - interspeech-2025-0948 - interspeech-2025-1192 - '2509.15845' open_questions: - How should description encoders represent local timing and compositional attributes that global embeddings routinely blur? - id: structured_intermediate_control name: Structured intermediate control summary: An LLM or analysis stage converts free-form requests into emotion captions, style tags, SSML, or numerical vocal plans before a separate speech renderer applies them. papers: - 2025.findings-naacl.38 - 2025.findings-acl.101 - 2025.findings-acl.1226 - '2508.17494' - '2505.17093' - '2509.26514' open_questions: - Which intermediate representation best balances user accessibility, precision, and portability across renderers? - id: context_conditioned_situated_tts name: Context-conditioned situated TTS summary: Dialogue history, narrative context, scene descriptions, or inferred interlocutor state condition speech generation beyond the current sentence. papers: - 2025.acl-long.911 - 2025.findings-acl.101 - 2025.findings-acl.1226 - '2509.17516' open_questions: - Why does additional dialogue context help only up to a bounded window, and how should longer context be selected or compressed? - id: instruction_data_and_annotation_pipeline name: Instruction-data and annotation pipelines summary: Automatic acoustic labels, LLM rewriting or captioning, synthetic speech, and filtering pipelines create instruction-speech supervision at scales impractical for manual annotation. papers: - '2305.11000' - '2402.01912' - 2025.acl-long.681 - 2025.findings-acl.101 - 2025.findings-acl.1226 - 2025.findings-acl.534 - '2508.11326' - '2509.24570' open_questions: - How much of the observed gain comes from instruction diversity, raw scale, label quality, or the filtering policy? - id: instruction_following_benchmark_and_audit name: Instruction-following benchmarks and audits summary: Benchmarks and audits test compositional acoustic control, human alignment, robustness, demographic bias, and the validity of automatic judges rather than proposing a new renderer. papers: - 2025.findings-naacl.298 - '2506.16381' - '2508.07273' - '2508.08095' - '2504.12867' - '2509.09716' - '2509.13989' - '2505.17093' open_questions: - Can a benchmark jointly measure instruction adherence, naturalness, identity preservation, fairness, and robustness without relying on the evaluated model family? - id: prompt_conditioned_identity_and_privacy name: Prompt-conditioned identity and privacy control summary: Natural-language speaker descriptions or attributes are used for anonymisation, voice creation, cloning, and identity-preserving style transfer. papers: - '2507.07799' - 2025.acl-long.346 - interspeech-2025-0047 - '2509.15845' open_questions: - How can prompt-defined identity remain consistent across utterances without reproducing protected speaker information or demographic stereotypes? reassessment_queue: - id: automatic_judges_do_not_fully_replace_human_evaluation type: benchmark_validity reason: The current evidence is genuinely mixed between high judge-human agreement and systematic perceptual blind spots. trigger: An independently validated multilingual benchmark reports per-attribute judge-human agreement and naturalness correlations across open and closed systems. due: 2026-10 current_assessment: contested watch_for: - Independent human validation of audio-language-model judges - Attribute-level disagreement analyses - id: direct_prompt_conditioned_speech_lm type: method_family reason: The family spans direct TTS, spoken dialogue, editing, and domain adaptation and may become too broad as evidence grows. trigger: At least three additional papers form a coherent architectural subgroup with distinct conditioning placement. due: 2026-10 current_assessment: active_evidence watch_for: - Stable separation between instruction encoders, prompt tokens, and in-context conditioning - id: prompt_interpretation_can_amplify_demographic_bias type: claim_status reason: Three papers support the risk, but model-specific bias profiles and mitigation behavior vary. trigger: A cross-model, multilingual audit separates training-data bias from LLM interpretation bias and tests mitigation. due: 2026-10 current_assessment: strongly_supported watch_for: - Intersectional and multilingual bias audits - Evidence that mitigation generalizes without compensatory bias - id: global_embeddings_limit_local_and_compositional_control type: claim_status reason: The claim is supported by only two direct studies and may depend on the embedding architecture or data. trigger: A controlled comparison holds renderer and data fixed while varying global, sequence-level, and structured conditioning. due: 2026-10 current_assessment: emerging watch_for: - Position-aware instruction encoders - Compositional multi-attribute ablations - id: instruction_diversity_improves_generalization type: claim_status reason: Instruction diversity is repeatedly beneficial but remains confounded with dataset size and filtering. trigger: A factorial ablation separates instruction diversity, hours, domain breadth, and filtering quality. due: 2026-10 current_assessment: strongly_supported watch_for: - Matched-scale template-versus-free-form comparisons - Cross-domain instruction-following tests open_questions: - Which instruction representation provides the best balance of free-form usability, local temporal precision, and compositional control? - How can instruction-conditioned systems preserve speaker identity while independently changing style, emotion, and non-verbal events? - Which automatic evaluation protocol predicts human judgments across abstract prompts, atypical speech, and multilingual settings? - How much instruction diversity is needed before gains saturate, and can it be separated from raw data scale? - Can open systems close the commercial-system gap on abstract and degree-scaled control without relying on proprietary data or judge models? trend_notes: - The evidence shifts from broad multimodal instruction tuning in 2023–2024 toward dedicated controllable-TTS datasets, benchmarks, and audits in 2025. - By 2025, instruction conditioning increasingly uses LLM-generated descriptions or structured intermediate plans rather than fixed categorical style labels alone. - Evaluation work in 2025 moves beyond emotion accuracy toward compositional attributes, human alignment, demographic bias, and domain robustness. - A recurring 2025 pattern separates instruction interpretation from rendering, allowing stronger language models or explicit vocal plans to improve a fixed speech generator.