concept: streaming-tts last_updated: '2026-07-30' paper_count: 54 papers: - id: '2402.08093' published_date: "2024-02-12" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_streaming_generation claims: - claim_id: scaling_autoregressive_codec_tts_to_500m_parameters_and_10k_hours role: supports claim: Scaling autoregressive codec TTS to 500M+ parameters and 10K+ hours of training data produces qualitatively different prosody rendering on linguistically complex inputs compared to smaller models trained on less data. source: §4.3, Figure 4, Table 5 evidence: Scaling autoregressive codec TTS to 500M+ parameters and 10K+ hours of training data produces qualitatively different prosody rendering on linguistically complex inputs compared to smaller models trained on less data. confidence: high relevance: low - claim_id: ssl_based_speech_representations_with_explicit_speaker_disentanglement_outperform_purely role: supports claim: SSL-based speech representations with explicit speaker disentanglement outperform purely acoustic codec representations for zero-shot TTS, particularly in lower-resource languages. source: §4.1, Table 3 evidence: SSL-based speech representations with explicit speaker disentanglement outperform purely acoustic codec representations for zero-shot TTS, particularly in lower-resource languages. confidence: high relevance: low - claim_id: a_streamable_convolutional_decoder_can_match_or_exceed_a_diffusion role: supports claim: A streamable convolutional decoder can match or exceed a diffusion-based spectrogram decoder in subjective naturalness while reducing synthesis compute by approximately 3x and enabling low-latency streaming. source: §4.2, §4.5, Table 4 evidence: A streamable convolutional decoder can match or exceed a diffusion-based spectrogram decoder in subjective naturalness while reducing synthesis compute by approximately 3x and enabling low-latency streaming. confidence: high relevance: high - claim_id: applying_bpe_to_discrete_speech_tokens_reduces_autoregressive_sequence_length role: supports claim: Applying BPE to discrete speech tokens reduces autoregressive sequence length by approximately 40% without degrading downstream synthesis quality, enabling longer-context training. source: §2.2.3 evidence: Applying BPE to discrete speech tokens reduces autoregressive sequence length by approximately 40% without degrading downstream synthesis quality, enabling longer-context training. confidence: high relevance: low - claim_id: autoregressive_tts_trained_at_scale_generalises_to_a_wide_range role: supports claim: Autoregressive TTS trained at scale generalises to a wide range of textual phenomena without any explicit prosody annotation or task-specific supervision. source: §4.3, §6 evidence: Autoregressive TTS trained at scale generalises to a wide range of textual phenomena without any explicit prosody annotation or task-specific supervision. confidence: high relevance: low limitations: - Model weights are not released, and evaluation uses proprietary test speakers. The MUSHRA baselines (YourTTS, Bark, TortoiseTTS) are not trained on comparable data or compute, making architecture-level conclusions difficult to separate from scale effects. - The speechcode decoder is tightly coupled to a specific frozen SpeechGPT checkpoint via hidden-state conditioning, preventing modular updates and complicating experimentation. The paper identifies this as a limitation requiring future work. - Hallucinations and cutoffs remain an inherent issue of the autoregressive formulation, worsened by misalignment between noisy web audio and ASR-generated transcripts. The authors avoid denoising during training to test robustness but acknowledge this makes the alignment problem harder. - Emotions and paralinguistics remain below ceiling even for BASE-large, suggesting that 100K hours and 980M parameters are not sufficient for reliable rendering of these categories. Formal scaling laws for TTS (analogous to Chinchilla for text LMs) are proposed as future work but not established here. - The "emergent abilities" phenomenon is characterised across only three data-scale points and assessed by a single expert linguist, leaving open whether it is a smooth or discontinuous function of scale and how sensitive it is to tokenization and architecture choices. caveats: [] - id: '2408.02622' published_date: "2024-08-05" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation claims: - claim_id: full_duplex_speech_generation_requires_that_the_listening_channel_information role: supports claim: Full-duplex speech generation requires that the listening channel information be injected at intermediate representation layers, not at the input or output level, to avoid degrading speech generation quality. source: §6.2, Table 2 evidence: Full-duplex speech generation requires that the listening channel information be injected at intermediate representation layers, not at the input or output level, to avoid degrading speech generation quality. confidence: high relevance: high - claim_id: a_single_layer_discrete_token_autoregressive_tts_backbone_can_integrate role: supports claim: A single-layer discrete token autoregressive TTS backbone can integrate real-time audio input streams with minimal WER degradation relative to a no-listening baseline in controlled conditions. source: §6.2, Table 2 evidence: A single-layer discrete token autoregressive TTS backbone can integrate real-time audio input streams with minimal WER degradation relative to a no-listening baseline in controlled conditions. confidence: high relevance: high - claim_id: robustness_to_noise_and_sensitivity_to_unseen_speaker_interruptions_are role: supports claim: Robustness to noise and sensitivity to unseen speaker interruptions are distinct challenges in full-duplex SLMs, and voice-based generalisation introduces significantly higher error rates than command-based triggering. source: §6.2, Table 3 evidence: Robustness to noise and sensitivity to unseen speaker interruptions are distinct challenges in full-duplex SLMs, and voice-based generalisation introduces significantly higher error rates than command-based triggering. confidence: high relevance: high - claim_id: joint_fine_tuning_of_both_the_speech_generation_backbone_and role: supports claim: Joint fine-tuning of both the speech generation backbone and the streaming SSL encoder is necessary for full-duplex models to reach peak interactive capability; freezing either component degrades turn-taking recall. source: §6.3, Table 4 evidence: Joint fine-tuning of both the speech generation backbone and the streaming SSL encoder is necessary for full-duplex models to reach peak interactive capability; freezing either component degrades turn-taking recall. confidence: high relevance: high limitations: - 'The system produces speech tokens but not semantic speech responses: it stops speaking when interrupted but does not generate a contextually appropriate spoken reply. The paper evaluates TTS output and turn-taking accuracy, not full dialogue capability. The "full duplex" claim is therefore limited to interruption detection and cessation, not conversational back-and-forth.' - Voice-based FDM shows a meaningful WER increase (5.33% vs 4.28% baseline in clean conditions, 8.50% under noise), suggesting real costs from the dual-channel architecture in harder generalisation settings. The evaluation relies entirely on automatic metrics (WER, precision/recall for turn-taking); no subjective MOS or naturalness assessment is reported, making quality comparisons to cascade systems difficult. The 106M parameter model trained on LibriTTS is modest in scale, and it is unclear whether the fusion findings generalise to larger LLM backbones. Speaker-following (identifying which interrupting speaker to respond to) and speech-in/speech-out dialogue generation with full-duplex capability are explicitly left as future work. caveats: [] - id: '2408.16725' published_date: "2024-08-29" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation claims: - claim_id: simultaneous_text_and_audio_generation_conditioned_on_text_tokens_generated role: supports claim: Simultaneous text and audio generation, conditioned on text tokens generated in parallel, enables streaming speech output without the latency penalty of sequential text-then-audio decoding. source: §3.2 evidence: Simultaneous text and audio generation, conditioned on text tokens generated in parallel, enables streaming speech output without the latency penalty of sequential text-then-audio decoding. confidence: high relevance: high - claim_id: audio_reasoning_quality_in_end_to_end_speech_lms_lags role: supports claim: Audio reasoning quality in end-to-end speech LMs lags behind text reasoning quality when trained on similar data volumes, and batch inference strategies can partially bridge this gap. source: §3.2, §4.4 evidence: Audio reasoning quality in end-to-end speech LMs lags behind text reasoning quality when trained on similar data volumes, and batch inference strategies can partially bridge this gap. confidence: high relevance: medium - claim_id: a_three_stage_adapter_based_training_curriculum_can_integrate_speech role: supports claim: A three-stage adapter-based training curriculum can integrate speech input and output into a frozen language model backbone with minimal degradation to text capabilities. source: §3.3 evidence: A three-stage adapter-based training curriculum can integrate speech input and output into a frozen language model backbone with minimal degradation to text capabilities. confidence: high relevance: medium - claim_id: multi_codebook_audio_codecs_with_high_token_rates_require_parallel role: supports claim: Multi-codebook audio codecs with high token rates require parallel decoding schemes to maintain practical streaming throughput in autoregressive speech LMs. source: §3.1, §3.2 evidence: Multi-codebook audio codecs with high token rates require parallel decoding schemes to maintain practical streaming throughput in autoregressive speech LMs. confidence: high relevance: high limitations: - The paper reports no MOS or naturalness metrics for speech output, making it impossible to quantitatively compare audio quality against TTS or SCA baselines. The claim that quality is "on par with common TTS systems" is unsupported. - Evaluation is restricted to ASR performance on LibriSpeech; there is no evaluation of conversational quality, response coherence, or latency. The 0.5B model size limits reasoning depth, and the paper acknowledges that audio reasoning remains weaker than text reasoning. The VoiceAssistant-400K dataset is entirely synthesized by GPT-4o, which may introduce systematic biases in prosody and topic coverage. The model supports only English. The paper is described as a work-in-progress technical report, with some experiments deferred to a future version. caveats: [] - id: '2409.06666' published_date: "2024-09-10" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA architecture: - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - hybrid_modular_streaming_systems claims: - claim_id: end_to_end_speech_llms_with_parallel_text_and_speech role: supports claim: End-to-end speech LLMs with parallel text and speech generation can achieve lower response latency than cascaded ASR-LLM-TTS pipelines without sacrificing prosody coherence under low-latency streaming conditions. source: §4.5 evidence: End-to-end speech LLMs with parallel text and speech generation can achieve lower response latency than cascaded ASR-LLM-TTS pipelines without sacrificing prosody coherence under low-latency streaming conditions. confidence: high relevance: high - claim_id: aligning_llm_output_to_speech_interaction_conventions_through_targeted_instruction role: supports claim: Aligning LLM output to speech interaction conventions through targeted instruction data rewriting substantially improves response style suitability, independently of model architecture. source: §3, §4.4 evidence: Aligning LLM output to speech interaction conventions through targeted instruction data rewriting substantially improves response style suitability, independently of model architecture. confidence: high relevance: medium - claim_id: non_autoregressive_ctc_decoding_from_llm_hidden_states_enables_streaming role: supports claim: Non-autoregressive CTC decoding from LLM hidden states enables streaming speech synthesis whose speech rate and naturalness are robust to chunk size variation, unlike word-level streaming TTS cascades. source: §4.5, Table 4 evidence: Non-autoregressive CTC decoding from LLM hidden states enables streaming speech synthesis whose speech rate and naturalness are robust to chunk size variation, unlike word-level streaming TTS cascades. confidence: high relevance: high - claim_id: training_an_end_to_end_speech_interaction_model_on_a role: supports claim: Training an end-to-end speech interaction model on a small, carefully curated speech instruction dataset is sufficient to significantly close the gap with models trained on orders of magnitude more data, provided the LLM backbone is sufficiently capable. source: §4.4, §5 evidence: Training an end-to-end speech interaction model on a small, carefully curated speech instruction dataset is sufficient to significantly close the gap with models trained on orders of magnitude more data, provided the LLM backbone is sufficiently capable. confidence: high relevance: medium limitations: - The ASR-WER of 10.82% is notably higher than cascaded baselines (3.78% for SALMONN+Orca), reflecting that the speech decoder is trained on only approximately 1K hours of response speech — far below industrial TTS scale. Intelligibility limitations restrict applicability in domains requiring precise spoken content. - The evaluation benchmark (InstructS2S-Eval) is derived from AlpacaEval with math and code questions removed, which skews toward conversational helpfulness and may not represent more demanding speech interaction tasks. The speech encoder relies on Whisper, which is optimised for ASR rather than general speech understanding, potentially limiting response to prosodic or para-linguistic cues in the user's speech. The current architecture does not support full-duplex interaction (interruption, turn-taking) — speech responses are generated after the full instruction is received. The training data is synthesised from text corpora, which may not capture the naturalness and variability of real spoken dialogue. caveats: [] - id: '2410.00037' published_date: "2024-09-17" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: foundational method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: eliminating_the_text_bottleneck_in_spoken_dialogue_requires_modeling_acoustic role: supports claim: Eliminating the text bottleneck in spoken dialogue requires modeling acoustic tokens jointly with semantic tokens in a single generative model, as purely semantic approaches cannot capture paralinguistic information or generate in arbitrary voices. source: §3.4, §5.4 evidence: Eliminating the text bottleneck in spoken dialogue requires modeling acoustic tokens jointly with semantic tokens in a single generative model, as purely semantic approaches cannot capture paralinguistic information or generate in arbitrary voices. confidence: high relevance: low - claim_id: predicting_time_aligned_text_tokens_as_a_per_frame_prefix role: supports claim: Predicting time-aligned text tokens as a per-frame prefix to audio tokens substantially improves the linguistic quality and factual accuracy of speech generated by audio language models, with minimal inference overhead. source: §3.4.4, §5.3, Table 6 evidence: Predicting time-aligned text tokens as a per-frame prefix to audio tokens substantially improves the linguistic quality and factual accuracy of speech generated by audio language models, with minimal inference overhead. confidence: high relevance: medium - claim_id: modeling_conversation_as_parallel_autoregressive_streams_for_each_speaker_without role: supports claim: Modeling conversation as parallel autoregressive streams for each speaker, without explicit turn boundaries, enables full-duplex spoken interaction and allows training on naturally overlapping speech. source: §3.4.3, §5.6, Table 9 evidence: Modeling conversation as parallel autoregressive streams for each speaker, without explicit turn boundaries, enables full-duplex spoken interaction and allows training on naturally overlapping speech. confidence: high relevance: high - claim_id: adversarial_only_training_of_neural_audio_codecs_substantially_improves_subjectively role: supports claim: Adversarial-only training of neural audio codecs substantially improves subjectively rated audio quality relative to mixed reconstruction-adversarial objectives, despite degrading objective metrics such as VisQOL. source: §3.3, §5.2, Table 4 evidence: Adversarial-only training of neural audio codecs substantially improves subjectively rated audio quality relative to mixed reconstruction-adversarial objectives, despite degrading objective metrics such as VisQOL. confidence: high relevance: medium - claim_id: standard_objective_audio_quality_metrics_visqol_mosnet_are_unreliable_proxies role: supports claim: Standard objective audio quality metrics (VisQOL, MOSNet) are unreliable proxies for perceived quality when the training objective changes, making human evaluation indispensable for codec comparison. source: §5.2, §5.8 evidence: Standard objective audio quality metrics (VisQOL, MOSNet) are unreliable proxies for perceived quality when the training objective changes, making human evaluation indispensable for codec comparison. confidence: high relevance: low limitations: - 'Moshi''s spoken factual question answering performance lags substantially behind its Helium text baseline, particularly on multi-sentence or syntactically complex questions (TriviaQA: 22.8 vs. 56.4 for text-only Helium). This indicates that audio training causes significant forgetting of factual knowledge, and the instruct fine-tuning data does not cover the syntactic diversity needed to recover it.' - 'Signal-based watermarking (Audioseal) is ineffective against codec compression: Mimi''s own lossy coding removes the watermark to below detection threshold. The generative watermarking alternatives explored in §6.4 are blocked by the non-idempotence of audio codecs, leaving no robust content attribution mechanism available at release.' - The instruction fine-tuning pipeline relies heavily on synthetic TTS-generated speech for both conversation transcripts and user voice diversity. This introduces a distribution mismatch with real conversational speech that likely limits robustness to unusual acoustic conditions and speaking styles. The paper notes this but leaves more realistic instruct data collection as future work. - Quantization below 4-bit precision causes noticeable audio artifacts (repetitive generation, noisy voice) that current automatic metrics fail to detect, requiring entropy-spectrum analysis as a surrogate. This underscores a general gap in speech quality evaluation tooling for generative dialogue models. caveats: [] - id: '2410.11190' published_date: "2024-10-15" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: staged_adapter_training_encoder_alignment_before_language_model_fine_tuning role: supports claim: Staged adapter training (encoder alignment before language model fine-tuning) enables tri-modal extension of a compact language model with minimal data without catastrophic forgetting of the base model's text capabilities. source: §3.3 evidence: Staged adapter training (encoder alignment before language model fine-tuning) enables tri-modal extension of a compact language model with minimal data without catastrophic forgetting of the base model's text capabilities. confidence: high relevance: medium - claim_id: using_continuous_encoder_features_whisper_rather_than_discrete_audio_tokens role: supports claim: Using continuous encoder features (Whisper) rather than discrete audio tokens for speech input yields more stable and semantically consistent representations, reducing ASR loss instability during training. source: §3.1, "Audio Encoder" evidence: Using continuous encoder features (Whisper) rather than discrete audio tokens for speech input yields more stable and semantically consistent representations, reducing ASR loss instability during training. confidence: high relevance: medium - claim_id: adding_a_third_modality_vision_to_an_audio_text_spoken role: supports claim: Adding a third modality (vision) to an audio-text spoken conversational agent modestly degrades ASR performance, likely due to diluted training data proportion rather than architectural interference. source: §4.4, Table 2 evidence: Adding a third modality (vision) to an audio-text spoken conversational agent modestly degrades ASR performance, likely due to diluted training data proportion rather than architectural interference. confidence: high relevance: medium - claim_id: command_based_semantic_interruption_intent_token_classification_provides_a_viable role: supports claim: Command-based semantic interruption (intent token classification) provides a viable alternative to VAD-based full-duplex detection, with the advantage of robustness to noise and unrelated background sounds. source: §3.4 evidence: Command-based semantic interruption (intent token classification) provides a viable alternative to VAD-based full-duplex detection, with the advantage of robustness to noise and unrelated background sounds. confidence: high relevance: high limitations: - 'Evaluation coverage is incomplete: no naturalness MOS, SMOS, or intelligibility metrics for speech output are reported in this version, and vision benchmark results are explicitly deferred. Claims about speech quality and vision understanding capability cannot be independently verified from this paper alone.' - The interruption mechanism is demonstrated on a single synthesised phrase ("Stop Omni") with a narrow distribution of noise conditions. Whether the approach generalises to arbitrary semantic interrupt commands or real conversational interruption patterns is an open question. The model is trained and evaluated exclusively on English data despite using multilingual Whisper; cross-lingual transfer to speech output is not evaluated. The 0.5B model scale is deliberately small, and the authors note that scaling data and compute would likely yield substantial capability gains, but this is not demonstrated. Synthetic data is used for spoken question-answering and interruption training, and the effect of synthetic-to-real domain mismatch on deployment robustness is not assessed. caveats: [] - id: '2410.17799' published_date: "2024-10-23" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: full_duplex_spoken_dialogue_can_be_achieved_by_flattening_interleaved role: supports claim: Full-duplex spoken dialogue can be achieved by flattening interleaved speech and text token streams into a single autoregressive sequence, without modifying the backbone LLM architecture. source: §3, §3.3.2 evidence: Full-duplex spoken dialogue can be achieved by flattening interleaved speech and text token streams into a single autoregressive sequence, without modifying the backbone LLM architecture. confidence: high relevance: high - claim_id: progressive_curriculum_training_modality_alignment_followed_by_half_duplex_then role: supports claim: Progressive curriculum training (modality alignment followed by half-duplex, then full-duplex) improves final full-duplex dialogue quality compared to training directly on full-duplex data. source: §4.3, Table 3 evidence: Progressive curriculum training (modality alignment followed by half-duplex, then full-duplex) improves final full-duplex dialogue quality compared to training directly on full-duplex data. confidence: high relevance: high - claim_id: eliminating_intermediate_text_output_from_dialogue_models_substantially_reduces_response role: complicates claim: Eliminating intermediate text output from dialogue models substantially reduces response latency but causes a significant drop in semantic coherence, indicating a fundamental trade-off between speed and content quality in speech-to-speech generation. source: §3.3.2, §4.3, Table 3 evidence: Eliminating intermediate text output from dialogue models substantially reduces response latency but causes a significant drop in semantic coherence, indicating a fundamental trade-off between speed and content quality in speech-to-speech generation. confidence: high relevance: high - claim_id: turn_taking_response_latency_in_full_duplex_speech_models_can role: supports claim: Turn-taking response latency in full-duplex speech models can be reduced by chunked interleaved sequence training, with practical response times under 200 ms achievable at 0.5B parameter scale. source: §4.3, Table 4 evidence: Turn-taking response latency in full-duplex speech models can be reduced by chunked interleaved sequence training, with practical response times under 200 ms achievable at 0.5B parameter scale. confidence: high relevance: high limitations: - The 0.5B backbone is substantially smaller than comparators (LLaMA-Omni 8B, GLM-Voice 9B, Moshi 7B), making LLM-score comparisons in Table 3 not directly attributable to the method alone. The paper acknowledges GLM-Voice results may reflect test-set leakage. Dialogue quality scores remain well below the ground-truth ceiling. - Training data is entirely synthesised from text dialogues via a TTS pipeline; real conversational dynamics (natural prosody, disfluencies, real interruption patterns) are not represented. The model does not handle backchannels from either speaker, a basic feature of natural human conversation. User turn-taking accuracy at 25 tokens remains below 55% for both models evaluated, leaving interruption handling far from reliable. The paper does not report naturalness MOS, making direct quality comparison to TTS-oriented systems difficult. All evaluation uses simulated test data matching the training distribution, raising questions about real-world robustness. caveats: [] - id: '2411.00774' published_date: "2024-11-01" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant current_role: influential method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: freezing_the_llm_backbone_during_speech_modality_alignment_reduces_the role: supports claim: Freezing the LLM backbone during speech-modality alignment reduces the intelligence gap between spoken and text question-answering performance compared to fine-tuned approaches. source: §3.4, Table 3 evidence: Freezing the LLM backbone during speech-modality alignment reduces the intelligence gap between spoken and text question-answering performance compared to fine-tuned approaches. confidence: high relevance: medium - claim_id: a_three_stage_training_curriculum_using_large_asr_corpora_for role: supports claim: A three-stage training curriculum, using large ASR corpora for encoder pretraining followed by small-scale multi-modal Q&A fine-tuning, is sufficient to achieve competitive spoken dialogue quality without updating backbone LLM parameters. source: §2.2.2, §2.3.2 evidence: A three-stage training curriculum, using large ASR corpora for encoder pretraining followed by small-scale multi-modal Q&A fine-tuning, is sufficient to achieve competitive spoken dialogue quality without updating backbone LLM parameters. confidence: high relevance: low - claim_id: chunk_level_state_classification_integrated_into_the_llm_s_prefill role: supports claim: Chunk-level state classification integrated into the LLM's prefill stage enables duplex interruption detection without requiring a separate monitoring model or additional LLM context. source: §2.4 evidence: Chunk-level state classification integrated into the LLM's prefill stage enables duplex interruption detection without requiring a separate monitoring model or additional LLM context. confidence: high relevance: high - claim_id: decoupling_encoder_and_llm_kv_cache_per_user_session_allows role: supports claim: Decoupling encoder and LLM KV-cache per user session allows a server-side pool of model replicas to handle concurrent users with chunk-granular scheduling. source: §2.4 evidence: Decoupling encoder and LLM KV-cache per user session allows a server-side pool of model replicas to handle concurrent users with chunk-granular scheduling. confidence: high relevance: medium limitations: - The spoken Q&A benchmarks used for intelligence comparison (Web Questions, LlaMA Questions, Trivia QA) were synthesised from text using edge-tts rather than collected from real speakers. Results on naturally spoken or noisy input are not reported, limiting generalisability claims about real-world speech understanding. - Speech output quality is evaluated primarily through CER on 1000 utterances using a single speaker, not through subjective MOS ratings or speaker naturalness benchmarks. It is therefore difficult to assess voice quality relative to other systems. The system supports a limited number of output speakers and does not yet support style or voice instruct-following, which the authors flag as future work. Emotion understanding and audio captioning are also deferred to a planned encoder upgrade. The duplex state classifier operates at chunk boundaries (approximately 160-320 ms non-statistical latency), which may be perceptible in fast-paced dialogue. caveats: [] - id: '2411.13577' published_date: "2024-11-15" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - transformer-enc-dec - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - evaluation_caution - infrastructure current_role: influential method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: the_cascaded_and_end_to_end_paradigms_for_spoken_dialogue role: supports claim: The cascaded and end-to-end paradigms for spoken dialogue impose fundamentally different trade-offs between latency, paralinguistic fidelity, and intelligibility, and the appropriate choice depends on the target interaction scenario. source: §2.2, §2.3 evidence: The cascaded and end-to-end paradigms for spoken dialogue impose fundamentally different trade-offs between latency, paralinguistic fidelity, and intelligibility, and the appropriate choice depends on the target interaction scenario. confidence: high relevance: high - claim_id: semantic_speech_representations_offer_higher_compression_rates_and_better_llm role: complicates claim: Semantic speech representations offer higher compression rates and better LLM compatibility than acoustic representations, but sacrifice expressiveness, timbre, and style fidelity, necessitating additional vocoders in pipeline-based generation. source: §3.3.1, Table 1 evidence: Semantic speech representations offer higher compression rates and better LLM compatibility than acoustic representations, but sacrifice expressiveness, timbre, and style fidelity, necessitating additional vocoders in pipeline-based generation. confidence: high relevance: medium - claim_id: achieving_genuine_full_duplex_spoken_dialogue_simultaneous_listening_and_speaking role: supports claim: Achieving genuine full-duplex spoken dialogue (simultaneous listening and speaking with interrupt handling) requires architecturally causal models throughout the full pipeline, a constraint that current systems largely satisfy only on the output side. source: §5.1, §5.2 evidence: Achieving genuine full-duplex spoken dialogue (simultaneous listening and speaking with interrupt handling) requires architecturally causal models throughout the full pipeline, a constraint that current systems largely satisfy only on the output side. confidence: high relevance: high - claim_id: speech_text_modality_alignment_in_current_spoken_dialogue_systems_relies role: supports claim: Speech-text modality alignment in current spoken dialogue systems relies heavily on paired data, introducing catastrophic forgetting risk and creating a structural dependency on the availability of labelled speech corpora. source: §4.4.1 evidence: Speech-text modality alignment in current spoken dialogue systems relies heavily on paired data, introducing catastrophic forgetting risk and creating a structural dependency on the availability of labelled speech corpora. confidence: high relevance: low - claim_id: evaluation_infrastructure_for_spoken_dialogue_lags_substantially_behind_system_capabilities role: supports claim: 'Evaluation infrastructure for spoken dialogue lags substantially behind system capabilities: interaction, streaming latency, and audio generation are either absent from or severely underrepresented in existing benchmarks.' source: §6.2, §6.3, Table 3 evidence: 'Evaluation infrastructure for spoken dialogue lags substantially behind system capabilities: interaction, streaming latency, and audio generation are either absent from or severely underrepresented in existing benchmarks.' confidence: high relevance: high limitations: - The survey was produced concurrently with the primary wave of open-source spoken dialogue models it covers (late 2024), meaning that some systems are described in early form and the field will have evolved by the time readers encounter the paper. The coverage of music and sound understanding and generation within dialogue systems is acknowledged as thin (the authors defer to an appendix), and security evaluation for spoken dialogue receives less treatment than its importance warrants. A number of prominent systems (Westlake-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) lack published papers and are excluded from the timeline figure, which may leave gaps for practitioners interested in the deployed-systems landscape. The SuperCLUE benchmark, one of the more comprehensive interaction evaluations listed, is not open-source and focuses on Mandarin, limiting its utility for the broader research community. - 'Open questions surfaced include: whether speech tokenisers can be designed to enforce text-space alignment during encoding, eliminating the need for large paired corpora; what granularity of temporal alignment priors (sentence, word, phoneme level) is optimal for spoken dialogue training; and how preference optimisation techniques can be adapted for the joint text-speech output space.' caveats: [] - id: '2411.18803' published_date: "2024-11-27" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - architecture_variant current_role: influential method_family: - gan_causal_streaming_vocoders claims: - claim_id: transformer_architectures_can_match_convolutional_neural_audio_codecs_in_streaming role: supports claim: Transformer architectures can match convolutional neural audio codecs in streaming reconstruction quality while requiring substantially lower multiply-accumulate operations at similar parameter counts. source: §3.2, §5.1, Table 3 evidence: Transformer architectures can match convolutional neural audio codecs in streaming reconstruction quality while requiring substantially lower multiply-accumulate operations at similar parameter counts. confidence: high relevance: high - claim_id: a_single_codebook_design_is_compatible_with_full_streaming_operation role: supports claim: A single-codebook design is compatible with full streaming operation, showing that the prior tradeoff between single-codebook simplicity and streaming capability is not fundamental. source: §2.3, §3.1 evidence: A single-codebook design is compatible with full streaming operation, showing that the prior tradeoff between single-codebook simplicity and streaming capability is not fundamental. confidence: high relevance: high - claim_id: semantic_distillation_as_used_in_mimi_and_speechtokenizer_provides_consistent role: supports claim: Semantic distillation (as used in Mimi and SpeechTokenizer) provides consistent word error rate benefits over codecs trained without it, even when those codecs achieve higher perceptual quality scores. source: §5.1, §5.2, Table 3, Table 4 evidence: Semantic distillation (as used in Mimi and SpeechTokenizer) provides consistent word error rate benefits over codecs trained without it, even when those codecs achieve higher perceptual quality scores. confidence: high relevance: medium - claim_id: at_equivalent_computational_budgets_transformer_based_codec_architectures_outperform_their role: supports claim: At equivalent computational budgets, transformer-based codec architectures outperform their causal convolutional counterparts across intelligibility, distortion, and naturalness metrics. source: §5.1, Figure 2, Figure 3 evidence: At equivalent computational budgets, transformer-based codec architectures outperform their causal convolutional counterparts across intelligibility, distortion, and naturalness metrics. confidence: high relevance: high limitations: - All naturalness evaluations rely on UTMOS rather than human MOS. While UTMOS correlates well with human judgements on codec-reconstructed speech, the paper presents no subjective listening test to confirm its quality claims. - The codec is evaluated only on English speech (LibriSpeech). Generalisation to other languages, accents, and non-speech audio is untested. The training set (Libri-light) is entirely read speech; performance on conversational, emotional, or noisy speech is unknown. Code and checkpoints are not released (as of the preprint), limiting reproducibility. The paper does not evaluate latency (time-to-first-byte or algorithmic delay) in a real streaming deployment, only computational complexity in offline MACs. The effect of the large codebook sizes (65K, 131K) on downstream speech language model training and inference has not been demonstrated. caveats: [] - id: '2412.02612' published_date: "2024-12-03" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders claims: - claim_id: speech_text_interleaved_pre_training_at_trillion_token_scale_enables role: supports claim: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. source: §4.1, Table 4 evidence: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. confidence: high relevance: medium - claim_id: single_codebook_supervised_speech_tokenizers_derived_from_asr_models_can role: supports claim: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. source: §3.1, Table 1 evidence: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. confidence: high relevance: medium - claim_id: streaming_interleaved_generation_templates_alternating_text_and_speech_token_output role: supports claim: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. source: §3.3 evidence: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. confidence: high relevance: high - claim_id: end_to_end_speech_language_models_that_include_dedicated_speech role: supports claim: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. source: §5.2, Table 6 evidence: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. confidence: high relevance: medium - claim_id: decoupling_the_text_and_speech_output_subtasks_during_fine_tuning role: supports claim: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. source: §4.2.2 evidence: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. confidence: high relevance: medium limitations: - The paper reports no subjective listening test (MOS/MUSHRA) on the chat model output; UTMOS is used as a proxy for speech naturalness, and the chat evaluation relies on GPT-4o scoring of ASR transcriptions, introducing cascaded error from both the vocoder quality and the Whisper transcription step. - 'The 175bps tokenizer trades acoustic fidelity for compactness: VisQOL at 12.5Hz (2.52) is lower than SpeechTokenizer variants and the 50Hz variant of the same system. This may limit voice cloning quality and the fidelity of paralinguistic feature reproduction (accent, fine-grained emotion), though the paper does not directly evaluate these.' - Instruction-following for speech style (emotion, dialect, rate) is described and demonstrated qualitatively but not evaluated quantitatively; it is unclear how reliably the model follows complex or combined style instructions. - 'The streaming thoughts ratio (13 text : 26 speech tokens) and block size (b=0.8s) are empirically chosen hyperparameters; their sensitivity and generalisability to other tokenizer frame rates or LLM sizes is not studied.' caveats: [] - id: '2412.10117' published_date: "2024-12-13" entry_date: '2026-07-30' year: 2024 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - hybrid_modular_streaming_systems claims: - claim_id: finite_scalar_quantization_achieves_full_codebook_utilization_in_supervised_speech role: supports claim: Finite scalar quantization achieves full codebook utilization in supervised speech tokenizers, capturing substantially more semantic content than vector quantization at equivalent bitrates. source: §2.2, §4.1, Table 4 evidence: Finite scalar quantization achieves full codebook utilization in supervised speech tokenizers, capturing substantially more semantic content than vector quantization at equivalent bitrates. confidence: high relevance: medium - claim_id: replacing_a_randomly_initialized_custom_lm_with_a_pre_trained role: supports claim: Replacing a randomly initialized custom LM with a pre-trained LLM backbone improves content consistency in hybrid TTS systems without requiring a separate text encoder. source: §2.3, §4.3, Table 7 evidence: Replacing a randomly initialized custom LM with a pre-trained LLM backbone improves content consistency in hybrid TTS systems without requiring a separate text encoder. confidence: high relevance: low - claim_id: streaming_and_non_streaming_synthesis_can_be_unified_in_a role: supports claim: Streaming and non-streaming synthesis can be unified in a single autoregressive model through interleaved text-speech token sequences, with virtually lossless quality on typical inputs relative to offline mode. source: §2.3, §4.2, Table 8 evidence: Streaming and non-streaming synthesis can be unified in a single autoregressive model through interleaved text-speech token sequences, with virtually lossless quality on typical inputs relative to offline mode. confidence: high relevance: high - claim_id: training_a_flow_matching_model_simultaneously_on_multiple_causal_mask role: complicates claim: Training a flow matching model simultaneously on multiple causal mask types — from non-causal to full-causal — enables a single model to span the latency-quality trade-off continuum at inference time, with masks providing implicit self-distillation. source: §2.4, §4.3, Table 8 evidence: Training a flow matching model simultaneously on multiple causal mask types — from non-causal to full-causal — enables a single model to span the latency-quality trade-off continuum at inference time, with masks providing implicit self-distillation. confidence: high relevance: high - claim_id: differentiable_asr_reward_optimization_generalizes_better_to_out_of_domain role: supports claim: Differentiable ASR reward optimization generalizes better to out-of-domain and hard-case inputs than preference-based DPO in TTS speaker fine-tuning. source: §2.8, §4.7, Table 11 evidence: Differentiable ASR reward optimization generalizes better to out-of-domain and hard-case inputs than preference-based DPO in TTS speaker fine-tuning. confidence: high relevance: low limitations: - '- EN quality still lags CosyVoice 2 behind Seed-TTS and F5-TTS on SEED test-en (WER 2.57% vs. 2.25% and 1.83%), reflecting data imbalance toward Chinese. - Japanese synthesis degrades due to character set overlap with Chinese (CER 18.79% test-ja vs. 7.98% test-ko). - Cannot control timbre through text instructions. - Singing not supported. - Streaming still incurs a hard degradation on test-hard, suggesting that contextual information from future text is important for difficult patterns.' caveats: [] - id: '2502.11946' published_date: "2025-02-17" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: influential method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - hybrid_modular_streaming_systems claims: - claim_id: a_dual_codebook_interleaved_tokenizer_that_combines_linguistic_and_semantic role: supports claim: A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality. source: §4.4, §6.2.1 evidence: A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality. confidence: high relevance: medium - claim_id: scaling_autoregressive_llm_backbone_size_from_3b_to_130b_parameters role: supports claim: Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive. source: §6.2.2, Table 3 evidence: Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive. confidence: high relevance: low - claim_id: rlhf_reward_models_trained_on_speech_interaction_data_can_exhibit role: supports claim: RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training. source: §5.2.6 evidence: RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training. confidence: high relevance: medium - claim_id: speculative_response_generation_triggered_by_voice_activity_detection_can_reduce role: supports claim: Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems. source: §3.4 evidence: Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems. confidence: high relevance: high - claim_id: synthetic_speech_data_generated_by_a_large_multi_modal_model role: supports claim: Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles. source: §5.1.1 evidence: Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles. confidence: high relevance: low limitations: - The StepEval-Audio-360 benchmark is proprietary and created by the same team; human evaluation results on it cannot be independently reproduced. Open-source benchmark comparisons mix locally re-run models with results taken from original publications, complicating direct numerical comparison. - 'Speaker similarity scores for the distilled Step-Audio-TTS-3B are noticeably lower than CosyVoice 2 on both Chinese and English SEED-TTS tests, suggesting that the dual-codebook approach trades some acoustic identity preservation for intelligibility gains. The AQTA+TTS design still relies on a cascade: errors in ASR transcription of history or in text generation propagate to the TTS stage. The paper''s future work section acknowledges that purely end-to-end audio-in/audio-out (AQAA) remains unsolved. Evaluation for singing, RAP, and dialect control is limited to instruction following scores without reference audio; absolute quality in these dimensions is difficult to assess from the reported numbers alone.' caveats: [] - id: iclr-2025-tQ1PmLfPBL published_date: "2025-04-24" entry_date: '2026-07-30' year: 2025 venue: ICLR task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: influential method_family: - flow_matching_streaming_decoders claims: - claim_id: flow_matching_enables_higher_quality_waveform_generation_than_diffusion_with role: supports claim: Flow matching enables higher-quality waveform generation than diffusion with fewer inference steps. source: §4.6, Table 8 evidence: PeriodWave with CFM and 6 steps achieves UTMOS 3.628 versus PeriodWave with DDPM at 50 steps achieving UTMOS 3.377 on LibriTTS; the CFM model also reaches comparable quality to 50-step DDPM at only 16 steps. confidence: high relevance: low - claim_id: explicit_multi_period_decomposition_in_the_generator_architecture_improves_pitch role: supports claim: Explicit multi-period decomposition in the generator architecture improves pitch accuracy and periodicity over GAN and diffusion vocoders. source: §4.2, §4.6, Table 1, Table 7 evidence: PeriodWave achieves pitch error of 15.04 cents and periodicity 0.0744 on LJSpeech, substantially below BigVGAN (19.02 cents, 0.0782) and all diffusion baselines; ablation shows monotonic improvement as more distinct prime-number periods are added. confidence: high relevance: low - claim_id: single_step_gan_vocoders_achieve_significantly_faster_inference_than_iterative role: complicates claim: Single-step GAN vocoders achieve significantly faster inference than iterative flow-matching vocoders. source: §E, Table 15, Table 16 evidence: PeriodWave at 16 steps runs at 7.48× real-time; HiFi-GAN runs at 166.70× real-time; even PeriodWave at 2 steps (56.36×) is slower than HiFi-GAN, though it already outperforms HiFi-GAN on all quality metrics. confidence: high relevance: low - claim_id: iterative_waveform_generation_reduces_train_inference_mismatch_artefacts_in_two role: supports claim: Iterative waveform generation reduces train-inference mismatch artefacts in two-stage TTS relative to one-step GAN vocoders. source: §4.7, Table 9, §G evidence: In zero-shot TTS with ARDiT-TTS acoustic features, PeriodWave+FreeU achieves MOS 4.07 versus BigVGAN's 4.03 and BigVSAN's 3.99; the iterative refinement allows the vocoder to correct imperfections in generated Mel-spectrograms rather than propagating them. confidence: high relevance: low - claim_id: flow_matching_vocoders_can_decode_neural_codec_tokens_with_streaming role: supports claim: Flow-matching vocoders can decode neural codec tokens with streaming generation and minimal quality degradation. source: §5, Table 10, Table 11 evidence: PeriodWave trained for parallel generation from Mimi (Q=8) tokens achieves CER 2.5% versus Mimi decoder's 3.07%; streaming with single-token delay and 2-step sampling maintains comparable quality (CER 2.45%, UTMOS 3.85 versus parallel 3.93). confidence: high relevance: high limitations: - 'Synthesis speed is the principal limitation: at 16 steps, PeriodWave runs at 7.48× real-time, substantially slower than one-step GAN vocoders (HiFi-GAN: 166×, BigVGAN: 38×). For latency-sensitive applications, the 2-step variant (56×) is practical but still 3× slower than HiFi-GAN.' - High-frequency reproduction remains challenging even with multi-band modeling and FreeU. The single-loss (CFM objective only) training means the model lacks the spectral feedback that GAN discriminators provide, and M-STFT metrics are generally worse than GAN baselines despite better perceptual scores. The authors plan to incorporate short-time Fourier convolution blocks or modified spectral objectives to address this. - The codec streaming mode uses a non-causal architecture with a one-token look-ahead, which introduces a small latency penalty. In-context streaming generation for longer sequences is identified as future work. Evaluation is confined to English speech and music; generalisation to other languages, accents, and audio domains remains untested in this paper. caveats: [] - id: 2025.naacl-srw.6 published_date: "2025-04-29" entry_date: '2026-07-30' year: 2025 venue: NAACL task: - TTS - codec architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: non_overlapping_encoder_receptive_fields_in_rvq_codecs_improve_downstream role: supports claim: Non-overlapping encoder receptive fields in RVQ codecs improve downstream language model likelihood and end-to-end TTS metrics relative to the standard causal overlapping setup. source: §3.1, Table 1 evidence: Replacing the overlapping causal encoder with a framewise encoder on a DAC-based codec reduces LM NLL by more than 8% and improves WER, NISQA, and speaker similarity on LibriTTS-R test-clean, despite slightly worsening Mel-L1 reconstruction. confidence: high relevance: high - claim_id: better_codec_audio_reconstruction_quality_does_not_reliably_predict_better role: complicates claim: Better codec audio reconstruction quality does not reliably predict better end-to-end speech generation quality in codec-LM systems. source: §4, Table 1 evidence: The framewise encoder achieves higher NLL and better TTS metrics than the causal baseline while scoring slightly worse on Mel-spectral L1 reconstruction distance, demonstrating that reconstruction-optimised codecs can be suboptimal for downstream LM training. confidence: high relevance: low - claim_id: increasing_rvq_codec_frame_duration_can_substantially_reduce_codec_lm role: supports claim: Increasing RVQ codec frame duration can substantially reduce codec-LM inference latency with little or no degradation in TTS intelligibility and speaker similarity, provided the bitrate is held approximately constant by adjusting codebook depth. source: §3.3, §4, Table 2 evidence: Doubling frame duration from 11ms to 22ms yields a 1.94x inference speedup with WER 4.21%, NISQA 4.42, and speaker similarity 81.0%, matching or improving on the 11ms framewise baseline. Quadrupling to 44ms further accelerates inference (3.2-3.8x) but substantially degrades WER and speaker similarity. confidence: high relevance: high - claim_id: a_single_lm_trained_with_codebook_level_dropout_can_efficiently role: supports claim: A single LM trained with codebook level dropout can efficiently approximate the performance profile of training one LM per candidate RVQ level count. source: §3.2, §4, Figure 2 evidence: Training with 90%-full CL drop on a 12-level codec produces per-level performance curves for WER, NISQA, and speaker similarity that closely track those of 12 independently trained LMs across all Q' values 1-12. confidence: high relevance: medium - claim_id: the_optimal_number_of_rvq_codebook_levels_for_end_to role: refines claim: The optimal number of RVQ codebook levels for end-to-end codec-LM TTS differs across evaluation dimensions, and more levels are not universally better for end-to-end performance even when they monotonically improve codec reconstruction. source: Appendix B, Figure 3 evidence: End-to-end FAD reaches a global minimum at 9 levels before degrading, while WER reaches its best at 3-4 levels and NISQA and speaker similarity peak at approximately 9 levels, in contrast to codec Mel-L1 which improves monotonically. confidence: high relevance: low limitations: - The evaluation is conducted on a single English TTS corpus (LibriTTS-R test-clean) using automatic metrics only (WER via Whisper, NISQA, cosine speaker similarity). No human listening test is reported, so perceptual quality gains are estimated rather than directly validated. - The codec is trained on proprietary in-house podcast data (1.7K hours), which limits reproducibility for the codec training stage specifically. The LM training does use the public LibriTTS-R dataset. The codebook size hyperparameter (|V|) remains outside the scope of CL drop, requiring separate trial-and-error search. The paper does not investigate multilingual or noisy speech settings. The optimal frame duration finding (22ms being the sweet spot) is specific to this codec architecture and training data and may not generalise to other codec families. caveats: [] - id: '2505.02625' published_date: "2025-05-05" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - hybrid_modular_streaming_systems claims: - claim_id: autoregressive_streaming_tts_decoders_in_modular_spoken_conversational_agents_produce role: supports claim: Autoregressive streaming TTS decoders in modular spoken conversational agents produce higher-quality speech and smaller S2T-to-S2S accuracy gaps than non-autoregressive streaming decoders, at a modest latency cost. source: §5.1, Table 1 evidence: Autoregressive streaming TTS decoders in modular spoken conversational agents produce higher-quality speech and smaller S2T-to-S2S accuracy gaps than non-autoregressive streaming decoders, at a modest latency cost. confidence: high relevance: high - claim_id: modular_spoken_conversational_agents_trained_on_tens_of_thousands_of role: supports claim: Modular spoken conversational agents trained on tens of thousands of hours of synthesized speech-to-speech dialogue can match the performance of native SpeechLMs trained on millions of hours of unsupervised speech data. source: §5.1, Table 1 evidence: Modular spoken conversational agents trained on tens of thousands of hours of synthesized speech-to-speech dialogue can match the performance of native SpeechLMs trained on millions of hours of unsupervised speech data. confidence: high relevance: low - claim_id: jointly_conditioning_a_tts_language_model_on_llm_hidden_states role: supports claim: Jointly conditioning a TTS language model on LLM hidden states and text token embeddings via a learned gate fusion improves both semantic consistency (WER) and instruction-following quality over hidden-state-only conditioning. source: §5.2, Table 2 evidence: Jointly conditioning a TTS language model on LLM hidden states and text token embeddings via a learned gate fusion improves both semantic consistency (WER) and instruction-following quality over hidden-state-only conditioning. confidence: high relevance: low - claim_id: tts_language_model_pretraining_on_text_speech_pairs_is_a role: supports claim: TTS language model pretraining on text-speech pairs is a critical prerequisite for stable convergence in modular SpeechLMs; initializing from a language model alone is insufficient. source: §5.2, Table 3 evidence: TTS language model pretraining on text-speech pairs is a critical prerequisite for stable convergence in modular SpeechLMs; initializing from a language model alone is insufficient. confidence: high relevance: low - claim_id: in_streaming_speech_generation_the_write_chunk_size_w_primarily role: supports claim: In streaming speech generation, the write chunk size (W) primarily determines speech naturalness while the read chunk size (R) primarily determines text-speech alignment, with latency jointly determined by both. source: §5.2, Table 4 evidence: In streaming speech generation, the write chunk size (W) primarily determines speech naturalness while the read chunk size (R) primarily determines text-speech alignment, with latency jointly determined by both. confidence: high relevance: high limitations: - All evaluations are conducted in English only. The model is trained on a single fixed output voice, and no multilingual or voice-diversity experiments are reported. Generalization to other languages or to emotionally expressive speech is untested. - The model cannot modulate speech style (emotion, speaking rate, dialect) in response to paralinguistic cues in the input, because training data contains only conventional speech-to-speech dialogue. The authors note this as planned future work. The benchmarks used (SpokenQA accuracy, ChatGPT score) are narrow and do not cover naturalness in unconstrained conversational settings or robustness to noisy or accented input. Comparisons to Minmo (a concurrent work using 1.4M hours) are mentioned in related work but not included in the main experimental table, leaving the data efficiency claim partially unverified against the most directly comparable system. caveats: [] - id: '2507.12197' published_date: "2025-07-16" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - singing architecture: - autoregressive-LM - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_streaming_generation - gan_causal_streaming_vocoders claims: - claim_id: scaling_the_number_of_rvq_codebooks_in_a_discrete_speech role: supports claim: Scaling the number of RVQ codebooks in a discrete speech codec reduces information loss and improves reconstruction quality for expressive and challenging vocal content. source: §3.2, Table 5 evidence: QDAC reconstruction improves monotonically from 1 to 16 codebooks across PESQ, STOI, and SI-SDR; 16-codebook QDAC at 50Hz achieves PESQ 3.83 versus PESQ 2.98 for 8 codebooks, with similar gains on Mel distance and speaker similarity. confidence: high relevance: low - claim_id: higher_multi_codebook_reconstruction_fidelity_in_a_codec_does_not role: complicates claim: Higher multi-codebook reconstruction fidelity in a codec does not necessarily translate into better speaker identity preservation in downstream zero-shot TTS. source: §3.2, Table 6 evidence: QTTS achieves Spk Sim 0.75 on SeedTTS-Easy compared to 0.82 and 0.81 for single-codebook CosyVoice v1 and v2, despite QDAC's superior reconstruction metrics in Table 5. confidence: high relevance: low - claim_id: semantic_disentanglement_in_audio_codecs_can_be_achieved_by_backpropagating role: supports claim: Semantic disentanglement in audio codecs can be achieved by backpropagating ASR loss exclusively through the first RVQ codebook, enforcing content isolation without relying on general-purpose self-supervised representations. source: §2.1.2, Table 5 evidence: QDAC trains an AR-ASR module conditioned only on first-codebook tokens; the resulting WER at reconstruction (6.42 for 8cb/25Hz) is close to ground truth (6.01), indicating the first codebook encodes phoneme-level content while residual codebooks capture acoustic detail. confidence: high relevance: medium - claim_id: multi_codebook_autoregressive_tts_admits_a_principled_speed_quality_trade role: refines claim: Multi-codebook autoregressive TTS admits a principled speed-quality trade-off by choosing between strict hierarchical inter-codebook conditioning and a delayed multi-head parallel prediction scheme. source: §2.2, §2.3, Tables 1, 3, 4 evidence: Hierarchy Parallel (200M, dual-AR) and Multihead Delay (120M, parallel with fixed delay) achieve comparable TTFT at 512 tokens (26ms vs 24ms) but differ substantially in decode throughput; the Multihead variant reaches over 196K codebook tokens/s at short output lengths versus 105K for Hierarchy. confidence: high relevance: low - claim_id: mos_evaluations_in_zero_shot_tts_can_yield_above_reference role: complicates claim: MOS evaluations in zero-shot TTS can yield above-reference scores for synthesised speech, undermining direct absolute comparisons across studies. source: §3.2, Table 6 evidence: Ground truth speech achieves MOS 2.7 while QTTS, CosyVoice, and CosyVoice2 all score between 3.01 and 3.03 on the same test set, producing a ranking inconsistent with naturalness expectations. confidence: high relevance: low limitations: - Training data is not disclosed, the PGC-hard benchmark is proprietary, and no code or demo is available, making results difficult to reproduce or build upon. - The evaluation compares only against the CosyVoice v1/v2 family, leaving open how QTTS performs relative to flow-matching systems, other multi-codebook approaches, or stronger autoregressive baselines. Speaker similarity is lower than both single-codebook baselines (0.75 vs 0.81-0.82), suggesting the multi-codebook generation pipeline needs further work to fully leverage the improved codec for speaker transfer. The paper positions singing and music synthesis as motivating use cases but does not evaluate on these tasks. Only the 8-codebook QTTS model is evaluated for TTS synthesis, leaving open whether 16-codebook generation would further improve or introduce new training challenges. caveats: [] - id: 2025.acl-demo.37 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - VC architecture: - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: articulatory_feature_spaces_enable_interpretable_content_speaker_disentanglement_in_voice role: supports claim: Articulatory feature spaces enable interpretable content-speaker disentanglement in voice conversion without sacrificing intelligibility relative to SSL-based approaches. source: §5.3, Table 1 evidence: Articulatory feature spaces enable interpretable content-speaker disentanglement in voice conversion without sacrificing intelligibility relative to SSL-based approaches. confidence: high relevance: medium - claim_id: real_time_zero_shot_voice_conversion_on_cpu_hardware_is role: supports claim: Real-time zero-shot voice conversion on CPU hardware is achievable below 70 ms end-to-end latency while maintaining naturalness MOS above 3.8. source: §3.6, Table 1 evidence: Real-time zero-shot voice conversion on CPU hardware is achievable below 70 ms end-to-end latency while maintaining naturalness MOS above 3.8. confidence: high relevance: high - claim_id: causal_ddsp_vocoders_conditioned_on_articulatory_features_provide_competitive_synthesis role: supports claim: Causal DDSP vocoders conditioned on articulatory features provide competitive synthesis quality compared to GAN-based alternatives at substantially lower computational cost. source: §2.3, §3.5 evidence: Causal DDSP vocoders conditioned on articulatory features provide competitive synthesis quality compared to GAN-based alternatives at substantially lower computational cost. confidence: high relevance: high - claim_id: voice_conversion_systems_trained_with_static_noise_augmentation_degrade_gracefully role: complicates claim: Voice conversion systems trained with static noise augmentation degrade gracefully down to approximately 20 dB SNR input but fail at 10 dB, suggesting a practical noise floor for real-time deployment. source: §5.4 evidence: Voice conversion systems trained with static noise augmentation degrade gracefully down to approximately 20 dB SNR input but fail at 10 dB, suggesting a practical noise floor for real-time deployment. confidence: high relevance: high limitations: - '- EMA representation omits nasal cavity and laryngeal dynamics, limiting modeling of nasal sounds and vocal fry. - Pseudo-EMA labels come from a WavLM model pretrained on English; cross-lingual performance is limited. - Sensitivity to input quality below 20 dB SNR, especially for white noise. - Model weights will not be open-sourced due to misuse concerns. - Future work: prompt-free conversion by offline target speaker design (gender, age, emotion, accent).' caveats: [] - id: 2025.acl-long.817 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: boundary_aware_speech_representations_are_critical_for_enabling_offline_trained role: supports claim: Boundary-aware speech representations are critical for enabling offline-trained speech LLMs to perform simultaneous inference via wait-k strategies. source: §5.1, §5.2, Tables 4, 11 evidence: CIF-based boundary-aware prompts outperform fixed downsampling by approximately 4 ASR-BLEU points at equivalent latency on Es-En, Fr-En, and De-En CVSS-C test sets, with a parallel ~4 BLEU gap on text output confirming the bottleneck is LLM prediction rather than speech synthesis. confidence: high relevance: high - claim_id: offline_training_combined_with_test_time_simultaneous_inference_policies_can role: supports claim: Offline training combined with test-time simultaneous inference policies can match or outperform systems trained specifically for streaming in speech-to-speech translation. source: §5.1, Fig. 4, Table 1 evidence: SimulS2S-LLM, trained offline, consistently outperforms the streaming-trained StreamSpeech model across all three language pairs at comparable latency, achieving up to 4 ASR-BLEU improvement on Es-En. confidence: high relevance: high - claim_id: aggregating_llm_hidden_states_across_multiple_layers_improves_discrete_speech role: supports claim: Aggregating LLM hidden states across multiple layers improves discrete speech token prediction compared to using only the final layer. source: §5.3, Fig. 6 evidence: Multi-layer hidden state weighting yields approximately 1 ASR-BLEU improvement over last-layer-only decoding on CVSS-C Es-En Simul-S2ST, attributed to the final layer's focus on semantic text information at the expense of acoustic richness needed for speech token generation. confidence: high relevance: medium - claim_id: llm_based_approaches_to_simultaneous_speech_generation_face_a_latency role: complicates claim: LLM-based approaches to simultaneous speech generation face a latency penalty from LLM inference overhead that narrows the practical quality-latency advantage over non-LLM methods. source: §D, Tables 8-10, Limitations evidence: Computation-aware ATD for SimulS2S-LLM is substantially higher than standard ATD (e.g., 4239ms vs. 3440ms at k=8 on Es-En), and the system is not evaluated at very low latency regimes (AL < 1s) where reordering requirements make offline-trained models unsuitable. confidence: high relevance: high - claim_id: shallow_fusion_of_n_gram_language_models_with_ctc_decoding role: supports claim: Shallow fusion of n-gram language models with CTC decoding of discrete speech tokens improves simultaneous speech translation quality without increasing latency. source: §5.4, Table 2 evidence: n-gram LM fusion over greedy CTC search improves ASR-BLEU from 24.7 to 26.3 on CVSS-C Es-En at identical ATD of 3439ms. confidence: high relevance: high limitations: - The system is not evaluated at very low latency (AL < 1s), a regime the authors identify as unsuitable for offline-trained models due to reordering requirements. This excludes SimulS2S-LLM from the most latency-critical applications. Computation-aware latency is substantially higher than the reported ATD, and all experiments use 7B/8B open-source LLMs on small datasets (70-174 hours per language pair), leaving scalability to larger models and data unverified. - The evaluation is limited to three European language pairs in a single translation direction each. Language pairs with greater structural divergence or more extensive reordering would stress the wait-k assumption more severely. The system has not been evaluated on offline inference or tasks other than S2ST and S2TT, despite the claim that offline training preserves such capabilities. Long-form simultaneous speech translation is also untested due to lack of suitable data. caveats: [] - id: 2025.acl-long.912 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - hybrid_modular_streaming_systems claims: - claim_id: autoregressive_streaming_speech_decoders_in_modular_spoken_conversational_agents_substantially role: supports claim: Autoregressive streaming speech decoders in modular spoken conversational agents substantially improve naturalness over non-autoregressive alternatives with comparable latency. source: §5.1, Table 1 evidence: Autoregressive streaming speech decoders in modular spoken conversational agents substantially improve naturalness over non-autoregressive alternatives with comparable latency. confidence: high relevance: high - claim_id: a_gate_fusion_mechanism_that_adaptively_blends_llm_hidden_states role: supports claim: A gate fusion mechanism that adaptively blends LLM hidden states with text token embeddings as input to the TTS language model improves both instruction following quality and text-speech consistency over simpler additive fusion. source: §5.2, Table 2 evidence: A gate fusion mechanism that adaptively blends LLM hidden states with text token embeddings as input to the TTS language model improves both instruction following quality and text-speech consistency over simpler additive fusion. confidence: high relevance: low - claim_id: streaming_tts_pretraining_on_speech_dialogue_data_is_critical_for role: complicates claim: 'Streaming TTS pretraining on speech dialogue data is critical for quality: initializing from a text-only pretrained model degrades performance substantially, and training from scratch fails to converge.' source: §5.2, Table 3 evidence: 'Streaming TTS pretraining on speech dialogue data is critical for quality: initializing from a text-only pretrained model degrades performance substantially, and training from scratch fails to converge.' confidence: high relevance: high - claim_id: multi_turn_dialogue_training_data_consistently_outperforms_single_turn_data role: supports claim: Multi-turn dialogue training data consistently outperforms single-turn data of the same total size for modular speech language models across spoken QA and instruction-following benchmarks. source: §5.3, Table 5 evidence: Multi-turn dialogue training data consistently outperforms single-turn data of the same total size for modular speech language models across spoken QA and instruction-following benchmarks. confidence: high relevance: low - claim_id: the_s2t_to_s2s_accuracy_gap_in_spoken_question_answering role: supports claim: The S2T-to-S2S accuracy gap in spoken question answering is a meaningful indicator of speech generation quality, and autoregressive TTS decoders reduce this gap relative to non-autoregressive alternatives. source: §5.1, Table 1 evidence: The S2T-to-S2S accuracy gap in spoken question answering is a meaningful indicator of speech generation quality, and autoregressive TTS decoders reduce this gap relative to non-autoregressive alternatives. confidence: high relevance: low limitations: - The model generates speech in a single fixed output style; it cannot modulate emotion, speaking rate, or dialect in response to the content or paralinguistic cues of the input speech. All evaluation is in English. The output voice is fixed during training (a single uniform voice for all responses), which limits expressiveness and speaker diversity. The system is inherently a response-after-input architecture and does not support full-duplex conversation. Latency, while adequate for real-time interaction, leaves room for further reduction via engineering optimization. Whether the gate fusion approach generalizes to multilingual settings or to more expressive speech styles is not explored. caveats: [] - id: 2025.findings-acl.1051 published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: ACL task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: decoupling_speech_synthesis_from_llm_text_generation_via_a_lightweight role: supports claim: Decoupling speech synthesis from LLM text generation via a lightweight autoregressive module can preserve the base LLM's reasoning quality while achieving competitive streaming latency. source: §6.4, Table 1 evidence: Decoupling speech synthesis from LLM text generation via a lightweight autoregressive module can preserve the base LLM's reasoning quality while achieving competitive streaming latency. confidence: high relevance: high - claim_id: end_to_end_speech_enabled_llms_that_fine_tune_or role: supports claim: End-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen. source: §6.4, Table 1 evidence: End-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen. confidence: high relevance: medium - claim_id: a_single_layer_rvq_codec_is_sufficient_for_low_latency role: supports claim: A single-layer RVQ codec is sufficient for low-latency autoregressive TTS generation when paired with a compact decoder transformer, avoiding the complexity of multi-codebook prediction. source: §3.1 evidence: A single-layer RVQ codec is sufficient for low-latency autoregressive TTS generation when paired with a compact decoder transformer, avoiding the complexity of multi-codebook prediction. confidence: high relevance: high - claim_id: streaming_tts_quality_improves_with_larger_decode_chunk_sizes_with role: supports claim: Streaming TTS quality improves with larger decode chunk sizes, with WER and UTMOS gains achievable without substantially increasing end-to-end latency. source: §6.4, Figure 6 evidence: Streaming TTS quality improves with larger decode chunk sizes, with WER and UTMOS gains achievable without substantially increasing end-to-end latency. confidence: high relevance: high - claim_id: language_adaptation_of_a_codec_based_tts_module_can_be role: supports claim: Language adaptation of a codec-based TTS module can be achieved by replacing training data alone, without architectural changes or explicit G2P conversion for the new language. source: §6.5, Table 3 evidence: Language adaptation of a codec-based TTS module can be achieved by replacing training data alone, without architectural changes or explicit G2P conversion for the new language. confidence: high relevance: low limitations: - LLMVoX is single-speaker — no voice cloning or speaker reference support. The Arabic model was trained on XTTS-synthesized data, so XTTS acts as an upper bound (CER 1.7% vs. LLMVoX 8.2%). The streaming pipeline does not yet extend to the ASR front-end. Latency with 70B LLMs exceeds 1.9s, making real-time use marginal. The quality improvement from larger chunk sizes (UTMOS 3.75→4.41) suggests that the 475ms latency figure is somewhat optimistic for maximum quality operation. caveats: [] - id: '2507.17527' published_date: "2025-07-27" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: end_to_end_speech_to_speech_translation_trained_with_reinforcement role: supports claim: End-to-end speech-to-speech translation trained with reinforcement learning can achieve substantially lower latency than cascaded commercial systems while surpassing them in translation quality. source: §3.3, Table 1 evidence: On RealSI (zh-en), the system achieves VIP 79.5 and SVIP 67.8 at AL 5.18 seconds, versus the best commercial competitor at VIP 53.2 and SVIP 3.0 at AL 48.21. confidence: high relevance: high - claim_id: two_stage_rl_training_that_separates_fine_grained_process_rewards role: supports claim: Two-stage RL training that separates fine-grained process rewards from holistic sequence-level outcome rewards outperforms applying either strategy in isolation for streaming speech generation. source: §4.3, Table 6 evidence: The combined two-stage scheme achieves higher VIP and lower AL than both single-turn-only and multi-turn-only variants on internal benchmarks; multi-turn-only trades translation quality for latency, while single-turn-only cannot exploit global latency constraints. confidence: high relevance: high - claim_id: rl_reward_design_for_speech_generation_is_prone_to_exploitation role: complicates claim: RL reward design for speech generation is prone to exploitation when individual reward signals are applied without complementary constraints. source: §4.2, Table 5 evidence: Training with the Time Compliance Reward alone caused the model to reduce audio duration by ~35% and text token count by ~15%, yielding an 8-point BLEURT drop while satisfying the temporal constraint; adding an adversarial quality reward restored translation fidelity. confidence: high relevance: medium - claim_id: reinforcement_learning_applied_after_sft_improves_latency_more_substantially_than role: supports claim: Reinforcement learning applied after SFT improves latency more substantially than translation quality in simultaneous speech translation. source: §4.1, Tables 3, 4 evidence: RL reduces FLAL from 3.90 to 2.37 and AL from 2.82 to 2.58 on RealSI, while VIP improves more modestly from 75.1 to 79.5 compared to the SFT baseline. confidence: high relevance: high - claim_id: automated_mt_metrics_such_as_bleurt_and_comet_do_not role: complicates claim: Automated MT metrics such as BLEURT and COMET do not fully capture the quality of simultaneous speech interpretation in realistic long-form scenarios. source: §3.2, Appendix A evidence: The paper introduces SVIP as a composite human evaluation metric combining translation quality, delivery latency, speech rate, pronunciation, and fluency, arguing that automated metrics fail to assess the pragmatic requirements of live spoken interpretation. confidence: high relevance: high limitations: - Training data, model weights, and all commercial baseline identities are proprietary. The headline results cannot be independently reproduced or verified. - Evaluation covers only Chinese-to-English and English-to-Chinese translation; generalisation to other language pairs is untested. The paper does not isolate the contribution of voice cloning quality from translation quality within the SVIP metric, making it difficult to assess how each component affects the composite score. Future work identified by the authors includes broader language coverage, improved voice personalization, and enhanced speech stability and expressiveness. caveats: [] - id: 2025.sigdial-1.51 published_date: "2025-08-01" entry_date: '2026-07-30' year: 2025 venue: workshop task: - SCA architecture: [] relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: minor method_family: [] claims: - claim_id: modular_incremental_architectures_built_on_the_iu_framework_enable_flexible role: supports claim: Modular, incremental architectures built on the IU framework enable flexible composition of heterogeneous speech and perception components for real-time spoken dialogue. source: §2, §3 evidence: rrSDS 2.0 supports seamless swapping of ASR modules (Whisper vs. Wav2vec) and vision modules (YOLOv11, YOLOv12, RT-DETR, SAM, DINOv2) through a shared IU interface, demonstrated in three multi-component robot pipelines. confidence: high relevance: high - claim_id: real_time_multimodal_spoken_dialogue_for_physical_and_simulated_robotic role: complicates claim: Real-time multimodal spoken dialogue for physical and simulated robotic platforms introduces synchronization challenges not present in text-only or speech-only systems. source: §4 evidence: The authors report facing challenges balancing real-time, multimodal interaction when integrating rrSDS 2.0 across physical and simulated robots, without providing quantitative results characterizing the trade-offs. confidence: high relevance: high - claim_id: open_source_framework_level_integration_of_state_of_the_art role: supports claim: Open-source, framework-level integration of state-of-the-art ASR and vision components reduces the engineering overhead for building multimodal robotic dialogue systems. source: §1, §2 evidence: rrSDS 2.0 wraps Whisper, Wav2vec 2.0, YOLO variants, SAM, DINOv2, MediaPipe, RASA 3.0, and HuggingFace text generation behind a common IU interface, with pypi distribution and improved documentation to lower the setup barrier. confidence: high relevance: low limitations: - No quantitative evaluation is reported, making it impossible to compare rrSDS 2.0 against other dialogue system frameworks on latency, accuracy, or robustness. The demo paper format means claims about incremental processing quality and real-time performance rest on qualitative demonstration rather than systematic measurement. Integration with additional benchmarks beyond ALFRED is noted as future work, as is compatibility with the Remdis framework for LLM-driven incremental dialogue. caveats: [] - id: '2507.22746' published_date: "2025-08-01" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: composing_autoregressive_generation_across_fixed_length_token_blocks_with_parallel role: supports claim: Composing autoregressive generation across fixed-length token blocks with parallel flow-matching denoising within each block can simultaneously provide KV-cache efficiency and bidirectional contextual refinement. source: §3.1, Table 3 evidence: Composing autoregressive generation across fixed-length token blocks with parallel flow-matching denoising within each block can simultaneously provide KV-cache efficiency and bidirectional contextual refinement. confidence: high relevance: low - claim_id: neural_codecs_using_finite_scalar_quantisation_can_preserve_speaker_similarity role: supports claim: Neural codecs using finite scalar quantisation can preserve speaker similarity and intelligibility at frame rates (12.5 Hz) where STFT-based vocoders suffer significant quality degradation. source: §4.3.3, Table 4 evidence: Neural codecs using finite scalar quantisation can preserve speaker similarity and intelligibility at frame rates (12.5 Hz) where STFT-based vocoders suffer significant quality degradation. confidence: high relevance: medium - claim_id: continuous_denoising_models_can_implicitly_classify_discrete_token_targets_through role: supports claim: Continuous denoising models can implicitly classify discrete token targets through appropriate embedding design, without requiring a separate discrete language model head. source: §3.1 evidence: Continuous denoising models can implicitly classify discrete token targets through appropriate embedding design, without requiring a separate discrete language model head. confidence: high relevance: medium - claim_id: reducing_the_token_frame_rate_is_a_more_tractable_path role: supports claim: Reducing the token frame rate is a more tractable path to low-latency hybrid AR-diffusion TTS than increasing diffusion step efficiency alone, given the quadratic scaling of self-attention with sequence length. source: §3.1, §4.3.2 evidence: Reducing the token frame rate is a more tractable path to low-latency hybrid AR-diffusion TTS than increasing diffusion step efficiency alone, given the quadratic scaling of self-attention with sequence length. confidence: high relevance: high limitations: - No MOS or SMOS listening test results are reported. All quality comparisons use SPK-SIM, WER, and FAD on an internal podcast dataset. The absence of subjective evaluation and fair comparison against published baselines (VALL-E 2, E2 TTS, NaturalSpeech 3) on a public benchmark makes it impossible to independently verify naturalness claims. - The model is trained and evaluated on English podcast data only. Generalisation to other languages, controlled studio-quality TTS, and expressive speech domains is untested. The podcast use-case naturally emphasises diversity and disfluency tolerance over precise prosody control, so the evaluation protocol may not transfer to production TTS settings. - The codec and acoustic model are proprietary (Microsoft internal), with no public code or demo reported. Reproducibility relies entirely on the architectural description in the paper. - Mean flow optimisation for step reduction is borrowed from concurrent work (Geng et al. 2025); the sensitivity of Dragon-FM quality to NFE count at scale is not fully characterised — ablations cover only 2, 4, 6, 12, and 24 steps. caveats: [] - id: '2508.02849' published_date: "2025-08-04" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - codec architecture: - VAE relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - vae_compressed_streaming_latents claims: - claim_id: distillation_from_self_supervised_models_such_as_hubert_or_wavlm role: supports claim: Distillation from self-supervised models such as HuBERT or WavLM does not achieve true semantic disentanglement in speech codecs, as these representations inherently retain paralinguistic content. source: §II.A, §V.B evidence: Distillation from self-supervised models such as HuBERT or WavLM does not achieve true semantic disentanglement in speech codecs, as these representations inherently retain paralinguistic content. confidence: high relevance: medium - claim_id: frame_level_cross_modal_contrastive_learning_between_phoneme_and_speech role: supports claim: Frame-level cross-modal contrastive learning between phoneme and speech representations produces cleaner semantic-paralinguistic separation than phoneme classification loss in neural codecs. source: §V.D, Table IV evidence: Frame-level cross-modal contrastive learning between phoneme and speech representations produces cleaner semantic-paralinguistic separation than phoneme classification loss in neural codecs. confidence: high relevance: medium - claim_id: vae_augmented_finite_scalar_quantization_vae_fsq_achieves_substantially_higher role: supports claim: VAE-augmented finite scalar quantization (VAE+FSQ) achieves substantially higher codebook utilisation and reduced long-tail token distribution compared to VQ-VAE in single-codebook speech codecs. source: §V.C, Figure 4, Table II evidence: VAE-augmented finite scalar quantization (VAE+FSQ) achieves substantially higher codebook utilisation and reduced long-tail token distribution compared to VQ-VAE in single-codebook speech codecs. confidence: high relevance: medium - claim_id: explicit_modeling_of_paralinguistic_information_as_a_reconstruction_bridge_between role: supports claim: Explicit modeling of paralinguistic information as a reconstruction bridge between semantic and acoustic encodings improves both semantic completeness and reconstruction fidelity in low-bitrate streaming codecs. source: §III.B, §V.B evidence: Explicit modeling of paralinguistic information as a reconstruction bridge between semantic and acoustic encodings improves both semantic completeness and reconstruction fidelity in low-bitrate streaming codecs. confidence: high relevance: high - claim_id: staged_training_that_freezes_acoustic_modules_before_introducing_semantic_and role: supports claim: Staged training that freezes acoustic modules before introducing semantic and KL losses is necessary for stable convergence in multi-objective codec training. source: §III.E, §V.C, Table II evidence: Staged training that freezes acoustic modules before introducing semantic and KL losses is necessary for stable convergence in multi-objective codec training. confidence: high relevance: low limitations: - The contrastive learning objective requires duration-aligned phoneme-level text labels during training. The authors acknowledge this as a key limitation and flag unsupervised disentanglement as the primary future direction. - Evaluation is conducted only on English (LibriTTS) and Mandarin (AISHELL-3) data; generalisation to other languages, especially low-resource or tonal languages with different phoneme structures, is unverified. The model uses HiFi-GAN as its vocoder, which introduces separate dependencies, vocoder artefacts, and the bulk of the decoding RTF — excluding vocoder time, the decoder RTF drops to 0.001. Codebook utilisation of 98.06% is reported for the VAE-FSQ variant but without comparison to the final SecoustiCodec configuration in the ablation table directly; the connection between codebook saturation and downstream LM training quality is asserted but not demonstrated empirically in TTS or dialogue tasks. caveats: [] - id: '2504.10352' published_date: "2025-08-05" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: influential method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: pseudo_autoregressive_generation_which_commits_spans_left_to_right_within role: supports claim: Pseudo-autoregressive generation, which commits spans left-to-right within a bidirectional masked transformer, achieves constant inference steps regardless of target speech duration while maintaining temporal coherence. source: §3, §5.4 evidence: Pseudo-autoregressive generation, which commits spans left-to-right within a bidirectional masked transformer, achieves constant inference steps regardless of target speech duration while maintaining temporal coherence. confidence: high relevance: low - claim_id: a_model_trained_on_580_hours_of_english_speech_can role: supports claim: A model trained on 580 hours of English speech can match or exceed the intelligibility of NAR flow-matching systems trained on 100,000+ hours when temporal ordering is explicitly enforced during generation. source: §5.3, Table 1 evidence: A model trained on 580 hours of English speech can match or exceed the intelligibility of NAR flow-matching systems trained on 100,000+ hours when temporal ordering is explicitly enforced during generation. confidence: high relevance: medium - claim_id: confidence_guided_iterative_nar_refinement_of_an_initial_par_generation role: supports claim: Confidence-guided iterative NAR refinement of an initial PAR generation substantially reduces word error rate with only a small number of additional inference steps. source: §5.5, Figure 4 evidence: Confidence-guided iterative NAR refinement of an initial PAR generation substantially reduces word error rate with only a small number of additional inference steps. confidence: high relevance: medium - claim_id: temporally_unordered_nar_generation_produces_higher_alignment_errors_than_span role: supports claim: Temporally unordered NAR generation produces higher alignment errors than span-level causal generation across both continuation and cross-sentence evaluation tasks. source: §5.4, Table 3 evidence: Temporally unordered NAR generation produces higher alignment errors than span-level causal generation across both continuation and cross-sentence evaluation tasks. confidence: high relevance: high - claim_id: separate_model_capacity_for_each_generation_stage_is_necessary_unifying role: supports claim: Separate model capacity for each generation stage is necessary; unifying PAR and NAR refinement into a single multitask model degrades cross-sentence intelligibility by approximately 20%. source: §5.5 evidence: Separate model capacity for each generation stage is necessary; unifying PAR and NAR refinement into a single multitask model degrades cross-sentence intelligibility by approximately 20%. confidence: high relevance: medium limitations: - PALLE is evaluated only on English (LibriTTS). The 100-step inference (with 7 refinement steps) may still be too slow for the most latency-sensitive streaming applications despite the 10x speedup. Duration estimation for the cross-sentence task relies on a simple linear heuristic; errors in duration estimation lead to modest quality degradation (WER-H 2.83 vs. 2.62 with GT duration). The shared architecture between stage one and stage two (joint multitask fine-tuning) causes stage two loss to degrade stage one performance, suggesting that separate model capacity is required. caveats: [] - id: '2508.05207' published_date: "2025-08-07" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - gan_causal_streaming_vocoders claims: - claim_id: operating_in_the_time_frequency_domain_enables_neural_codecs_to role: supports claim: Operating in the time-frequency domain enables neural codecs to achieve higher perceptual quality for full-band audio than equivalent waveform-domain architectures, especially at low bit rates. source: §2, §4, Table 1 evidence: Operating in the time-frequency domain enables neural codecs to achieve higher perceptual quality for full-band audio than equivalent waveform-domain architectures, especially at low bit rates. confidence: high relevance: medium - claim_id: cross_channel_phase_coherence_in_multi_channel_neural_codecs_requires role: supports claim: Cross-channel phase coherence in multi-channel neural codecs requires joint processing of audio channels in at least some encoder layers, and neither fully independent nor fully joint encoding is optimal. source: §2 evidence: Cross-channel phase coherence in multi-channel neural codecs requires joint processing of audio channels in at least some encoder layers, and neither fully independent nor fully joint encoding is optimal. confidence: high relevance: medium - claim_id: multi_scale_spectral_discriminators_are_effective_for_training_high_quality role: supports claim: Multi-scale spectral discriminators are effective for training high-quality neural codecs without requiring waveform-domain discriminators. source: §3 evidence: Multi-scale spectral discriminators are effective for training high-quality neural codecs without requiring waveform-domain discriminators. confidence: high relevance: medium - claim_id: biased_quantizer_dropout_towards_low_codebook_counts_during_training_improves role: supports claim: Biased quantizer dropout towards low codebook counts during training improves codec quality at the low bit rate end without sacrificing high bit rate performance. source: §3.1.1 evidence: Biased quantizer dropout towards low codebook counts during training improves codec quality at the low bit rate end without sacrificing high bit rate performance. confidence: high relevance: low - claim_id: real_time_streaming_neural_codec_inference_at_48_khz_stereo role: supports claim: Real-time streaming neural codec inference at 48 kHz stereo is achievable on a desktop CPU with an 80 ms architectural latency when using causal convolutions and a minimal look-ahead. source: §1, §2 evidence: Real-time streaming neural codec inference at 48 kHz stereo is achievable on a desktop CPU with an 80 ms architectural latency when using causal convolutions and a minimal look-ahead. confidence: high relevance: high limitations: - Training data is proprietary and the only baseline is DAC; results cannot be reproduced and the comparison does not include SoundStream, EnCodec, or Mimi, leaving SpectroStream's position in the broader codec landscape unclear. - Evaluation is restricted to music (MUSDB18) despite the paper's "general audio" framing. Speech quality at 48 kHz stereo is not reported. The A/B preference protocol does not include a MUSHRA-style anchor, making absolute quality judgements difficult. The latency of 80 ms is described as suitable for streaming but is not benchmarked against real-time constraints in actual deployment. The delayed-fusion fusion point is treated as a design choice found empirically — no ablation is provided to quantify the quality/coherence trade-off as a function of fusion layer depth. caveats: [] - id: '2508.06262' published_date: "2025-08-08" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: token_verification_is_necessary_for_multi_token_prediction_to_be role: supports claim: 'Token verification is necessary for multi-token prediction to be effective in autoregressive TTS: without it, WER increases from 3.07% to 14.37% and speaker similarity drops from 0.570 to 0.463.' source: §V, Table III evidence: 'Token verification is necessary for multi-token prediction to be effective in autoregressive TTS: without it, WER increases from 3.07% to 14.37% and speaker similarity drops from 0.570 to 0.463.' confidence: high relevance: low - claim_id: plug_and_play_mtp_modules_trained_on_a_modest_dataset role: supports claim: Plug-and-play MTP modules trained on a modest dataset can accelerate a frozen autoregressive TTS backbone by up to 1.48x without sacrificing generation quality on standard benchmarks. source: §IV.B, Table I evidence: Plug-and-play MTP modules trained on a modest dataset can accelerate a frozen autoregressive TTS backbone by up to 1.48x without sacrificing generation quality on standard benchmarks. confidence: high relevance: low - claim_id: under_quality_maximizing_inference_settings_mtp_with_verification_can_improve role: supports claim: Under quality-maximizing inference settings, MTP with verification can improve intelligibility beyond the backbone baseline, likely due to extended look-ahead context from the cascaded hidden states. source: §IV.A, Table I evidence: Under quality-maximizing inference settings, MTP with verification can improve intelligibility beyond the backbone baseline, likely due to extended look-ahead context from the cascaded hidden states. confidence: high relevance: medium - claim_id: converting_a_non_causal_codec_decoder_to_a_causal_streaming role: supports claim: Converting a non-causal codec decoder to a causal streaming architecture via lightweight fine-tuning preserves approximately 95% of reconstruction quality, making streaming reconstruction viable without full retraining. source: §IV.C, Table II evidence: Converting a non-causal codec decoder to a causal streaming architecture via lightweight fine-tuning preserves approximately 95% of reconstruction quality, making streaming reconstruction viable without full retraining. confidence: high relevance: high - claim_id: attention_based_mtp_modules_substantially_outperform_mlp_based_equivalents_of role: supports claim: Attention-based MTP modules substantially outperform MLP-based equivalents of similar parameter count in both intelligibility and speaker similarity for TTS acceleration. source: §IV.B, Table I evidence: Attention-based MTP modules substantially outperform MLP-based equivalents of similar parameter count in both intelligibility and speaker similarity for TTS acceleration. confidence: high relevance: low limitations: - Evaluation is English-only (LibriTTS training, Seed-TTS-eval-en test), and speaker generalization to out-of-distribution languages or accents is untested. The verification overhead (one additional LM forward pass for verification at each step) partially offsets the MTP speedup, especially at strict topk values. The 1.48x figure assumes topk=500, which allows some quality degradation; the fully lossless speedup (topk=100) is closer to 1.42x. Scaling MTP to larger models (Llasa-3B, Llasa-8B) is not investigated. caveats: [] - id: '2508.07375' published_date: "2025-08-10" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - evaluation_caution current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: aggregating_text_tokens_at_the_dialogue_turn_level_provides_stronger role: supports claim: Aggregating text tokens at the dialogue-turn level provides stronger semantic guidance for full-duplex speech generation than word-level token-by-token text insertion. source: §2.2, §4.2.1, Table 2 evidence: Aggregating text tokens at the dialogue-turn level provides stronger semantic guidance for full-duplex speech generation than word-level token-by-token text insertion. confidence: high relevance: high - claim_id: the_timing_of_text_insertion_into_full_duplex_dialogue_sequences role: supports claim: The timing of text insertion into full-duplex dialogue sequences is more sensitive to error than the length of inserted text, with mistimed insertions causing larger semantic degradation than incorrectly sized text chunks. source: §4.2.3, Table 4 evidence: The timing of text insertion into full-duplex dialogue sequences is more sensitive to error than the length of inserted text, with mistimed insertions causing larger semantic degradation than incorrectly sized text chunks. confidence: high relevance: high - claim_id: increasing_the_training_loss_weight_of_text_tokens_relative_to role: supports claim: Increasing the training loss weight of text tokens relative to speech tokens in a text-speech interleaved model improves semantic quality without sacrificing turn-taking naturalness. source: §4.2.1, Table 2 evidence: Increasing the training loss weight of text tokens relative to speech tokens in a text-speech interleaved model improves semantic quality without sacrificing turn-taking naturalness. confidence: high relevance: high - claim_id: fine_grained_turn_taking_benchmarks_are_more_informative_than_corpus role: supports claim: Fine-grained turn-taking benchmarks are more informative than corpus-level statistical correlations for evaluating full-duplex spoken dialogue models. source: §4.2.2 evidence: Fine-grained turn-taking benchmarks are more informative than corpus-level statistical correlations for evaluating full-duplex spoken dialogue models. confidence: high relevance: high - claim_id: gpt_based_automated_semantic_evaluation_of_spoken_dialogue_aligns_closely role: supports claim: GPT-based automated semantic evaluation of spoken dialogue aligns closely with human preference judgements when score differences exceed one point, but reliability degrades for near-tied comparisons. source: Appendix A, Table 5 evidence: GPT-based automated semantic evaluation of spoken dialogue aligns closely with human preference judgements when score differences exceed one point, but reliability degrades for near-tied comparisons. confidence: high relevance: low limitations: - All results are on the Fisher telephone conversation corpus only. Fisher's conversational style and acoustic conditions (telephone, English, spontaneous) are narrow, and the paper provides no evidence that TurnGuide generalises to other languages, speaking styles, or domains. - The model has not undergone safety alignment (RLHF or equivalent), which the authors acknowledge is a prerequisite for deployment. The theoretical first-package latency is 1.05 seconds — workable for some interactive applications but not near real-time; the authors note the vocoder imposes the dominant cost and that an optimised decoder would reduce this. - The evaluation uses GPT-4o as both the semantic evaluator and implicitly as an oracle for dialogue quality, which introduces a potential circularity if the trained model's outputs are biased toward patterns GPT-4o scores favourably. The corpus-level Pearson correlation analysis (Table 7) shows TurnGuide is at parity with baselines on turn-taking statistics but does not establish whether the improvement in GPT-score translates to user preference in live interaction. caveats: [] - id: interspeech-2025-0669 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - codec - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: supervised_phonetic_data_ctc_and_phoneme_classification_can_replace_ssl role: supports claim: Supervised phonetic data (CTC and phoneme classification) can replace SSL pseudo-label distillation as the phonetic supervision signal in hybrid speech tokenizers, achieving superior phonetic representation without requiring a pretrained SSL model. source: §3.3, §5.1, Table 1 evidence: Supervised phonetic data (CTC and phoneme classification) can replace SSL pseudo-label distillation as the phonetic supervision signal in hybrid speech tokenizers, achieving superior phonetic representation without requiring a pretrained SSL model. confidence: high relevance: medium - claim_id: ctc_character_match_loss_is_the_dominant_driver_of_phonetic role: supports claim: CTC character-match loss is the dominant driver of phonetic encoding quality in RVQ-based tokenizers, contributing more than phoneme classification alone. source: §5.3, Table 4 evidence: CTC character-match loss is the dominant driver of phonetic encoding quality in RVQ-based tokenizers, contributing more than phoneme classification alone. confidence: high relevance: medium - claim_id: a_transformer_encoder_inserted_before_the_rvq_quantizer_improves_phonetic role: supports claim: A transformer encoder inserted before the RVQ quantizer improves phonetic representation, but requires stochastic skip-connection dropout during training to prevent the network from bypassing it. source: §3.2, §5.3, Table 5 evidence: A transformer encoder inserted before the RVQ quantizer improves phonetic representation, but requires stochastic skip-connection dropout during training to prevent the network from bypassing it. confidence: high relevance: medium - claim_id: hybrid_tokenizers_that_optimize_phonetic_encoding_via_direct_supervision_can role: supports claim: Hybrid tokenizers that optimize phonetic encoding via direct supervision can approach pure acoustic codecs in reconstruction quality while substantially surpassing SSL-distilled baselines. source: §5.1, Table 2 evidence: Hybrid tokenizers that optimize phonetic encoding via direct supervision can approach pure acoustic codecs in reconstruction quality while substantially surpassing SSL-distilled baselines. confidence: high relevance: medium - claim_id: speech_tokenizers_that_better_encode_phonetic_structure_yield_stronger_downstream role: supports claim: Speech tokenizers that better encode phonetic structure yield stronger downstream speech language model performance on lexical discrimination benchmarks. source: §5.2, Table 3 evidence: Speech tokenizers that better encode phonetic structure yield stronger downstream speech language model performance on lexical discrimination benchmarks. confidence: high relevance: medium limitations: - PAST requires labeled phoneme/character data, which constrains multilingual scalability — the paper explicitly acknowledges this and targets it as future work. PAST's reconstruction quality (SISNR=4.84) falls below pure EnCodec (SISNR=7.49), reflecting the phonetic-acoustic trade-off. The model is 185M parameters, larger than some baseline tokenizers, partly due to the transformer encoder. Evaluation is English-only on clean speech; robustness to noise, accents, and spontaneous speech is not assessed. caveats: [] - id: interspeech-2025-0874 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: separating_user_and_agent_stream_representations_using_a_pretrained_speech role: supports claim: Separating user and agent stream representations — using a pretrained speech encoder for input and a neural codec for generation — allows full-duplex S2S models to bypass LLM speech pretraining without sacrificing conversation quality. source: §3, §6.1, §6.2 evidence: Separating user and agent stream representations — using a pretrained speech encoder for input and a neural codec for generation — allows full-duplex S2S models to bypass LLM speech pretraining without sacrificing conversation quality. confidence: high relevance: high - claim_id: codec_personalisation_through_fine_tuning_on_target_speaker_data_can role: supports claim: Codec personalisation through fine-tuning on target-speaker data can recover audio quality at half the bitrate of an untuned codec, as measured by MOS, CER, and speaker similarity. source: §6.3, Table 4 evidence: Codec personalisation through fine-tuning on target-speaker data can recover audio quality at half the bitrate of an untuned codec, as measured by MOS, CER, and speaker similarity. confidence: high relevance: low - claim_id: turn_level_alignment_between_text_and_speech_tokens_in_duplex role: supports claim: Turn-level alignment between text and speech tokens in duplex training is sufficient to learn barge-in behaviour; word-level alignment provides no measurable improvement. source: §3.1 evidence: Turn-level alignment between text and speech tokens in duplex training is sufficient to learn barge-in behaviour; word-level alignment provides no measurable improvement. confidence: high relevance: high - claim_id: full_duplex_end_to_end_models_remain_at_a_reasoning role: supports claim: Full-duplex end-to-end models remain at a reasoning disadvantage compared to cascaded oracle systems, though the gap narrows as backbone LLM quality increases. source: §6.2, Table 3 evidence: Full-duplex end-to-end models remain at a reasoning disadvantage compared to cascaded oracle systems, though the gap narrows as backbone LLM quality increases. confidence: high relevance: high - claim_id: open_source_availability_of_training_code_and_model_weights_is role: supports claim: Open-source availability of training code and model weights is a critical bottleneck for research progress in full-duplex spoken dialogue, given the historical concentration of such systems in closed industrial labs. source: §1 evidence: Open-source availability of training code and model weights is a critical bottleneck for research progress in full-duplex spoken dialogue, given the historical concentration of such systems in closed industrial labs. confidence: high relevance: high limitations: - The backbone is TinyLlama-1.1B, a relatively small LLM. The reasoning gap between the end-to-end model and the GT+LLM cascaded oracle is real and acknowledged, particularly on QA tasks. Scaling to a larger backbone remains untested and its interaction with the duplex architecture is an open question. - Training data is entirely synthetic (TTS-generated user and agent speech) except for the ASR-QA portion, which introduces a domain mismatch with natural conversation. The fixed 0.64-second silence inserted before agent turns is a hard-coded heuristic that will affect latency in practice and may not generalise to more varied conversational pacing. The evaluation does not include a listening test (MOS via human raters) for duplex conversation quality — UTMOS and GPT score are proxies. The first-response latency metric is not comparable to Moshi because Moshi's proactive interruption behaviour makes the metric inapplicable, which limits direct system comparison. caveats: [] - id: interspeech-2025-1084 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: mamba_based_sequence_models_can_match_or_exceed_the_quality role: supports claim: Mamba-based sequence models can match or exceed the quality of larger Transformer-based TTS systems while enabling real-time streaming inference on CPU hardware. source: §4.5, §4.6, Table 1 evidence: SMAM+MLM (26M params) achieves MOS 4.02 and CER 2.73%, matching Lee et al. (2024) at 263M params (MOS 4.00, CER 4.01%) while reducing first-token latency from 26.5s to 0.065s on a single-threaded CPU. confidence: high relevance: high - claim_id: iterative_depthwise_refinement_of_rvq_tokens_substantially_improves_codec_tts role: supports claim: Iterative depthwise refinement of RVQ tokens substantially improves codec TTS quality over single-pass parallel depth prediction. source: §4.7, Table 1 evidence: Replacing MLM depthwise decoding with a single-pass no-masking baseline (SMAM+noMLM) causes a significant drop in all quality metrics (MOS from 4.02 to 3.89, CER from 2.73% to 4.12%, UTMOS from 4.13 to 3.83) with negligible change in RTF and latency. confidence: high relevance: low - claim_id: objective_speaker_similarity_metrics_based_on_embedding_cosine_distance_do role: complicates claim: Objective speaker similarity metrics based on embedding cosine distance do not reliably predict subjective speaker similarity as judged by human listeners. source: §4.6, Table 1 evidence: SMAM+MLM scores SECS 0.816 (below Lee et al.'s 0.863) but achieves higher SMOS of 3.36 vs. 3.27, indicating a divergence between embedding-space distance and perceptual similarity that has practical implications for zero-shot TTS evaluation. confidence: high relevance: medium - claim_id: depthwise_decoding_strategies_for_rvq_present_an_explicit_quality_speed role: supports claim: Depthwise decoding strategies for RVQ present an explicit quality-speed trade-off that system designers can exploit based on deployment constraints. source: §3.3, §4.5, §4.6, Table 1 evidence: SMAM+MLM (iterative, 3 passes) achieves MOS 4.02 and RTF 0.701, while SMAM+INR (single forward pass) achieves MOS 3.97 and RTF 0.568, demonstrating a consistent quality-speed trade-off across both objective and subjective evaluations. confidence: high relevance: medium limitations: - Evaluation is limited to LibriTTS test-clean (English, read speech), leaving performance on spontaneous speech, noisy environments, and non-English languages uncharacterized. The SECS speaker similarity scores for the proposed models fall below the strongest baseline (Lee et al. 2024), indicating room for improvement in speaker faithfulness despite strong subjective SMOS scores. The paper does not release code, limiting reproducibility and adoption. RTF comparisons are not fully apples-to-apples since baselines generate complete utterances in batch mode while SMAM operates incrementally. Future directions mentioned include a fully streaming pipeline covering codec processing and applying depthwise decoding strategies to decoder-only speech language models. caveats: [] - id: interspeech-2025-1289 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - codec architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: constant_frame_rate_coding_introduces_temporal_redundancy_in_neural_speech role: supports claim: Constant-frame-rate coding introduces temporal redundancy in neural speech codecs by allocating equal resolution to silence and phonetically dense regions alike. source: §1, §2.1 evidence: Constant-frame-rate coding introduces temporal redundancy in neural speech codecs by allocating equal resolution to silence and phonetically dense regions alike. confidence: high relevance: medium - claim_id: dynamically_allocating_coarser_temporal_frames_to_low_entropy_speech_regions role: supports claim: Dynamically allocating coarser temporal frames to low-entropy speech regions reduces token sequence length without proportional degradation in reconstruction quality. source: §4.2, Table 1 evidence: Dynamically allocating coarser temporal frames to low-entropy speech regions reduces token sequence length without proportional degradation in reconstruction quality. confidence: high relevance: medium - claim_id: reducing_the_number_of_encoded_frames_at_equivalent_bitrate_can role: supports claim: Reducing the number of encoded frames at equivalent bitrate can improve intelligibility, suggesting that sequence compactness benefits autoregressive downstream models independently of bitrate. source: §4.2, Table 1 evidence: Reducing the number of encoded frames at equivalent bitrate can improve intelligibility, suggesting that sequence compactness benefits autoregressive downstream models independently of bitrate. confidence: high relevance: low - claim_id: variable_frame_rate_allocation_and_variable_bitrate_control_are_orthogonal role: supports claim: Variable frame rate allocation and variable bitrate control are orthogonal axes in neural codec design and can be combined additively. source: §3, §5 evidence: Variable frame rate allocation and variable bitrate control are orthogonal axes in neural codec design and can be combined additively. confidence: high relevance: low limitations: - Evaluation is restricted to a single codec backbone (DAC) and a single dataset (LibriTTS). No human listening study is reported; all quality judgements rest on UTMOS, STOI, WER, and spectral distances. Generalisability to other architectures or acoustic conditions is untested. - The paper does not report downstream TTS or speech LM experiments, so the claimed benefit of shorter token sequences for generation quality and latency is prospective rather than demonstrated. The entropy-based frame allocation heuristic uses fixed hyperparameters (bin count N, smoothing σ) without ablation; sensitivity to these choices is unknown. Training with mixed granularity ratios uses a fixed heuristic allocation that may not be optimal. The granularity ratios at inference must be set by the user; no automatic target-rate optimisation is described. caveats: [] - id: interspeech-2025-1819 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - gan_causal_streaming_vocoders claims: - claim_id: input_side_multi_stream_decomposition_via_analysis_filters_can_improve role: supports claim: Input-side multi-stream decomposition via analysis filters can improve both quality and efficiency in harmonic-prior 2D convolutional vocoders. source: §2.2, §4.2 evidence: MS-Wavehax splits the input waveform into four subscale signals before STFT, producing sub-spectrograms with locally consistent spatial structure; it achieves the highest MOS in both analysis-synthesis and TTS tasks while requiring roughly half the MACs of standard Wavehax and only 0.332M parameters. confidence: high relevance: medium - claim_id: batch_inference_real_time_factor_is_a_poor_proxy_for role: supports claim: Batch-inference real-time factor is a poor proxy for streaming throughput; architecture choices interact with chunk size and cache behaviour in ways that can reverse the efficiency ranking. source: §3.2, Table 2, Figure 2 evidence: At small chunk sizes MS-Wavehax achieves the lowest RTF among all vocoders; at larger chunk sizes Vocos overtakes it because 1D filter matrices are constant-sized and cache-resident, while Wavehax's 2D im2col matrices grow with chunk size T, causing more frequent main-memory access. confidence: high relevance: high - claim_id: multi_stream_decomposition_does_not_uniformly_improve_quality_across_all role: complicates claim: Multi-stream decomposition does not uniformly improve quality across all frame-level vocoder architectures. source: §4.2 evidence: MS-Vocos underperforms standard Vocos in MOS evaluation; the authors attribute this to a latent representation narrower than the output complex spectrum, which makes estimating inter-stream dependencies particularly difficult. confidence: high relevance: low - claim_id: replacing_layer_normalization_with_batch_normalization_in_gan_vocoders_enables role: supports claim: Replacing layer normalization with batch normalization in GAN vocoders enables streaming compatibility and can also yield objective quality improvements. source: §2.3 evidence: Layer normalization computes statistics over the full sequence and is incompatible with incremental streaming; switching to batch normalization removed this constraint and improved PESQ and UTMOS scores in internal experiments on all tested vocoder configurations. confidence: high relevance: high - claim_id: causal_vocoder_processing_with_minimal_lookahead_can_match_non_causal role: supports claim: Causal vocoder processing with minimal lookahead can match non-causal quality in single-speaker TTS settings. source: §4.2, Figure 4 evidence: Causal MS-Wavehax with a one-frame lookahead achieves nearly identical MOS to non-causal MS-Wavehax on the JSUT TTS task; the gap observed in the multi-speaker A/S task disappears, suggesting that causal constraints function as an effective regularizer in single-speaker conditions. confidence: high relevance: high limitations: - Evaluations are conducted exclusively on Japanese speech corpora (JVS and JSUT) on a single CPU architecture (AMD EPYC), so generalisation to multilingual or GPU-accelerated deployment is not demonstrated. All models were conditioned on mel-spectrograms extracted from clean studio recordings; behaviour on noisy or spontaneous speech is not assessed. The paper does not include a real-time demo or end-to-end latency measurement with an acoustic model frontend, so the practical time-to-first-audio-byte in a complete TTS pipeline remains uncharacterised. caveats: [] - id: interspeech-2025-2447 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: speculative_decoding_adapted_for_speech_can_reduce_autoregressive_inference_latency role: supports claim: Speculative decoding adapted for speech can reduce autoregressive inference latency without measurable degradation in subjective naturalness or speaker similarity. source: §4.1, §4.2 evidence: Speculative decoding adapted for speech can reduce autoregressive inference latency without measurable degradation in subjective naturalness or speaker similarity. confidence: high relevance: high - claim_id: speech_token_sequences_exhibit_many_to_one_mappings_to_perceived role: supports claim: Speech token sequences exhibit many-to-one mappings to perceived quality, enabling relaxed acceptance criteria that improve decoding throughput over strict token-distribution matching. source: §2.2, §4.3 evidence: Speech token sequences exhibit many-to-one mappings to perceived quality, enabling relaxed acceptance criteria that improve decoding throughput over strict token-distribution matching. confidence: high relevance: medium - claim_id: initialising_a_lightweight_draft_model_from_the_upper_layers_of role: supports claim: Initialising a lightweight draft model from the upper layers of the target model provides immediate vocabulary alignment and reduces the data requirements for draft model training. source: §2.3 evidence: Initialising a lightweight draft model from the upper layers of the target model provides immediate vocabulary alignment and reduces the data requirements for draft model training. confidence: high relevance: medium - claim_id: inference_stage_acceleration_of_autoregressive_tts_is_achievable_without_fine role: supports claim: Inference-stage acceleration of autoregressive TTS is achievable without fine-tuning the target model, preserving deployment flexibility for frozen production systems. source: §2, §4.1 evidence: Inference-stage acceleration of autoregressive TTS is achievable without fine-tuning the target model, preserving deployment flexibility for frozen production systems. confidence: high relevance: low limitations: - The WER increase (3.67% → 5.70%) is unexplained beyond a data-scale hypothesis. The draft model's limited training data (LibriTTS, ~580h vs. CosyVoice 2's proprietary corpus) is identified as the likely cause, but this is not verified experimentally — e.g., by scaling draft training data or by ablating with matched data. - Evaluation is restricted to a single target model (CosyVoice 2) on a single English benchmark. Generalisation of SSD to multilingual systems, streaming inference contexts, or multi-codebook AR models (e.g., VALL-E-style RVQ decoding) is not explored. The tolerance factor β is treated as a fixed hyperparameter tuned on objective metrics; its interaction with speaker diversity and domain shift is unexamined. The reported 1.4× speedup measures LM-RTF only and does not account for the draft model's own compute overhead in the total pipeline time. caveats: [] - id: interspeech-2025-2449 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - flow_matching_streaming_decoders claims: - claim_id: non_uniform_ode_time_step_schedules_can_reduce_flow_matching role: supports claim: Non-uniform ODE time-step schedules can reduce flow-matching TTS inference to 7 steps with minimal quality degradation when steps are pruned from the later, near-linear portion of the trajectory. source: §3.2, §4.3, Table 1 evidence: Non-uniform ODE time-step schedules can reduce flow-matching TTS inference to 7 steps with minimal quality degradation when steps are pruned from the later, near-linear portion of the trajectory. confidence: high relevance: low - claim_id: the_early_high_curvature_phase_of_flow_matching_inference_in role: supports claim: The early, high-curvature phase of flow-matching inference in TTS is disproportionately critical for quality, and uniform step reduction degrades performance primarily by under-sampling this phase. source: §3.1, §4.4, Table 3 evidence: The early, high-curvature phase of flow-matching inference in TTS is disproportionately critical for quality, and uniform step reduction degrades performance primarily by under-sampling this phase. confidence: high relevance: low - claim_id: training_free_step_scheduling_methods_generalise_across_flow_matching_tts role: supports claim: Training-free step scheduling methods generalise across flow-matching TTS architectures without requiring model-specific retraining. source: §4.3, Table 1 evidence: Training-free step scheduling methods generalise across flow-matching TTS architectures without requiring model-specific retraining. confidence: high relevance: low - claim_id: at_very_low_nfe_below_6_steps_quality_degradation_in role: supports claim: At very low NFE (below 6 steps), quality degradation in flow-matching TTS becomes abrupt rather than gradual, suggesting a practical floor for inference reduction without distillation. source: §4.4, Table 3 evidence: At very low NFE (below 6 steps), quality degradation in flow-matching TTS becomes abrupt rather than gradual, suggesting a practical floor for inference reduction without distillation. confidence: high relevance: low limitations: - Evaluation uses no subjective listening tests (MOS); quality assessment relies entirely on UTMOS, WER, and speaker cosine similarity — automatic proxies that may not fully capture perceptual degradation at low NFE. - EPSS is validated only on F5-TTS and E2 TTS, both mel-spectrogram-based flow-matching systems sharing similar backbone design. Applicability to token-based or vocoder-free flow-matching systems is unexplored. The time-step configurations are determined empirically by inspection rather than through a principled search procedure, which limits reproducibility on new models. The paper does not explore combining EPSS with consistency distillation or rectified-flow distillation, which could potentially push quality-preserving inference below 7 steps. caveats: [] - id: interspeech-2025-2564 published_date: "2025-08-17" entry_date: '2026-07-30' year: 2025 venue: Interspeech task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: large_scale_monolingual_pre_training_followed_by_stereo_dialogue_fine role: supports claim: Large-scale monolingual pre-training followed by stereo dialogue fine-tuning enables spoken dialogue models to acquire language-specific conversational behaviors. source: §5, Table 3 evidence: J-Moshi, trained on J-CHAT and stereo Japanese dialogue, exhibits more speech overlaps (5.0s/min) and more IPUs (53.2/min) than English Moshi (1.2s overlap, 35.1 IPUs), consistent with Japanese conversational norms. confidence: high relevance: low - claim_id: synthetic_spoken_dialogue_generated_by_multi_stream_tts_improves_language role: supports claim: Synthetic spoken dialogue generated by multi-stream TTS improves language capability in full-duplex dialogue models when added to fine-tuning data. source: §4.3, Table 2 evidence: J-Moshi-ext (trained with 602 hours of TTS-synthesized dialogue added) achieves meaningfulness 2.30 versus J-Moshi's 2.19, a statistically distinguishable improvement, with no degradation in naturalness. confidence: high relevance: high - claim_id: neural_audio_codecs_pre_trained_on_one_language_can_transfer role: complicates claim: Neural audio codecs pre-trained on one language can transfer to another with minimal acoustic degradation, but the autoregressive language model component requires substantial retraining to achieve acceptable dialogue quality. source: §4.3, Table 2 evidence: Mimi re-synthesis of Japanese speech degrades by approximately 0.5 MOS from ground truth, while J-Moshi (with RQ-Transformer adapted) degrades by more than 1 MOS, identifying the language model as the primary quality bottleneck. confidence: high relevance: low - claim_id: morphological_density_differences_across_languages_affect_full_duplex_dialogue_model role: complicates claim: 'Morphological density differences across languages affect full-duplex dialogue model training dynamics: languages with higher phoneme-to-token ratios produce sparser text-to-audio token alignments that may require adjusted training objectives.' source: §5 evidence: Japanese data preprocessing results in 88% PAD tokens in text sequences versus 65% for English in Moshi, reflecting that kanji characters encode more phonemes per token, and the authors flag this as a design consideration for future Japanese-specific training objectives. confidence: high relevance: high limitations: - Both J-Moshi and J-Moshi-ext score above 1 MOS below the Mimi re-synthesis ceiling, indicating the autoregressive RQ-Transformer component is a major quality bottleneck in Japanese. Mimi itself degrades by approximately 0.5 MOS from ground truth when applied to Japanese without adaptation, suggesting codec fine-tuning for Japanese will be necessary for production-quality systems. - The comparison with English Moshi in Table 3 is not conducted under identical experimental conditions (different test sets, possibly different prompt lengths), so turn-taking statistics should be interpreted as indicative rather than rigorously controlled. The 24.6% overall WER of the TTS-synthesized augmentation data introduces noise, and the effect of this noise on specific error categories is not analyzed. The paper does not evaluate spoken dialogue content quality beyond naturalness and meaningfulness, leaving turn-taking appropriateness and response coherence unmeasured. caveats: [] - id: '2508.15827' published_date: "2025-08-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: in_speech_models_reasoning_depth_and_response_latency_are_not role: supports claim: In speech models, reasoning depth and response latency are not fundamentally in conflict when the model's token generation rate substantially exceeds the real-time audio playback rate. source: §1, §3.1 evidence: In speech models, reasoning depth and response latency are not fundamentally in conflict when the model's token generation rate substantially exceeds the real-time audio playback rate. confidence: high relevance: high - claim_id: interleaving_silent_reasoning_tokens_with_spoken_response_tokens_at_a role: supports claim: Interleaving silent reasoning tokens with spoken response tokens at a fixed ratio can improve accuracy on structured reasoning tasks while reducing audible output length. source: §2.3, §4.4, Table 2 evidence: Interleaving silent reasoning tokens with spoken response tokens at a fixed ratio can improve accuracy on structured reasoning tasks while reducing audible output length. confidence: high relevance: medium - claim_id: the_thinking_before_speaking_paradigm_when_applied_directly_to_speech role: supports claim: The "thinking-before-speaking" paradigm, when applied directly to speech, produces user-facing latency or verbosity that impairs conversational quality independently of reasoning correctness. source: §1, §2.2 evidence: The "thinking-before-speaking" paradigm, when applied directly to speech, produces user-facing latency or verbosity that impairs conversational quality independently of reasoning correctness. confidence: high relevance: high - claim_id: multi_stage_training_separating_modality_alignment_reasoning_transfer_and_acoustic role: supports claim: Multi-stage training — separating modality alignment, reasoning transfer, and acoustic synthesis — is an effective strategy for progressively adapting an existing speech LLM to a new generation paradigm. source: §3.3 evidence: Multi-stage training — separating modality alignment, reasoning transfer, and acoustic synthesis — is an effective strategy for progressively adapting an existing speech LLM to a new generation paradigm. confidence: high relevance: medium - claim_id: synthetic_speech_based_mathematical_reasoning_datasets_constructed_from_text_corpora role: supports claim: Synthetic speech-based mathematical reasoning datasets constructed from text corpora via TTS can provide sufficient training signal for spoken reasoning capabilities. source: §3.2, §4.4 evidence: Synthetic speech-based mathematical reasoning datasets constructed from text corpora via TTS can provide sufficient training signal for spoken reasoning capabilities. confidence: high relevance: low limitations: - The entire evaluation uses a single benchmark (Spoken-MQA) focused on mathematics. There is no assessment of speech naturalness, intelligibility, or reasoning accuracy on open-domain conversational tasks. Reported latency claims refer to the absence of a pre-speech reasoning phase rather than to measured real-time performance metrics. - The fixed 2:8 interleaving ratio is derived from a throughput estimate for a specific GPU configuration; it is not adaptive and may be suboptimal for different deployment environments or model sizes. The training data is entirely synthetic — both the audio (produced by CosyVoice2-0.5B) and the reasoning traces (constructed algorithmically from text datasets). Whether the model generalises to naturalistic spoken queries beyond maths problems is untested. The GPT-based verification stage for dataset quality introduces a dependency on a proprietary closed model that is not reproducible. caveats: [] - id: '2508.19098' published_date: "2025-08-26" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching - VAE relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: influential method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - vae_compressed_streaming_latents claims: - claim_id: replacing_discrete_neural_codec_tokens_with_continuous_vae_latents_in role: supports claim: Replacing discrete neural codec tokens with continuous VAE latents in an autoregressive TTS framework reduces the number of AR decoding steps and lowers inference RTF without sacrificing speech quality. source: §5.1, Table 2 evidence: Replacing discrete neural codec tokens with continuous VAE latents in an autoregressive TTS framework reduces the number of AR decoding steps and lowers inference RTF without sacrificing speech quality. confidence: high relevance: low - claim_id: a_per_token_mlp_rectified_flow_head_that_operates_independently role: supports claim: A per-token MLP rectified flow head that operates independently on each AR hidden state enables streaming speech synthesis with latency competitive with non-streaming systems. source: §5.3, Table 4 evidence: A per-token MLP rectified flow head that operates independently on each AR hidden state enables streaming speech synthesis with latency competitive with non-streaming systems. confidence: high relevance: high - claim_id: high_compression_ratio_vae_latents_2048x_downsampling_can_achieve_reconstruction role: supports claim: High-compression-ratio VAE latents (2048x downsampling) can achieve reconstruction quality comparable to ground truth on standard benchmarks while producing sequences short enough to benefit AR sequence modeling. source: §3.4, Appendix C.1 evidence: High-compression-ratio VAE latents (2048x downsampling) can achieve reconstruction quality comparable to ground truth on standard benchmarks while producing sequences short enough to benefit AR sequence modeling. confidence: high relevance: medium - claim_id: continuous_ar_tts_systems_trained_with_a_joint_language_model role: supports claim: Continuous AR TTS systems trained with a joint language model and flow-matching head in a single stage can match or exceed the WER of two-stage cascaded discrete AR systems trained on comparable data. source: §5.1, Table 1 evidence: Continuous AR TTS systems trained with a joint language model and flow-matching head in a single stage can match or exceed the WER of two-stage cascaded discrete AR systems trained on comparable data. confidence: high relevance: low - claim_id: objective_speaker_similarity_metrics_underestimate_the_perceptual_voice_fidelity_of role: supports claim: Objective speaker similarity metrics underestimate the perceptual voice fidelity of AR systems that rely entirely on in-context learning from an audio prompt, as shown by divergence between WavLM-TDNN scores and subjective S-MOS ratings. source: §5.2, Table 3, Appendix D.2 evidence: Objective speaker similarity metrics underestimate the perceptual voice fidelity of AR systems that rely entirely on in-context learning from an audio prompt, as shown by divergence between WavLM-TDNN scores and subjective S-MOS ratings. confidence: high relevance: medium limitations: - 'The paper explicitly acknowledges: (1) objective speaker similarity (WavLM-TDNN) lags behind models that inject speaker embeddings directly into the generator (e.g., CosyVoice, NaturalSpeech3 with FACodec); CLEAR relies entirely on in-context learning from the audio prompt with no explicit speaker conditioning, which is a design choice that limits speaker fidelity under high compression. (2) Evaluation is English-only; multilingual generalization is untested. (3) Sequential single-token AR generation; multi-token prediction could improve throughput further. Additionally, training data for VAE (LibriTTS, 585h) is much smaller than TTS training data (LibriHeavy, 50kh), and the interaction between VAE compression ratio and downstream TTS quality could be investigated more rigorously in full-scale settings.' caveats: [] - id: '2507.14534' published_date: "2025-08-30" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - VC architecture: - GAN - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - gan_causal_streaming_vocoders - hybrid_modular_streaming_systems claims: - claim_id: streaming_voice_conversion_quality_degrades_significantly_when_offline_hubert_representations role: supports claim: Streaming voice conversion quality degrades significantly when offline HuBERT representations are naively replaced by causal alternatives, but knowledge distillation into an Emformer backbone can recover content accuracy with acceptable latency. source: §III.B, Table III evidence: Streaming voice conversion quality degrades significantly when offline HuBERT representations are naively replaced by causal alternatives, but knowledge distillation into an Emformer backbone can recover content accuracy with acceptable latency. confidence: high relevance: high - claim_id: causal_temporal_upsampling_via_pixel_shuffle_eliminates_the_checkerboard_artifacts role: supports claim: Causal temporal upsampling via pixel shuffle eliminates the checkerboard artifacts introduced by zero-padding non-causal vocoders, without sacrificing subjective quality. source: §III.D, Table III evidence: Causal temporal upsampling via pixel shuffle eliminates the checkerboard artifacts introduced by zero-padding non-causal vocoders, without sacrificing subjective quality. confidence: high relevance: high - claim_id: explicit_style_modeling_with_clustering_based_vector_quantization_improves_zero role: supports claim: Explicit style modeling with clustering-based vector quantization improves zero-shot speaker similarity in streaming VC beyond what timbre embeddings alone provide. source: §III.C, Table I evidence: Explicit style modeling with clustering-based vector quantization improves zero-shot speaker similarity in streaming VC beyond what timbre embeddings alone provide. confidence: high relevance: high - claim_id: online_voice_conversion_systems_can_achieve_speaker_similarity_comparable_to role: supports claim: Online voice conversion systems can achieve speaker similarity comparable to offline systems when style transfer is modeled at chunk level rather than at the global utterance level. source: §IV.B, Table I evidence: Online voice conversion systems can achieve speaker similarity comparable to offline systems when style transfer is modeled at chunk level rather than at the global utterance level. confidence: high relevance: high limitations: - '- Evaluation is in English only; cross-lingual style transfer is untested. - The reference speaker must be fully available before streaming begins, limiting applications where reference is also captured in real-time. - CER is slightly higher than StreamVC because StreamVC reuses source pitch; Conan introduces some pitch variation that ASR penalizes as CER. - Model size is not reported; the cost of the Emformer + main model + CSV in production deployment is unclear. - Perceptual evaluation used 15 listeners per pair on Amazon Mechanical Turk style tasks; larger-scale evaluation would be needed to establish statistical robustness.' caveats: [] - id: '2509.02020' published_date: "2025-09-02" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation claims: - claim_id: reducing_speech_tokenizer_frame_rate_to_12_5hz_with_explicit role: supports claim: Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection. source: §2.1, §4.1, Table 1 evidence: Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection. confidence: high relevance: low - claim_id: a_dual_transformer_architecture_for_multi_layer_rvq_prediction_achieves role: supports claim: A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns. source: §2.2 evidence: A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns. confidence: high relevance: high - claim_id: autoregressive_tts_systems_trained_on_multi_speaker_dialogue_data_with role: supports claim: Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels. source: §3.2, §4.3, Table 3 evidence: Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels. confidence: high relevance: low - claim_id: sentence_by_sentence_multi_speaker_dialogue_tts_systems_produce_more role: supports claim: Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track. source: §4.4, Table 4 evidence: Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track. confidence: high relevance: low - claim_id: fine_tuning_a_post_trained_dialogue_tts_model_on_as role: supports claim: Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials. source: §4.4, Figure 4 evidence: Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials. confidence: high relevance: low limitations: - '- Currently limited to 3-minute dialogues with up to 4 speakers; scaling requires extending training corpus. - English speaker similarity (SIM 0.665) lags Mandarin (0.736), attributed to limited English voice diversity in training data — a data rather than architectural limitation. - Trails Mimi on PESQ metrics, likely because Mimi was trained on a massive English-only corpus closely matching LibriSpeech. - Emotion fine-tuning is demonstrated for a single distinctive female voice; generalisation to arbitrary voices and more nuanced emotional transitions is not evaluated. - No ablation of the semantic supervision contribution vs. the lower frame rate independently.' caveats: [] - id: '2509.04667' published_date: "2025-09-04" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - VC architecture: - GAN - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - gan_causal_streaming_vocoders - hybrid_modular_streaming_systems claims: - claim_id: limited_lookahead_in_causal_speech_encoders_substantially_improves_linguistic_content role: supports claim: Limited lookahead in causal speech encoders substantially improves linguistic content preservation with minimal latency penalty compared to purely causal encoders. source: §V.A, §V.B, Tables I–II evidence: Wave+CL accuracy improves from 53.16% at zero lookahead to 78.99% at 140ms, while end-to-end latency increases from 84.3ms to 203ms; extending to 280ms adds only 0.7pp accuracy with 120ms additional delay. confidence: high relevance: high - claim_id: better_content_encoding_in_streaming_anonymization_introduces_a_fundamental_tension role: complicates claim: 'Better content encoding in streaming anonymization introduces a fundamental tension: improved linguistic clarity reduces speaker anonymization strength under adversarial threat models.' source: §V.E, Table III evidence: Adding the contextual layer drops lazy-informed EER from 36.61% to 20.35% at zero lookahead, meaning the cleaner representations are more discriminative for speaker recognition attacks, directly trading anonymization quality for intelligibility. confidence: high relevance: high - claim_id: token_quantization_via_k_means_clustering_achieves_near_chance_speaker role: supports claim: Token quantization via k-means clustering achieves near-chance speaker verification performance in streaming anonymization by removing fine-grained speaker cues from content representations. source: §V.C, §V.D, §V.E, Tables IV–V evidence: Applying a 256-centroid k-means bottleneck raises lazy-informed EER from ~12% to ~47% (Wave+CL, 140ms lookahead), at a cost of WER rising from 2.09% to 9.52% and MOS falling from 3.79 to 3.22. confidence: high relevance: high - claim_id: streaming_voice_anonymization_systems_can_approach_offline_pipeline_anonymization_performance role: supports claim: Streaming voice anonymization systems can approach offline-pipeline anonymization performance when evaluated under the lazy-informed threat scenario, while retaining real-time latency. source: §V.F, Table VI evidence: DarkStream achieves 22.68% semi-informed EER in streaming mode (140ms lookahead), matching VoicePrivacy 2024 baselines B3 (26.28%) and B5a (22.09%) that require full-utterance processing. confidence: high relevance: high - claim_id: direct_waveform_synthesis_in_streaming_voice_conversion_maintains_acceptable_naturalness role: complicates claim: Direct waveform synthesis in streaming voice conversion maintains acceptable naturalness without mel-spectrogram intermediate representations, but k-means quantization introduced for privacy causes perceivable quality degradation beyond what objective metrics capture. source: §V.D, Table V evidence: MOS drops from 3.79 (Wave+CL) to 3.22 (Wave+CL+KMeans) with quantization; WER degrades only modestly, indicating that intelligibility metrics underestimate the perceptual impact of quantization artifacts. confidence: high relevance: high limitations: - DarkStream does not explicitly disentangle static speaker traits (accent, age, sex) from dynamic attributes (emotion, speaking style), leaving indirect identity cues potentially intact. The semi-informed EER of 22.68% remains well above chance, indicating meaningful residual linkability for well-resourced adversaries. - 'The privacy/quality trade-off exposed by the quantization ablation is steep: each MOS point recovered (by disabling k-means) costs roughly 30pp EER under the lazy-informed scenario. Systems requiring both high quality and robust anonymization against semi-informed attackers have no current solution in this architecture. Comparison of perceptual quality against the offline VoicePrivacy baselines is not reported, so whether DarkStream''s naturalness advantage over batch-processing pipelines is real remains an open question. Evaluation is limited to English (LibriTTS), and generalization to accented or code-switched speech is untested.' caveats: [] - id: '2509.06502' published_date: "2025-09-08" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: speaker_conditioned_vad_substantially_reduces_false_barge_in_rates_in role: supports claim: Speaker-conditioned VAD substantially reduces false barge-in rates in full-duplex spoken conversational agents compared to standard VAD, at the cost of slightly higher target-speaker response latency. source: §3.1, Table 2 evidence: pVAD achieves a 10.2% false barge-in rate versus 33.4% (LiveKit SileroVAD) and 78.1% (Ten VAD), with T90 of 170 ms versus 140 ms and 90 ms respectively. confidence: high relevance: high - claim_id: compact_task_specific_classifiers_for_semantic_end_of_turn_detection role: supports claim: Compact task-specific classifiers for semantic end-of-turn detection can match parameter-heavy general-purpose language models while imposing negligible latency overhead. source: §3.2, Table 3 evidence: A 170M-parameter BERT-based EoT model achieves 96.0%/94.9% average accuracy on Chinese/English, on par with a 7B-parameter model (95.8%/94.4%), while using far fewer parameters and thus adding minimal latency to the pipeline. confidence: high relevance: high - claim_id: semi_cascaded_architectures_that_replace_the_asr_llm_cascade_with role: supports claim: Semi-cascaded architectures that replace the ASR+LLM cascade with a single AudioLLM offer paralinguistic awareness and reduced error propagation while maintaining competitive end-to-end latency. source: §2.3.2 evidence: The semi-cascaded variant conditioned on user speech emotion achieves a simpler pipeline with potential latency benefits, mitigates ASR error propagation, and enables paralinguistic-consistent TTS synthesis, without sacrificing the modularity needed for independent component optimisation. confidence: high relevance: high - claim_id: end_to_end_speech_to_speech_models_face_training_and role: complicates claim: End-to-end speech-to-speech models face training and architectural challenges that limit their use as a foundation for full-duplex interaction systems. source: §1 evidence: The authors identify catastrophic forgetting in LLM backbones and token length mismatch between text and speech sequences as key drawbacks of end-to-end approaches, motivating the pluggable modular controller design. confidence: high relevance: high - claim_id: system_level_evaluation_of_full_duplex_voice_interaction_requires_metrics role: supports claim: System-level evaluation of full-duplex voice interaction requires metrics beyond component-level quality scores, covering barge-in accuracy, semantic end-of-turn detection, and end-to-end latency jointly. source: §3 evidence: FireRedChat introduces three interaction-level metrics (barge-in T90, false barge-in rate, and end-to-first-response P50/P95) that capture naturalness, control accuracy, and efficiency independently of individual ASR, LLM, or TTS scores. confidence: high relevance: high limitations: - All evaluation uses proprietary test data for barge-in and latency experiments. The barge-in test set is 1,000 utterances per language constructed by the authors, and latency is measured on 25 synthetic utterances. External reproducibility of these results is not possible. - Latency remains above industrial-grade systems (DouBao) by 0.3 s (P50) and 0.6 s (P95). The authors identify three concrete engineering improvements, streaming ASR output, streaming TTS input, and reduced VAD chunk size, as a path to closing this gap, but these are not demonstrated. The semi-cascaded pipeline's paralinguistic advantages are described qualitatively; no objective metric (MOS, emotion accuracy) quantifies the improvement in response coherence or emotional consistency. The AudioLLM and FireRedTTS-2 components are in-house proprietary models not released, limiting direct reproducibility of the semi-cascaded results. caveats: [] - id: '2509.09174' published_date: "2025-09-11" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: decoupling_semantic_training_objectives_from_acoustic_token_prediction_substantially_reduces role: supports claim: Decoupling semantic training objectives from acoustic token prediction substantially reduces knowledge degradation in speech-to-speech LLMs. source: §5.1, Table 4 evidence: EchoX's Echo training, which generates speech targets from the model's own semantic hidden states, raises average QA accuracy from 24.3 (T2C without Echo) to 37.1 (EchoX-3B) on the same data, compared to 12.8 for direct interleaved training. confidence: high relevance: medium - claim_id: unit_based_speech_token_compression_via_language_model_segmentation_improves role: supports claim: Unit-based speech token compression via language-model segmentation improves downstream accuracy and reduces sequence length without sacrificing audio quality. source: §5.3, Table 5, Figure 7 evidence: Unit language achieves 4.57 length ratio vs. 9.31 for raw units while improving accuracy on all three QA benchmarks and maintaining comparable audio quality in spectral comparison. confidence: high relevance: medium - claim_id: streaming_inference_in_speech_llms_can_be_achieved_with_minimal role: supports claim: Streaming inference in speech LLMs can be achieved with minimal accuracy degradation when the segmentation boundary is determined by semantic similarity rather than fixed length. source: §5.4, Table 6 evidence: EchoX's cosine-similarity trigger reduces first-token latency from 138 to 27 tokens at 3B scale with less than 1.5 percentage points of accuracy drop on any benchmark. confidence: high relevance: high - claim_id: training_data_efficiency_in_speech_llms_may_depend_more_on role: complicates claim: Training data efficiency in speech LLMs may depend more on the training paradigm than on data volume. source: §4.2, Table 2 evidence: EchoX achieves competitive performance on spoken QA against models trained on millions of hours using only approximately 6,200 hours, but this result holds specifically for factual QA and has not been tested on broader spoken dialogue tasks. confidence: high relevance: medium - claim_id: speech_naturalness_and_response_helpfulness_are_not_jointly_optimised_by role: complicates claim: Speech naturalness and response helpfulness are not jointly optimised by the same training signal in speech-to-speech LLMs. source: §Appendix C, Figure 8 evidence: Human evaluation shows EchoX wins clearly on helpfulness but performs only competitively on naturalness, reflecting a training objective focused on semantic correctness rather than prosodic quality. confidence: high relevance: medium limitations: - Speech quality is assessed only via brief spectral comparison (Figure 7) and a 5-rater human study. No perceptual quality metric (MOS, DNSMOS) or automatic speech recognition accuracy on generated audio is reported as a primary evaluation result, making it difficult to characterise the system's output quality independently of QA accuracy. - The evaluation benchmarks are limited to factual knowledge QA (Llama Questions, Web Questions, TriviaQA). It is unclear whether Echo training retains its advantage on open-ended dialogue, instruction following, or longer-form conversational tasks. The human evaluation was conducted with only five raters on one dataset (AlpacaEval), which limits statistical confidence in the naturalness comparison. The model trains on synthesised assistant audio from GPT-SoVITS, which may introduce a fixed timbre bias and limit voice diversity. The streaming threshold and window size are fixed hyperparameters with no ablation reported on their sensitivity. caveats: [] - id: '2509.13667' published_date: "2025-09-17" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - GAN relevance: medium evidence_role: - architecture_variant current_role: active_evidence method_family: - gan_causal_streaming_vocoders claims: - claim_id: knowledge_distillation_from_a_non_causal_teacher_can_substantially_recover role: supports claim: Knowledge distillation from a non-causal teacher can substantially recover the quality degradation introduced by causal convolutions in GAN-based neural vocoders. source: §V-A, §V-B, Table I, Table II evidence: DLL-APNet with full distillation reaches UTMOS 3.98, compared to 3.90 for causal APNet2 trained without distillation and 4.00 for non-causal APNet2; distillation depth ablation shows monotonic improvement from 3.73 (no blocks) to 3.98 (all 8 blocks) on VCTK unseen speakers. confidence: high relevance: high - claim_id: causal_adaptation_of_gan_based_vocoders_is_not_universally_applicable role: complicates claim: Causal adaptation of GAN-based vocoders is not universally applicable across all architectures. source: §V-A evidence: The causal variant of BigVGAN failed to converge during training and could not produce intelligible speech under the same causal substitution procedure applied to HiFi-GAN, iSTFTNet, APNet2, and Vocos, all of which converged. confidence: high relevance: high - claim_id: causal_constraints_in_neural_vocoders_disproportionately_degrade_pitch_accuracy_relative role: complicates claim: Causal constraints in neural vocoders disproportionately degrade pitch accuracy relative to spectral quality. source: §V-A, Table I evidence: 'Replacing standard convolutions with causal counterparts caused substantially larger relative degradations in F0-RMSE and V/UV error than in MCD and LAS-RMSE across all causal baseline vocoders on VCTK (e.g., causal HiFi-GAN: F0-RMSE 58.06 vs. 31.61 non-causal; causal iSTFTNet: F0-RMSE 54.43 vs. 32.87 non-causal).' confidence: high relevance: high - claim_id: all_frame_level_vocoders_that_predict_spectral_features_before_waveform role: supports claim: All-frame-level vocoders that predict spectral features before waveform reconstruction are more resilient to causal adaptation than waveform-upsampling vocoders. source: §V-A, Table I evidence: Under the same causal substitution procedure on VCTK, causal APNet2 (UTMOS 3.90), causal Vocos (3.87), and DLL-APNet (3.98) outperform causal HiFi-GAN (3.88) and causal iSTFTNet (3.75), and computational cost (GFLOPS) is unchanged by causality conversion in all cases. confidence: high relevance: high limitations: - Evaluation is entirely objective; no human listening test is reported. All perceptual quality claims rest on UTMOS (an automated MOS predictor), which may not fully capture quality degradation under causality constraints or differences between causal and non-causal models. - Evaluation uses a single English dataset (VCTK at 16 kHz) with 8 unseen speakers, so generalisation to other languages, higher sampling rates, or diverse speaking conditions is untested. Model size (parameter count) is not reported. The comparison against causal BigVGAN is limited to reporting convergence failure, with no diagnostic on why the architecture is more fragile to causal substitution. Future work is directed at reducing model size and computational overhead. caveats: [] - id: '2509.13670' published_date: "2025-09-17" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - gan_causal_streaming_vocoders claims: - claim_id: knowledge_distillation_from_a_non_causal_high_complexity_teacher_can role: supports claim: Knowledge distillation from a non-causal, high-complexity teacher can effectively recover reconstruction quality degraded by model causalization and channel pruning in low-latency streamable neural codecs. source: §IV.A, Table I evidence: StreamCodec2 (NH→CL direct KD) achieves PESQ 2.744 and ViSQOL 4.313 vs. 2.650 and 4.290 for the undistilled student at identical 910 MFLOPs and 5.4 M parameters, with all gains p < 0.01. confidence: high relevance: high - claim_id: multi_stage_distillation_pipelines_do_not_provide_additive_quality_gains role: complicates claim: Multi-stage distillation pipelines do not provide additive quality gains over direct teacher-to-student distillation in neural codec compression. source: §IV.A, Table I evidence: Both indirect distillation schemes (NH→CH→CL and NH→NL→CL) underperform direct distillation across all objective metrics, with ViSQOL 4.294 and 4.305 vs. 4.313 for direct, suggesting that intermediate steps dilute rather than refine the knowledge transferred. confidence: high relevance: low - claim_id: fully_causal_neural_codec_architectures_incur_a_meaningful_quality_penalty role: complicates claim: Fully causal neural codec architectures incur a meaningful quality penalty relative to non-causal counterparts at the same bitrate, which knowledge distillation only partially closes. source: §IV.A, Table I evidence: Even with the best distillation strategy, StreamCodec2 (NH→CL) scores PESQ 2.744 vs. the teacher's 3.132 and ViSQOL 4.313 vs. 4.463, leaving a gap that reflects the fundamental constraint of causal-only processing. confidence: high relevance: high - claim_id: distillation_loss_weighting_in_codec_training_requires_careful_calibration_excessively role: refines claim: Distillation loss weighting in codec training requires careful calibration; excessively large weights degrade reconstruction quality by shifting the learning objective toward teacher imitation. source: §IV.B, Figure 3 evidence: ViSQOL for StreamCodec2 (NH→CL) peaks at lambda_KD = 0.01 and declines for weights above this value (tested at 0.002, 0.005, 0.01, 0.02, 0.05), consistent with the reconstruction objective being displaced by over-fitting the teacher's intermediate representations. confidence: high relevance: low limitations: - The evaluation uses only objective metrics (LSD, STOI, PESQ, ViSQOL); it is unclear whether the statistically significant gains over the undistilled student are perceptually meaningful. Comparisons are limited to the authors' own student and teacher variants with no benchmarking against published competing streamable codecs (SoundStream, EnCodec, or others) under matched latency and bitrate conditions, making it difficult to assess where StreamCodec2 stands in the broader landscape. Experiments use a single dataset (LibriTTS at 16 kHz), leaving generalisation to other languages, domains, or sampling rates unconfirmed. Future directions noted by the authors include improving reconstruction quality further and evaluating on additional audio datasets. caveats: [] - id: '2509.14784' published_date: "2025-09-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: aligning_autoregressive_decoder_representations_with_pre_trained_asr_encoder_semantics role: supports claim: Aligning autoregressive decoder representations with pre-trained ASR encoder semantics improves content consistency in continuous-feature TTS. source: §2.3, §3.2, Table 1, Table 2 evidence: Representation alignment reduces WER 6.3 to 5.3 on LibriTTS ablation and produces 25%, 40%, and 30% relative CER/WER reductions on seed-tts-eval test-zh, test-en, and test-hard at 170K-hour scale. Training convergence accelerates by over 3.3x. confidence: high relevance: low - claim_id: using_the_target_acoustic_domain_mel_spectrogram_as_an_intermediate role: contradicts claim: Using the target acoustic domain (mel-spectrogram) as an intermediate alignment target helps continuous AR TTS systems. source: §3.2, Table 1 evidence: Aligning transformer decoder outputs to mel-spectrograms (Exp 2, Table 1) raises WER from 6.3 to 6.7 and degrades SS1/SS2, while alignment to SenseVoice-Large ASR representations (Exp 1) reduces WER to 5.3. confidence: high relevance: low - claim_id: continuous_feature_ar_tts_achieves_competitive_content_accuracy_but_lags role: complicates claim: Continuous-feature AR TTS achieves competitive content accuracy but lags behind discrete-token systems on speaker similarity in zero-shot voice cloning. source: §3.3, Table 2 evidence: MELA-TTS achieves best CER on seed-tts-eval test-zh but lower SS1 (0.59 vs. F5-TTS 0.65) and SS2 (0.68 vs. F5-TTS 0.74) on test-en; attributed to the diffusion module's local context constraint versus discrete-token systems that condition on all prior tokens and the full prompt. confidence: high relevance: low - claim_id: unified_interleaved_training_on_streaming_and_offline_sequences_enables_continuous role: supports claim: Unified interleaved training on streaming and offline sequences enables continuous AR TTS to operate in both modes at comparable quality without separate model variants. source: §2.4, §3.2, §3.3, Table 1, Table 2 evidence: Streaming mode achieves WER 5.0 versus offline 5.2, and SS1/SS2 parity, on LibriTTS. At 170K-hour scale, streaming WER is 2.5 versus offline 2.4 on test-en. confidence: high relevance: high - claim_id: continuous_feature_tts_without_speech_tokenization_scales_effectively_with_training role: supports claim: Continuous-feature TTS without speech tokenization scales effectively with training data, reducing content error rates and approaching discrete-token system performance. source: §3.3, Table 2 evidence: Scaling from LibriTTS (585h) to 170K-hour in-house data reduces English WER from 5.2 to 2.4 and improves SS1 from 0.48 to 0.59, bringing MELA-TTS close to CosyVoice 2.0 (2.6 WER) and CosyVoice 3.0 (2.5 WER). confidence: high relevance: low limitations: - The diffusion module operates on local context only (the immediately adjacent latent vector and the previous mel chunk), preventing it from attending to the full prompt speech. This is identified as the likely structural cause of the consistent speaker similarity gap versus discrete-token systems across test-en and test-hard. The limitation is left as future work with no proposed fix. - Main comparison results rely on a proprietary 170K-hour multilingual dataset that is not publicly available, limiting reproducibility of the top-line numbers. Ablations are conducted on LibriTTS only (585 hours), which may not reflect behavior at full scale. No comparison to DiTAR on test-hard is available, making it unclear whether MELA-TTS's robustness advantage on challenging inputs holds relative to the closest continuous AR baseline. Code and demo are not released with the preprint. caveats: [] - id: '2509.15085' published_date: "2025-09-18" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - flow-matching relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - flow_matching_streaming_decoders claims: - claim_id: flow_matching_vocoders_can_match_or_exceed_gan_vocoders_in role: supports claim: Flow matching vocoders can match or exceed GAN vocoders in intrusive perceptual quality metrics even under a streaming constraint that reduces inference steps. source: §5, Table 1 evidence: MelFlow at N=5 (streaming) achieves PESQ 4.12 vs HiFi-GAN's 2.99 and SI-SDR -8.8 dB vs -29.9 dB on EARS-WHAM v2, while maintaining RTF < 1 on a consumer GPU. confidence: high relevance: high - claim_id: low_step_count_budgets_in_streaming_flow_diffusion_inference_degrade role: complicates claim: Low step-count budgets in streaming flow/diffusion inference degrade spectral fidelity even when perceptual quality remains competitive. source: §5, Table 1 evidence: MelFlow at N=5 shows LSD 1.0 and MCD 3.28 on EARS-WHAM v2, worse than HiFi-GAN (LSD 0.77, MCD 2.41), whereas the non-streaming N=25 variant recovers to LSD 0.7 and MCD 1.7. confidence: high relevance: high - claim_id: streaming_inference_for_multi_step_flow_and_diffusion_models_can role: supports claim: Streaming inference for multi-step flow and diffusion models can be achieved without train-test mismatch by maintaining N independent sets of per-layer causal cache buffers. source: §3.2 evidence: The N-buffer scheme produces outputs identical to offline batch processing (up to floating-point precision) because cached past activations never change, enabling standard parallel training alongside streaming deployment. confidence: high relevance: high - claim_id: high_algorithmic_latency_is_a_significant_barrier_for_diffusion_buffer role: complicates claim: High algorithmic latency is a significant barrier for diffusion-buffer streaming approaches, limiting practical streaming to methods that avoid coupling diffusion time to physical time. source: §2.2 evidence: The Diffusion Buffer achieves sub-second latency but requires at least 20 frames per buffer, giving a minimum effective algorithmic latency of ~340 ms at 16 ms hop size, which MelFlow avoids by using the per-layer cache scheme. confidence: high relevance: high limitations: - MelFlow operates at 16 kHz only; extension to 24 or 44.1 kHz is not evaluated and the step-count constraint tightens at higher sample rates. Spectral detail metrics (LSD, MCD) at N=5 are weaker than HiFi-GAN, suggesting a quality-latency tradeoff that few-step distillation might address (noted as future work). The training dataset is relatively small (~87 hours); behavior on highly diverse or noisy speech is untested. Comparison vocoders operating at 22-24 kHz are evaluated after downsampling to 16 kHz, introducing a resampling artifact that may affect fairness. caveats: [] - id: '2509.15969' published_date: "2025-09-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation claims: - claim_id: fully_autoregressive_streaming_tts_can_achieve_first_packet_latencies_under role: supports claim: Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation. source: §4, Table 1, Table 3 evidence: Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation. confidence: high relevance: high - claim_id: training_data_scale_is_the_primary_driver_of_speaker_similarity role: supports claim: Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive. source: §4, Table 1, Table 2 evidence: Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive. confidence: high relevance: low - claim_id: borrowing_frozen_depth_transformer_weights_from_a_large_scale_pretrained role: supports claim: Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training. source: §3, Table 4 evidence: Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training. confidence: high relevance: medium - claim_id: full_stream_input_processing_introduces_only_marginal_quality_degradation_relative role: supports claim: Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS. source: §4, Table 1 evidence: Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS. confidence: high relevance: high - claim_id: non_autoregressive_flow_matching_decoders_used_in_chunk_based_streaming role: supports claim: Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents. source: §4, Table 3 evidence: Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents. confidence: high relevance: high limitations: - Speaker similarity remains lower than large-scale systems trained on hundreds of thousands of hours, particularly in full-stream mode (SPK-SIM 0.458 on LibriSpeech test-clean vs. 0.587 for CosyVoice2 trained on 167k hours). The system is English-only; multilingual extension is not addressed. Prosody and speaking rate are not explicitly controllable at inference. The use of a frozen CSM depth transformer introduces a dependency on an external large-scale model. Long-form streaming beyond 10-15 second utterances is identified as future work. Performance in adverse or spontaneous speech conditions (outside the training domain) is untested. caveats: [] - id: '2509.16195' published_date: "2025-09-19" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - codec - VC architecture: - hybrid relevance: medium evidence_role: - architecture_variant - control_evidence - evaluation_caution current_role: active_evidence method_family: - hybrid_modular_streaming_systems claims: - claim_id: single_codebook_binary_quantization_at_sub_1_kbps_bitrates_can role: supports claim: Single-codebook binary quantization at sub-1 kbps bitrates can match or exceed multi-codebook streaming codecs on naturalness and intelligibility in speech resynthesis. source: §4.1, Table 2 evidence: FocalCodec-S@50-65k achieves UTMOS 3.85 and dWER 3.68% at 0.80 kbps with a single codebook of 65,536 entries, outperforming Mimi6 (0.83 kbps, 6 codebooks, UTMOS 3.44, dWER 4.77%) on both metrics. confidence: high relevance: high - claim_id: multi_stage_causal_distillation_of_self_supervised_speech_encoders_preserves role: supports claim: Multi-stage causal distillation of self-supervised speech encoders preserves hybrid acoustic-semantic representations for downstream tasks under streaming constraints. source: §4.2, Table 3 evidence: FocalCodec-Stream variants trained via four-stage WavLM distillation outperform acoustic streaming codecs (EnCodec, AudioDec, HILCodec) on ASR, keyword spotting, and intent classification despite operating at lower bitrates; the 65k variant matches or surpasses PAST on all discriminative and generative tasks except ASR. confidence: high relevance: high - claim_id: supervised_domain_specific_fine_tuning_of_hybrid_codecs_achieves_strong role: complicates claim: Supervised domain-specific fine-tuning of hybrid codecs achieves strong in-domain intelligibility at the cost of multilingual generalization. source: §4.1 evidence: PAST, fine-tuned on English data, achieves the lowest English dWER among streaming codecs (4.04%) but degrades severely on multilingual MLS (49.35% dWER), whereas FocalCodec-Stream (trained on English-only Libri-Light but without supervised task fine-tuning) retains competitive multilingual performance (19.88% dWER). confidence: high relevance: medium - claim_id: voice_conversion_quality_in_streaming_codecs_requires_joint_optimization_of role: supports claim: Voice conversion quality in streaming codecs requires joint optimization of intelligibility and speaker fidelity; gains on one metric alone are insufficient for practical use. source: §4.1, Table 2 evidence: Mimi6 achieves competitive speaker similarity (91.3%) in one-shot VC on VCTK but at 110% dWER, while PAST achieves lower dWER (18.28%) at only 68.5% speaker similarity; FocalCodec-S@50-65k is the only streaming codec to simultaneously achieve high values on both (dWER 22.71%, Sim 92.5%). confidence: high relevance: high - claim_id: a_lightweight_refiner_module_bridging_causal_and_full_context_feature role: supports claim: A lightweight refiner module bridging causal and full-context feature distributions substantially improves perceptual quality in distilled streaming codecs. source: §4.3, Table 4 evidence: Ablation on FocalCodec-S@50-4k shows that removing the refiner degrades UTMOS from 3.87 to 3.84 and dWER from 4.39% to 4.65%; omitting Stage 4 fine-tuning (which jointly trains the refiner) has a larger effect, raising dWER to 5.05% and reducing speaker similarity from 96.3% to 95.8%. confidence: high relevance: high limitations: - Training data is limited to English (LibriTTS, Libri-Light), so the multilingual robustness observed on MLS reflects generalization rather than explicit multilingual training. A performance gap with the non-streaming FocalCodec@50 remains across most metrics, particularly in ASR WER (17% vs. 15.33%) and SI error rate (2.18% vs. 0.35%), reflecting the inherent cost of the 80 ms latency budget. Downstream generative tasks (TTS, speech language modeling) are left for future work, so performance of the codec's discrete representations in autoregressive modeling pipelines is not yet demonstrated. - The paper does not provide listening tests or crowd-sourced MOS, relying instead on the automatic UTMOS predictor for naturalness assessment. caveats: [] - id: '2509.17765' published_date: "2025-09-22" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence - control_evidence current_role: influential method_family: - autoregressive_streaming_generation - hybrid_modular_streaming_systems claims: - claim_id: replacing_a_block_wise_diffusion_vocoder_with_a_lightweight_causal role: supports claim: Replacing a block-wise diffusion vocoder with a lightweight causal convolutional decoder, driven by a multi-codebook autoregressive token predictor, can substantially reduce first-packet latency in streaming speech generation without sacrificing competitiveness on content-consistency metrics. source: §2.4, §2.5, Table 1, Table 13 evidence: The Talker's multi-codebook AR scheme plus a 200M-parameter causal ConvNet Code2Wav stage achieves a 234ms end-to-end first-packet latency at 1x concurrency and the lowest reported content-consistency error on SEED test-en (1.39) among all compared zero-shot TTS systems including flow-matching and diffusion-based baselines. confidence: high relevance: high - claim_id: mixing_unimodal_and_cross_modal_training_data_from_the_earliest role: supports claim: Mixing unimodal and cross-modal training data from the earliest stage of pretraining allows a language model to add new input/output modalities without degrading its original text, vision, or audio-specific capabilities relative to matched unimodal baselines. source: §6, Table 16 evidence: A controlled comparison of parameter-matched text-only, vision-only, and Omni models trained on identical corpora, schedules, and compute shows the Omni model matches or exceeds the unimodal baselines on general, math/STEM, coding, and multilingual text benchmarks as well as vision and video benchmarks. confidence: high relevance: medium - claim_id: strong_zero_shot_voice_cloning_performance_in_one_or_two role: complicates claim: Strong zero-shot voice-cloning performance in one or two conditioning languages does not guarantee comparable speaker-similarity performance uniformly across all supported languages. source: §5.2.2, Table 14 evidence: Against MiniMax-Speech and ElevenLabs Multilingual v2 on a 10-language multilingual test set, the system leads by a substantial margin on Chinese, English, and French but reports only "competitive," non-leading, speaker-similarity or content-consistency scores on several other languages such as Portuguese and Russian. confidence: high relevance: medium - claim_id: a_large_scale_purpose_built_supervised_audio_encoder_trained_from role: supports claim: A large-scale, purpose-built supervised audio encoder trained from scratch for a multimodal LLM's audio pathway can outperform reusing a general pretrained ASR encoder (e.g., Whisper) as the perceptual front-end for both speech understanding and downstream speech generation. source: §1, §2.2, Table 6, Table 7 evidence: Replacing the Whisper-based audio encoder from the predecessor system with AuT, trained from scratch on 20 million hours of supervised audio at a 12.5 Hz token rate, is cited as a key driver of gains across ASR, lyric-ASR, and voice-interaction benchmarks relative to Qwen2.5-Omni. confidence: high relevance: medium limitations: - Speech generation quality is evaluated exclusively with automatic metrics (WER/CER-style content consistency, embedding-based speaker similarity, BLEU for translation); no human MOS or listening-test results are reported anywhere in the paper for the Talker's synthesized speech, so claims of "stable, naturalistic speech synthesis" in the conclusion are not directly supported by subjective evidence in this report. - The paper acknowledges suboptimal performance on long-video benchmarks, attributed to limited positional extrapolation and restricted context length, as an explicit architectural limitation left for future work. The reported 234ms first-packet latency is described as "theoretical," measured under a specific vLLM/torch.compile/CUDA-Graph deployment configuration rather than as an end-user-measured figure across arbitrary hardware or network conditions, and latency degrades substantially under higher concurrency (up to 1172ms at 6-way concurrency in the audio case). Several baselines used for comparison (ElevenLabs, MiniMax-Speech, Gemini-2.5-Pro, GPT-4o variants) are closed proprietary systems, so exact reproduction of the comparative numbers is not possible outside the authors' own evaluation pipeline. The non-degradation ablation study, while methodologically rigorous, was run at limited model scales due to computational cost, and the authors explicitly caution that they could not sweep across all model sizes. caveats: [] - id: '2509.23147' published_date: "2025-09-27" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - TTS architecture: [] relevance: medium evidence_role: - architecture_variant - control_evidence current_role: active_evidence method_family: [] claims: - claim_id: neural_ctc_based_forced_aligners_can_achieve_substantial_inference_speedups role: supports claim: Neural CTC-based forced aligners can achieve substantial inference speedups over classical HMM-GMM aligners while retaining competitive recall at practical tolerance levels. source: §3, Table 1, Table 2 evidence: The proposed aligner processes a 10-second clip in ~0.2-0.25s versus roughly 45-60 minutes for a comparable Buckeye clip under MFA, a measured 45-242x total-time speedup, while matching or exceeding MFA's recall at 40ms and 60ms tolerance on both TIMIT and Buckeye. confidence: high relevance: medium - claim_id: aligners_that_predict_both_phoneme_onset_and_offset_boundaries_trade role: complicates claim: Aligners that predict both phoneme onset and offset boundaries trade precision for recall relative to onset-only baselines, because doubling the predicted boundary count increases the chance of spurious matches against onset-only ground truth. source: §4 "End Boundary", Table 2 evidence: At 20ms tolerance on TIMIT, the proposed aligner's onset precision drops to 55.6% versus MFA's 81.2%, even though its recall is comparable, because the model predicts roughly double the boundary count that onset-only annotations penalize as insertions. confidence: high relevance: medium - claim_id: multilingual_phoneme_representations_trained_without_target_language_data_can_generalize role: supports claim: Multilingual phoneme representations trained without target-language data can generalize to unseen-language alignment tasks with only modest performance degradation. source: §4 "Cross-lingual Performance", Table 2 evidence: Aligner variants trained exclusively on non-English multilingual corpora (7-language and 35-language configurations) achieve recall within 1-11 percentage points of an English-only trained variant when evaluated on English test sets, with the 7-language variant nearly matching it (71.0% vs. 71.4% recall at 20ms on TIMIT). confidence: high relevance: medium - claim_id: natural_speech_contains_measurable_silent_gaps_between_phonemes_that_conventional role: refines claim: Natural speech contains measurable silent gaps between phonemes that conventional onset-only forced alignment discards by assuming acoustic contiguity. source: §4 "End Boundary", Table 3 evidence: Explicit dual boundary prediction shows 30-40% of phonemes on both TIMIT and Buckeye have a measurable inter-phoneme gap, with median gap durations of 39-41ms on TIMIT and 39-40ms on Buckeye. confidence: high relevance: medium limitations: - Cross-lingual generalization is validated only on English test sets. The multilingual and universal variants are trained on non-English corpora but are never evaluated on non-English alignment accuracy in this paper, so the claim that the universal phoneme representation "generalizes across languages" rests on training-data composition, not on measured performance in those languages. - The dual boundary formulation roughly doubles predicted boundary count relative to onset-only ground truth annotations, which mechanically depresses precision scores at tight tolerances and complicates direct precision comparison with onset-only aligners. Evaluation is restricted to two English-language benchmarks (TIMIT, a clean read-speech corpus, and Buckeye, spontaneous conversational speech); performance on tonal languages and other typologically distant languages is left to future work, as the authors themselves note. caveats: [] - id: '2509.25131' published_date: "2025-09-29" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: medium evidence_role: - architecture_variant - acceleration_evidence current_role: active_evidence method_family: - autoregressive_streaming_generation - flow_matching_streaming_decoders - hybrid_modular_streaming_systems claims: - claim_id: chunking_text_into_aligned_segments_with_a_short_token_delay role: supports claim: Chunking text into aligned segments with a short token-delay before speech decoding reduces error accumulation in long-form autoregressive speech generation. source: §4.2, Table 6 evidence: Removing chunk-based decoding raises Long-TTS-Eval error rates above those of concurrent long-form TTS baselines, and with it enabled MGM-Omni-TTS-2B achieves EN-hard WER 26.26 versus 42.48-98.61 for CosyVoice2, MOSS-TTSD-v0.5, and Higgs-Audio-v2. confidence: high relevance: low - claim_id: multi_token_parallel_decoding_is_not_restricted_to_rvq_speech role: supports claim: Multi-token parallel decoding is not restricted to RVQ speech tokenizers and can be applied effectively to finite scalar quantization (FSQ) tokenizers. source: §4.2, Table 6 evidence: Increasing parallel decoding size on the CosyVoice2 FSQ tokenizer maintains TTS quality on Seed-TTS-Eval while cutting inference RTF by roughly 3x at parallel size 4. confidence: high relevance: medium - claim_id: increasing_the_parallel_decoding_size_trades_off_synthesis_error_rate role: complicates claim: Increasing the parallel decoding size trades off synthesis error rate against inference speed rather than improving both simultaneously. source: §4.2 evidence: Larger parallel sizes in the ablation slightly raise audio error rate even as they substantially accelerate inference, leading the authors to select a parallel size of 4 as a balance point. confidence: high relevance: medium - claim_id: separating_multimodal_reasoning_from_speech_synthesis_into_distinct_model_components role: refines claim: Separating multimodal reasoning from speech synthesis into distinct model components can improve long-form audio understanding without sacrificing speech generation efficiency. source: §4.1.1, §4.1.3, Figure 5, Table 5b evidence: The dual-track brain-mouth design lets the MLLM handle needle-in-the-haystack audio inputs up to 4,500 seconds while the SpeechLM independently achieves the lowest RTF among compared long-form TTS systems. confidence: high relevance: medium limitations: - 'The long-form evaluation itself is partly self-authored: Long-TTS-Eval is introduced by this paper, and while its construction and normalized-text scoring procedure are documented, results on it cannot yet be cross-checked against independent replications. The comparison in Table 5b is limited to three baseline systems, and the qualitative long-speech examples in the appendix (a classical Chinese poem and a code-switched English-Chinese poem) are illustrative rather than a systematic error analysis. The paper does not report results on emotion or prosody control, nor does it evaluate robustness to reference audio recorded in noisy or far-field conditions. The 32B MLLM variant''s long-form and vision-speech results are mixed relative to the 7B variant (e.g., lower TextVQA-Speech and EN-hard performance context is not directly reported for 32B TTS), suggesting scaling benefits are not uniform across all sub-tasks.' caveats: [] - id: '2509.26542' published_date: "2025-09-30" entry_date: '2026-07-30' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: medium evidence_role: - architecture_variant current_role: influential method_family: [] claims: - claim_id: a_substantial_reasoning_accuracy_gap_exists_between_text_and_voice role: supports claim: A substantial reasoning-accuracy gap exists between text and voice modalities within the same model family, and the gap widens with the number of reasoning steps a task requires. source: §5.1, Table 3 evidence: GPT-5 text reaches 74.8% accuracy on competition math versus 6.1% for GPT-realtime voice (a 68.7-point gap), while the same model pair differs by only ~21 points on factual recall (48.3% vs. 27.4%); macro-averaged across five tracks, text models reach 54.0% vs. 11.3% for voice. confidence: high relevance: medium - claim_id: increasing_inference_time_deliberation_thinking_time_before_a_voice_system role: complicates claim: Increasing inference-time deliberation ("thinking time") before a voice system speaks is not, by itself, sufficient to close a voice-native reasoning gap. source: §5.2 evidence: Enabling Audio Flamingo 3's thinking mode raises time-to-first-response by 530% (2.40s → 15.14s) but overall accuracy decreases from 1.7% to 1.5%, with Context-track accuracy dropping from 3.8% to 1.8%. confidence: high relevance: medium - claim_id: decoupling_reasoning_from_speech_narration_in_a_cascade_architecture_narrows role: complicates claim: Decoupling reasoning from speech narration in a cascade architecture narrows a voice-native reasoning gap but introduces new grounding and logical-consistency failures that prevent full parity with text. source: §5.2, Figure 5 evidence: A cascade pairing a strong text reasoner with a separate narration model still trails the text-only upper bound by 15.7 points on Math (59.1% vs. 74.8%) and collapses to 0.2% accuracy on the Context track, with elevated rates of UNSUPPORTED_FACT and LOGICAL_CONTRADICTION errors attributed to the narration stage. confidence: high relevance: medium - claim_id: speech_output_intelligibility_is_not_the_primary_bottleneck_limiting_reasoning role: supports claim: Speech output intelligibility is not the primary bottleneck limiting reasoning performance in voice-interactive systems. source: §5.2, Table 3 evidence: Word Error Rate across evaluated voice systems ranges from 7.9% (Gemini-2.5-Flash-audio) to 19.8% (Freeze-Omni) with no corresponding difference in reasoning accuracy; all systems in this WER range perform uniformly poorly on VERA's reasoning tracks. confidence: high relevance: medium - claim_id: different_voice_system_architectures_fail_on_reasoning_tasks_in_systematically role: refines claim: Different voice system architectures fail on reasoning tasks in systematically different, architecture-linked ways rather than through a single uniform error pattern. source: §5.3, Figure 6 evidence: Native streaming models (GPT-realtime, Gemini-2.5-Flash-Audio) under-produce NO_FINAL_ANSWER and OFF_TARGET errors, favoring fluent-but-incorrect completions, while the LiveAnswer cascade over-produces UNSUPPORTED_FACT (+0.27), OFF_TARGET (+0.31), and LOGICAL_CONTRADICTION (+0.22) deviations, and end-to-end models (Moshi, Qwen2.5-Omni) show yet another bimodal pattern of extreme over- or under-completion. confidence: high relevance: medium limitations: - 'The study is an observational, cross-architecture diagnostic rather than a controlled experiment: because the 12 evaluated voice systems differ in training data, objectives, and proprietary implementation details, the authors explicitly state they cannot isolate the causal impact of modality alone, only document a consistent cross-model pattern.' - 'Additional caveats: accuracy grading relies on an LLM-as-a-judge protocol (validated at 97.8% human agreement, but with track-dependent reliability as low as 84.3% on Science, where answers require more nuanced interpretation); several evaluated systems are closed commercial APIs, limiting exact reproducibility of their internal behavior even though the benchmark and prompts are released; and the LiveAnswer cascade, while informative as a diagnostic, is a single hand-built configuration rather than a systematic sweep over possible decoupled designs.' caveats: [] claim_clusters: - id: incremental_generation_reduces_first_packet_latency claim: Incremental acoustic or speech-token generation can substantially reduce first-packet and response latency. status: strongly_supported confidence: high supporting_papers: - '2402.08093' - '2409.06666' - '2410.17799' - '2411.13577' - '2412.02612' - 2025.acl-long.912 - 2025.findings-acl.1051 - '2508.15827' - '2509.13670' - '2509.15969' - '2509.17765' contradicting_papers: [] refining_papers: - '2410.17799' caveats: - Reported latency depends on hardware, buffering policy, network assumptions, and whether upstream processing is included. last_reviewed: '2026-07-30' - id: chunk_size_controls_latency_quality_tradeoff claim: Chunk size and read–write policy create a direct trade-off among latency, alignment, and acoustic quality. status: emerging confidence: medium supporting_papers: - '2505.02625' - 2025.findings-acl.1051 contradicting_papers: [] refining_papers: [] caveats: - Optimal chunks are architecture-, language-, and metric-dependent. last_reviewed: '2026-07-30' - id: causality_and_lookahead_trade_latency_for_quality claim: Causal constraints and limited lookahead reduce delay but can degrade quality unless the model is adapted explicitly. status: strongly_supported confidence: high supporting_papers: - '2411.18803' - 2025.acl-demo.37 - '2508.06262' - interspeech-2025-1819 - '2507.14534' - '2509.04667' - '2509.13667' - '2509.13670' - '2509.16195' - '2509.17765' contradicting_papers: [] refining_papers: - '2412.10117' - '2509.13667' - '2509.13670' caveats: - Pitch, phase, and boundary artifacts may respond differently to the same lookahead budget. last_reviewed: '2026-07-30' - id: distillation_recovers_causal_decoder_quality claim: Teacher distillation can recover quality lost when codecs, acoustic decoders, or vocoders are made causal. status: strongly_supported confidence: high supporting_papers: - '2411.18803' - '2509.13667' - '2509.13670' contradicting_papers: [] refining_papers: [] caveats: - The benefit may depend on access to a strong non-causal teacher and matched training data. last_reviewed: '2026-07-30' - id: interleaved_text_speech_generation_supports_streaming claim: Interleaving or jointly generating text and speech tokens enables streaming output while retaining semantic guidance. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2409.06666' - '2412.02612' contradicting_papers: [] refining_papers: [] caveats: - Interleaved decoding can inherit exposure errors from both text and acoustic token streams. last_reviewed: '2026-07-30' - id: parallel_and_speculative_decoding_accelerate_streaming claim: Parallel, multi-token, and speculative decoding reduce serial speech-token generation cost. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2502.11946' - interspeech-2025-2447 - '2509.16195' - '2509.17765' contradicting_papers: [] refining_papers: - '2509.25131' caveats: - Speedups can shrink when verification, codec decoding, or small-batch deployment is included. last_reviewed: '2026-07-30' - id: low_rate_tokenizers_reduce_streaming_sequence_cost claim: Low-frame-rate and compact speech tokenizers shorten sequences and improve streaming efficiency. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2411.18803' - '2412.02612' - '2507.22746' - '2509.16195' contradicting_papers: [] refining_papers: [] caveats: - Aggressive token compression can discard speaker, prosodic, or transient detail. last_reviewed: '2026-07-30' - id: one_model_can_support_streaming_and_offline_modes claim: Shared models can support both streaming and offline speech generation through masks, chunk schedules, or mode conditioning. status: emerging confidence: medium supporting_papers: - '2412.10117' - '2509.14784' contradicting_papers: [] refining_papers: - '2412.10117' caveats: - Joint training may underperform separately optimized models at either extreme. last_reviewed: '2026-07-30' - id: full_duplex_requires_simultaneous_listening_and_speaking claim: Full-duplex speech systems require generation to incorporate an incoming audio stream while speaking. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2410.00037' - '2410.17799' - '2411.13577' - '2508.07375' - interspeech-2025-0874 contradicting_papers: [] refining_papers: [] caveats: - Many systems demonstrate interruption handling rather than unrestricted simultaneous dialogue. last_reviewed: '2026-07-30' - id: turn_taking_and_barge_in_need_explicit_modeling claim: Natural streaming dialogue requires explicit modeling of turn boundaries, interruptions, barge-in, and response timing. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2410.00037' - '2410.11190' - '2410.17799' - '2411.00774' - '2411.13577' - '2508.07375' - interspeech-2025-0874 - interspeech-2025-2564 - '2509.06502' contradicting_papers: [] refining_papers: - interspeech-2025-2564 caveats: - Turn-taking benchmarks often use constrained speakers, prompts, or synthetic overlap. last_reviewed: '2026-07-30' - id: curricula_and_alignment_enable_full_duplex_training claim: Staged curricula, aligned duplex data, and joint component adaptation improve full-duplex training. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2408.16725' - '2410.17799' - '2508.15827' contradicting_papers: [] refining_papers: [] caveats: - Synthetic or weakly aligned interaction data may encode unrealistic timing patterns. last_reviewed: '2026-07-30' - id: modular_and_end_to_end_systems_have_distinct_tradeoffs claim: Modular, cascaded, and end-to-end streaming systems make different trade-offs among latency, semantics, controllability, and paralinguistic fidelity. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2409.06666' - '2411.13577' - '2412.02612' - '2505.02625' - 2025.acl-demo.37 - 2025.acl-long.912 - 2025.findings-acl.1051 - '2507.17527' - interspeech-2025-0874 - '2509.06502' contradicting_papers: [] refining_papers: - '2509.26542' caveats: - Comparisons frequently confound architecture with model scale, data, and proprietary components. last_reviewed: '2026-07-30' - id: speech_reasoning_lags_text_and_cascaded_systems claim: End-to-end speech systems often lag text-mediated or cascaded systems in reasoning and semantic reliability. status: emerging confidence: medium supporting_papers: - '2408.16725' - '2509.26542' contradicting_papers: [] refining_papers: - '2509.26542' caveats: - Reasoning results are sensitive to backbone scale and the amount of speech instruction data. last_reviewed: '2026-07-30' - id: causal_codecs_and_vocoders_can_approach_offline_quality claim: Causal codecs and vocoders can approach offline reconstruction quality with suitable architecture and training. status: strongly_supported confidence: high supporting_papers: - 2025.acl-demo.37 - '2508.06262' - interspeech-2025-1819 - '2507.14534' - '2509.16195' contradicting_papers: [] refining_papers: - '2509.13667' - '2509.13670' caveats: - Near-offline averages can conceal boundary, pitch, and long-context failures. last_reviewed: '2026-07-30' - id: continuous_generators_trade_steps_for_streaming_quality claim: Flow and diffusion streaming decoders trade integration steps, caching, and algorithmic delay against generation quality. status: strongly_supported confidence: high supporting_papers: - '2402.08093' - '2408.16725' - '2410.00037' - '2410.17799' - '2411.00774' - '2411.18803' - '2412.02612' - '2412.10117' - '2502.11946' - iclr-2025-tQ1PmLfPBL - 2025.naacl-srw.6 - '2505.02625' - 2025.acl-demo.37 - 2025.acl-long.817 - 2025.acl-long.912 - 2025.findings-acl.1051 - '2507.22746' - '2508.02849' - '2508.05207' - '2508.06262' - '2508.07375' - interspeech-2025-0669 - interspeech-2025-0874 - interspeech-2025-1084 - interspeech-2025-1819 - '2508.15827' - '2507.14534' - '2509.04667' - '2509.06502' - '2509.09174' - '2509.13667' - '2509.13670' - '2509.14784' - '2509.15085' - '2509.15969' - '2509.16195' - '2509.17765' - '2509.26542' contradicting_papers: [] refining_papers: - '2410.17799' - '2412.10117' - '2509.13667' - '2509.13670' - '2509.15085' caveats: - Nominal step counts do not directly determine end-to-end latency across implementations. last_reviewed: '2026-07-30' - id: streaming_evaluation_requires_interaction_metrics claim: Streaming evaluation should measure first-packet delay, interaction timing, chunk throughput, and cache behavior in addition to batch real-time factor. status: emerging confidence: medium supporting_papers: - '2509.02020' - '2509.17765' contradicting_papers: [] refining_papers: [] caveats: - Latency definitions and measurement boundaries remain inconsistent across papers. last_reviewed: '2026-07-30' - id: multilingual_and_cross_domain_streaming_remain_uneven claim: Multilingual, noisy, unseen-speaker, and cross-domain streaming performance remains uneven. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2408.16725' - '2412.02612' - 2025.acl-long.817 - interspeech-2025-0669 - interspeech-2025-2564 - '2509.06502' - '2509.09174' - '2509.17765' - '2509.23147' contradicting_papers: [] refining_papers: - 2025.acl-demo.37 - interspeech-2025-2564 - '2509.16195' - '2509.17765' caveats: - Coverage is sparse outside high-resource languages and read or assistant-style speech. last_reviewed: '2026-07-30' method_families: - id: autoregressive_streaming_generation name: Autoregressive streaming generation summary: Autoregressive streaming systems emit codec or acoustic tokens incrementally, using interleaving, delayed streams, or parallel heads to control response latency. papers: - '2402.08093' - '2408.02622' - '2408.16725' - '2410.00037' - '2410.11190' - '2410.17799' - '2411.00774' - '2411.13577' - '2412.02612' - '2412.10117' - '2502.11946' - 2025.naacl-srw.6 - '2505.02625' - '2507.12197' - 2025.acl-long.817 - 2025.acl-long.912 - 2025.findings-acl.1051 - '2507.17527' - '2504.10352' - '2508.06262' - '2508.07375' - interspeech-2025-0874 - interspeech-2025-1084 - interspeech-2025-2447 - interspeech-2025-2564 - '2508.15827' - '2508.19098' - '2509.02020' - '2509.09174' - '2509.15969' - '2509.17765' - '2509.25131' open_questions: - How can autoregressive speech generation reduce serial decoding cost without losing semantic or acoustic coherence? - id: hybrid_modular_streaming_systems name: Hybrid and modular streaming systems summary: Hybrid systems combine streaming ASR, language modeling, TTS, codecs, VAD, or turn-taking modules to preserve controllability while reducing end-to-end delay. papers: - '2409.06666' - '2410.00037' - '2411.00774' - '2411.13577' - '2412.10117' - '2502.11946' - 2025.naacl-srw.6 - '2505.02625' - 2025.acl-demo.37 - 2025.acl-long.912 - '2507.22746' - '2504.10352' - interspeech-2025-0669 - interspeech-2025-0874 - interspeech-2025-1289 - '2507.14534' - '2509.04667' - '2509.06502' - '2509.14784' - '2509.16195' - '2509.17765' - '2509.25131' open_questions: - When does a modular or semi-cascaded design outperform end-to-end generation under matched latency and quality budgets? - id: flow_matching_streaming_decoders name: Flow-matching streaming decoders summary: Flow-matching systems use causal masking, cached state, distillation, or few-step generation to produce continuous speech chunks with bounded latency. papers: - '2412.02612' - '2412.10117' - '2502.11946' - iclr-2025-tQ1PmLfPBL - '2505.02625' - 2025.acl-long.912 - interspeech-2025-2449 - '2508.19098' - '2509.15085' - '2509.25131' open_questions: - Can causal flow systems retain offline quality at one-step or very small-chunk streaming budgets? - id: gan_causal_streaming_vocoders name: GAN-based causal streaming vocoders summary: Causal GAN vocoders and codec decoders generate waveform chunks with limited lookahead and one-pass inference. papers: - '2411.18803' - '2507.12197' - '2508.05207' - interspeech-2025-1819 - '2507.14534' - '2509.04667' - '2509.13667' - '2509.13670' open_questions: - Which causalization and distillation techniques best preserve pitch, phase, and naturalness at low lookahead? - id: vae_compressed_streaming_latents name: VAE-based compressed streaming latents summary: VAE and variational codec systems compress speech into short latent streams that support incremental transmission or generation. papers: - '2508.02849' - '2508.19098' open_questions: - How should latent rate, causality, and decoder capacity be balanced for interactive speech generation? reassessment_queue: - id: incremental_generation_reduces_first_packet_latency type: benchmark_validity reason: Latency boundaries and deployment hardware vary substantially across studies. trigger: Matched end-to-end studies report first packet, sustained throughput, and listener quality on common hardware. due: 2026-10 current_assessment: strongly_supported watch_for: - Matched end-to-end latency benchmarks - Hardware-normalized first-packet measurements - id: speech_reasoning_lags_text_and_cascaded_systems type: claim_status reason: Rapid scaling and speech-instruction data growth may narrow the semantic reliability gap. trigger: End-to-end speech models match text-mediated systems on controlled reasoning and conversational evaluations. due: 2026-10 current_assessment: emerging watch_for: - Controlled speech-versus-text reasoning tests - Larger native speech models - id: causal_codecs_and_vocoders_can_approach_offline_quality type: claim_status reason: Aggregate quality may hide boundary and pitch failures at very small lookahead. trigger: Matched subjective tests show parity across multiple codecs, speakers, and chunk sizes. due: 2026-10 current_assessment: strongly_supported watch_for: - Chunk-boundary listener studies - Pitch-sensitive causal decoder tests - id: full_duplex_requires_simultaneous_listening_and_speaking type: definition_scope reason: The term full duplex is applied both to interruption handling and unconstrained simultaneous dialogue. trigger: Benchmarks standardize duplex capability levels and test semantic responses during overlap. due: 2026-10 current_assessment: strongly_supported watch_for: - Full-duplex benchmark definitions - Semantic overlap-response evaluation - id: continuous_generators_trade_steps_for_streaming_quality type: claim_status reason: Distillation and solver improvements are changing the quality–step frontier quickly. trigger: One-step causal flow or diffusion decoders match autoregressive and GAN systems under matched latency. due: 2026-10 current_assessment: strongly_supported watch_for: - One-step causal flow evaluations - Matched algorithmic-latency comparisons open_questions: - Which latency definition and measurement boundary should be standard for streaming speech generation? - How small can chunks and lookahead become before intelligibility, prosody, or speaker identity fails? - Can native speech systems match text-mediated reasoning while preserving paralinguistic information? - What training data and objectives support genuine simultaneous listening, speaking, interruption, and recovery? - How should token rate, codebook structure, and parallel decoding be co-designed for interactive generation? - Which architectures provide the best quality–latency–energy trade-off across multilingual and noisy deployment? trend_notes: - Streaming research is moving from component-level causal TTS toward full speech-in/speech-out interactive systems. - Autoregressive systems increasingly interleave semantic and acoustic streams or generate multiple codec tokens in parallel. - Flow-matching decoders are adopting causal masks, caching, distillation, and few-step inference for bounded-latency synthesis. - Full-duplex work increasingly separates interruption detection, turn timing, and semantic response generation as distinct capabilities. - Evaluation is broadening from batch real-time factor toward first-packet latency, chunk behavior, and interactive timing.