concept: spoken-language-model last_updated: '2026-07-27' paper_count: 127 papers: - id: '2209.03143' published_date: "2022-09-07" entry_date: '2026-07-27' year: 2022 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - historical_context current_role: foundational method_family: - autoregressive_speech_language_models claims: - claim_id: combining_self_supervised_semantic_tokens_with_codec_acoustic role: supports claim: Combining self-supervised semantic tokens with codec acoustic tokens in a hierarchical language model resolves the quality-versus-coherence tension that affects single-tokenizer audio language models. source: §III-B, §III-C, Table I evidence: AudioLM treats audio generation as a language modeling problem over a hybrid vocabulary of discrete tokens, and solves the quality-versus-coherence tension by separating the representation into two complementary token streams. **Semantic tokens** are extracted by applying k-means clustering (K=1024) to the intermediate representations of w2v-BERT XL (0.6B parameters), a self-supervised model trained with contrastive and masked language modeling objectives. confidence: high relevance: high - claim_id: semantic_and_acoustic_tokens_in_speech_carry_complementary role: supports claim: 'Semantic and acoustic tokens in speech carry complementary information: semantic tokens primarily encode linguistic content and prosody, while acoustic tokens primarily encode speaker identity and recording conditions.' source: §IV-C, §IV-D, Tables II–III evidence: This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)* confidence: high relevance: low - claim_id: autoregressive_language_modeling_over_discrete_audio_tokens_can role: supports claim: Autoregressive language modeling over discrete audio tokens can produce speech continuations indistinguishable from real speech to human listeners in an unpaired forced-choice test. source: §IV-G evidence: This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)* confidence: high relevance: high - claim_id: the_semantic_to_acoustic_hierarchical_generation_pattern_transfers role: supports claim: 'The semantic-to-acoustic hierarchical generation pattern transfers across audio domains: a model trained on piano music without symbolic notation also benefits from the two-tier tokenization.' source: §IV-I evidence: This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)* confidence: high relevance: low - claim_id: self_supervised_speech_representations_trained_with_masked_language role: supports claim: Self-supervised speech representations trained with masked language modeling objectives encode sufficient lexical and syntactic information to outperform earlier causal spoken language models on zero-resource linguistic benchmarks. source: §IV-E, Table IV evidence: AudioLM treats audio generation as a language modeling problem over a hybrid vocabulary of discrete tokens, and solves the quality-versus-coherence tension by separating the representation into two complementary token streams. **Semantic tokens** are extracted by applying k-means clustering (K=1024) to the intermediate representations of w2v-BERT XL (0.6B parameters), a self-supervised model trained with contrastive and masked language modeling objectives. confidence: high relevance: high limitations: - 'AudioLM is a continuation model only: it generates continuations of an audio prompt but cannot synthesise speech from a specified transcript.' - The paper explicitly frames TTS integration (encoder-decoder with text conditioning) as future work, which means the WER/CER results reflect acoustic fidelity to a given semantic token sequence, not instruction-following capability. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2301.02111' published_date: "2023-01-05" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - TTS - codec architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - historical_context current_role: foundational method_family: - autoregressive_speech_language_models claims: - claim_id: treating_tts_as_conditional_language_modeling_over_discrete role: supports claim: Treating TTS as conditional language modeling over discrete codec tokens enables zero-shot speaker generalisation as in-context learning, without speaker-specific fine-tuning or engineered speaker encoders. source: §4.1, §5.2 evidence: VALL-E treats TTS as a conditional language modeling problem over discrete acoustic tokens. confidence: high relevance: high - claim_id: training_on_large_scale_semi_supervised_speech_data role: supports claim: Training on large-scale semi-supervised speech data, even with noisy transcriptions and diverse acoustic conditions, yields stronger generalisation to unseen speakers than training on smaller clean corpora. source: §1, §5.2 evidence: On LibriSpeech test-clean (zero-shot, 3-second prompt, 40 unseen speakers), VALL-E achieves SMOS 4.38 ± 0.10 versus YourTTS's 3.45 ± 0.09, a +0.93 gap that closes most of the distance to ground truth (4.50 ± 0.10). confidence: high relevance: low - claim_id: the_hierarchical_structure_of_residual_vector_quantization_supports role: supports claim: The hierarchical structure of residual vector quantization supports a two-stage AR+NAR generation pipeline in which first-codebook tokens carry speaker identity and subsequent codebooks refine fine acoustic detail. source: §4.2 evidence: 'Audio is tokenised using EnCodec, a convolutional encoder-decoder with residual vector quantization (RVQ): the encoder produces embeddings at 75 Hz, and each frame is modelled by eight hierarchical quantizers with 1024 entries each, corresponding to a 6K bitrate.' confidence: high relevance: low - claim_id: speaker_similarity_in_zero_shot_codec_tts_improves role: supports claim: Speaker similarity in zero-shot codec TTS improves monotonically with acoustic prompt length, suggesting that speaker identity modelling does not saturate within a few seconds. source: §5.3, Table 6 evidence: Speaker similarity improves monotonically with longer prompts, and the gap between systems widens when YourTTS is evaluated on its truly unseen 11 speakers (Table 6). confidence: high relevance: low - claim_id: stochastic_sampling_in_autoregressive_codec_generation_introduces_output role: supports claim: Stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation. source: §4.3, §5.4 evidence: The paper presents evidence that stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation. confidence: high relevance: low limitations: - 'Synthesis robustness is a material constraint: the autoregressive first-stage LM exhibits attention alignment failures that cause word deletions, insertions, and repetitions.' - WER on LibriSpeech test-clean is 5.9%, nearly three times the ground-truth rate of 2.2%. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2301.11325' published_date: "2023-01-26" entry_date: '2026-07-27' year: 2023 venue: arXiv task: [] architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: hierarchical_autoregressive_modeling_over_semantic_and_acoustic_tokens role: supports claim: Hierarchical autoregressive modeling over semantic and acoustic tokens enables long-form music generation (several minutes) with temporal coherence at 24 kHz. source: §3.2, §6 evidence: An ablation removing the semantic modeling stage shows that generating acoustic tokens directly from MuLan tokens preserves FAD scores but degrades KLD (1.01 to 1.05) and MCC (0.51 to 0.49), confirming that the semantic intermediary improves text adherence and long-term structure. confidence: high relevance: low - claim_id: a_joint_audio_text_embedding_space_can_substitute role: supports claim: A joint audio-text embedding space can substitute for paired text-audio supervision at training time, allowing generative models to be trained on audio-only corpora and conditioned on text at inference. source: §3.1, §4.2 evidence: 'The system uses three independently pretrained and frozen models: SoundStream (a neural audio codec operating at 24 kHz, 6 kbps, with 12-level RVQ yielding 600 acoustic tokens per second), w2v-BERT (a 600M-parameter self-supervised model whose intermediate representations are discretized into 25 semantic tokens per second via k-means clustering), and MuLan (a contrastive music-text joint embedding model trained on weakly paired music-text data).' confidence: high relevance: low - claim_id: semantic_token_intermediaries_improve_adherence_to_text_descriptions role: supports claim: Semantic token intermediaries improve adherence to text descriptions in hierarchical audio generation beyond what direct acoustic token prediction achieves. source: §5, Table 1 evidence: An ablation removing the semantic modeling stage shows that generating acoustic tokens directly from MuLan tokens preserves FAD scores but degrades KLD (1.01 to 1.05) and MCC (0.51 to 0.49), confirming that the semantic intermediary improves text adherence and long-term structure. confidence: high relevance: low - claim_id: for_text_conditioned_music_generation_perceptual_audio_quality role: supports claim: For text-conditioned music generation, perceptual audio quality (FAD) and semantic text alignment (MCC, KLD) are complementary evaluation axes that do not always correlate with each other. source: §4.4, Table 1 evidence: 'The system uses three independently pretrained and frozen models: SoundStream (a neural audio codec operating at 24 kHz, 6 kbps, with 12-level RVQ yielding 600 acoustic tokens per second), w2v-BERT (a 600M-parameter self-supervised model whose intermediate representations are discretized into 25 semantic tokens per second via k-means clustering), and MuLan (a contrastive music-text joint embedding model trained on weakly paired music-text data).' confidence: high relevance: low - claim_id: large_autoregressive_audio_lms_trained_on_extensive_unlabeled role: supports claim: Large autoregressive audio LMs trained on extensive unlabeled corpora memorize only a small fraction of training sequences exactly, but approximate semantic matches affect a higher proportion of generated outputs under targeted prompting. source: §5, Figure 3 evidence: Memorization analysis shows exact token matches below 0.2% even when prompting with 10 seconds of training audio, and approximate matches (based on optimal transport between token histograms) below 1%, with detected approximate matches largely corresponding to low-entropy, repetitive token sequences. confidence: high relevance: high limitations: - MCC, one of the two primary text-adherence metrics, is computed using MuLan itself, the same model used for conditioning. - This circularity biases the metric in MusicLM's favour relative to baselines that do not use MuLan representations. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2305.09636' published_date: "2023-05-16" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: parallel_iterative_masked_decoding_adapted_to_rvq_structure role: supports claim: Parallel, iterative masked decoding adapted to RVQ structure enables acoustic token generation two orders of magnitude faster than autoregressive generation at matched perceptual quality. source: §4.3, Figure 3 evidence: 'The runtime gain is two orders of magnitude: SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4 versus AudioLM''s acoustic stages which scale prohibitively with sequence length.' confidence: high relevance: low - claim_id: non_autoregressive_rvq_level_by_level_decoding_maintains role: supports claim: Non-autoregressive RVQ-level-by-level decoding maintains better voice and acoustic consistency over long sequences than autoregressive chunk-and-prompt approaches. source: §4.2, Table 1, Figure 2 evidence: On long continuations, SoundStorm maintains higher prompt-to-generation acoustic consistency than AudioLM's slide-and-prompt chunking strategy. confidence: high relevance: low - claim_id: fine_level_rvq_tokens_are_conditionally_independent_given role: supports claim: Fine-level RVQ tokens are conditionally independent given coarser tokens and can be decoded greedily in a single pass without measurable quality loss. source: §3.3, §4.3 evidence: For subsequent levels, greedy decoding in a single pass suffices, exploiting the conditional independence of fine-level tokens given coarser ones. confidence: high relevance: low - claim_id: confidence_based_iterative_decoding_provides_a_meaningful_quality role: supports claim: Confidence-based iterative decoding provides a meaningful quality gain over greedy decoding at the coarsest RVQ level, but additional iterations at finer levels yield no significant improvement for speech. source: §4.3, Figure 4 evidence: An ablation on decoding iterations (Figure 4) shows that 16 iterations in the first RVQ level provides approximately 0.1-0.2 quality gain over greedy decoding; additional iterations on finer levels yield no statistically significant improvement, validating the greedy-at-fine-levels strategy. confidence: high relevance: low - claim_id: coupling_a_text_to_semantic_token_model_with role: supports claim: Coupling a text-to-semantic token model with an efficient acoustic generator enables real-time synthesis of controllable multi-speaker dialogue at 30-second horizons. source: §5 evidence: SoundStorm is a 350M-parameter Conformer trained on the acoustic generation task, taking as conditioning the time-aligned semantic tokens from AudioLM (originating from w2v-BERT quantised with k-means) and producing SoundStream codec tokens (50 frames/second, 12 RVQ levels, codebook size 1024 per level). confidence: high relevance: low limitations: - Evaluation uses a DNSMOS-style estimator rather than human listening tests for audio quality comparisons, and the subjective baseline is carried over from earlier AudioLM papers rather than re-run. - Direct perceptual comparisons between SoundStorm and AudioLM on the same conditions by human raters are not reported. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2305.11000' published_date: "2023-05-18" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - historical_context current_role: foundational method_family: - autoregressive_speech_language_models claims: - claim_id: expanding_an_llm_s_token_vocabulary_with_discretised role: supports claim: Expanding an LLM's token vocabulary with discretised speech units enables a single model to perform both speech comprehension and speech generation without a cascade pipeline. source: §4.1 evidence: Stage 1 (modality-adaptation pre-training) trains LLaMA-13B on LibriLight (60K hours) with a next-token prediction objective over discrete unit sequences, teaching the model to handle the speech modality at scale (96 A100 GPUs, 900 steps). confidence: high relevance: high - claim_id: a_multi_stage_training_curriculum_separating_modality_adaptation role: supports claim: A multi-stage training curriculum, separating modality adaptation, cross-modal instruction tuning, and chain-of-modality alignment, is necessary to acquire reliable cross-modal instruction-following from an LLM backbone. source: §4.2 evidence: Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability. confidence: high relevance: high - claim_id: the_chain_of_modality_pattern_generating_a_text role: supports claim: The chain-of-modality pattern, generating a text intermediate before the speech response, is a practical mechanism for transferring LLM reasoning capability to speech output. source: §3.2, §4.2 evidence: This chain-of-modality pattern serialises cross-modal reasoning as an explicit text intermediate, mitigating the need for direct speech-to-speech reasoning. confidence: high relevance: high - claim_id: large_scale_instruction_dataset_construction_via_gpt_4 role: supports claim: Large-scale instruction dataset construction via GPT-4-generated task descriptions applied to existing ASR corpora is a scalable approach to bootstrapping cross-modal training data. source: §3.1 evidence: Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability. confidence: high relevance: high limitations: - 'The paper provides no quantitative evaluation: no MOS, WER, or speaker similarity scores are reported, and no comparison to cascade baselines is made.' - All results are case studies. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2305.15255' published_date: "2023-05-24" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: spectron_continuous_specs_competitive_with_discrete role: supports claim: Continuous spectrogram representations can support spoken language modelling at quality levels competitive with discrete-token systems, without requiring a separate quantisation stage. source: §4.2.1, §4.2.2, Tables 1-3 evidence: Spectron (350M) reaches log-perplexity 126.08, outperforming AudioLM 3-RVQ (138.96) and 12-RVQ (140.28) on LibriSpeech speech continuation. confidence: high relevance: high - claim_id: spectron_joint_supervision_preserves_world_knowledge role: supports claim: Jointly supervising ASR, text continuation, and speech synthesis within a single decoding pass preserves LLM world knowledge better than separate components or warm-start initialisation alone. source: §4.2.3, Table 4 evidence: Spectron (1B) reaches 22.9% accuracy on LLaMA-Questions versus 0.5-1% for TWIST models that only warm-start from a text LM without joint ASR/synthesis supervision. confidence: high relevance: low - claim_id: spectron_derivative_loss_improves_coherence role: supports claim: Penalising temporal and spectral derivatives of the predicted spectrogram improves semantic coherence in speech continuation more than the basic regression loss alone. source: §4.2.4, Table 5 evidence: Removing the spectrogram derivative loss increases log-perplexity by more than 580 points in the ablation. confidence: medium relevance: low - claim_id: spectron_compact_lm_matches_larger_on_qa role: supports claim: A compact LM (1B parameters) that tightly couples audio encoding with text decoding can match a 7B instruction-tuned spoken LM on zero-shot spoken question answering tasks. source: §4.2.3, Table 4 evidence: Spectron (1B) matches SpeechGPT-7B on LLaMA-Questions (22.9% vs. 21.9%) and WebQuestions (6.1% vs. 6.5%) at one-seventh the parameter count. confidence: high relevance: low limitations: - Frame-by-frame spectrogram decoding at 12.5 ms resolution makes generating long utterances computationally prohibitive and precludes streaming inference. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2306.12925' published_date: "2023-06-22" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - historical_context current_role: foundational method_family: - autoregressive_speech_language_models claims: - claim_id: initializing_a_speech_text_llm_from_a_pretrained role: supports claim: Initializing a speech-text LLM from a pretrained text-only checkpoint substantially outperforms training from scratch at equivalent model scale. source: §5.4.2, Table 6 evidence: At the same 8B model scale, fine-tuning from a pretrained PaLM-2 checkpoint substantially outperforms training the speech-text model from scratch on the reported translation evaluation. confidence: high relevance: low - claim_id: audio_tokenizer_quality_is_a_primary_bottleneck_in role: supports claim: 'Audio tokenizer quality is a primary bottleneck in LLM-based speech generation and understanding: stronger semantic tokenizers yield large downstream gains independent of LM scale.' source: §5.4.3, Table 7 evidence: 'Ablation over tokenizers shows a large gap: USM-v2 tokens reach 30.5 BLEU on CoVoST2 AST while w2v-BERT tokens yield only 15.2 BLEU at the same model scale, establishing audio tokenizer quality as the primary performance bottleneck (§5.4.3, Table 7).' confidence: high relevance: high - claim_id: a_unified_multimodal_vocabulary_that_interleaves_text_and role: supports claim: A unified multimodal vocabulary that interleaves text and audio tokens enables zero-shot speech translation to language pairs not seen during speech training, by inheriting translation capability from text pretraining. source: §5.2, Table 3 evidence: Zero-shot AST on FLEURS shows AudioPaLM-2 achieving 20.7 BLEU on language pairs where only ASR data was seen during training, outperforming Whisper (19.6) despite Whisper having seen speech-to-text translation data for all those languages. confidence: high relevance: high - claim_id: training_on_combined_tasks_that_decompose_complex_speech role: supports claim: Training on combined tasks that decompose complex speech operations into intermediate text steps improves performance over direct end-to-end decoding. source: §5.4.4, Table 8 evidence: For complex tasks, the model can also be trained on combined tasks, emitting intermediate outputs (transcription, then translation, then translated audio) in a single autoregressive pass, which mirrors chain-of-thought prompting and improves performance relative to direct decoding. confidence: high relevance: low - claim_id: voice_identity_preservation_in_cross_lingual_speech_synthesis role: supports claim: Voice identity preservation in cross-lingual speech synthesis can exceed high-quality TTS-based references when an audio LM is conditioned on a short spoken prompt. source: §5.3, Table 4 evidence: Voice conditioning is provided as a 3-second reference clip encoded as both audio tokens and SoundStream tokens, enabling cross-lingual voice transfer. confidence: high relevance: high limitations: - The entire system depends on the quality of the audio tokenizer, which is not released and requires access to Google-internal USM models. - The best-performing configuration (AudioPaLM-2 with USM-v2 tokens) is not reproducible externally; the published ablations use the multilingual w2v-BERT tokenizer as the weakest condition, suggesting that reported performance at USM-v2 quality cannot be independently verified. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2310.00704' published_date: "2023-10-01" entry_date: '2026-07-27' year: 2023 venue: arXiv task: - TTS - VC - singing architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: influential method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: training_a_single_audio_language_model_across_diverse role: supports claim: Training a single audio language model across diverse generation tasks (TTS, voice conversion, sound synthesis, music, singing) produces consistent performance improvements over task-specific models trained on the same data. source: §3.4.1, Appendix C.1, Table 17 evidence: Multi-task training consistently outperforms single-task training across all 11 tasks (Appendix C.1, Table 17), confirming that cross-task sharing is beneficial rather than neutral. confidence: high relevance: high - claim_id: the_autoregressive_property_is_critical_for_audio_generation role: supports claim: 'The autoregressive property is critical for audio generation quality: parallel and delay-based codec prediction approaches yield measurably lower naturalness than fully autoregressive methods when codec quantization levels are held constant.' source: §3.4.2, Tables 4–5 evidence: With codec quantization levels held constant, fully autoregressive prediction receives higher naturalness ratings than the parallel and delayed prediction variants in the architecture ablation. confidence: high relevance: low - claim_id: hierarchical_factorisation_of_rvq_codec_token_sequences_into role: supports claim: Hierarchical factorisation of RVQ codec token sequences into inter-frame and intra-frame modeling substantially reduces training memory and time relative to flat-sequence autoregressive prediction, with comparable generation quality. source: §2.3, §3.4.2, Table 4 evidence: The multi-scale Transformer matches flat-sequence autoregressive prediction in TTS quality while reducing the memory and training-time burden of modeling RVQ sequences. confidence: high relevance: low - claim_id: pre_training_on_a_broad_multi_task_audio role: supports claim: Pre-training on a broad multi-task audio corpus enables strong adaptation to unseen audio generation tasks via fine-tuning on small datasets, outperforming task-specific models trained from scratch on those tasks. source: §3.3, Appendix B.5–B.8, Table 17 evidence: 'Training proceeds in two stages: joint pretraining on 7 tasks (165K hours), followed by fine-tuning to add 4 new tasks (audio editing, speech dereverberation, instructed TTS, speech editing).' confidence: high relevance: low - claim_id: signal_level_metrics_such_as_pesq_are_poorly role: supports claim: 'Signal-level metrics such as PESQ are poorly suited for evaluating generative audio models: systems achieving higher perceptual MOS scores routinely score lower on PESQ than discriminative baselines.' source: §3.2, §3.4.2, Table 11 evidence: On speech enhancement, UniAudio obtains higher DNSMOS and human MOS than the compared systems while its PESQ remains below discriminative baselines. confidence: high relevance: low limitations: - Model checkpoints are not released due to misuse concerns, limiting reproducibility. - Only code and demos are public. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2402.05755' published_date: "2024-02-08" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: influential method_family: - autoregressive_speech_language_models claims: - claim_id: continuous_pretraining_of_a_text_llm_on_interleaved role: supports claim: Continuous pretraining of a text LLM on interleaved speech and text tokens transfers the text model's few-shot learning and semantic reasoning abilities to the speech modality. source: §4.2, §4.3, Table 4 evidence: Crucially, Spirit LM BASE achieves 36.9% accuracy on MMLU (5-shot, text-only), demonstrating that the text LLM's capabilities are largely preserved despite pretraining on 30B+ speech tokens. confidence: high relevance: high - claim_id: word_level_interleaving_of_speech_and_text_during role: supports claim: Word-level interleaving of speech and text during training is more effective than parallel ASR/TTS training or speech-only fine-tuning for cross-modal semantic understanding. source: §4.2, Table 6 evidence: 'Ablations confirm that interleaving is the critical training ingredient: removing interleaved data while keeping speech and text separately drops cross-modal StoryCloze performance by more than 10 points.' confidence: high relevance: high - claim_id: expressive_speech_properties_sentiment_pitch_style_can_be role: supports claim: Expressive speech properties (sentiment, pitch, style) can be modeled in a language model through discrete token streams that supplement phonetic tokens, enabling cross-modal sentiment preservation. source: §5, Table 3 evidence: 'The Expressive version supplements HuBERT tokens with two additional token streams: pitch tokens from a VQ-VAE trained on F0 (64-unit codebook at 12.5 Hz), and style tokens from speaker-normalized SpeechProp features clustered into 100 units at 1 Hz.' confidence: high relevance: high - claim_id: adding_expressive_speech_tokens_to_a_speech_lm role: supports claim: Adding expressive speech tokens to a speech LM improves expressivity at the cost of moderate degradation in lexical and grammatical speech understanding. source: §4.2, Table 4 evidence: On standard speech comprehension benchmarks, Spirit LM Base outperforms prior speech-only LMs (GSLM, TWIST, AudioLM) on StoryCloze, which requires high-level semantic understanding, while remaining competitive on WUGGY (lexical) and BLIMP (grammatical). confidence: high relevance: high - claim_id: cascade_speech_pipelines_remain_substantially_stronger_than_end role: supports claim: Cascade speech pipelines remain substantially stronger than end-to-end unified models on task-specific metrics such as ASR WER and TTS intelligibility at equivalent model scale. source: §4.3, Table 5 evidence: Adding explicit ASR+TTS parallel training data to the recipe dramatically improves these numbers (6.0% WER, 6.7% CER), showing the base interleaved model is not optimized for task-specific generation. confidence: high relevance: high limitations: - Spirit LM's vocoder is conditioned on only 4 speaker voices from the Expresso dataset, which severely constrains the diversity and quality of synthesized speech; the model cannot generalize to arbitrary target speakers at inference without retraining the vocoder. - The STSP benchmark is evaluated using fine-tuned automatic classifiers rather than human listeners, which may not capture perceptual sentiment fidelity accurately. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2402.13236' published_date: "2024-02-20" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - TTS - SCA - codec architecture: [] relevance: high evidence_role: - core_evidence - infrastructure - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: residual_vector_quantisation_is_the_dominant_quantisation_strategy role: supports claim: Residual vector quantisation is the dominant quantisation strategy across neural audio codec models, with variation concentrated in discriminator design, bitrate, and semantic token integration rather than in the core compression mechanism. source: §II.A, Table II evidence: 'Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion).' confidence: high relevance: low - claim_id: codec_based_audio_language_models_increasingly_target_multi role: supports claim: Codec-based audio language models increasingly target multi-task coverage rather than single-task specialisation, with several systems spanning TTS, voice conversion, speech editing, speech enhancement, and translation in a single framework. source: §III.B, Table III evidence: 'Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion).' confidence: high relevance: high - claim_id: integrating_semantic_tokens_from_self_supervised_speech_representations role: supports claim: Integrating semantic tokens from self-supervised speech representations into the codec quantisation process improves audio quality at low bitrates, with HuBERT-guided RVQ being the most common approach. source: §II.B evidence: 'Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion).' confidence: high relevance: low - claim_id: discrete_units_derived_from_self_supervised_representations_enable role: supports claim: Discrete units derived from self-supervised representations enable textless speech language modelling but sacrifice speaker and paralinguistic information relative to codec-based approaches. source: §III.A evidence: 'Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion).' confidence: high relevance: high limitations: - The survey covers only open-source codec models and does not include proprietary codecs used in industry systems. - 'Evaluation methodology is not addressed: the paper does not compare codecs on shared benchmarks or report reproduction numbers, making it difficult to assess quality claims from the original papers in a unified way.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2408.02622' published_date: "2024-08-05" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: influential method_family: - autoregressive_speech_language_models claims: - claim_id: full_duplex_speech_generation_requires_that_the_listening role: supports claim: Full-duplex speech generation requires that the listening channel information be injected at intermediate representation layers, not at the input or output level, to avoid degrading speech generation quality. source: §6.2, Table 2 evidence: Early fusion degrades severely (WER 33.56%), likely because fusing at the input embedding level conflates speaking and listening signals. confidence: high relevance: high - claim_id: a_single_layer_discrete_token_autoregressive_tts_backbone role: supports claim: A single-layer discrete token autoregressive TTS backbone can integrate real-time audio input streams with minimal WER degradation relative to a no-listening baseline in controlled conditions. source: §6.2, Table 2 evidence: At each autoregressive step, the model conditions its next-token prediction on both the speaking channel history and the real-time listening channel embeddings up to that point. confidence: high relevance: low - claim_id: robustness_to_noise_and_sensitivity_to_unseen_speaker role: supports claim: Robustness to noise and sensitivity to unseen speaker interruptions are distinct challenges in full-duplex SLMs, and voice-based generalisation introduces significantly higher error rates than command-based triggering. source: §6.2, Table 3 evidence: The higher error rates reflect the diversity of interruption commands and the speaker independence test condition. confidence: high relevance: high - claim_id: joint_fine_tuning_of_both_the_speech_generation role: supports claim: Joint fine-tuning of both the speech generation backbone and the streaming SSL encoder is necessary for full-duplex models to reach peak interactive capability; freezing either component degrades turn-taking recall. source: §6.3, Table 4 evidence: Ablation results show that fine-tuning both the TTS backbone and SSL encoder jointly (rather than freezing either) is essential for peak performance. confidence: high relevance: high limitations: - 'The system produces speech tokens but not semantic speech responses: it stops speaking when interrupted but does not generate a contextually appropriate spoken reply.' - The paper evaluates TTS output and turn-taking accuracy, not full dialogue capability. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2408.16532' published_date: "2024-08-29" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - codec architecture: - GAN - VAE relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - adversarial_codec_speech_models - variational_latent_speech_models claims: - claim_id: a_single_large_codebook_quantizer_can_achieve_higher role: supports claim: A single large-codebook quantizer can achieve higher perceptual reconstruction quality than multi-quantizer residual VQ systems at substantially higher bitrates, when supported by a strong decoder. source: §4.2, Table 1, Table 2 evidence: When comparing models constrained to a single quantizer, WavTokenizer (40 or 75 tokens) substantially outperforms DAC's 1-quantizer variant (100 tokens) across all metrics, where DAC at 1 kbps achieves only UTMOS 1.49. confidence: high relevance: low - claim_id: codec_tokens_produced_by_a_single_quantizer_enable role: supports claim: Codec tokens produced by a single quantizer enable better downstream autoregressive speech generation than tokens from multi-quantizer codecs, as measured by intelligibility and speaker similarity. source: §4.2, Appendix I, Table 12 evidence: In downstream TTS evaluation (using an autoregressive language model backbone following the MusicGen/ParlerTTS paradigm on LibriTTS), WavTokenizer-based synthesis achieves WER 5.1% and speaker similarity 0.61, versus 6.9% WER and 0.59 SPK-SIM for DAC at 9 kbps (Table 12), demonstrating that the single-quantizer representations transfer better to generation tasks despite lower bitrate. confidence: high relevance: high - claim_id: incorporating_attention_mechanisms_in_the_codec_decoder_and role: supports claim: Incorporating attention mechanisms in the codec decoder and extending the training context window improves the semantic richness of discrete audio tokens without requiring distillation from a semantic model. source: §3.3, §4.3, Table 9 evidence: This improvement is achieved without distillation from a semantic model, relying instead on the attention decoder and extended context window. confidence: high relevance: high - claim_id: expanding_the_vq_codebook_space_beyond_the_conventional role: supports claim: Expanding the VQ codebook space beyond the conventional 1024 entries improves reconstruction quality under extreme compression, but excessively large codebooks reduce codebook utilisation and yield diminishing returns. source: §3.2, §4.3, Table 5 evidence: Standard codebooks of size 2^10 are insufficient to represent the full speech space under single-quantizer constraints; WavTokenizer uses 4096 entries (2^12) as the default after empirical analysis showing that codebook usage concentrates below 2^12 even at large training scale. confidence: high relevance: low - claim_id: inverse_fourier_transform_decoding_significantly_outperforms_mirrored_transposed role: supports claim: Inverse Fourier transform decoding significantly outperforms mirrored transposed-convolution upsampling in high-compression codec settings. source: §3.3, §4.3, Table 7 evidence: The decoder replaces the conventional transposed-convolution upsampling stack with an inverse Fourier transform structure borrowed from Vocos. confidence: high relevance: low limitations: - The downstream TTS evaluation uses only LibriTTS (~960 hours) and a single model configuration; the claimed advantages of WavTokenizer over multi-quantizer codecs in generative modelling have not been validated at the scale of systems like VALL-E or Voicebox, where the codec is a fixed component in a much larger pipeline. - Acoustic codecs including WavTokenizer lack ASR-level speech understanding capabilities; the authors note this constrains use in unified multimodal understanding-and-generation frameworks (GPT-4o paradigm). caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2408.16725' published_date: "2024-08-29" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: influential method_family: - autoregressive_speech_language_models claims: - claim_id: simultaneous_text_and_audio_generation_conditioned_on_text role: supports claim: Simultaneous text and audio generation, conditioned on text tokens generated in parallel, enables streaming speech output without the latency penalty of sequential text-then-audio decoding. source: §3.2 evidence: This avoids the latency penalty of sequential text-then-audio generation while retaining the reasoning structure of the text modality. confidence: high relevance: low - claim_id: audio_reasoning_quality_in_end_to_end_speech role: supports claim: Audio reasoning quality in end-to-end speech LMs lags behind text reasoning quality when trained on similar data volumes, and batch inference strategies can partially bridge this gap. source: §3.2, §4.4 evidence: A second inference strategy, batch parallel decoding, addresses the observed gap between text and audio reasoning quality. confidence: high relevance: high - claim_id: a_three_stage_adapter_based_training_curriculum_can role: supports claim: A three-stage adapter-based training curriculum can integrate speech input and output into a frozen language model backbone with minimal degradation to text capabilities. source: §3.3 evidence: 'Mini-Omni builds on Qwen2-0.5B as its backbone language model and extends it with two lightweight adapters: a Whisper-small encoder adapter for speech input and a TTS adapter consisting of six additional transformer blocks for speech output.' confidence: high relevance: high - claim_id: multi_codebook_audio_codecs_with_high_token_rates role: supports claim: Multi-codebook audio codecs with high token rates require parallel decoding schemes to maintain practical streaming throughput in autoregressive speech LMs. source: §3.1, §3.2 evidence: The high codebook depth makes flattened token sequences impractically long, so the model instead uses parallel decoding across all eight heads (one text head plus seven SNAC codebook heads) with a one-step delay between adjacent layers. confidence: high relevance: high limitations: - The paper reports no MOS or naturalness metrics for speech output, making it impossible to quantitatively compare audio quality against TTS or SCA baselines. - The claim that quality is "on par with common TTS systems" is unsupported. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2409.06666' published_date: "2024-09-10" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: end_to_end_speech_llms_with_parallel_text role: supports claim: End-to-end speech LLMs with parallel text and speech generation can achieve lower response latency than cascaded ASR-LLM-TTS pipelines without sacrificing prosody coherence under low-latency streaming conditions. source: §4.5 evidence: Critically, unlike cascaded systems where reducing the word chunk size degrades speech rate and naturalness, LLaMA-Omni maintains stable prosody across latency settings because streaming unit generation is end-to-end — only the final vocoder step is cascaded. confidence: high relevance: high - claim_id: aligning_llm_output_to_speech_interaction_conventions_through role: supports claim: Aligning LLM output to speech interaction conventions through targeted instruction data rewriting substantially improves response style suitability, independently of model architecture. source: §3, §4.4 evidence: 'To provide suitable training data, the authors construct InstructS2S-200K: 200K speech instruction-response pairs derived from Alpaca and UltraChat text data by rewriting instructions for spoken style (via Llama-3-70B-Instruct) and synthesising responses to speech using VITS trained on LJSpeech.' confidence: high relevance: high - claim_id: non_autoregressive_ctc_decoding_from_llm_hidden_states role: supports claim: Non-autoregressive CTC decoding from LLM hidden states enables streaming speech synthesis whose speech rate and naturalness are robust to chunk size variation, unlike word-level streaming TTS cascades. source: §4.5, Table 4 evidence: Critically, unlike cascaded systems where reducing the word chunk size degrades speech rate and naturalness, LLaMA-Omni maintains stable prosody across latency settings because streaming unit generation is end-to-end — only the final vocoder step is cascaded. confidence: high relevance: low - claim_id: training_an_end_to_end_speech_interaction_model role: supports claim: Training an end-to-end speech interaction model on a small, carefully curated speech instruction dataset is sufficient to significantly close the gap with models trained on orders of magnitude more data, provided the LLM backbone is sufficiently capable. source: §4.4, §5 evidence: 'To provide suitable training data, the authors construct InstructS2S-200K: 200K speech instruction-response pairs derived from Alpaca and UltraChat text data by rewriting instructions for spoken style (via Llama-3-70B-Instruct) and synthesising responses to speech using VITS trained on LJSpeech.' confidence: high relevance: high limitations: - The ASR-WER of 10.82% is notably higher than cascaded baselines (3.78% for SALMONN+Orca), reflecting that the speech decoder is trained on only approximately 1K hours of response speech — far below industrial TTS scale. Intelligibility limitations restrict applicability in domains requiring precise spoken content. - The evaluation benchmark (InstructS2S-Eval) is derived from AlpacaEval with math and code questions removed, which skews toward conversational helpfulness and may not represent more demanding speech interaction tasks. The speech encoder relies on Whisper, which is optimised for ASR rather than general speech understanding, potentially limiting response to prosodic or para-linguistic cues in the user's speech. The current architecture does not support full-duplex interaction (interruption, turn-taking) — speech responses are generated after the full instruction is received. The training data is synthesised from text corpora, which may not capture the naturalness and variability of real spoken dialogue. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2410.00037' published_date: "2024-09-17" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: influential method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: eliminating_the_text_bottleneck_in_spoken_dialogue_requires role: supports claim: Eliminating the text bottleneck in spoken dialogue requires modeling acoustic tokens jointly with semantic tokens in a single generative model, as purely semantic approaches cannot capture paralinguistic information or generate in arbitrary voices. source: §3.4, §5.4 evidence: Moshi's RQ-Transformer jointly predicts text, semantic speech, and acoustic speech tokens for both dialogue participants at each timestep. confidence: high relevance: high - claim_id: predicting_time_aligned_text_tokens_as_a_per role: supports claim: Predicting time-aligned text tokens as a per-frame prefix to audio tokens substantially improves the linguistic quality and factual accuracy of speech generated by audio language models, with minimal inference overhead. source: §3.4.4, §5.3, Table 6 evidence: Inner Monologue inserts time-aligned text tokens as a prefix to the semantic token at each frame. confidence: high relevance: high - claim_id: modeling_conversation_as_parallel_autoregressive_streams_for_each role: supports claim: Modeling conversation as parallel autoregressive streams for each speaker, without explicit turn boundaries, enables full-duplex spoken interaction and allows training on naturally overlapping speech. source: §3.4.3, §5.6, Table 9 evidence: The multi-stream design models both sides of the conversation as parallel autoregressive token streams with no explicit turn boundary; Moshi generates silence tokens when the user is speaking. confidence: high relevance: high - claim_id: adversarial_only_training_of_neural_audio_codecs_substantially role: supports claim: Adversarial-only training of neural audio codecs substantially improves subjectively rated audio quality relative to mixed reconstruction-adversarial objectives, despite degrading objective metrics such as VisQOL. source: §3.3, §5.2, Table 4 evidence: Mimi's adversarial-only training raises MUSHRA from 58.8 with mixed losses to 81.0, despite worsening objective reconstruction metrics. confidence: high relevance: low - claim_id: standard_objective_audio_quality_metrics_visqol_mosnet_are role: supports claim: Standard objective audio quality metrics (VisQOL, MOSNet) are unreliable proxies for perceived quality when the training objective changes, making human evaluation indispensable for codec comparison. source: §5.2, §5.8 evidence: Mimi's adversarial-only configuration receives substantially higher MUSHRA despite objective metrics such as VisQOL moving in the opposite direction. confidence: high relevance: low limitations: - Moshi's spoken factual question answering lags substantially behind its Helium text baseline, including 22.8 versus 56.4 on TriviaQA. - Audio training causes factual-knowledge degradation that the available instruction-tuning data does not fully recover. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2410.03751' published_date: "2024-10-01" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA - TTS architecture: [] relevance: high evidence_role: - core_evidence - infrastructure - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: end_to_end_speech_generation_models_avoid_the role: supports claim: End-to-end speech generation models avoid the information loss, latency, and cumulative error introduced by cascaded ASR-LLM-TTS pipelines, but require integrating speech tokenisation and synthesis into a unified training regime. source: §I, §II evidence: 'Text-speech representation alignment is identified as a key training strategy: interleaved token sequences (text and speech tokens mixed in a single stream), multi-sequence parallel generation (simultaneous text and speech token decoding), and cross-modal supervision all appear in the literature with different latency/quality trade-offs.' confidence: high relevance: high - claim_id: semantic_tokenizers_and_acoustic_tokenizers_impose_an_inherent role: supports claim: 'Semantic tokenizers and acoustic tokenizers impose an inherent trade-off: semantic tokens produce coherent content but poor acoustic quality, while acoustic tokens enable high-fidelity reconstruction but risk content inaccuracies.' source: §III-A, §IV-A1 evidence: Semantic tokenizers (HuBERT, wav2vec 2.0, W2v-BERT) excel at content fidelity but produce speech that lacks fine acoustic detail. confidence: high relevance: low - claim_id: initialising_a_speech_lm_from_a_text_pretrained role: supports claim: Initialising a speech LM from a text-pretrained checkpoint accelerates convergence and improves speech understanding, whereas initialisation from image-pretrained checkpoints yields worse results than random initialisation. source: §IV-B1 evidence: 'Key findings from the literature reviewed include: - HuBERT consistently outperforms CPC and wav2vec 2.0 as a semantic tokenizer in speech generation quality (GSLM). - Text-pretrained LM checkpoints yield faster convergence and better speech understanding than cold initialisation; image-pretrained checkpoints perform worse than cold initialisation, confirming that cross-modal transfer requires semantic proximity. - Interleaved speech-text pre-training (as in SPIRIT-LM) demonstrably improves alignment between text and speech representations, measurable through feature similarity visualisations. - Mixed-objective tokenizers (SpeechTokenizer, Mimi) show promise in balancing semantic and acoustic trade-offs, but the area is described as early-stage. - Post-alignment for SpeechLMs (Align-SLM, SpeechAlign) remains underexplored relative to its counterpart in text LMs. - Full-duplex modelling is identified as an open frontier; most evaluated systems still use streaming architectures without true simultaneous bidirectional capability.' confidence: high relevance: high - claim_id: interleaving_speech_and_text_tokens_during_pre_training role: supports claim: Interleaving speech and text tokens during pre-training measurably improves cross-modal representation alignment compared to training on speech tokens alone. source: §IV-B1 evidence: 'Text-speech representation alignment is identified as a key training strategy: interleaved token sequences (text and speech tokens mixed in a single stream), multi-sequence parallel generation (simultaneous text and speech token decoding), and cross-modal supervision all appear in the literature with different latency/quality trade-offs.' confidence: high relevance: high - claim_id: post_alignment_techniques_rlhf_dpo_for_speech_lms role: supports claim: Post-alignment techniques (RLHF, DPO) for speech LMs remain substantially underexplored relative to their established role in text LM development, leaving semantic consistency and acoustic quality gaps in deployed systems. source: §IV-B3, §VII evidence: 'Key findings from the literature reviewed include: - HuBERT consistently outperforms CPC and wav2vec 2.0 as a semantic tokenizer in speech generation quality (GSLM). - Text-pretrained LM checkpoints yield faster convergence and better speech understanding than cold initialisation; image-pretrained checkpoints perform worse than cold initialisation, confirming that cross-modal transfer requires semantic proximity. - Interleaved speech-text pre-training (as in SPIRIT-LM) demonstrably improves alignment between text and speech representations, measurable through feature similarity visualisations. - Mixed-objective tokenizers (SpeechTokenizer, Mimi) show promise in balancing semantic and acoustic trade-offs, but the area is described as early-stage. - Post-alignment for SpeechLMs (Align-SLM, SpeechAlign) remains underexplored relative to its counterpart in text LMs. - Full-duplex modelling is identified as an open frontier; most evaluated systems still use streaming architectures without true simultaneous bidirectional capability.' confidence: high relevance: high limitations: - The survey's arXiv version was submitted in October 2024 and the rapidly evolving SpeechLM landscape means several systems surveyed (notably Moshi, Mini-Omni, Llama-Omni) were still very recent preprints without peer-reviewed evaluation. - The coverage of full-duplex systems and post-alignment techniques is acknowledged by the authors as incomplete. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2410.11190' published_date: "2024-10-15" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: staged_adapter_training_encoder_alignment_before_language_model role: supports claim: Staged adapter training (encoder alignment before language model fine-tuning) enables tri-modal extension of a compact language model with minimal data without catastrophic forgetting of the base model's text capabilities. source: §3.3 evidence: Stage 1 trains only the adapter weights on speech recognition and image captioning data, aligning multimodal feature distributions to the text embedding space without modifying the language model. confidence: high relevance: high - claim_id: using_continuous_encoder_features_whisper_rather_than_discrete role: supports claim: Using continuous encoder features (Whisper) rather than discrete audio tokens for speech input yields more stable and semantically consistent representations, reducing ASR loss instability during training. source: §3.1, "Audio Encoder" evidence: 'The backbone is Qwen2-0.5B, augmented with two pretrained encoders: CLIP ViT-B/32 for vision (producing a 50-element feature sequence) and Whisper-small for audio input (operating as a continuous feature extractor rather than a discrete tokenizer).' confidence: high relevance: high - claim_id: adding_a_third_modality_vision_to_an_audio role: supports claim: Adding a third modality (vision) to an audio-text spoken conversational agent modestly degrades ASR performance, likely due to diluted training data proportion rather than architectural interference. source: §4.4, Table 2 evidence: Mini-Omni2 builds on the Mini-Omni framework by adding a visual modality to an existing audio-text spoken conversational agent. confidence: high relevance: high - claim_id: command_based_semantic_interruption_intent_token_classification_provides role: supports claim: Command-based semantic interruption (intent token classification) provides a viable alternative to VAD-based full-duplex detection, with the advantage of robustness to noise and unrelated background sounds. source: §3.4 evidence: 'The duplex interaction mechanism uses a command-based approach rather than full-duplex VAD: the model receives streaming SNAC-encoded tokens of the user''s audio in real time and emits frame-level `irq` (interrupt) or `n-irq` (no interrupt) state tokens.' confidence: high relevance: high limitations: - 'Evaluation coverage is incomplete: no naturalness MOS, SMOS, or intelligibility metrics for speech output are reported in this version, and vision benchmark results are explicitly deferred.' - Claims about speech quality and vision understanding capability cannot be independently verified from this paper alone. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2410.17196' published_date: "2024-10-22" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - evaluation - SCA architecture: [] relevance: high evidence_role: - core_evidence - evaluation_caution current_role: influential method_family: [] claims: - claim_id: voicebench_pipeline_beats_e2e role: supports claim: Pipeline-based voice assistants (ASR plus LLM) substantially outperform end-to-end audio LLMs on spoken instruction understanding, with the gap persisting even for state-of-the-art proprietary end-to-end systems. source: §3.3, Table 3 evidence: The naive Whisper + LLaMA-3.1-8B pipeline scores 79.1 on the composite VoiceBench score versus 55.7 for the best open-source end-to-end model (DiVA); even GPT-4o-Audio (86.4) trails the Naive-4o pipeline (87.2). confidence: high relevance: high - claim_id: voicebench_e2e_training_degrades_llm_capability role: supports claim: End-to-end audio LLM training can degrade the underlying LLM's text-instruction capabilities, including safety alignment, compared to using the same base model in a pipeline configuration. source: §3.3 evidence: LLaMA-Omni's safety refusal rate drops from 98.5% (text) to 11.4% (speech), and its text-instruction performance falls over 11 points compared to the same base LLM used in a pipeline. confidence: high relevance: low - claim_id: voicebench_mispronunciation_worse_than_disfluency role: refines claim: Mispronunciation causes significantly greater performance degradation in voice assistants than disfluencies or grammatical errors, because it increases effective word error rate rather than merely adding irrelevant context. source: §6.2, Table 5 evidence: Mispronunciation causes an average 20.3% score reduction across models, versus 12.6% for repair disfluencies, while grammatical errors are largely tolerated. confidence: high relevance: high - claim_id: voicebench_clean_synthetic_speech_overestimates_robustness role: contradicts claim: Evaluations using clean synthetic speech substantially overestimate voice assistant robustness relative to real recorded speech, particularly for models with non-standard speech encoders. source: §3.3, Table 4 evidence: Real accent data (SD-QA) is shown to be substantially harder than synthetic accent data for the same models, particularly for low-resource accents such as Philippine and Indian English. confidence: high relevance: high - claim_id: voicebench_encoder_choice_dominates_robustness role: refines claim: The choice of speech encoder determines a voice assistant's robustness to accent and speaker variation more than the choice of base LLM. source: §4.2 evidence: VITA, which uses a non-Whisper CNN+Transformer speech encoder, shows the greatest accent sensitivity among all evaluated systems, a pattern the authors attribute to encoder choice rather than the base LLM. confidence: medium relevance: high limitations: - Evaluates only the text content of responses, not speech output quality; Moshi, the only tested model that natively outputs speech, is scored under the same text-only protocol as text-output models. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2410.17799' published_date: "2024-10-23" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: full_duplex_spoken_dialogue_can_be_achieved_by role: supports claim: Full-duplex spoken dialogue can be achieved by flattening interleaved speech and text token streams into a single autoregressive sequence, without modifying the backbone LLM architecture. source: §3, §3.3.2 evidence: OmniFlatten builds on Qwen2-0.5B as a backbone text LLM and extends it to full-duplex speech conversation through three progressive post-training stages, without modifying the backbone architecture. confidence: high relevance: high - claim_id: progressive_curriculum_training_modality_alignment_followed_by_half role: supports claim: Progressive curriculum training (modality alignment followed by half-duplex, then full-duplex) improves final full-duplex dialogue quality compared to training directly on full-duplex data. source: §4.3, Table 3 evidence: 'Ablations confirm each training stage contributes cumulatively: omitting modality alignment and half-duplex stages both hurt final full-duplex scores.' confidence: high relevance: high - claim_id: eliminating_intermediate_text_output_from_dialogue_models_substantially role: complicates claim: Eliminating intermediate text output from dialogue models substantially reduces response latency but causes a significant drop in semantic coherence, indicating a fundamental trade-off between speed and content quality in speech-to-speech generation. source: §3.3.2, §4.3, Table 3 evidence: The 2-stream model (speech-only output) drops significantly to 2.19/3.06, indicating that eliminating intermediate text harms semantic coherence. confidence: high relevance: high - claim_id: turn_taking_response_latency_in_full_duplex_speech role: supports claim: Turn-taking response latency in full-duplex speech models can be reduced by chunked interleaved sequence training, with practical response times under 200 ms achievable at 0.5B parameter scale. source: §4.3, Table 4 evidence: OmniFlatten builds on Qwen2-0.5B as a backbone text LLM and extends it to full-duplex speech conversation through three progressive post-training stages, without modifying the backbone architecture. confidence: high relevance: high limitations: - The 0.5B backbone is substantially smaller than comparators (LLaMA-Omni 8B, GLM-Voice 9B, Moshi 7B), making LLM-score comparisons in Table 3 not directly attributable to the method alone. The paper acknowledges GLM-Voice results may reflect test-set leakage. Dialogue quality scores remain well below the ground-truth ceiling. - Training data is entirely synthesised from text dialogues via a TTS pipeline; real conversational dynamics (natural prosody, disfluencies, real interruption patterns) are not represented. The model does not handle backchannels from either speaker, a basic feature of natural human conversation. User turn-taking accuracy at 25 tokens remains below 55% for both models evaluated, leaving interruption handling far from reliable. The paper does not report naturalness MOS, making direct quality comparison to TTS-oriented systems difficult. All evaluation uses simulated test data matching the training distribution, raising questions about real-world robustness. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2411.00774' published_date: "2024-11-01" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: freezing_the_llm_backbone_during_speech_modality_alignment role: supports claim: Freezing the LLM backbone during speech-modality alignment reduces the intelligence gap between spoken and text question-answering performance compared to fine-tuned approaches. source: §3.4, Table 3 evidence: 'For speech output: (1) training a single-codebook TiCodec on speech data alone; (2) training the NAR and AR speech decoders on ~3k hours of text-speech pairs using LLM token embeddings as semantic input (LLM embedding layer frozen); (3) prefix kv-cache fine-tuning of the NAR prefix decoder on Q&A data to close the distribution gap between LLM output text tokens and the training text used in stage 2.' confidence: high relevance: low - claim_id: a_three_stage_training_curriculum_using_large_asr role: supports claim: A three-stage training curriculum, using large ASR corpora for encoder pretraining followed by small-scale multi-modal Q&A fine-tuning, is sufficient to achieve competitive spoken dialogue quality without updating backbone LLM parameters. source: §2.2.2, §2.3.2 evidence: 'For speech input: (1) CTC-based ASR pretraining of the speech encoder; (2) connecting the encoder to the frozen LLM via the adapter, trained with ASR-style supervised objectives on 110k hours of Chinese/English speech; (3) prompt-embedding fine-tuning on 60k synthesised multi-round Q&A pairs with all components except trainable prompt embeddings frozen.' confidence: high relevance: high - claim_id: chunk_level_state_classification_integrated_into_the_llm role: supports claim: Chunk-level state classification integrated into the LLM's prefill stage enables duplex interruption detection without requiring a separate monitoring model or additional LLM context. source: §2.4 evidence: Duplex capability is achieved by adding a chunk-level state-prediction classification head to the LLM's output layer during stage-3 training. confidence: high relevance: high - claim_id: decoupling_encoder_and_llm_kv_cache_per_user role: supports claim: Decoupling encoder and LLM KV-cache per user session allows a server-side pool of model replicas to handle concurrent users with chunk-granular scheduling. source: §2.4 evidence: A server-side scheduling strategy keeps encoder and LLM KV-caches isolated per user, allowing any model replica to serve any chunk of any active session without coordination overhead. confidence: high relevance: low limitations: - The spoken Q&A benchmarks used for intelligence comparison (Web Questions, LlaMA Questions, Trivia QA) were synthesised from text using edge-tts rather than collected from real speakers. Results on naturally spoken or noisy input are not reported, limiting generalisability claims about real-world speech understanding. - Speech output quality is evaluated primarily through CER on 1000 utterances using a single speaker, not through subjective MOS ratings or speaker naturalness benchmarks. It is therefore difficult to assess voice quality relative to other systems. The system supports a limited number of output speakers and does not yet support style or voice instruct-following, which the authors flag as future work. Emotion understanding and audio captioning are also deferred to a planned encoder upgrade. The duplex state classifier operates at chunk boundaries (approximately 160-320 ms non-statistical latency), which may be perceptible in fast-paced dialogue. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2411.13577' published_date: "2024-11-15" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - transformer-enc-dec - hybrid relevance: high evidence_role: - core_evidence - infrastructure - evaluation_caution current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - transformer_encoder_decoder_speech_models claims: - claim_id: wavchat_cascade_vs_e2e_tradeoffs role: supports claim: The cascaded and end-to-end paradigms for spoken dialogue impose fundamentally different trade-offs between latency, paralinguistic fidelity, and intelligibility, and the appropriate choice depends on the target interaction scenario. source: §2.2, §2.3 evidence: The survey's taxonomy contrasts cascaded systems' reliance on text as an intermediary against end-to-end systems' direct speech understanding and generation, mapping each to distinct latency/fidelity trade-offs. confidence: medium relevance: high - claim_id: wavchat_semantic_repr_sacrifices_expressiveness role: supports claim: Semantic speech representations offer higher compression rates and better LLM compatibility than acoustic representations, but sacrifice expressiveness, timbre, and style fidelity, necessitating additional vocoders in pipeline-based generation. source: §3.3.1, Table 1 evidence: The survey's representation taxonomy documents that semantic tokens (HuBERT, WavLM) compress more aggressively and integrate better with LLMs, at the cost of requiring a separate vocoder to recover acoustic detail. confidence: medium relevance: low - claim_id: wavchat_full_duplex_requires_causal_architecture role: supports claim: Achieving genuine full-duplex spoken dialogue (simultaneous listening and speaking with interrupt handling) requires architecturally causal models throughout the full pipeline, a constraint that current systems largely satisfy only on the output side. source: §5.1, §5.2 evidence: The survey documents that most systems still use non-streaming encoders (Whisper-based) on the input side even when Moshi and OmniFlatten demonstrate genuine full-duplex output-side capability. confidence: medium relevance: high - claim_id: wavchat_alignment_relies_on_paired_data role: complicates claim: Speech-text modality alignment in current spoken dialogue systems relies heavily on paired data, introducing catastrophic forgetting risk and creating a structural dependency on the availability of labelled speech corpora. source: §4.4.1 evidence: The survey identifies paired text-speech data dependency as a near-universal property of surveyed SDM training pipelines, creating structural forgetting risk. confidence: medium relevance: high - claim_id: wavchat_eval_infrastructure_lags_capability role: complicates claim: 'Evaluation infrastructure for spoken dialogue lags substantially behind system capabilities: interaction, streaming latency, and audio generation are either absent from or severely underrepresented in existing benchmarks.' source: §6.2, §6.3, Table 3 evidence: Mapping eight existing benchmarks (VoiceBench, SUPERB, AudioBench, AIR-Bench, SpokenWOZ, SD-Eval, SuperCLUE, MMAU) against eleven capability dimensions reveals no benchmark covers audio generation by spoken dialogue models and that interaction-capability evaluation remains largely ad hoc. confidence: high relevance: high limitations: - Produced concurrently with the primary wave of open-source spoken dialogue models it covers (late 2024); some systems are described in early form. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2411.17607' published_date: "2024-11-26" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: synthetic_speech_text_interleaved_data_generated_by_converting role: supports claim: Synthetic speech-text interleaved data, generated by converting text spans to speech tokens using a learned text-to-token model, enables effective cross-modal knowledge transfer from pre-trained LLMs to the speech domain. source: §2.2, §3.3.1, Table 5 evidence: To generate 600B tokens of synthetic interleaved data without requiring any parallel speech-text corpus, a 1.5B text-to-token model is trained on TTS datasets to predict speech token sequences from text. confidence: high relevance: high - claim_id: lower_speech_tokenizer_frame_rates_improve_speech_language role: supports claim: Lower speech tokenizer frame rates improve speech language modelling performance within a fixed token budget, with gains plateauing around 12.5 Hz where information loss begins to outweigh efficiency benefits. source: §3.3.2, Figure 3a evidence: 'On pre-training benchmarks (Table 3), the 9B model outperforms all baselines on speech language modelling tasks, matching Spirit-LM and Moshi on Spoken TopicStoryCloze (S setting: 82.9) while substantially exceeding them on cross-modal settings (T→S: 85.0, S→T: 93.6).' confidence: high relevance: high - claim_id: supervised_speech_tokenization_derived_from_asr_model_fine role: supports claim: Supervised speech tokenization derived from ASR model fine-tuning achieves stronger semantic preservation at low frame rates than unsupervised codec-based tokenisers, while maintaining competitive speech reconstruction quality. source: §2.1, Table 1 evidence: Training uses ASR supervision rather than waveform reconstruction, yielding discrete tokens with strong semantic preservation even at low frame rates. confidence: high relevance: high - claim_id: speech_text_pre_training_with_interleaved_data_substantially role: supports claim: Speech-text pre-training with interleaved data substantially narrows the gap between speech-only and speech-to-text performance on spoken question answering, suggesting that cross-modal alignment transfers factual knowledge from text representations to speech decoding. source: §3.2, Table 3 evidence: 'On pre-training benchmarks (Table 3), the 9B model outperforms all baselines on speech language modelling tasks, matching Spirit-LM and Moshi on Spoken TopicStoryCloze (S setting: 82.9) while substantially exceeding them on cross-modal settings (T→S: 85.0, S→T: 93.6).' confidence: high relevance: high - claim_id: an_intermediate_text_response_in_speech_generation_text role: supports claim: An intermediate text response in speech generation (text-guided mode) provides meaningful accuracy gains for knowledge-intensive tasks, but a well-pre-trained model operating purely in the speech domain can still match text-guided baselines from prior work. source: §3.2, Table 4 evidence: Even in direct speech-to-speech mode (without text guidance), the model (scores 3.18 / 3.2) still performs comparably to text-guided baselines from prior work. confidence: high relevance: high limitations: - No code or model weights are publicly released with this paper, and the proprietary Chinese ASR dataset (10k hours) used in tokenizer training cannot be replicated by third parties, limiting reproducibility of the full pipeline. - The fine-tuning dataset (SpeechDialog-90K) is synthesised using MeloTTS for speech responses, so the chatbot's output speech may inherit MeloTTS quality characteristics rather than reflecting the model's own generative capacity. The evaluation of spoken chatbots relies on GPT-4 scoring, which is a reasonable but proxy measure that may not correlate perfectly with human judgements. The paper does not evaluate multilingual spoken QA or chatbot performance despite training on English and Chinese data. Full-duplex conversation capability, demonstrated by Moshi, is not explored. Scaling beyond 9B parameters is left as future work. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2411.18803' published_date: "2024-11-27" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - adversarial_codec_speech_models claims: - claim_id: ts3_transformer_codec_efficiency role: supports claim: Transformer-only architectures can match convolutional neural audio codecs in streaming reconstruction quality at substantially lower computational cost. source: §3.2, §5.1, Table 3 evidence: TS3-Codec matches or exceeds causal convolutional baselines on most reconstruction metrics while using a fraction of the compute of the largest BigCodec-S configuration. confidence: high relevance: low - claim_id: ts3_single_codebook_streaming_compatibility role: supports claim: Single-codebook audio tokenization is compatible with full streaming operation, removing a structural trade-off for spoken-language-model token prediction. source: §2.3, §3.1 evidence: A causal sliding-window transformer codec operates with one large codebook at 640–850 bps instead of the parallel residual-codebook streams used by earlier streaming codecs. confidence: high relevance: high - claim_id: ts3_semantic_distillation_wer_advantage role: supports claim: Semantic distillation gives speech tokenizers consistent intelligibility advantages even when reconstruction-oriented codecs score higher on perceptual-quality proxies. source: §5.1, §5.2, Table 3, Table 4 evidence: Mimi and SpeechTokenizer retain lower WER in several matched comparisons despite TS3-Codec achieving stronger UTMOS or distortion scores. confidence: medium relevance: high - claim_id: ts3_transformer_outperforms_causal_convolution role: supports claim: At matched compute, transformer codec backbones can outperform causal convolutional alternatives across intelligibility, distortion, and estimated naturalness. source: §5.1, Figure 2, Figure 3 evidence: The transformer configurations trace a stronger quality–compute frontier than the reproduced causal BigCodec-S baselines. confidence: medium relevance: low limitations: - Naturalness is evaluated with UTMOS rather than a human listening test. - Downstream speech-language-model training and inference with the large single codebook are not demonstrated. - Evaluation is limited to English read speech, and real deployment latency is not measured. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2411.19842' published_date: "2024-11-29" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - codec architecture: - transformer-enc-dec - VAE relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models - variational_latent_speech_models claims: - claim_id: scaling_transformer_architecture_parameter_count_in_neural_audio role: supports claim: Scaling transformer architecture parameter count in neural audio codecs produces consistent quality improvements across objective and subjective metrics. source: §4.6, Table 4 evidence: Scaling ablations (Table 4) confirm consistent quality improvement as parameter count grows from 240M to 540M to 950M, with all metrics improving monotonically. confidence: high relevance: low - claim_id: finite_scalar_quantization_achieves_near_perfect_codebook_utilization role: supports claim: Finite scalar quantization achieves near-perfect codebook utilization without explicit utilization regularization, simplifying downstream generative modeling compared to RVQ. source: §3.2, §A.8, Table 9 evidence: Near-perfect codebook utilization follows from this formulation without any explicit utilization regularization. confidence: high relevance: low - claim_id: a_neural_codec_trained_exclusively_on_english_speech role: supports claim: A neural codec trained exclusively on English speech can generalize effectively to unseen languages, outperforming multilingual-trained baselines of similar scale on most objective metrics. source: §A.5, Table 7 evidence: On multilingual LibriSpeech (Table 7), TAAE trained on English only outperforms all baselines across all seven evaluated languages on most metrics, including Mimi which was trained on 7M hours (roughly 70 times more data), though Mimi leads slightly on PESQ in several languages. confidence: high relevance: low - claim_id: perceptual_losses_derived_from_self_supervised_speech_models role: supports claim: Perceptual losses derived from self-supervised speech models (WavLM-Large features) are critical for achieving intelligible reconstruction at low bitrates, beyond what adversarial and spectral reconstruction losses alone provide. source: §3.4, §A.1, Table 3 evidence: Finetuning replaces the reconstruction losses with a WavLM-Large perceptual feature-matching loss, which the paper shows is essential for achieving intelligible, high-quality reconstruction (Table 3, Appendix A.1). confidence: high relevance: low - claim_id: systematic_spectral_bias_in_multi_resolution_stft_discriminators role: supports claim: Systematic spectral bias in multi-resolution STFT discriminators, arising from power-of-two FFT configurations, causes periodic reconstruction artifacts that disproportionately affect large-capacity codec architectures. source: §3.3, §B.5 evidence: Pretraining uses an adversarial feature-matching loss from a multi-resolution complex STFT discriminator, augmented with exponentially decaying L1 waveform and STFT reconstruction losses. confidence: high relevance: low limitations: - Training data is 16 kHz English audiobook speech only (105k hours). Multilingual generalization results are promising but the model was not trained or optimized for non-English data; claims about multilingual capability should be interpreted cautiously relative to models with dedicated multilingual training at scale. - The model has not been evaluated on noisy speech, overlapping speakers, or environmental audio, which are common real-world conditions. The large parameter count (950M) requires substantially more compute than lighter baselines (DAC at 76M, Mimi at 80M); while the RTF is acceptable on H100 GPUs for longer utterances, latency for short clips is roughly 3x that of smaller models, which matters for streaming applications. The post-hoc Residual FSQ decomposition is restricted to specific level configurations (L = 2^n + 1); arbitrary bitrate targets are not directly achievable without retraining. The systematic bias analysis in the discriminator (§B.5) raises open questions about whether similar biases appear in other convolutional discriminator architectures and how to address them in the general case. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2412.02612' published_date: "2024-12-03" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: speech_text_interleaved_pre_training_at_trillion_token role: supports claim: Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality. source: §4.1, Table 4 evidence: 'The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607).' confidence: high relevance: high - claim_id: single_codebook_supervised_speech_tokenizers_derived_from_asr role: supports claim: Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis. source: §3.1, Table 1 evidence: 'The design deliberately sacrifices full acoustic fidelity for semantic compactness: a single codebook at 12.5Hz avoids the multi-codebook complexity of RVQ-based codecs while preserving enough semantic content for accurate ASR (2.10 WER on LibriSpeech clean) and high-quality reconstruction (MOSNet 3.39).' confidence: high relevance: high - claim_id: streaming_interleaved_generation_templates_alternating_text_and_speech role: supports claim: Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech. source: §3.3 evidence: 'The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607).' confidence: high relevance: high - claim_id: end_to_end_speech_language_models_that_include role: supports claim: End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data. source: §5.2, Table 6 evidence: Training uses a mix of supervised ASR data (LibriSpeech, GigaSpeech, CommonVoice, AISHELL-1, MLS-Eng, Wenet) and 700k hours of pseudo-labelled unsupervised speech. confidence: high relevance: high - claim_id: decoupling_the_text_and_speech_output_subtasks_during role: supports claim: Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities. source: §4.2.2 evidence: Separate loss-masking passes train the text-output and speech-output subtasks at different rates (4 vs. 20 epochs) to account for their different learning speeds. confidence: high relevance: low limitations: - The paper reports no subjective listening test (MOS/MUSHRA) on the chat model output; UTMOS is used as a proxy for speech naturalness, and the chat evaluation relies on GPT-4o scoring of ASR transcriptions, introducing cascaded error from both the vocoder quality and the Whisper transcription step. - 'The 175bps tokenizer trades acoustic fidelity for compactness: VisQOL at 12.5Hz (2.52) is lower than SpeechTokenizer variants and the 50Hz variant of the same system. This may limit voice cloning quality and the fidelity of paralinguistic feature reproduction (accent, fine-grained emotion), though the paper does not directly evaluate these.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2412.15649' published_date: "2024-12-20" entry_date: '2026-07-27' year: 2024 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: decoupling_speaker_identity_from_semantic_content_in_spoken role: supports claim: Decoupling speaker identity from semantic content in spoken dialogue systems enables zero-shot timbre control without modifying the language model or adding speaker-conditioning layers. source: §3.4 evidence: Because the semantic tokens encode only linguistic content (not speaker identity), a conditional flow-matching vocoder conditioned on a speaker audio prompt converts semantic tokens to mel spectrograms, which HiFi-GAN then synthesizes to waveform. confidence: high relevance: high - claim_id: grouping_audio_tokens_to_reduce_the_frequency_mismatch role: supports claim: Grouping audio tokens to reduce the frequency mismatch between text and speech representations substantially improves speech-text alignment and training efficiency in parallel audio-text dialogue models. source: §3.3, Table 5 evidence: With G = 3, the audio sequence length is reduced by 3x, enabling the model to converge with less than half the GPU hours of the G = 1 baseline while achieving substantially better speech-text alignment (4.54% ASR-WER vs. 18.23% at G = 1). confidence: high relevance: high - claim_id: single_stage_fine_tuning_on_dialogue_data_can role: supports claim: Single-stage fine-tuning on dialogue data can match or outperform multi-stage pipelines that include ASR or TTS pre-training, because modality-specific pre-training degrades instruction-following and general knowledge retention. source: §5.3.2, Table 6 evidence: The entire system is trained in a single stage on dialogue data (primarily VoiceAssistant-400K, ~664 hours of instruction audio), with full fine-tuning of all components except the Whisper encoder, which is kept frozen. confidence: high relevance: low - claim_id: replacing_audio_token_history_with_text_only_history role: supports claim: Replacing audio-token history with text-only history in multi-turn spoken dialogue models improves the system's ability to handle longer conversation contexts while leveraging pre-trained LLM in-context learning. source: §3.5 evidence: For multi-turn dialogue, Historical Text Prompting replaces the typical approach of maintaining interleaved audio-text token histories. confidence: high relevance: high - claim_id: current_spoken_dialogue_models_consistently_underperform_text_only role: supports claim: Current spoken dialogue models consistently underperform text-only LLMs of similar scale on semantic content quality, even after dialogue fine-tuning. source: §5.1, Table 3 evidence: At 0.5B scale, SLAM-Omni substantially outperforms Mini-Omni and Mini-Omni2 on ChatGPT-based content scoring (39.32 vs. 22.58 and 26.56 overall), achieves the highest UTMOS across all SDMs including the larger 7-9B models (4.45 vs. 4.37 for Freeze-Omni), and the lowest ASR-WER (4.54% vs. 6.05% for Mini-Omni and 10.42% for LLaMA-Omni). confidence: high relevance: high limitations: - Historical text prompting discards all non-verbal information from prior dialogue turns (prosody, emotion, paralinguistic cues). In scenarios where voice-level context matters for dialogue coherence, this strategy may reduce response quality in ways not captured by the text-based evaluation metrics used. - The system is evaluated exclusively at 0.5B scale. The single-stage training advantage may not hold for larger LLMs, where the data volume required for joint audio-text modeling grows substantially. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2501.06282' published_date: "2025-01-10" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: large_scale_multi_task_training_across_heterogeneous_speech role: supports claim: Large-scale multi-task training across heterogeneous speech tasks substantially improves both comprehension and generation in aligned multimodal speech LMs without catastrophic forgetting of the base LLM's text capabilities. source: §3.4, §4.1–4.5 evidence: MinMo adopts an aligned multimodal architecture that adapts a pretrained text LLM by adding audio input and output pathways without fundamentally replacing the LLM's operating regime. confidence: high relevance: high - claim_id: aligned_multimodal_architectures_can_achieve_instruction_controlled_speech role: supports claim: Aligned multimodal architectures can achieve instruction-controlled speech style (emotion, dialect, speaking rate, voice identity) when trained with appropriate instruction data, contradicting prior claims that this capability is limited to native multimodal models. source: §3.2, §4.4, Table 18 evidence: End-to-end instruction control (emotion, dialect, speaking rate, voice identity) is achieved by conditioning on hidden embeddings from the LLM, which carry the user instruction context through to the audio output. confidence: high relevance: high - claim_id: full_duplex_spoken_dialogue_at_sub_second_latency role: supports claim: Full-duplex spoken dialogue at sub-second latency is achievable with a modular aligned architecture combining a semantic predictor with a streaming autoregressive decoder, without requiring joint speech-text pre-training. source: §3.1, §4.5, Table 21 evidence: The end-to-end full-duplex response latency is ~600ms in theory (250ms duplex decision + 150ms text generation + 70ms speech token generation + 130ms token-to-wave), and ~800ms in practice on L20 GPUs. confidence: high relevance: high - claim_id: an_autoregressive_streaming_voice_decoder_that_interleaves_text role: supports claim: An autoregressive streaming voice decoder that interleaves text hidden states with speech tokens outperforms non-autoregressive CTC-based decoders in naturalness and content consistency for aligned speech LMs. source: §3.2, §4.4, Table 17 evidence: 'The decoder operates on interleaved sequences of semantic vectors and speech tokens in a fixed 5:15 ratio: every five text tokens produce five semantic vectors via a linear output projector, and these drive fifteen autoregressive speech tokens from the Voice Token LM.' confidence: high relevance: high limitations: - The instruction-following voice generation evaluation uses a single in-house Chinese test set of 122 turns, making external validation of the 98.4% accuracy figure impossible. Code and model weights had not been released at time of publication. - LoRA-only updates to the text LLM during training limit the model's ability to follow diverse instructions; more comprehensive LLM updates with higher-quality text data remain unexplored. Long-tail pronunciation errors persist in end-to-end audio generation, partly due to special symbols that the decoder cannot reliably convert to speech. The full-duplex module still relies on external acoustic echo cancellation (AEC) and voice activity detection (VAD) modules, meaning a truly end-to-end duplex system has not been achieved. Performance on speech emotion recognition in low-resource languages shows mixed results (e.g., Polish at 55.9% F1), indicating that cross-lingual generalization is uneven despite the broad multilingual training. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.coling-industry.29 published_date: "2025-01-19" entry_date: '2026-07-27' year: 2025 venue: COLING Industry Track task: - SCA architecture: [] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [] claims: - claim_id: carmem_category_bounded_extraction_reduces_redundancy role: supports claim: Structured, category-bounded memory extraction reduces both redundant and contradictory preference storage compared to unstructured extraction in conversational agents. source: §5.2, Table 5 evidence: The category-bounded system achieves a 95% reduction in redundant preferences and a 92% reduction in contradictory ones on the CarMem maintenance benchmark; hierarchical schema constraints prevent out-of-category storage entirely. confidence: high relevance: high - claim_id: carmem_extraction_degrades_with_granularity role: complicates claim: LLM-based preference extraction from conversational context degrades significantly as category granularity increases. source: §5.1, Tables 2-3 evidence: Extraction F1 drops from 0.94 at the 4-category main level to 0.78 at the 41-category detail level, with over-extraction rates of 12-15% in the In-Schema condition driven by confusion among semantically similar categories. confidence: high relevance: high - claim_id: carmem_structured_metadata_improves_retrieval role: supports claim: Enriching retrieval embeddings with structured metadata (category label and attribute) substantially improves preference recall in personalised spoken conversational agents. source: §5.3, Table 6 evidence: Concatenating the detail category and attribute to the sentence-of-preference embedding raises top-n retrieval accuracy from 0.75 to 0.87 on CarMem. confidence: high relevance: high - claim_id: carmem_synthetic_benchmarks_overestimate_robustness role: complicates claim: Synthetic benchmarks evaluated on in-domain synthetic data overestimate system robustness for real-world deployment in spoken conversational agents. source: §7 evidence: The CarMem dataset is generated by GPT-4 and evaluated against GPT-4o, omitting speech recognition errors and multi-preference conversations common in real-world in-car interactions, a limitation the authors acknowledge. confidence: medium relevance: high limitations: - CarMem contains exactly one preference per conversation and is entirely synthetic, generated and evaluated with GPT-class models; no real user data or ASR errors are simulated. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2409.20007' published_date: "2025-01-27" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: desta2_broad_instruction_following_without_tuning_data role: supports claim: Speech language models can acquire broad instruction-following capabilities over paralinguistic attributes without task-specific speech instruction-tuning data. source: §V.A, Table I evidence: DeSTA2 achieves 56.78% overall accuracy on Dynamic-SUPERB across 48 tasks, surpassing all prior end-to-end systems that used extensive instruction-tuning, using only LLM-generated speech captions with a single generic prompt. confidence: high relevance: high - claim_id: desta2_constrained_prompts_degrade_general_capability role: complicates claim: Automated speech instruction-tuning pipelines that use constrained prompts or QA formatting may degrade the base LLM's general language capabilities. source: §V.B, Table III evidence: Ablations show training with open-ended QA pairs improves degradation-task performance but causes measurable drops in content and semantic dimensions relative to DeSTA2's unconstrained caption-generation approach. confidence: medium relevance: low - claim_id: desta2_distributional_alignment_reduces_forgetting role: supports claim: Distributional alignment between training targets and the base LLM's pretraining regime reduces catastrophic forgetting in speech-extended language models. source: §V.D, Table V evidence: DeSTA2 preserves chain-of-thought reasoning and output-format following (e.g., computing words-per-second, responding in all-caps on demand), capabilities absent from Qwen2-Audio despite its larger training budget. confidence: medium relevance: high - claim_id: desta2_paralinguistic_metadata_drives_performance role: refines claim: Rich paralinguistic metadata coverage, rather than task diversity, is the primary driver of broad spoken language model performance on multi-dimensional speech benchmarks. source: §IV.A, §V.A, Table I evidence: DeSTA2 uses a single captioning task with 12 paralinguistic attributes and outperforms systems with diverse task-specific instruction sets on the paralinguistic dimension of Dynamic-SUPERB (43.14% vs. 29.21% for Qwen2-Audio). confidence: medium relevance: high limitations: - Speaker verification performance remains below specialized baselines (42.5% vs. 47.9% for Qwen2-Audio on the Dynamic-SUPERB speaker dimension); the approach does not address multi-speaker comparison tasks. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2502.06490' published_date: "2025-02-10" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - VC - codec architecture: [] relevance: high evidence_role: - core_evidence - infrastructure - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: acoustic_tokens_and_semantic_tokens_occupy_fundamentally_distinct role: complicates claim: Acoustic tokens and semantic tokens occupy fundamentally distinct points in a reconstruction-versus-semantics trade-off space, and no single tokenization strategy currently achieves strong performance on both axes simultaneously. source: §VI-C, Table I evidence: 'General-purpose acoustic tokens achieve the strongest reconstruction fidelity (EnCodec Q=8: PESQ 2.83, WER 1.53%) but carry high bitrates (6 kbps) and relatively weak semantic probe performance (IC accuracy 34.8%).' confidence: high relevance: low - claim_id: speaker_disentanglement_in_acoustic_tokens_enables_voice_conversion role: supports claim: Speaker disentanglement in acoustic tokens enables voice conversion capability but consistently reduces reconstruction quality metrics at equivalent bitrates. source: §VI-D, Table I evidence: Semantic SSL tokens (HuBERT Large, WavLM Large) achieve the best voice conversion speaker similarity (SECS ~0.87–0.88) and superior intent classification (67–74%) at very low bitrates (0.55 kbps), but suffer severe prosody loss (GPE ~15–18%, versus ~1% for acoustic tokens) and poor signal quality (PESQ ~1.15). confidence: high relevance: low - claim_id: k_means_clustering_on_ssl_model_embeddings_discards role: supports claim: K-means clustering on SSL model embeddings discards prosody information more severely than supervised or internally-quantized semantic tokenizers, making offline clustering ill-suited for tasks requiring prosody fidelity. source: §VI-C, §VI-D, Table I evidence: 'The supervised S3 Tokenizer offers an intermediate profile: good VC speaker similarity (0.868) with better prosody retention than clustering-based SSL tokens, suggesting its ASR objective preserves some paralinguistic structure that k-means discards.' confidence: high relevance: low - claim_id: acoustic_byte_pair_encoding_achieves_greater_length_reduction role: supports claim: Acoustic byte-pair encoding achieves greater length reduction on tokens with lower information density, such as speaker-decoupled and semantic tokens, than on general-purpose acoustic tokens. source: §V-A, Figure 8 evidence: 'Acoustic BPE length reduction disproportionately compresses tokens with low information density: semantic tokens and speaker-decoupled LSCodec tokens compress more than general-purpose acoustic tokens, confirming that BPE gain is inversely related to information density.' confidence: high relevance: low - claim_id: single_codebook_tokens_at_very_low_frame_rates role: supports claim: Single-codebook tokens at very low frame rates improve compatibility with language model generation but currently exhibit measurable quality and intelligibility degradation relative to multi-codebook or higher frame-rate alternatives. source: §VIII.1, §VI-C evidence: Semantic SSL tokens (HuBERT Large, WavLM Large) achieve the best voice conversion speaker similarity (SECS ~0.87–0.88) and superior intent classification (67–74%) at very low bitrates (0.55 kbps), but suffer severe prosody loss (GPE ~15–18%, versus ~1% for acoustic tokens) and poor signal quality (PESQ ~1.15). confidence: high relevance: high limitations: - The experimental comparisons are conducted on English data only (LibriTTS, LibriSpeech), leaving multilingual tokenization trade-offs unexplored. The unified vocoder (CTX-vec2wav) is specifically designed for semantic tokens, which may introduce a systematic advantage for semantic token types in reconstruction experiments. Not all acoustic tokens support voice conversion in the paper's framework, so the VC comparison covers only a subset of systems. - 'Open questions identified by the survey include: the bitrate lower bound for single-codebook tokens with acceptable intelligibility; whether causal SSL architectures can match non-causal models for semantic token quality; how VFR tokens perform on generative tasks beyond ASR; and whether token vocoders trained at scale can match flow matching-based alternatives for timbre controllability.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2502.11946' published_date: "2025-02-17" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: a_dual_codebook_interleaved_tokenizer_that_combines_linguistic role: supports claim: A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality. source: §4.4, §6.2.1 evidence: 'This interleaving is the core tokenizer innovation: semantic tokens maintain semantic coherence in generated continuations, while linguistic tokens preserve the acoustic quality needed for intelligible reconstruction.' confidence: high relevance: high - claim_id: scaling_autoregressive_llm_backbone_size_from_3b_to role: supports claim: Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive. source: §6.2.2, Table 3 evidence: On the SEED-TTS benchmark for TTS quality, Step-Audio-TTS-3B achieves 1.31% CER (Chinese) and 2.31% WER (English), outperforming CosyVoice, MaskGCT, and FireRedTTS on intelligibility while remaining competitive on speaker similarity. confidence: high relevance: high - claim_id: rlhf_reward_models_trained_on_speech_interaction_data role: supports claim: RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training. source: §5.2.6 evidence: The AQTA system is post-trained with PPO-based RLHF using a two-stage reward model, where a known failure mode ("deaf hacking") was identified and mitigated by constructing explicit counter-examples in the reward training data. confidence: high relevance: high - claim_id: speculative_response_generation_triggered_by_voice_activity_detection role: supports claim: Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems. source: §3.4 evidence: Real-time inference uses speculative response generation triggered by voice activity detection pauses. confidence: high relevance: high - claim_id: synthetic_speech_data_generated_by_a_large_multi role: supports claim: Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles. source: §5.1.1 evidence: The same decoder architecture underlies the distilled standalone Step-Audio-TTS-3B model, which is trained on synthetic data produced by the 130B model rather than manually curated recordings. confidence: high relevance: low limitations: - The StepEval-Audio-360 benchmark is proprietary and created by the same team; human evaluation results on it cannot be independently reproduced. Open-source benchmark comparisons mix locally re-run models with results taken from original publications, complicating direct numerical comparison. - 'Speaker similarity scores for the distilled Step-Audio-TTS-3B are noticeably lower than CosyVoice 2 on both Chinese and English SEED-TTS tests, suggesting that the dual-codebook approach trades some acoustic identity preservation for intelligibility gains. The AQTA+TTS design still relies on a cascade: errors in ASR transcription of history or in text generation propagate to the TTS stage. The paper''s future work section acknowledges that purely end-to-end audio-in/audio-out (AQAA) remains unsolved. Evaluation for singing, RAP, and dialect control is limited to instruction following scores without reference audio; absolute quality in these dimensions is difficult to assess from the reported numbers alone.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2502.17239' published_date: "2025-02-24" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: multi_codebook_rvq_tokenizers_with_semantic_alignment_objectives role: supports claim: Multi-codebook RVQ tokenizers with semantic alignment objectives better preserve both acoustic and linguistic content than single-codebook designs, with each additional layer reducing ASR error substantially up to 8 layers. source: §3.1, Table 1 evidence: Layerwise dropout concentrates semantic content in the top codebook layers, analogous to the design principle in SpeechTokenizer. confidence: high relevance: low - claim_id: staged_pretraining_frozen_llm_first_then_joint_training role: supports claim: Staged pretraining (frozen LLM first, then joint training) measurably reduces intelligence degradation in end-to-end speech LMs relative to single-stage joint training. source: §3.3.1, Table 5 evidence: This staged approach is designed to prevent audio modality training from overwriting textual knowledge; ablations (Table 5) show it closes roughly 2 percentage points of the S→T accuracy gap on sStoryCloze relative to single-stage training. confidence: high relevance: high - claim_id: text_guided_aligned_generation_where_the_model_completes role: supports claim: Text-guided aligned generation, where the model completes text tokens before emitting the corresponding audio tokens, mitigates semantic incoherence in speech LM outputs. source: §3.3, §4.1 evidence: During inference the model alternates text and audio token generation, with modality-switching special tokens enforcing alignment, so text completion always precedes audio generation for the corresponding utterance. confidence: high relevance: high - claim_id: flow_matching_decoders_trained_as_post_vq_refinement role: supports claim: Flow-matching decoders trained as post-VQ refinement stages recover significant audio quality lost during quantisation, with UTMOS improvements of 0.6 points possible without retraining the LLM. source: §3.2, Table 3 evidence: 'Baichuan-Audio builds on a 7B-parameter LLM backbone (Baichuan) extended with three interconnected components: a custom audio tokenizer, an independent audio head, and a flow-matching audio decoder.' confidence: high relevance: high - claim_id: end_to_end_speech_lms_evaluated_in_s role: supports claim: End-to-end speech LMs evaluated in S→S mode exhibit notably lower benchmark performance than the same model evaluated in S→T mode, indicating that audio token generation itself introduces a quality penalty beyond the comprehension step. source: §4.3, Table 8 evidence: During inference the model alternates text and audio token generation, with modality-switching special tokens enforcing alignment, so text completion always precedes audio generation for the corresponding utterance. confidence: high relevance: high limitations: - The TTS quality evaluation is limited to an in-house test set (MED-TTS); no comparison against standard TTS benchmarks (VCTK, LJSpeech, LibriTTS test-clean) or against dedicated TTS systems is reported. The naturalness and speaker similarity of the generated speech relative to state-of-the-art TTS systems is therefore unknown. - The intelligence gap to GPT-4o-Audio remains large (roughly 15-20 percentage points on QA tasks), and the paper does not explain what architectural or data factors account for this difference. The OpenAudioBench evaluation uses GPT-4o as judge, which may introduce evaluation bias. The full-duplex and interruption-handling capabilities common in deployed spoken conversational agents are not evaluated. Training data mix decisions (e.g. dropping INTLV audio loss, ITTS loss design) are motivated empirically but without systematic ablation. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2503.14345' published_date: "2025-03-18" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: spontaneous_scripting_from_an_llm_is_roughly_as role: supports claim: Spontaneous scripting from an LLM is roughly as important as the acoustic modeling choice for perceived spontaneity in long-form dialogue TTS. source: §4.2.2, Table 3 evidence: The paper demonstrates empirically that spontaneous scripting alone accounts for a large fraction of the final audio spontaneity. confidence: high relevance: low - claim_id: full_sequence_interleaving_of_text_and_speech_codes role: supports claim: Full-sequence interleaving of text and speech codes, extended to 40,000-token context windows with speaker-change tokens, enables coherent long-form zero-shot multi-speaker synthesis that turn-level concatenation cannot match. source: §3.2.1, Tables 1–2 evidence: The sequence design interleaves full podcast text with full speech codes (rather than per-turn interleaving), using speaker-change tokens to signal speaker transitions. confidence: high relevance: high - claim_id: curriculum_learning_progressively_exposing_a_codec_lm_to role: supports claim: Curriculum learning, progressively exposing a codec LM to increasing dialogue complexity, is an effective strategy for developing long-context and spontaneous generation capability without requiring matched long-context data from the outset. source: §3.2.1 evidence: 'Training uses curriculum learning across three stages: single-turn single-speaker zero-shot TTS; long-context two-speaker non-conversational data (audiobooks, up to 40,000 tokens / 800 seconds); and long-context two-speaker conversational data (podcasts).' confidence: high relevance: high - claim_id: automatic_speaker_similarity_metrics_cosine_embedding_similarity_can role: supports claim: Automatic speaker similarity metrics (cosine embedding similarity) can disagree with subjective speaker similarity ratings in long-form generation settings, particularly when the acoustic model attends to prosodic rather than purely timbral features. source: §4.2.1 evidence: However, SIM-O (automatic speaker cosine similarity) is notably lower for MoonCast on English (0.53 vs. 0.75 for the concatenation baseline), attributed to insufficient audiobook data in the English training curriculum, which the authors acknowledge as a limitation. confidence: high relevance: low - claim_id: chunk_wise_autoregressive_decoding_with_a_causal_chunk role: supports claim: Chunk-wise autoregressive decoding with a causal chunk mask provides a practical solution to the continuity and memory constraints of mel-spectrogram reconstruction from long semantic code sequences. source: §3.2.2 evidence: A chunk-wise causal attention mask allows each chunk to attend to all previously generated chunks, preserving continuity across boundaries without loading the full sequence. confidence: high relevance: low limitations: - All evaluation is conducted on a small internal test set (4 knowledge sources for podcast, 7 podcasts for the script ablation), and training data is entirely proprietary (~515K hours). Results cannot be independently reproduced, and generalisability to other domains or languages beyond Chinese and English is unverified. - The notably lower SIM-O score for English (0.53 vs. 0.75 baseline) reveals unresolved speaker consistency challenges in the English long-context setting, which the authors attribute to an imbalanced training curriculum favouring audiobook over conversational English data. Hallucinations in speaker attribution (utterances assigned to the wrong speaker) emerge from the interplay of semantic token timbre leakage, diarization errors in training data, and ambiguous filler-word interpretations; no mitigation is proposed beyond discussion. The two-speaker restriction (host + guest) is a deliberate scope limitation; extension to three or more speakers is left as future work. Evaluation is subjective-only for multi-speaker interactions; no standardised zero-shot TTS benchmark (LibriSpeech, VCTK) is used, limiting direct comparison to single-speaker zero-shot systems. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2504.08528' published_date: "2025-04-11" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: [] relevance: high evidence_role: - core_evidence - infrastructure - evaluation_caution current_role: active_evidence method_family: [] claims: - claim_id: content_understanding_and_acoustic_reasoning_separate role: complicates claim: Models that excel at content-level spoken language understanding tend to underperform on tasks requiring reasoning about acoustic and paralinguistic details, indicating that these capabilities require separate optimisation. source: §7.2 evidence: The survey documents that VoiceBench (content-oriented) and MMAU (audio reasoning) scores are negatively correlated across surveyed models, indicating that current training pipelines do not jointly optimize both capabilities. confidence: medium relevance: high - claim_id: cascaded_and_e2e_complementary_strengths role: supports claim: 'Cascaded speech pipelines and end-to-end spoken language models have complementary strengths: cascades are stronger on semantic tasks, while end-to-end models better handle speaker and paralinguistic information.' source: §7.2 evidence: Cascaded ASR+LLM+TTS systems consistently outperform end-to-end SLMs on semantic and content-heavy benchmarks (Dynamic-SUPERB Phase 2, MMAU, VoiceBench) but underperform on paralinguistic and speaker-related tasks, a pattern observed across the surveyed literature. confidence: medium relevance: high - claim_id: joint_speech_llm_training_forgetting_risk role: complicates claim: Jointly training a speech encoder with an LLM backbone improves instruction following and paralinguistic understanding but introduces risk of catastrophic forgetting of pre-trained text capabilities. source: §7.2, §4.3 evidence: The survey notes that end-to-end joint training of speech encoders with LLM backbones is reported across surveyed systems to trade instruction-following gains against forgetting of the base LLM's text capabilities. confidence: low relevance: low - claim_id: full_duplex_requires_architectural_tradeoffs role: complicates claim: Full-duplex spoken dialogue requires architectural support beyond turn-taking assumptions, and current approaches (dual-channel and time-multiplexing) each involve significant trade-offs in latency, naturalness, and modelling complexity. source: §6 evidence: Reported first-packet latency for full-duplex SLMs spans 200ms (Moshi) to 5,588ms (SpeechGPT), a 28x range reflecting fundamentally different dual-channel versus time-multiplexing architectural choices (Table 3). confidence: medium relevance: high - claim_id: tokenisation_choice_determines_coherence_fidelity_tradeoff role: supports claim: The tokenisation choice (phonetic tokens versus audio codec tokens) determines the trade-off between linguistic coherence and speaker or acoustic fidelity in spoken language model outputs. source: §3.1.2 evidence: The survey's taxonomy in §3.1.2 contrasts phonetic (SSL-derived) tokens, which favor linguistic coherence, against audio codec tokens, which favor speaker and acoustic fidelity, as the two dominant tokenisation families for SLM sequence models. confidence: medium relevance: high limitations: - The survey is primarily a snapshot of English-centric, high-resource SLM research; most SLM evaluations use non-overlapping benchmarks, making direct performance comparisons across model families impossible, and scaling behaviour for speech+text and speech-aware text LMs is entirely unknown. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2504.10344' published_date: "2025-04-14" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec - TTS architecture: - autoregressive-LM - VAE relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - variational_latent_speech_models claims: - claim_id: frame_level_quantization_without_cross_frame_context_limits role: supports claim: Frame-level quantization without cross-frame context limits codec semantic richness and increases the difficulty of autoregressive LM training on the resulting tokens. source: §1, Figure 1 evidence: Semantic retention, measured by ASR WER on LibriSpeech features using the codec's quantized representations, is 18.3% for ALMTokenizer versus 44.6% for WavTokenizer and 44.1% for DAC; emotion classification accuracy (29%) approaches WavLM (29%) and HuBERT (31%). confidence: high relevance: low - claim_id: initialising_vq_codebook_entries_from_semantic_priors_derived role: supports claim: Initialising VQ codebook entries from semantic priors derived from self-supervised models improves downstream recognition accuracy without additional distillation overhead. source: §3.2, Table 6 evidence: Semantic priors from k-means clustering of wav2vec 2.0 (for speech) and BEATs (for general audio) features initialise and fix the codebook of the first VQ layer, injecting structured semantic content without requiring online distillation. confidence: high relevance: low - claim_id: lower_token_sequence_length_lower_hz_frame_rate role: supports claim: Lower token-sequence length (lower Hz frame rate) improves both training and inference efficiency for audio language models, independent of bitrate. source: Appendix B, Table 8 evidence: Comparison with SemantiCodec shows ALMTokenizer is 30x faster at inference (RTF 0.031 vs. 0.92) while achieving better reconstruction and semantic scores at lower bitrate (Table 16). confidence: high relevance: high - claim_id: reconstruction_quality_alone_is_an_insufficient_predictor_of role: supports claim: 'Reconstruction quality alone is an insufficient predictor of a codec''s suitability for autoregressive language modelling: codecs with stronger semantic content produce more robust LM-based TTS even at higher bitrates.' source: §4.6, Table 4 evidence: An AR prediction loss trains a lightweight autoregressive transformer to predict the quantized features of deeper RVQ layers from shallower ones, making the latent space more amenable to sequential LM modelling. confidence: high relevance: high - claim_id: a_masked_autoencoder_auxiliary_loss_during_codec_training role: supports claim: A masked autoencoder auxiliary loss during codec training increases semantic information in learned representations at a modest reconstruction cost. source: §3.3, Table 6 evidence: Semantic retention, measured by ASR WER on LibriSpeech features using the codec's quantized representations, is 18.3% for ALMTokenizer versus 44.6% for WavTokenizer and 44.1% for DAC; emotion classification accuracy (29%) approaches WavLM (29%) and HuBERT (31%). confidence: high relevance: low limitations: - All downstream LM experiments use the same 1B-parameter LLaMA backbone with limited training data (2000 hours speech, ~500 hours each sound/music). Gains observed may not transfer to larger-scale audio LM systems, where baseline tokenizers may reach ceiling performance. - The two-stage training adds complexity and discards large sub-networks (MAE encoder/decoder, AR prediction transformer) after training, increasing resource cost without reuse. The authors note this explicitly and flag it as future work. Sound and music reconstruction remains challenging at 0.41 kbps, with VISQOL scores substantially lower than speech. Semantic retention still lags SSL models for ASR (18.3% WER vs. 6.2% for WavLM), indicating the codec does not fully substitute for purpose-built SSL representations. Code and model weights were not released at submission time, limiting reproducibility. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: iclr-2025-868masI331 published_date: "2025-04-24" entry_date: '2026-07-27' year: 2025 venue: ICLR task: - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: reducing_the_first_layer_rvq_frame_rate_of role: supports claim: Reducing the first-layer RVQ frame rate of a neural audio codec through hierarchical multi-resolution distillation enables autoregressive TTS to generate minute-long speech with stable intelligibility. source: §4, §7.3, Table 3, Table 4 evidence: MReQ-Encodec at 8 Hz achieves WER 9.79% on MinutesSpeech test-90s where standard Encodec at 8 Hz produces 100% WER and naive VALL-E at 48 Hz with long training data yields 16.14%. confidence: high relevance: low - claim_id: lowering_the_codec_frame_rate_in_autoregressive_tts role: complicates claim: Lowering the codec frame rate in autoregressive TTS improves temporal coherence and intelligibility for long utterances but degrades speaker similarity. source: §7.3, Table 4, Table 6 evidence: HALL-E consistently lags VALL-E by 0.025-0.042 SIM points on MinutesSpeech tests; the paper attributes this to acoustic information loss when compressing from 48 Hz to 8 Hz in the first RVQ layer. confidence: high relevance: low - claim_id: post_training_from_a_pre_trained_lm_checkpoint role: supports claim: 'Post-training from a pre-trained LM checkpoint is essential for hierarchical codec TTS: training from scratch without pre-trained weights collapses quality.' source: §7.4, Table 10 evidence: Removing VALL-E pre-training from HALL-E increases WER from 9.79% to 49.8% on MinutesSpeech test-90s, and removing MReQ pre-training similarly degrades codec reconstruction. confidence: high relevance: low - claim_id: the_frame_rate_at_which_autoregressive_speech_token role: refines claim: The frame rate at which autoregressive speech token generation becomes unstable is approximately 8 Hz, consistent with average phoneme durations of around 100 ms. source: Appendix C.1, Table 18 evidence: Ablation at 4 Hz first-layer rate raises WER to 20.07%, while 8 Hz yields 9.79%; the paper notes that phoneme duration averaging ~100 ms corresponds to ~10 Hz, making 4 Hz fundamentally insufficient. confidence: high relevance: high - claim_id: length_balanced_training_data_covering_the_target_synthesis role: supports claim: Length-balanced training data covering the target synthesis duration is necessary for autoregressive models to generalize to long-form speech. source: §7.3, Table 4 evidence: VALL-E trained only on segments up to 28 seconds achieves WER 39.77% on test-90s, and training on longer data decreases but does not eliminate the gap; HALL-E's 8 Hz formulation resolves instability that remains even with longer VALL-E training data. confidence: high relevance: low limitations: - 'Speaker similarity consistently degrades with frame rate reduction: HALL-E''s SIM scores are 0.025-0.042 lower than VALL-E across test conditions. This reflects a fundamental trade-off between temporal compression and acoustic fidelity preservation that the paper does not resolve.' - The current NAR model processes audio at full 48 Hz resolution, which limits NAR input length to around 28-54 seconds during training and requires a sliding window at inference. Integrating cross-attention text conditioning addresses alignment but does not eliminate the NAR length bottleneck. The paper also does not compare against streaming or chunked autoregressive synthesis approaches, which represent a practical alternative. MinutesSpeech training data consists entirely of English podcast speech, and generalization to other languages, domains, or reading styles is untested. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: iclr-2025-dGSOn7sdWg published_date: "2025-04-24" entry_date: '2026-07-27' year: 2025 venue: ICLR task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: reducing_the_token_rate_of_speech_representations_below role: supports claim: Reducing the token rate of speech representations below 10 Hz is sufficient to preserve semantic content adequate for spoken language modelling, while substantially improving training and inference efficiency. source: §5.6, Table 6 evidence: SyllableLM at 6.25 Hz, 90M parameters, matches or exceeds TWIST models up to 13B parameters on sBLIMP semantic understanding benchmarks, with 30x less training compute and 4.5x faster inference than an equal-sized TWIST baseline. confidence: high relevance: high - claim_id: the_loss_surface_of_a_masked_prediction_ssl role: supports claim: The loss surface of a masked-prediction SSL model encodes latent syllabic segmentation boundaries discoverable without additional supervised signal or cross-modal supervision. source: §3.1, Table 1 evidence: LossPred, applied to a frozen HuBERT student-teacher pair without any training, achieves F1-50 of 59.6 on syllabic boundary detection, outperforming the feature-similarity-based baseline (47.3) while requiring no fine-tuning. confidence: high relevance: high - claim_id: iterative_student_teacher_distillation_over_pseudo_syllabic_boundaries role: supports claim: Iterative student-teacher distillation over pseudo-syllabic boundaries progressively sharpens SSL encoder representations toward syllable-level organisation. source: §5.3, Table 2 evidence: SylBoost applied to HuBERT improves boundary detection F1 from 60.1 (LossPred initialisation) to 70.2 after two iterations; applying it to Data2Vec2 reaches 73.2, each iteration producing a measurable gain over the previous. confidence: high relevance: low - claim_id: low_frequency_speech_units_that_improve_semantic_modelling role: complicates claim: Low-frequency speech units that improve semantic modelling efficiency may sacrifice robustness to speaker rate variation. source: Appendix A.5, Table 11 evidence: SylBoost unit counts collapse under audio speedups of 0.5x and 0.6x relative to original length, performing comparably to SD-HuBERT only at mild speedups (0.8x–0.9x range), while showing greater robustness to slowdowns. confidence: high relevance: low - claim_id: units_optimised_for_semantic_modelling_in_audiobook_speech role: complicates claim: Units optimised for semantic modelling in audiobook speech may lose paralinguistic information, limiting applicability to domains requiring prosodic or tonal fidelity. source: §6 evidence: The authors note that low-frequency SylBoost units may lose paralinguistic features such as tone, and the entire evaluation is conducted on audiobook data (LibriSpeech / LibriLight); performance on spontaneous or multi-speaker speech is not reported. confidence: high relevance: low limitations: - All training and evaluation uses English audiobook data (LibriSpeech, LibriLight). Generalisation to spontaneous conversational speech, other languages, or multi-speaker settings is not demonstrated. - The interleaved vocoder decoding pipeline introduces a dependency on the TWIST tokeniser and vocoder for resynthesis, meaning the system is not fully end-to-end and unit bitrate for the final waveform is partially bounded by TWIST's own quality ceiling (WER 6.3%). The efficiency gains in the SpeechLM do not fully apply to the decoding pipeline, which involves an additional language model. Scaling beyond 300M parameters was not attempted due to compute constraints, leaving open whether the efficiency advantage persists at very large model scales. The base encoder quality (Data2Vec2 vs newer models like w2v-BERT 2.0) is acknowledged as a confounding factor in cross-model comparisons. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: iclr-2025-tQ1PmLfPBL published_date: "2025-04-24" entry_date: '2026-07-27' year: 2025 venue: ICLR task: - TTS architecture: - flow-matching relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - flow_matching_speech_decoders claims: - claim_id: periodwave_fm_beats_diffusion_fewer_steps role: supports claim: Flow matching enables higher-quality waveform generation than diffusion with fewer inference steps. source: §4.6, Table 8 evidence: PeriodWave with CFM at 6 steps achieves UTMOS 3.628 versus DDPM at 50 steps achieving UTMOS 3.377 on the same architecture; CFM also reaches comparable quality to 50-step DDPM at only 16 steps. confidence: high relevance: low - claim_id: periodwave_multiperiod_pitch_periodicity role: supports claim: Explicit multi-period decomposition in the generator architecture improves pitch accuracy and periodicity over GAN and diffusion vocoders. source: §4.2, §4.6, Table 1, Table 7 evidence: PeriodWave achieves pitch error of 15.04 cents and periodicity 0.0744 on LJSpeech, substantially below BigVGAN (19.02 cents, 0.0782) and all diffusion baselines; ablation shows monotonic improvement as more distinct prime-number periods are added. confidence: high relevance: low - claim_id: periodwave_gan_faster_than_fm_vocoder role: complicates claim: Single-step GAN vocoders achieve significantly faster inference than iterative flow-matching vocoders. source: §E, Table 15, Table 16 evidence: PeriodWave at 16 steps runs at 7.48x real-time versus HiFi-GAN's 166.70x; even PeriodWave at 2 steps (56.36x) remains slower than HiFi-GAN despite already outperforming it on all quality metrics. confidence: high relevance: low - claim_id: periodwave_iterative_reduces_mismatch role: supports claim: Iterative waveform generation reduces train-inference mismatch artefacts in two-stage TTS relative to one-step GAN vocoders. source: §4.7, Table 9, §G evidence: In zero-shot TTS with ARDiT-TTS acoustic features, PeriodWave+FreeU achieves MOS 4.07 versus BigVGAN's 4.03, attributed to iterative refinement correcting imperfections in generated mel-spectrograms rather than propagating them. confidence: medium relevance: low - claim_id: periodwave_streaming_codec_decoding role: supports claim: Flow-matching vocoders can decode neural codec tokens with streaming generation and minimal quality degradation. source: §5, Table 10, Table 11 evidence: PeriodWave trained for parallel generation from Mimi (Q=8) tokens achieves CER 2.5% versus the Mimi decoder's 3.07%; streaming with single-token delay and 2-step sampling maintains comparable quality. confidence: high relevance: low limitations: - At 16 steps, PeriodWave runs at 7.48x real-time, substantially slower than one-step GAN vocoders (HiFi-GAN 166x, BigVGAN 38x). - High-frequency reproduction remains challenging even with multi-band modeling and FreeU; M-STFT metrics are generally worse than GAN baselines despite better perceptual scores. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.naacl-demo.12 published_date: "2025-04-29" entry_date: '2026-07-27' year: 2025 venue: NAACL task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: a_speech_language_model_can_be_initialized_from role: supports claim: A speech language model can be initialized from a pre-trained text LLM and jointly trained on speech recognition, speech synthesis, text continuation, and audio continuation without substantially degrading the text-only capability of the base model. source: §4.3, Table 5 evidence: The 1.7B multi-task model trained on ASR, TTS, TextLM, and AudioLM objectives scores MMLU 30.5, ARC-C 41.3, and HellaSwag 50.4, close to the text-only LLaMA-3.2-1B baseline (32.2, 32.8, 41.2) despite carrying three additional speech tasks. confidence: high relevance: high - claim_id: concatenating_neural_codec_tokens_with_self_supervised_speech role: supports claim: Concatenating neural codec tokens with self-supervised speech representations frame-by-frame is a viable tokenization strategy for both speech understanding and generation tasks within a single sequential model. source: §3.3 evidence: The "Codec_SSL" scheme (ESPnet-Codec combined with XEUS SSL tokens) is used for the headline ASR and TTS experiments and the paper reports it "behaves well in both speech understanding and generation" *(§3.3)*. confidence: medium relevance: high - claim_id: a_decoder_only_autoregressive_speech_language_model_can role: supports claim: A decoder-only autoregressive speech language model can match or exceed dedicated, larger ASR-only systems on English benchmarks while using substantially fewer parameters. source: §4.2, Table 3 evidence: A 442M-parameter ESPnet-SpeechLM ASR model reaches average WER 5.4% across six English test sets, matching OWSM v3.1-medium (1.02B, 5.4%) and beating Whisper-small (244M, 6.4%) and Whisper-medium (769M, 5.7%). confidence: high relevance: high - claim_id: cross_system_comparisons_of_speech_language_models_reported role: complicates claim: Cross-system comparisons of speech language models reported in the literature are frequently not run under matched conditions, limiting how much can be concluded from any single performance table. source: §4.3, Table 5 evidence: In the multi-task comparison (Table 5), competitor numbers for Moshi, VITA, GLM-4-Voice, and others are taken directly from their own published reports rather than reproduced by the authors, and the paper explicitly flags this with footnote markers. confidence: high relevance: high - claim_id: combining_a_codec_tokenizer_with_a_self_supervised role: complicates claim: Combining a codec tokenizer with a self-supervised tokenizer frame-by-frame is reported as an effective design choice but is not validated against single-tokenizer ablations in the same controlled setting. source: §3.3 evidence: The claim that Codec_SSL tokenization "behaves well" rests on a single line of justification without a paired ablation against codec-only or SSL-only tokenization on the same task and dataset. confidence: high relevance: low limitations: - The multi-task model's TTS quality (Proxy MOS 3.99, WER 6.0%) is noticeably weaker than the single-task TTS model trained on the same architecture (Proxy MOS 4.03, WER 3.1%), indicating a capacity or interference cost to joint multi-task training that the paper reports but does not analyze further. Most training and evaluation data is English-only (the multilingual text corpus is used only for the TextLM objective, not for speech tasks), so the demonstrated speech capabilities are not evidence of multilingual generalization. Competitor numbers in the multi-task comparison table are drawn from third-party reports under unmatched training data and conditions rather than reproduced by the authors, which the paper itself notes. As a system/demo paper, the contribution is the toolkit and its reference recipes rather than a novel architecture or training method; the headline numbers serve to validate functionality rather than push state of the art on any individual benchmark. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.naacl-demo.21 published_date: "2025-04-29" entry_date: '2026-07-27' year: 2025 venue: NAACL task: - SCA architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: cascaded_pipelines_higher_output_quality_than_e2e role: supports claim: Cascaded spoken dialogue pipelines built from dedicated, task-specific ASR and TTS components can produce substantially higher audio output quality than current end-to-end spoken dialogue systems that generate speech directly from audio input. source: §7, Table 4 evidence: On Switchboard Eval 2000, the directly-synthesized speech from the E2E system Mini-Omni scores UTMOS 2.88, versus 3.97-4.07 for cascaded pipelines using LJSpeech, LibriTTS, or VCTK VITS as the TTS module. confidence: medium relevance: high - claim_id: e2e_coherent_but_repetitive_across_turns role: complicates claim: End-to-end spoken dialogue systems that respond directly to audio input without explicit dialogue-context modeling can produce coherent individual responses while still exhibiting unnatural conversational behavior such as repeated or templated phrasing across turns. source: §7, Table 3 evidence: The text dialogue evaluation finds that Mini-Omni "produces highly coherent and contextually relevant responses" but "exhibits significant overlap in its responses, often generating template-like outputs," which the authors note could make conversations feel less natural in deployment. confidence: medium relevance: high - claim_id: domain_mismatched_asr_produces_misleading_wer role: complicates claim: Automatic ASR module benchmarking on conversational telephone speech corpora can produce highly misleading word error rates when the candidate ASR model's training domain does not match the conversational, disfluent register of the test data. source: §7, Table 2 evidence: An ESPnet LibriSpeech-trained ASR model evaluated on Switchboard Eval 2000 yields a WER of 342.2% and CER of 298%, far worse than domain-matched models like OWSM 3.1 (14.4% WER), illustrating a severe train/test domain mismatch effect rather than a genuine architecture comparison. confidence: high relevance: high - claim_id: transcript_quality_affects_downstream_tts_intelligibility role: supports claim: The choice of input transcript quality substantially affects downstream TTS intelligibility metrics independent of the TTS model itself, because more grammatically well-formed text is easier for TTS systems to synthesize accurately. source: §A.4, Table 8 evidence: Replacing ground-truth (disfluent, spontaneous) Switchboard transcripts with LLM-generated text responses as TTS input improves intelligibility across all tested TTS models, e.g. LJSpeech VITS WER drops from 16.1% to 6.0% under OWSM 3.1 hypotheses. confidence: high relevance: low - claim_id: dialogue_lm_sensitivity_to_asr_noise_varies_by_size role: complicates claim: Text dialogue response generators integrated into a cascaded spoken dialogue pipeline are not strongly degraded by upstream automatic speech recognition errors relative to using ground-truth transcripts, though smaller language models may be more sensitive to this noise than larger ones. source: §A.3, Table 7 evidence: When substituting OWSM 3.1 ASR transcripts for ground-truth transcripts as dialogue context, LLaMA-3.2-1B perplexity changes only slightly (48.2 to 47.5) while SmolLM v2-1.7B perplexity degrades substantially (113.7 to 285.1). confidence: medium relevance: high limitations: - The headline comparative analysis is conducted on a single dataset (11 hours of Switchboard Eval 2000) with only one example E2E system (Mini-Omni) and a 4-participant human pilot drawn from the author group; findings should be read as a toolkit demonstration rather than a generalizable benchmark. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.naacl-long.484 published_date: "2025-04-29" entry_date: '2026-07-27' year: 2025 venue: NAACL task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: single_stream_representation_matches_doubled_channel role: supports claim: A single-stream control-token representation for full-duplex dialogue can match or exceed a doubled-channel alternating representation in both naturalness and behavioral adherence, while using fewer tokens per unit of audio duration. source: §7.1, §7.2, Table 4, Table 5 evidence: On the same Llama3.2-1B backbone and Behavior-SD training data, the streamlined-unit variant scores 4.09 naturalness MOS and 0.58 interruption adherence vs. 3.90 MOS and 1.21 interruption adherence for the alternating-unit variant. confidence: high relevance: high - claim_id: first_utterance_prompting_improves_speaker_consistency role: supports claim: Conditioning a speech-prompted TTS synthesizer on a speaker's very first utterance, in addition to their most recent utterance, materially improves long-range speaker identity consistency in independently synthesized multi-turn dialogue. source: §7.3, Table 6 evidence: WavLM-Base+ cosine similarity between the prompt and synthesized speech rises from 0.680 (no prompt) to 0.889 when conditioning on the first utterance alone, and combining first-utterance and previous-utterance prompts yields the best transition smoothness (0.872) without degrading global identity (0.885). confidence: high relevance: low - claim_id: text_pretraining_helps_narrative_not_behavior_adherence role: complicates claim: Pretraining a dialogue generation model on text-only dialogue before fine-tuning on speech-unit sequences improves adherence to narrative content but is not necessary for behavioral adherence. source: §7.2, Table 5 evidence: Removing the text-dialogue pretraining stage drops narrative adherence (GPT-4o-rated) from 3.11 to 2.80, while behavioral adherence scores for filler words, backchannels, and interruptions remain comparable (0.10/0.87/0.64 vs. 0.15/0.87/0.58 with pretraining). confidence: medium relevance: low - claim_id: cascaded_pipelines_lack_behavioral_context_tracking role: complicates claim: Cascaded LLM-then-TTS pipelines for spoken dialogue generation can achieve strong narrative coherence and sound quality but struggle to maintain conversational meaningfulness because they lack any mechanism to track behavioral context (speaker turn identity, backchannel placement) across the full dialogue. source: §7.1, §6.2, Table 3 evidence: GPT-4o and Llama3-70B cascaded baselines achieve the highest narrative adherence scores (4.58 and 4.09) but their meaningfulness MOS (3.97 for both) is lower than the proposed model (4.04), attributed by the authors to speaker confusion and misplaced backchannels in cascaded outputs. confidence: medium relevance: high limitations: - All quantitative evaluation (human MOS, behavioral adherence, speaker consistency) is conducted entirely on Behavior-SD's own synthetic test split, generated by the same LLM/TTS pipeline used for training data; no evaluation on real human-recorded dialogue is reported. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2505.02625' published_date: "2025-05-05" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: autoregressive_streaming_tts_decoders_in_modular_spoken_conversational role: supports claim: Autoregressive streaming TTS decoders in modular spoken conversational agents produce higher-quality speech and smaller S2T-to-S2S accuracy gaps than non-autoregressive streaming decoders, at a modest latency cost. source: §5.1, Table 1 evidence: Larger R:W write chunks improve UTMOS (speech quality) at the cost of latency, while smaller values reduce latency at a modest quality cost (Table 4). confidence: high relevance: high - claim_id: modular_spoken_conversational_agents_trained_on_tens_of role: supports claim: Modular spoken conversational agents trained on tens of thousands of hours of synthesized speech-to-speech dialogue can match the performance of native SpeechLMs trained on millions of hours of unsupervised speech data. source: §5.1, Table 1 evidence: The 200K multi-turn dialogue samples are synthesized from Alpaca/UltraChat text data using CosyVoice 2 for responses and FishSpeech-1.5 for diverse instruction voices. confidence: high relevance: high - claim_id: jointly_conditioning_a_tts_language_model_on_llm role: supports claim: Jointly conditioning a TTS language model on LLM hidden states and text token embeddings via a learned gate fusion improves both semantic consistency (WER) and instruction-following quality over hidden-state-only conditioning. source: §5.2, Table 2 evidence: 'Rather than conditioning M_TTS on LLM hidden states alone, the model adaptively blends two representations per output position: a projected hidden state from the LLM (carrying contextual information) and a text token embedding (ensuring alignment with the textual response).' confidence: high relevance: high - claim_id: tts_language_model_pretraining_on_text_speech_pairs role: supports claim: TTS language model pretraining on text-speech pairs is a critical prerequisite for stable convergence in modular SpeechLMs; initializing from a language model alone is insufficient. source: §5.2, Table 3 evidence: 'LLaMA-Omni 2 is a modular SpeechLM consisting of three main components: a speech encoder (Whisper-large-v3), an LLM backbone (one of the Qwen2.5-0.5B/1.5B/3B/7B/14B-Instruct variants), and a streaming speech decoder composed of a TTS language model and a causal flow matching vocoder.' confidence: high relevance: high - claim_id: in_streaming_speech_generation_the_write_chunk_size role: supports claim: In streaming speech generation, the write chunk size (W) primarily determines speech naturalness while the read chunk size (R) primarily determines text-speech alignment, with latency jointly determined by both. source: §5.2, Table 4 evidence: 'Streaming is controlled by a Read-Write (R:W) strategy: after every R LLM text tokens are generated, W speech tokens are produced by M_TTS and synthesized as a waveform chunk.' confidence: high relevance: high limitations: - All evaluations are conducted in English only. The model is trained on a single fixed output voice, and no multilingual or voice-diversity experiments are reported. Generalization to other languages or to emotionally expressive speech is untested. - The model cannot modulate speech style (emotion, speaking rate, dialect) in response to paralinguistic cues in the input, because training data contains only conventional speech-to-speech dialogue. The authors note this as planned future work. The benchmarks used (SpokenQA accuracy, ChatGPT score) are narrow and do not cover naturalness in unconstrained conversational settings or robustness to noisy or accented input. Comparisons to Minmo (a concurrent work using 1.4M hours) are mentioned in related work but not included in the main experimental table, leaving the data efficiency claim partially unverified against the most directly comparable system. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2505.09558' published_date: "2025-05-14" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: text_only_eval_misses_acoustic_mismatches role: supports claim: Text-only evaluation of speech-to-speech dialogue systems systematically misses acoustic-level mismatches that human judges readily detect. source: §2.2, §4.3 evidence: Existing benchmarks (VoiceBench, VoxDialogue, SD-Eval, AirBench) transcribe audio to text before scoring, so a response with correct wording but a mismatched emotional tone (e.g. cheerful mockery of a fearful query) receives no penalty, while WavReward and human raters both detect the mismatch. confidence: high relevance: low - claim_id: group_rl_outperforms_sft_for_audio_reward_models role: supports claim: Reinforcement learning with group-based advantage estimates substantially outperforms supervised fine-tuning for training audio-domain reward models, consistent with patterns observed in text-domain RLHF. source: §4.2, Table 2 evidence: GRPO-trained WavReward outperforms full-parameter supervised fine-tuning, which in turn outperforms LoRA-based fine-tuning, on ChatReward-30K-test scoring accuracy. confidence: high relevance: low - claim_id: cot_improves_audio_judge_ood_generalization role: supports claim: Chain-of-thought reasoning improves scoring accuracy of audio language model evaluators, with the benefit amplified in out-of-domain settings. source: §4.4 evidence: Removing chain-of-thought reasoning from WavReward costs roughly 10 percentage points of accuracy across in-domain categories and up to 21.7 points on the out-of-domain RealDialogue set. confidence: high relevance: high - claim_id: nonlinear_reward_shaping_improves_calibration role: supports claim: Nonlinear reward shaping that penalises large prediction errors exponentially improves calibration across acoustic attributes compared to binary correct/incorrect rewards. source: §3.1, §4.4 evidence: Removing the nonlinear reward component from WavReward's training objective reduces performance particularly on implicit dialogue and generation-scoring tasks relative to the full model. confidence: medium relevance: low - claim_id: implicit_dialogue_harder_to_evaluate_than_explicit role: supports claim: Implicit dialogue scenarios, where the appropriate response depends on inferring the user's emotional state from speech, remain significantly harder to evaluate automatically than explicit instruction-following scenarios. source: §4.2, Table 2 evidence: WavReward's accuracy on implicit emotional dialogue scoring (74.3%) trails its accuracy on explicit paralinguistic and content scoring categories, and the paper notes no single ground truth exists for what constitutes an emotionally appropriate response. confidence: medium relevance: low limitations: - The entire ChatReward-30K training corpus is synthesised using TTS systems (GPT-4o-mini-TTS, Step-Audio-TTS-3B, CosyVoice2) rather than real human speech, and generalisation to real-world dialogue audio is validated only on a 120-sample RealDialogue set. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2506.10274' published_date: "2025-06-12" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec - TTS - evaluation architecture: [] relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: no_discrete_audio_tokenizer_consistently_outperforms_others_across role: supports claim: No discrete audio tokenizer consistently outperforms others across reconstruction, downstream discriminative tasks, and acoustic language modeling; the optimal tokenizer is task- and domain-dependent. source: §3.4, Figure 4 evidence: Downstream performance uses the DASB benchmark with frozen tokenizer representations and lightweight task-specific heads across 12 tasks (ASR, speaker ID/verification, emotion recognition, keyword spotting, intent classification, speech enhancement, speech separation, music genre classification, music source separation, sound event classification, audio source separation). confidence: high relevance: high - claim_id: increasing_the_number_of_codebooks_improves_signal_reconstruction role: supports claim: Increasing the number of codebooks improves signal reconstruction quality but often degrades downstream task performance by adding redundancy that burdens representation-level learning. source: §3.2, §3.2 "Impact of Codebook Size" evidence: Increasing codebook count (bitrate) often hurts downstream performance by adding redundancy. confidence: high relevance: low - claim_id: semantic_distillation_aligning_early_rvq_layers_with_ssl role: supports claim: Semantic distillation (aligning early RVQ layers with SSL features) improves phonetic content preservation in acoustic tokenizers but may reduce cross-domain generalization when the distillation source is speech-specific. source: §4.2 "Distillation Effect", Table 16 evidence: Acoustic tokenizers trained on multi-domain data underperform speech-specific or semantic tokenizers for TTS. confidence: high relevance: low - claim_id: domain_alignment_between_tokenizer_training_data_and_evaluation role: supports claim: Domain alignment between tokenizer training data and evaluation domain is the dominant factor in discrete audio codec performance, outweighing quantization method or bitrate choices. source: §4.2 "Data Domains" evidence: The ablation study (Section 4) uses ESPnet-Codec with a controlled DAC backbone to isolate the effects of quantization method (RVQ, SVQ, FSQ), training domain (speech, audio, music, multi-domain), sampling rate (16 kHz vs. 44.1 kHz), and semantic distillation from SSL features. confidence: high relevance: low - claim_id: continuous_speech_representations_e_g_wavlm_large_consistently role: supports claim: Continuous speech representations (e.g., WavLM-large) consistently outperform all discrete tokenizers on discriminative tasks, with the gap widening in low-resource conditions. source: §3.2, Table 7 evidence: Continuous WavLM-large representations remain the ceiling, outperforming all discrete tokenizers across most tasks, with the gap widening in low-resource settings. confidence: high relevance: low limitations: - The TTS and audio LM evaluations are conducted with constrained training budgets (VALL-E on LibriTTS only; 300M audio LM at half the original training compute), which limits the practical transferability of findings on TTS tokenizer ranking to large-scale production settings. - The acoustic LM evaluation uses a single architecture (Qwen-2.5 based, 357M) and a single dataset (LibriHeavy), so findings on which tokenizers support better SLMs may not generalise to other LM architectures or training scales. The ablation study (Section 4) is restricted to a DAC-backbone framework; FSQ and SVQ conclusions may not transfer to other encoder-decoder designs. Evaluation metrics for audio generation (FAD, KLD, CLAP) conflate vocoder quality with language model quality, making it difficult to attribute performance differences to the tokenizer's representational properties versus its decoder quality. The paper does not evaluate streaming tokenizers under actual latency constraints, limiting guidance for real-time deployment. Trustworthiness considerations (voice deepfakes, bias) are raised as open concerns but not empirically evaluated. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.02380' published_date: "2025-07-03" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: routing_llm_hidden_states_into_the_tts_module role: supports claim: Routing LLM hidden states into the TTS module's embedding space can enable voice cloning in an end-to-end spoken chatbot without a dedicated speaker encoder. source: §1.2 evidence: JoyTTS projects Qwen-7B hidden states via an MLP into the CosyVoice2-based LLM-TTS embedding, achieving SS of 0.73 on seed-tts-zh using prompt audio as the only speaker reference. confidence: high relevance: low - claim_id: jointly_training_an_llm_chat_module_with_a role: complicates claim: Jointly training an LLM-Chat module with a TTS module in a chatbot pipeline can degrade intelligibility relative to running the TTS component standalone, even when speaker similarity improves. source: §4, Table 1 evidence: JoyTTS achieves WER of 5.09 compared to 1.45 for standalone CosyVoice2 on seed-tts-zh, despite closing the speaker similarity gap (JoyTTS SS 0.73 vs. CosyVoice2 SS 0.748). confidence: high relevance: low - claim_id: end_to_end_spoken_chatbot_systems_pairing_a role: supports claim: End-to-end spoken chatbot systems pairing a 7B-parameter LLM with an autoregressive TTS module can achieve sub-2-second response latency on a single consumer GPU without specialised inference optimisations. source: §4 evidence: JoyTTS reports 1.8-second end-to-end latency on a single NVIDIA 4090D with no engineering optimisation applied. confidence: high relevance: low limitations: - The evaluation is limited to a single Chinese benchmark (seed-tts-zh), leaving performance on English, multilingual, or spontaneous conversational speech uncharacterised. The WER gap between JoyTTS (5.09) and standalone CosyVoice2 (1.45) is large and unexplained; the paper does not ablate whether the regression originates from the joint training procedure, the conversational data distribution, or the MLP projection coupling. No subjective listening tests are reported, making it impossible to assess naturalness or perceived quality beyond intelligibility and speaker similarity metrics. Emotion control, identified as a target for future work, is not implemented in the current system. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2506.23325' published_date: "2025-07-09" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec architecture: - GAN - hybrid relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models - adversarial_codec_speech_models claims: - claim_id: minimizing_shared_parameters_between_semantic_and_acoustic_encoding role: supports claim: Minimizing shared parameters between semantic and acoustic encoding pathways in a speech codec reduces task conflict and enables simultaneous strong text alignment and high acoustic fidelity at low bitrates. source: §3.1, §4.4, Table 4 evidence: XY-Tokenizer's dual-tower architecture (parameters shared only in the RVQ module) achieves WER 0.13 and SPK-SIM 0.83 at 1 kbps; a matched single-channel model sharing encoder and RVQ across both tasks achieves SPK-SIM 0.77 with the same WER, confirming that parameter sharing degrades reconstruction. confidence: high relevance: low - claim_id: llm_based_asr_supervision_during_codec_pre_training role: supports claim: LLM-based ASR supervision during codec pre-training provides stronger text alignment than SSL representation distillation for low-bitrate codecs. source: §4.3, Table 3 evidence: At approximately 1 kbps, XY-Tokenizer (LLM-based ASR, WER 0.13) substantially outperforms SpeechTokenizer variants (distillation-based, WER 0.18-0.34) and Mimi-8 (distillation-based, WER 0.28) on the ASR probing task, while maintaining comparable or better reconstruction quality than those baselines. confidence: high relevance: low - claim_id: representation_distillation_from_ssl_models_for_semantic_alignment role: complicates claim: Representation distillation from SSL models for semantic alignment introduces reconstruction-quality penalties that worsen with distillation strength at low bitrates. source: §4.3, Table 3 evidence: SpeechTokenizer-x3 (5x stronger distillation than the official setting) achieves WER 0.18 but SPK-SIM only 0.48 at 1.5 kbps, compared to SpeechTokenizer-x1 (WER 0.34, SPK-SIM 0.65), demonstrating that stronger distillation systematically degrades acoustic fidelity. confidence: high relevance: low - claim_id: allowing_the_llm_decoder_to_train_freely_during role: complicates claim: Allowing the LLM decoder to train freely during multi-task codec pre-training improves autoregressive text generation but degrades the encoder's transferable text-alignment representations. source: §4.4, Table 5 evidence: A trainable-LLM variant achieves lower LLM decoder WER (0.03 vs. 0.06 at 200K steps) but higher ASR probing WER (0.18 vs. 0.13), with the probing WER worsening progressively over 800K training steps, suggesting the encoder's text alignment capacity migrates into the flexible decoder. confidence: high relevance: low - claim_id: whisper_s_supervised_asr_pre_training_enables_better role: supports claim: Whisper's supervised ASR pre-training enables better paralinguistic information preservation in codec encoder architectures than self-supervised alternatives. source: §2, Table 1 evidence: Preliminary auto-encoder experiments with frozen encoders show Whisper achieves SPK-SIM 0.68, STOI 0.88, and PESQ-NB 2.03 on LibriSpeech test-clean, compared to WavLM (SPK-SIM 0.53) and HuBERT (SPK-SIM 0.42), motivating Whisper as the codec encoder initialization. confidence: high relevance: low limitations: - Evaluation uses only objective metrics (WER, SPK-SIM, STOI, PESQ). No subjective listening tests or MOS scores are reported, making it impossible to assess perceptual audio quality relative to competing systems. - Achieving further reductions in bitrate below 1 kbps without performance degradation remains an open challenge noted by the authors. The scaling behavior of the two-stage training approach with respect to parameter count and dataset size is not characterized, leaving open questions about how to optimize training efficiency for larger variants. The LLM-based ASR decoder (Qwen2.5-0.5B) adds substantial parameter overhead during pre-training that is absent during inference, but the computational cost and training complexity of this stage relative to distillation-based approaches is not discussed. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2412.18603' published_date: "2025-07-10" entry_date: '2026-07-27' year: 2025 venue: ICML task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: ssm_enables_stable_long_form_generation role: supports claim: Linear-time state-space sequence models enable stable long-form speech generation that Transformer spoken LMs cannot sustain at comparable scale. source: §7.1, §7.2, Tables 3-4 evidence: SpeechSSM-2B maintains N-MOS of 4.12-4.16 across all time strata up to 4+ minutes and stable semantic coherence in 16-minute generation, while Transformer-based baselines and windowed extensions of prior spoken LMs degrade rapidly beyond their training lengths. confidence: high relevance: high - claim_id: separating_speaker_conditioning_preserves_identity_long_form role: supports claim: Separating speaker conditioning into a dedicated acoustic synthesis stage preserves voice identity more effectively in long-form speech generation than encoding speaker information in semantic tokens. source: §6.1, §7.1, Tables 2-3 evidence: SpeechSSM achieves SpkrSim of 0.85 in long-form evaluation versus 0.33-0.41 for GSLM and TWIST, attributed to SoundStorm's speaker-prompted acoustic stage handling identity independently of the semantic LM. confidence: medium relevance: high - claim_id: standard_spoken_lm_metrics_unreliable role: complicates claim: Standard evaluation metrics for spoken language models are unreliable or saturating for modern systems, particularly in long-form settings. source: §6.1, §4 evidence: sWUGGY scores negatively correlate with generation quality at large token vocabularies (32k vs. hundreds); short-form holistic MOS approaches ground truth at 7 seconds while qualitative failures persist; transcript perplexity favors repetitive outputs at default temperatures. confidence: high relevance: high - claim_id: long_form_generation_remains_below_human_quality role: complicates claim: Long-form speech generation remains far below human quality even for state-of-the-art spoken language models. source: §7.1, Table 3 evidence: No model achieves any wins against LibriSpeech-Long ground truth in 4-minute side-by-side comparisons judged by an LLM on fluency, coherence, logicality, and interestingness. confidence: high relevance: high limitations: - Model weights are not released and inference requires TPU infrastructure; independent reproducibility of the long-form results is not currently possible. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.16632' published_date: "2025-07-22" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: interleaving_discrete_audio_tokens_with_text_tokens_in role: supports claim: Interleaving discrete audio tokens with text tokens in a shared language model vocabulary enables end-to-end spoken dialogue systems that respond coherently to paralinguistic cues without requiring separate speech synthesis pipelines. source: §3.1, §4.6 evidence: Step-Audio 2 is a multi-modal LLM that processes raw audio as input and generates interleaved sequences of discrete text and audio tokens. confidence: high relevance: high - claim_id: reinforcement_learning_applied_to_audio_language_models_can role: supports claim: Reinforcement learning applied to audio language models can improve reasoning efficiency in complex acoustic scenarios while preserving generation quality. source: §3.4 evidence: Supervised fine-tuning follows on 4B tokens, succeeded by multi-stage reinforcement learning using proximal policy optimization (PPO) and group relative policy optimization (GRPO) to improve reasoning efficiency and audio perceptual ability. confidence: high relevance: high - claim_id: retrieval_augmented_generation_and_external_tool_calling_can role: supports claim: Retrieval-augmented generation and external tool calling can substantially reduce hallucination and expand capability (timbre switching, web-grounded responses) in large audio language models without architectural redesign. source: §3.1, §4.5 evidence: A key capability addition is tool calling, including a novel audio search tool that maintains a library of hundreds of thousands of speech samples with descriptions. confidence: high relevance: high - claim_id: comprehensive_multi_task_pre_training_across_asr_tts role: supports claim: Comprehensive multi-task pre-training across ASR, TTS, translation, and conversation significantly improves spoken dialogue performance in low-resource languages and accented speech. source: §3.2, §4.1 evidence: 'Training proceeds in four stages: adaptor alignment on 100B ASR tokens with frozen encoder and LLM; vocabulary expansion and joint training on 256B mixed tokens; main pre-training on 800B tokens across ASR, TTS, translation, and speech-to-speech conversation data; and a cooldown phase on 200B high-quality tokens.' confidence: high relevance: high - claim_id: existing_audio_language_model_benchmarks_fail_to_capture role: complicates claim: Existing audio language model benchmarks fail to capture fine-grained paralinguistic comprehension and voice-triggered tool invocation, leaving important capability dimensions systematically unmeasured. source: §4.2, §4.5 evidence: On the authors' StepEval-Audio-Paralinguistic benchmark evaluating 11 paralinguistic dimensions, Step-Audio 2 achieves 83.09% average accuracy compared to 49.64% for Kimi-Audio and 44.18% for Qwen-Omni (Table 2). confidence: high relevance: high limitations: - Two of the main evaluation benchmarks (StepEval-Audio-Paralinguistic, StepEval-Audio-Toolcall) are introduced by the authors themselves and have not been validated independently. Results on these benchmarks may overstate absolute capability levels even if relative comparisons are informative. - Model size is not reported, which makes parameter-count comparisons with other open-source systems (Kimi-Audio, Qwen2.5-Omni) difficult to interpret fairly. The open-source mini variant uses Qwen2.5-7B as its backbone, but the full model's parameter count remains undisclosed. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2505.15670' published_date: "2025-07-25" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: fullduplex_without_speech_pretraining role: supports claim: Full-duplex spoken dialogue systems can be built without speech-text pretraining by routing user audio through a pretrained streaming encoder rather than requiring the LLM to learn audio representations end-to-end. source: §3, §6.1, §6.2, Tables 2-3 evidence: SALM-Duplex skips speech pretraining entirely and instead uses a 100M streaming CTC encoder for user input; it still outperforms Moshi on barge-in success rate (94.5% vs. 55.1%) and reasoning GPT scores across all five evaluation sets. confidence: high relevance: high - claim_id: asymmetric_duplex_enables_specialization role: supports claim: Asymmetric duplex architectures that separate user and agent speech pathways enable independent specialization, including speaker-specific codec fine-tuning without affecting user comprehension. source: §3.2, §6.3, Table 4 evidence: Personalized 0.6 kbps NanoCodec (fine-tuned on 21k target-speaker utterances) outperforms Moshi's Mimi at 1.1 kbps and untuned NanoCodec at 1.2 kbps on MOS, CER, and SECS, while operating at roughly half the bitrate. confidence: high relevance: high - claim_id: e2e_s2s_not_consistently_matching_cascaded role: complicates claim: End-to-end speech-to-speech models do not consistently match optimal cascaded systems in reasoning quality, even when the cascaded oracle has access to ground-truth ASR transcriptions of user speech. source: §6.2, Table 3 evidence: SALM-Duplex outperforms GT+LLM on Roleplay and ASR-QA but underperforms on UltraChat (3.5 vs. 6.4), Topic, and Alpaca; the gap reflects compounding ASR error and limited backbone reasoning capacity at 1.1B parameters. confidence: medium relevance: high - claim_id: bargein_latency_more_discriminative_than_utmos role: supports claim: Barge-in success rate and latency together constitute more discriminative signals for evaluating full-duplex systems than speech quality metrics such as UTMOS. source: §5.2, §6.1, Table 2 evidence: On the Impatient set, SALM-Duplex and Moshi have zero false alarms each and UTMOS within 0.2 points (4.0 vs. 3.8), yet differ by 39.4 percentage points in barge-in success rate, making success rate the dominant differentiating metric. confidence: high relevance: high limitations: - Quantitative comparison is restricted to Moshi; other contemporaneous duplex systems (OmniFlatten, SALMONN-Omni, MinMo) are discussed but not benchmarked, limiting how broadly the evaluation protocol's conclusions generalize. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.18119' published_date: "2025-07-25" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: goatslm_paralinguistic_internalised_without_instruction role: supports claim: Modality-alignment spoken language models can internalize paralinguistic cues (dialect, age, emotion, non-speech vocalizations) through staged training without requiring explicit user instructions at inference time. source: §5.2, Table 7, Table 8 evidence: GOAT-SLM achieves 50.73% average dialect-following accuracy across five Chinese dialects without explicit prompting, versus 18.91% for the next-best model, and 40.91% on non-speech vocal response where all other models score below 10%. confidence: high relevance: high - claim_id: goatslm_shared_head_preserves_linguistic_competence role: supports claim: Initializing a speech generation head from a pretrained LLM text head promotes parameter reuse and can preserve linguistic competence in the speech branch without full retraining. source: §3, §5.1, Table 6 evidence: The Speak module is initialized from the Write module's weights and retains multi-turn dialogue accuracy of 84% despite not training on multi-turn data, attributed to this parameter-sharing initialization strategy. confidence: medium relevance: high - claim_id: goatslm_paralinguistic_tradeoff_general_qa role: complicates claim: Incorporating paralinguistic awareness into a spoken language model introduces a trade-off with general semantic question-answering performance. source: §5.1, Table 4 evidence: GOAT-SLM's general audio question-answering performance declines to average level across eight QA datasets compared to models without paralinguistic training, a trade-off the authors attribute to the expanded training objectives. confidence: medium relevance: high - claim_id: goatslm_dialect_generation_uneven role: complicates claim: Dialectal speech generation quality is uneven across dialects, with closely related varieties more reliably acquired than typologically distant ones. source: §5.2, Table 10 evidence: GOAT-SLM exceeds 90% subjective consistency for Cantonese, Henan, Shanghainese, and Sichuanese dialect generation, but achieves only 57.2% for northeastern Mandarin, which is lexically close to standard Mandarin. confidence: high relevance: high - claim_id: goatslm_gradient_masking_improves_pronunciation role: supports claim: Confidence-based gradient masking during speech token training can improve pronunciation stability by selectively updating tokens where the model's predictions are reliable. source: §4.3 evidence: GOAT-SLM applies confidence-based gradient masking during Stage 3 speech token training, masking low-confidence tokens during backpropagation, credited with enhanced pronunciation stability. confidence: medium relevance: high limitations: - Evaluation is restricted to Chinese-language dialects and Mandarin-English bilingual content; generalization to non-Chinese languages is untested. - The paper does not ablate the impact of LLM-initialized speech branch weights versus random initialization in a controlled experiment. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.18897' published_date: "2025-07-25" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - adversarial_codec_speech_models claims: - claim_id: single_quantizer_neural_codecs_can_match_multi_quantizer role: supports claim: Single-quantizer neural codecs can match multi-quantizer systems in perceived speech quality at far lower bitrates when combined with stabilized VQ spaces and asymmetric decoder architectures. source: §4.3, Table 1 evidence: HH-Codec achieves UTMOS 3.61 on LibriTTS test-clean at 0.3 kbps with a single quantizer, competitive with DAC's 4-quantizer configuration at 4 kbps (UTMOS 3.41) and markedly above SpeechTokenizer's single-quantizer variant at 0.75 kbps (UTMOS 1.26). confidence: high relevance: low - claim_id: multi_layer_vq_training_with_single_layer_inference role: supports claim: Multi-layer VQ training with single-layer inference, where auxiliary quantizer layers act as regularizers, substantially improves codebook utilization in high-compression single-quantizer settings. source: §4.4, Tables 2-3 evidence: SLM-VQ achieves 98% codebook utilization at 8192 entries versus 56% for Classic VQ and 92% for single-layer SLM-VQ, while improving UTMOS from 2.76 (Classic VQ) to 3.07 on LibriTTS test-other. confidence: high relevance: low - claim_id: dual_domain_supervision_combining_intermediate_mel_spectrogram_and role: supports claim: Dual-domain supervision combining intermediate mel-spectrogram and final audio reconstruction objectives is critical for stable high-compression neural codec training. source: §4.4, Table 2 evidence: Reducing to single audio-domain supervision drops UTMOS from 3.07 to 1.85 and SPK-SIM from 0.64 to 0.33 on LibriTTS test-other, the largest degradation across all ablation variants. confidence: high relevance: low - claim_id: standard_adversarial_codec_training_recipes_break_down_at role: complicates claim: Standard adversarial codec training recipes break down at extreme compression ratios, requiring architectural and procedural modifications to avoid collapse. source: §1 evidence: Below 0.3 kbps with existing methods, the paper documents adversarial training collapse, a 63% UTMOS drop below 30 tokens/s, 43% codebook utilization at 8192 entries, and minimal benefit from expanding training data, all addressed in HH-Codec through SLM-VQ and progressive training. confidence: high relevance: low limitations: - HuBERT-based semantic distillation is trained on English, so multilingual performance of SLM-VQ is unknown. The downstream spoken language modeling experiment measures only training loss reduction rather than end-to-end TTS quality, leaving it unclear how the 24-token-per-second compression affects downstream synthesis intelligibility and naturalness. Training data conditions differ across compared baselines (WavTokenizer and DAC use larger or different datasets), which limits direct attribution of gains to architecture versus data. The ablation study is conducted on a subset of training data only (LibriTTS train-100/360, not the full training set including Emilia), which may underestimate some component contributions. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.1252 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS - SCA architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: acoustic_accent_match_improves_chatbot_reception role: supports claim: Incorporating an acoustic accent that matches a user's linguistic community improves chatbot warmth, similarity, and engagement preference even when the underlying text remains in standard dialect. source: §4.4.2, Figure 4 evidence: The AA-accent + SAE-text spoken chatbot configuration outperforms the SAE baseline across all evaluated Likert dimensions (comprehension, communication ease, similarity, role appropriateness, engagement preference) among 8 AAE-speaking evaluators. confidence: medium relevance: low - claim_id: high_intensity_text_dialect_produces_caricature role: complicates claim: Text-based dialect generation at high intensity produces caricature-like representations that degrade perceived inoffensiveness and role appropriateness with speakers of that dialect. source: §4.4.1, Figure 3 evidence: High-AAE text chatbots receive the worst inoffensiveness scores among all 9 AAE text variants, with near-neutral or negative scores for trustworthiness and role appropriateness, which the authors attribute to caricature-like over-modification rather than authentic AAE. confidence: medium relevance: low - claim_id: phonetic_modification_dominates_llm_generated_aae role: supports claim: Phonetic modification dominates LLM-generated AAE output, while semantic feature generation remains sparse across all tested models and intensity levels. source: §4.3, Figure 2 evidence: Claude-Sonnet-3.5-based AAE feature tagging (91% accuracy) shows phonetic features dominate across all three LLMs' generated AAE text, while semantic AAE features remain sparse regardless of intensity level. confidence: medium relevance: low - claim_id: tts_trained_on_standard_dialect_degrades_on_minority_accent role: complicates claim: TTS models trained predominantly on standard-dialect data exhibit degraded speech clarity when reproducing minority accent features, limiting the fidelity of accent-conditioned spoken chatbots. source: §5 evidence: F5-TTS, trained predominantly on Standard American English data, shows decreasing speech clarity ratings as AAE dialect intensity increases in the spoken chatbot condition, even though it is applied only via a single CORAAL reference clip for accent. confidence: medium relevance: low limitations: - The evaluator pool is small (12 text, 8 spoken) from a single university demographic, limiting the statistical reliability of the 15-metric Likert evaluation framework. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.388 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: diva_distillation_without_instruction_responses role: supports claim: Cross-modal context distillation using ASR transcription pairs can transfer instruction-following capabilities to a speech encoder without annotated speech responses. source: §3.2, §5.1.2 evidence: DiVA trains on CommonVoice speech-transcript pairs while its frozen text LLM supplies the distillation target, avoiding annotated response labels. confidence: high relevance: high - claim_id: diva_distillation_avoids_capability_forgetting role: supports claim: Distillation-based speech-language-model training can avoid the instruction-following capability loss observed after narrow supervised fine-tuning. source: §5.1.2, Table 3 evidence: DiVA produced no instruction-ignoring outputs in the reported QA analysis, while Qwen Audio transcribed rather than answered 30% of questions. confidence: high relevance: low - claim_id: diva_joint_alignment_and_distillation_required role: supports claim: Input-token alignment and output-distribution distillation provide complementary supervision and are both required for stable cross-modal instruction transfer. source: §6, Table 5 evidence: Output distillation alone failed on translation and emotion tasks, whereas input alignment alone yielded mostly incoherent generations. confidence: high relevance: high - claim_id: diva_whisper_qformer_initialization role: supports claim: Initialising cross-attention from a pretrained speech decoder provides a transferable audio-to-text alignment bias for spoken question answering. source: §6, Table 5 evidence: Random Q-Former initialization reduced QA performance by 13.7% relative to initialization from Whisper decoder cross-attention weights. confidence: high relevance: low - claim_id: diva_human_preference_diverges_from_automatic_scores role: complicates claim: Human preference can diverge materially from automatic task benchmarks when evaluating spoken assistants. source: §5.2 evidence: Across 522 judgments from 53 Prolific participants, DiVA won 72% of pairwise preferences against Qwen 2 Audio despite trailing it on several automatic benchmarks. confidence: high relevance: low limitations: - The preference test used crowd workers rather than domain experts, and its prompts were not released. - Evaluation was single-turn, so the preference result does not establish multi-turn conversational quality. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.65 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS architecture: - autoregressive-LM - VAE - hybrid relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - variational_latent_speech_models claims: - claim_id: discrete_codec_representations_introduce_a_quantifiable_fidelity_loss role: supports claim: Discrete codec representations introduce a quantifiable fidelity loss relative to continuous mel-spectrogram representations even at high codebook counts, measurable in both WER and speaker similarity. source: §5, Table 1 evidence: 'Objective (Table 1, LibriSpeech test-clean, continuation task): - MELLE WER_C/WER_H: 1.47/1.98, SIM: 0.508 - VALL-E 2 WER_C/WER_H: 1.6/2.32, SIM: 0.504 - VALL-E WER_H: 3.8, SIM: 0.508 - Ground truth WER_H: 2.15, SIM: 0.668 (mel resynthesis: WER_H 2.24, SIM 0.617; EnCodec 8CB: WER_H 2.33, SIM 0.593) — confirming codec fidelity loss hypothesis MELLE achieves 47.9% relative WER reduction vs. VALL-E and 8.1% vs. VALL-E 2 on continuation.' confidence: high relevance: low - claim_id: continuous_valued_autoregressive_speech_synthesis_can_achieve_robustness role: supports claim: Continuous-valued autoregressive speech synthesis can achieve robustness and naturalness on par with codec-based two-stage systems when paired with appropriate regularization objectives. source: §5.1, §5.2, Table 1, Table 3 evidence: 'This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling.' confidence: high relevance: low - claim_id: a_variational_latent_sampling_module_applied_to_continuous role: supports claim: A variational latent sampling module applied to continuous spectrogram prediction provides diversity and robustness benefits analogous to top-p sampling for discrete tokens, without the instability caused by the high similarity of consecutive acoustic codes. source: §3.2.2, §5.3, Table 4 evidence: 'This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling.' confidence: high relevance: low - claim_id: bypassing_the_non_autoregressive_second_stage_in_codec role: supports claim: Bypassing the non-autoregressive second stage in codec language model pipelines reduces inference time while maintaining competitive output quality. source: §5.4, Table 5 evidence: 'Subjective (Table 3, 40 samples cross-sentence): - MELLE MOS: 4.20 ± 0.20 vs. GT 4.29 ± 0.16 (CMOS -0.032, p > 0.1 → not significantly different) - MELLE SMOS: 4.40 ± 0.22 vs. GT 3.94 ± 0.25 — higher than ground truth, suggesting the model''s speaker reproduction is more consistent than inter-utterance variation in the reference - VALL-E 2 MOS: 4.08, CMOS: -0.085 Efficiency (Table 5): - MELLE: 5.49s for 10s speech vs. VALL-E 2: 7.32s (no NAR second pass) - MELLE-R2: 2.76s, MELLE-R4: 1.40s — 5x speedup at r=4 while still competitive Ablation (Table 4): Removing both LSM and SFL degrades cross-sentence WER_C from 1.47 to 23.21 — catastrophic failure.' confidence: high relevance: high - claim_id: prediction_quality_in_continuous_valued_autoregressive_tts_degrades role: complicates claim: Prediction quality in continuous-valued autoregressive TTS degrades gracefully with reduction factor increases, enabling a controllable quality-efficiency trade-off unavailable in discrete-token systems. source: §5.1, Table 1, Table 2 evidence: 'This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling.' confidence: high relevance: low limitations: - '- English-only evaluation; multilingual extension not attempted. - Vocoder quality bottleneck: uses open-source HiFi-GAN trained on 585h LibriTTS; Voicebox''s proprietary vocoder trained on 60Kh provides higher quality ceiling. - Mel-spectrogram as the only continuous representation explored; VAE latent states suggested as future work. - SMOS exceeding ground truth may partly reflect the test setup''s limitation (inter-speaker/inter-session variation in the reference set rather than genuine quality superiority). - No streaming or low-latency inference analysis.' caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.654 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS - codec architecture: - GAN - VAE relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - adversarial_codec_speech_models - variational_latent_speech_models claims: - claim_id: codec_token_distributions_in_the_first_rvq_channel role: complicates claim: Codec token distributions in the first RVQ channel are a meaningful bottleneck for autoregressive generation from text, independent of reconstruction quality. source: §1, §3.3 evidence: The net effect is that no single early channel dominates information content, making the first-channel token distribution lower-entropy and therefore more tractable for a text-conditioned autoregressive model to predict. confidence: high relevance: low - claim_id: redistributing_information_load_uniformly_across_the_first_few role: supports claim: Redistributing information load uniformly across the first few RVQ codebook channels via parallel masked quantization consistently improves speaker similarity in downstream autoregressive TTS. source: §4.4, Table 3 evidence: 'In the first Nq=3 layers the quantizers operate in parallel: each receives only 1/Nq of the latent frame (the rest is masked), so each of the first three codebooks carries an equal, reduced share of the total information.' confidence: high relevance: low - claim_id: a_fourier_based_decoder_with_a_self_attention role: supports claim: A Fourier-based decoder with a self-attention module achieves better codec reconstruction quality than a transposed-convolution upsampler, without length extrapolation issues. source: §3.2, Appendix G, Table 9 evidence: The decoder replaces EnCodec's transposed-convolution upsampler with a Vocos-style Fourier-based decoder. confidence: high relevance: low - claim_id: codec_reconstruction_quality_does_not_scale_substantially_with role: supports claim: Codec reconstruction quality does not scale substantially with training data volume beyond a few hundred hours, while domain generalization does benefit from larger and more diverse datasets. source: Appendix A, Table 5 evidence: 'Codec reconstruction (LibriTTS Test-Clean, Table 1): - Language-Codec with 4 codebooks (3.0 kbps): UTMOS 3.79, PESQ 3.27, STOI 0.949 — outperforms all 4-codebook baselines and several 8-codebook baselines (eg., Encodec-8 PESQ 2.72, STOI 0.939). - Language-Codec with 8 codebooks (6.0 kbps): UTMOS 4.04, PESQ 3.88, STOI 0.972 — best across all models on all metrics. - Results hold under noisy conditions (LibriTTS Test-Other, Table 6) and out-of-domain (LJSpeech, Table 7).' confidence: high relevance: low - claim_id: the_choice_of_underlying_codec_has_a_larger role: supports claim: The choice of underlying codec has a larger impact on zero-shot TTS speaker similarity than on intelligibility, with codec swaps producing 10–15% SPK-SIM gains while WER differences remain within noise. source: §4.3, Table 2 evidence: 'Downstream zero-shot TTS (LibriSpeech Test-Clean, Table 2): - Replacing EnCodec with Language-Codec in VALL-E raises SPK-SIM from 0.612 to 0.700 (+14%) with negligible WER change. - MobileSpeech with Language-Codec: SPK-SIM 0.771, WER 2.9%, MOS-Q 4.20 vs. EnCodec baseline MOS-Q 3.91.' confidence: high relevance: low limitations: - '- Language-Codec is trained and evaluated exclusively on speech; audio, music, and environmental sound domains are explicitly left as future work. The codec''s suitability for general audio language models is therefore unvalidated. - MCRVQ prediction accuracy drops when more than 4 codebook channels are used in downstream models, suggesting the information-redistribution benefit weakens at higher bitrates. The mechanism for this degradation is not fully explained. - The paper evaluates downstream quality only with VALL-E and MobileSpeech; it is unclear whether the SPK-SIM gains extend to flow-matching or diffusion-based TTS backends. - No demo page is linked in the paper, making it difficult to subjectively verify the quality claims beyond the crowd-sourced MOS. - The internal 20,000-hour Chinese dataset is not publicly available, limiting full reproducibility of the 50k-hour training run.' caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.681 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: large_scale_instruction_datasets_improve_generalization role: supports claim: Large-scale instruction datasets that span diverse acoustic and paralinguistic dimensions improve speech-text LLM generalization on instruction-following benchmarks beyond what smaller task-specific corpora achieve. source: §5.2, Table 3 evidence: SIFT-LLM outperforms O-ASQA-LLM (same backbone, fine-tuned on 2.7M-example OpenASQA) across all benchmarks, including 46.1% vs. 22.9% on EvalSIFT closed-ended and 57.4% vs. 45.9% on Dynamic-Superb. confidence: high relevance: low - claim_id: instruction_finetuning_reduces_specialized_performance role: complicates claim: Instruction fine-tuning on diverse speech tasks reduces specialized performance on foundational tasks relative to the pre-trained model checkpoint. source: §5.3, Table 4 evidence: SIFT-LLM WER on LibriSpeech test-clean rises from 2.5% (50K pre-training checkpoint) to 3.5% after instruction fine-tuning, consistent with behavior observed in Qwen2-Audio and its instruction fine-tuned variant. confidence: medium relevance: low - claim_id: joint_semantic_acoustic_codecs_enable_better_controllable_generation role: supports claim: Codec representations that jointly encode acoustic and semantic content enable more accurate controllable speech generation than semantic-only representations. source: §5.4, Table 6 evidence: X-codec2 (fused semantic and acoustic codes) achieves QWK of 0.69 for pitch variation versus 0.15 for HuBERT codes (semantic-only) on SIFT-LLM GEN controllable generation evaluations. confidence: high relevance: high - claim_id: semantic_only_codes_insufficient_for_speaker_dependent_generation role: complicates claim: Semantic-only discrete speech representations are insufficient for reliable speaker-dependent controllable generation, as they primarily encode content rather than prosodic and acoustic identity. source: §5.4, §F.3 evidence: The HuBERT-code vocoder setup achieves approximately 50% gender accuracy against instruction-specified gender, compared to 95.8% for X-codec2; pitch variation QWK from original audio degrades substantially in HuBERT re-synthesis (0.11 QWK, Table 18 Appendix F.3). confidence: high relevance: low limitations: - Controllable-generation evaluation relies on automatically extracted acoustic features (QWK, MAE) compared against instruction-specified targets, which measures categorical agreement rather than perceptual quality, and the LLM-as-a-judge methodology for open-ended tasks introduces judge-model dependency. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.682 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS - SCA - evaluation architecture: - autoregressive-LM - GAN - hybrid relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - adversarial_codec_speech_models claims: - claim_id: gan_based_vocoders_occupy_the_dominant_position_in role: supports claim: GAN-based vocoders occupy the dominant position in production speech generation systems because their computational efficiency advantage over autoregressive and diffusion alternatives is orders of magnitude, at acceptable perceptual quality cost. source: §3.3.1, §F.3, Table 8 evidence: GAN-based vocoders (HiFi-GAN, BigVGAN) dominate in practice due to their computational efficiency and real-time factor orders of magnitude better than autoregressive or diffusion alternatives. confidence: high relevance: high - claim_id: continued_pretraining_from_a_text_language_model_checkpoint role: supports claim: Continued pretraining from a text language model checkpoint improves SpeechLM convergence and downstream task performance compared to random initialization. source: §4.2.1 evidence: 'The synthesised empirical picture includes: HuBERT outperforms CPC and wav2vec 2.0 as a semantic tokenizer (GSLM); continued pretraining from TextLMs improves convergence and task performance over cold initialization (TWIST); interleaving text and speech tokens during pretraining improves cross-modal performance (SPIRIT-LM); larger TextLM checkpoints and more data continue to improve SpeechLMs (AudioPaLM, GLM-4-Voice with 13M hours of synthetic data).' confidence: high relevance: high - claim_id: semantic_tokenizers_and_acoustic_tokenizers_represent_complementary_capability role: supports claim: Semantic tokenizers and acoustic tokenizers represent complementary capability profiles — strong semantic content fidelity versus strong acoustic reconstruction fidelity — and no single tokenizer type dominates both dimensions. source: §3.1, §F.2, Table 6 evidence: 'The survey distinguishes three objective families: semantic understanding tokenizers (HuBERT, WavLM, w2v-BERT) produce tokens that align well with text but sacrifice acoustic detail; acoustic generation tokenizers (neural codecs: SoundStream, EnCodec, DAC) preserve high-fidelity audio characteristics but often produce semantically inconsistent outputs; mixed-objective tokenizers (SpeechTokenizer, Mimi) distill semantic representations into the first quantizer layer of a codec architecture to capture both.' confidence: high relevance: low - claim_id: post_alignment_of_speechlms_via_preference_optimization_addresses role: supports claim: Post-alignment of SpeechLMs via preference optimization addresses qualitatively different failure modes (semantic inconsistency, token distribution mismatch) than post-alignment of text LLMs. source: §4.2.3 evidence: 'Training is covered across three stages: large-scale pretraining (with or without TextLM initialization), instruction tuning (cross-modal fine-tuning, chain-of-modality tuning), and post-alignment (DPO for semantic consistency in AlignSLM; token distribution alignment in SpeechAlign).' confidence: high relevance: high - claim_id: full_duplex_spoken_interaction_simultaneous_bidirectional_speech_with role: complicates claim: Full-duplex spoken interaction — simultaneous bidirectional speech with interruption support — requires joint modeling of both speaker streams and remains an open research challenge. source: §4.3 evidence: Full-duplex systems that support simultaneous bidirectional communication — including user interruption and simultaneous response — are surveyed as an emerging frontier, with dGSLM, Parrot, Moshi, and LSLM as representative systems. confidence: high relevance: high limitations: - 'Coverage necessarily lags the field: systems published after mid-2024 receive limited treatment, and the overall corpus skews heavily toward English and Mandarin. The safety section identifies toxicity and speaker privacy risks but does not analyze them quantitatively. End-to-end training that backpropagates gradients from vocoder output to tokenizer input is flagged as a potentially high-value research direction but remains unexplored. The question of whether incorporating text modality fundamentally benefits or constrains speech intelligence is left open.' caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.817 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: simuls2sllm_boundary_aware_critical_for_streaming role: supports claim: Boundary-aware speech representations are critical for enabling offline-trained speech LLMs to perform simultaneous inference via wait-k strategies. source: §5.1, §5.2, Tables 4, 11 evidence: CIF-based boundary-aware prompts outperform fixed downsampling by approximately 4 ASR-BLEU points at equivalent latency on Es-En, Fr-En, and De-En CVSS-C test sets, with a parallel gap on text output confirming the bottleneck is LLM prediction rather than speech synthesis. confidence: high relevance: low - claim_id: simuls2sllm_offline_training_matches_streaming_trained role: supports claim: Offline training combined with test-time simultaneous inference policies can match or outperform systems trained specifically for streaming in speech-to-speech translation. source: §5.1, Fig. 4, Table 1 evidence: SimulS2S-LLM, trained offline, consistently outperforms the streaming-trained StreamSpeech model across all three language pairs at comparable latency, achieving up to 4 ASR-BLEU improvement on Es-En. confidence: high relevance: low - claim_id: simuls2sllm_multilayer_aggregation_improves_prediction role: supports claim: Aggregating LLM hidden states across multiple layers improves discrete speech token prediction compared to using only the final layer. source: §5.3, Fig. 6 evidence: Multi-layer hidden state weighting yields approximately 1 ASR-BLEU improvement over last-layer-only decoding, attributed to the final layer's focus on semantic text information at the expense of acoustic richness. confidence: medium relevance: high - claim_id: simuls2sllm_llm_latency_overhead_narrows_advantage role: complicates claim: LLM-based approaches to simultaneous speech generation face a latency penalty from LLM inference overhead that narrows the practical quality-latency advantage over non-LLM methods. source: §D, Tables 8-10, Limitations evidence: Computation-aware average token delay for SimulS2S-LLM is substantially higher than standard ATD (e.g., 4239ms vs. 3440ms at k=8 on Es-En), and the system is not evaluated at very low latency regimes (AL < 1s). confidence: medium relevance: high - claim_id: simuls2sllm_ngram_fusion_improves_quality role: supports claim: Shallow fusion of n-gram language models with CTC decoding of discrete speech tokens improves simultaneous speech translation quality without increasing latency. source: §5.4, Table 2 evidence: n-gram LM fusion over greedy CTC search improves ASR-BLEU from 24.7 to 26.3 on CVSS-C Es-En at identical average token delay. confidence: medium relevance: high limitations: - Not evaluated at very low latency (AL < 1s), a regime the authors identify as unsuitable for offline-trained models due to reordering requirements. - Evaluated only on three European language pairs (Es-En, Fr-En, De-En) on relatively small datasets (70-174 hours per language pair). caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.912 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: autoregressive_streaming_speech_decoders_in_modular_spoken_conversational role: supports claim: Autoregressive streaming speech decoders in modular spoken conversational agents substantially improve naturalness over non-autoregressive alternatives with comparable latency. source: §5.1, Table 1 evidence: 'Speech naturalness (UTMOS): LLaMA-Omni2 models score 4.19–4.22 across sizes, substantially above GLM-4-Voice (3.48) and LLaMA-Omni (3.67).' confidence: high relevance: high - claim_id: a_gate_fusion_mechanism_that_adaptively_blends_llm role: supports claim: A gate fusion mechanism that adaptively blends LLM hidden states with text token embeddings as input to the TTS language model improves both instruction following quality and text-speech consistency over simpler additive fusion. source: §5.2, Table 2 evidence: 'M_TTS receives a gate-fused combination of LLM hidden states and text token embeddings: a learned sigmoid gate adaptively blends the two representations element-wise, allowing the model to exploit both broad contextual information from hidden states and precise textual content from embeddings.' confidence: high relevance: high - claim_id: streaming_tts_pretraining_on_speech_dialogue_data_is role: complicates claim: 'Streaming TTS pretraining on speech dialogue data is critical for quality: initializing from a text-only pretrained model degrades performance substantially, and training from scratch fails to converge.' source: §5.2, Table 3 evidence: Ablation studies (Tables 2–5) confirm the importance of the gate fusion module, streaming TTS pretraining over offline or scratch initialization, and that multi-turn data consistently improves over single-turn data of the same size. confidence: high relevance: low - claim_id: multi_turn_dialogue_training_data_consistently_outperforms_single role: supports claim: Multi-turn dialogue training data consistently outperforms single-turn data of the same total size for modular speech language models across spoken QA and instruction-following benchmarks. source: §5.3, Table 5 evidence: Ablation studies (Tables 2–5) confirm the importance of the gate fusion module, streaming TTS pretraining over offline or scratch initialization, and that multi-turn data consistently improves over single-turn data of the same size. confidence: high relevance: high - claim_id: the_s2t_to_s2s_accuracy_gap_in_spoken role: supports claim: The S2T-to-S2S accuracy gap in spoken question answering is a meaningful indicator of speech generation quality, and autoregressive TTS decoders reduce this gap relative to non-autoregressive alternatives. source: §5.1, Table 1 evidence: 'Spoken QA accuracy (S2S setting): LLaMA-Omni2-7B achieves 60.7% on Llama Questions and 31.3% on Web Questions, versus GLM-4-Voice''s 50.7%/15.9% and LLaMA-Omni''s 49.0%/23.7%.' confidence: high relevance: high limitations: - The model generates speech in a single fixed output style; it cannot modulate emotion, speaking rate, or dialect in response to the content or paralinguistic cues of the input speech. All evaluation is in English. The output voice is fixed during training (a single uniform voice for all responses), which limits expressiveness and speaker diversity. The system is inherently a response-after-input architecture and does not support full-duplex conversation. Latency, while adequate for real-time interaction, leaves room for further reduction via engineering optimization. Whether the gate fusion approach generalizes to multilingual settings or to more expressive speech styles is not explored. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.acl-long.997 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: alignslm_preference_opt_improves_semantic_coherence role: supports claim: Preference optimization with LLM-based semantic feedback improves long-range semantic coherence in textless spoken language models beyond next-token prediction training. source: §5.2, Table 2 evidence: DPO with Mistral-score preference data raises T-StoryCloze from 69.7% to 74.2% (1.3B model) and from 75.4% to 83.8% (7B model) over the pre-trained TWIST baseline; curriculum learning further improves both to 76.1% and 85.6% respectively. confidence: high relevance: high - claim_id: alignslm_llm_scoring_beats_perplexity role: supports claim: LLM-based semantic scoring provides more effective alignment signal for speech continuation than perplexity-based preference selection. source: §5.1, Table 1 evidence: On the 1.3B model, Align-SLM with Mistral score improves T-StoryCloze by +4.5 over the pre-trained baseline, whereas the PPL-based variant degrades T-StoryCloze by 2.0. confidence: high relevance: low - claim_id: alignslm_curriculum_learning_progressive_gains role: supports claim: Curriculum learning with iteratively tightened preference thresholds yields progressive semantic improvements in preference-optimized spoken language models. source: §5.3, Appendix E, Table 2 evidence: Two curriculum learning iterations on the 7B model improve T-StoryCloze from 83.8% (Align-SLM) to 85.6% (Align-SLM+CL) and GPT4-o score from 3.50 to 3.56. confidence: medium relevance: high - claim_id: alignslm_gains_not_uniform_grammar role: complicates claim: Semantic gains from preference optimization do not transfer uniformly to lexical and grammatical aspects of spoken language modeling. source: §5.5, Table 2 evidence: Align-SLM shows only marginal sBLIMP improvement (+1.3 for the 1.3B Mistral variant) while models with no preference training but better token designs (AudioLM, SyllableLM) outperform on grammatical correctness. confidence: medium relevance: high - claim_id: alignslm_gap_to_cascaded_topline role: complicates claim: Textless end-to-end spoken language models remain substantially behind cascaded ASR+LLM pipelines in semantic understanding even with preference optimization. source: §5.5, Table 2 evidence: The best Align-SLM variant achieves 77.9% sWUGGY and 86.8% T-StoryCloze, compared to 79.2% and 94.8% for the ASR+LLM cascade topline, a gap that preference optimization alone does not close. confidence: high relevance: high limitations: - Training and evaluation are restricted to English audiobook speech (LibriSpeech, MLS); generalisation to spontaneous, noisy, or multi-speaker speech is untested. - The paper does not compare against text-injecting SLMs (SPIRITLM, Moshi, VoxtLM) under matched data and compute budgets. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.101 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: decomposing_conversational_speech_synthesis_into_sequential_emotion_understandin role: supports claim: Decomposing conversational speech synthesis into sequential emotion-understanding and speech-generation stages improves expressiveness over direct speech token prediction. source: §6.5, Table 2 evidence: Chain-Talker (DMOS-E 4.239) outperforms GPT-Talker's direct token prediction (3.913) and GPT-Talker_c, which adds emotion understanding without fully chaining it through caption prediction (4.102); the ablation w/o captions (4.084) confirms the caption conditioning is the key factor. confidence: high relevance: high - claim_id: natural_language_emotion_captions_are_more_effective_conditioning role: supports claim: Natural language emotion captions are more effective conditioning signals for empathetic speech synthesis than discrete emotion category labels. source: §6.4, §6.5, Tables 1 and 2 evidence: Chain-Talker using empathetic captions (DMOS-E 4.239) outperforms Chain-Talker_e using emotion labels (4.127) and Chain-Talker_s using style labels (4.015); DMOS-C scores also show captions exceed ground-truth label quality (4.462 vs 4.327). confidence: high relevance: low - claim_id: supervised_asr_derived_semantic_speech_tokens_provide_more role: supports claim: Supervised ASR-derived semantic speech tokens provide more interpretable and expressive conversational speech generation than unsupervised tokens that mix semantic and acoustic content. source: §2.3, §6.5, Table 2 evidence: The paper argues that HuBERT tokens in GPT-Talker contain entangled acoustic information that limits emotional comprehension; Chain-Talker's ASR-VQ tokens improve DDTW from 44.625 to 38.784 and ACC_m from 0.562 to 0.612. confidence: high relevance: high - claim_id: llm_driven_automatic_speech_emotion_captioning_with_multi role: supports claim: LLM-driven automatic speech emotion captioning with multi-level attribute extraction produces higher-quality annotations than single-modality or keyword-based description approaches. source: §6.4, Table 1 evidence: CSS-EmCap (DMOS-C 4.462, SIM_G 0.694, DIS-2 0.296) outperforms Qwen2-Audio (4.212, 0.534, 0.174) and SECap (4.268, 0.617, 0.186); ablations confirm that removing either sentence-level style factors or dialogue-level emotion significantly degrades both quality and diversity. confidence: high relevance: low - claim_id: autoregressive_chain_modeling_for_empathetic_conversational_speech_synthesis role: complicates claim: Autoregressive chain modeling for empathetic conversational speech synthesis introduces latency that does not yet support real-time interaction requirements. source: §Limitations evidence: Chain-Talker generates average empathetic responses of 2.5 seconds duration on an RTX 4080 with 32 GB VRAM; authors describe this as a gap relative to real-time dialogue and identify streaming inference as a necessary future direction. confidence: high relevance: high limitations: - Chain-Talker is fine-tuned on only 384 hours of dialogue data drawn from daily conversational domains and predominantly young adult speakers. The authors explicitly note it may not capture the speaking styles of children or the elderly, and domain generalisation to non-conversational or non-English settings is untested. - 'Inference latency is a practical bottleneck: the three-stage autoregressive pipeline produces responses averaging 2.5 seconds, which is insufficient for low-latency real-time dialogue. Streaming inference is identified as future work but not yet implemented.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.1051 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: decoupling_speech_synthesis_from_llm_text_generation_via role: supports claim: Decoupling speech synthesis from LLM text generation via a lightweight autoregressive module can preserve the base LLM's reasoning quality while achieving competitive streaming latency. source: §6.4, Table 1 evidence: LLMVoX is a plug-and-play autoregressive TTS frontend that converts streaming LLM text output into speech tokens in real time, decoupled from the LLM. confidence: high relevance: high - claim_id: end_to_end_speech_enabled_llms_that_fine role: supports claim: End-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen. source: §6.4, Table 1 evidence: The paper reports evidence that end-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen. confidence: high relevance: low - claim_id: a_single_layer_rvq_codec_is_sufficient_for role: supports claim: A single-layer RVQ codec is sufficient for low-latency autoregressive TTS generation when paired with a compact decoder transformer, avoiding the complexity of multi-codebook prediction. source: §3.1 evidence: WavTokenizer (single-layer RVQ, 4096-entry codebook) encodes speech at 40–75 tokens/second at 24 kHz. confidence: high relevance: low - claim_id: streaming_tts_quality_improves_with_larger_decode_chunk role: supports claim: Streaming TTS quality improves with larger decode chunk sizes, with WER and UTMOS gains achievable without substantially increasing end-to-end latency. source: §6.4, Figure 6 evidence: 'On the joint General+Knowledge QA streaming benchmark (Table 1), Whisper+LLaMA3.1-8B+LLMVoX achieves: - GPT-4o General QA score: 6.14 (highest among streaming models; Freeze-Omni: 3.48; Moshi: 2.71) - GPT-4o Knowledge QA score: 7.62 (highest; vs. 4.98 for Freeze-Omni) - WER: 3.70% (lowest; Freeze-Omni: 14.05%; LLaMA-Omni: 9.18%; CosyVoice2: 17.2%) - UTMOS: 4.05 (competitive; Freeze-Omni: 4.38; MiniCPM-o 2.6: 3.87) - End-to-end latency: 475ms (vs. Whisper+XTTS: 4200ms; LLaMA-Omni: 220ms) For Arabic (Table 3), LLMVoX streaming CER of 8.2% competes with non-streaming XTTS (CER 1.7%) while operating ~10× faster.' confidence: high relevance: low - claim_id: language_adaptation_of_a_codec_based_tts_module role: supports claim: Language adaptation of a codec-based TTS module can be achieved by replacing training data alone, without architectural changes or explicit G2P conversion for the new language. source: §6.5, Table 3 evidence: No architectural changes needed for new languages — only data. confidence: high relevance: low limitations: - LLMVoX is single-speaker — no voice cloning or speaker reference support. The Arabic model was trained on XTTS-synthesized data, so XTTS acts as an upper bound (CER 1.7% vs. LLMVoX 8.2%). The streaming pipeline does not yet extend to the ASR front-end. Latency with 70B LLMs exceeds 1.9s, making real-time use marginal. The quality improvement from larger chunk sizes (UTMOS 3.75→4.41) suggests that the 475ms latency figure is somewhat optimistic for maximum quality operation. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.115 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA - TTS architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: slamomni_grouped_tokens_reduce_mismatch role: supports claim: Grouping discrete semantic tokens during autoregressive generation reduces training and inference costs by alleviating the frequency mismatch between text and audio token streams. source: §5.4.1, Table 6 evidence: Semantic Group Modeling with G=3 reduces SLAM-Omni training from 126 to 60 GPU hours and ASR-WER from 18.23% (G=1) to 4.54%. confidence: high relevance: high - claim_id: slamomni_pretraining_hurts_instruction_following role: refines claim: Multi-stage pre-training on modality-specific tasks for spoken dialogue systems can degrade instruction-following ability despite improving modality alignment metrics. source: §5.4.2, Table 7 evidence: ASR pre-training reduces ChatGPT Score from 39.32 to 34.02 and TTS pre-training to 27.22, while ASR-WER improves only marginally. confidence: high relevance: high - claim_id: slamomni_timbre_control_data_constrained role: complicates claim: Zero-shot timbre control in spoken dialogue systems achieves competitive speaker similarity but remains constrained by training data volume relative to dedicated TTS systems. source: §5.2, Table 5 evidence: SLAM-Omni reaches SIM-o of 0.517, comparable to FireRedTTS (0.486) but below CosyVoice2 (0.684), with the gap attributed to approximately 50x less training data. confidence: medium relevance: high - claim_id: slamomni_text_history_sacrifices_paralinguistics role: complicates claim: Compressing multi-turn dialogue history to text representation sacrifices non-verbal paralinguistic context that may be important for maintaining dialogue coherence across turns. source: §3.5, §6 Limitations evidence: Historical Text Prompting stores only text history, explicitly trading away emotional and prosodic signals from previous dialogue turns for computational efficiency. confidence: medium relevance: low - claim_id: slamomni_semantic_tokens_tighter_alignment role: supports claim: Semantic token-based speech generation in spoken dialogue systems provides tighter speech-text alignment than acoustic codec-based approaches, as measured by word error rate between generated speech and corresponding text. source: §5.1, Table 3 evidence: SLAM-Omni achieves the lowest ASR-WER (4.54%) among evaluated spoken dialogue models, outperforming Moshi (7.18%), GLM-4-Voice (12.71%), and Freeze-Omni (16.32%), using single-layer semantic tokens. confidence: high relevance: high limitations: - The single-stage training finding is demonstrated only at 0.5B scale with limited data (400K utterances); generalisation to larger backbones is untested. - The custom 8-task benchmark measures general spoken interaction but not domain-specific or emotion-aware dialogue quality. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.470 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [] claims: - claim_id: text_llm_speech_extension_degrades_context_recall role: supports claim: Extending a pre-trained text LLM to process speech input substantially degrades its ability to recall and utilize long-range conversational context. source: §4.1, Table 2 evidence: All four evaluated open-source voice interaction models score 2.00-2.83 on ContextDialog (5-point GPT Score), compared to 4.03-4.10 for their text-only backbone LLMs; the gap is largest when the LLM backbone is frozen during speech adaptation. confidence: high relevance: high - claim_id: modality_specific_recall_asymmetry role: supports claim: Voice interaction models exhibit a modality-specific recall asymmetry, retrieving information from model-generated text responses more reliably than from user-spoken utterances. source: §4.1, Table 2, Figure 3 evidence: All four speech models score significantly higher on questions about the model's own past utterances than past user utterances (p < 0.01); attention maps confirm systematically lower attention weight on user-spoken supporting utterances. confidence: high relevance: low - claim_id: rag_no_consistent_improvement_context_recall role: complicates claim: Retrieval-augmented generation provides a practical route to improving multi-turn context recall in spoken dialogue models. source: §4.2, Figure 4, Table 3, Table 5 evidence: Augmenting four open-source voice interaction models with a top-k text retriever (e5-large-v2) yields no consistent improvement and often degrades performance; top-1 retrieval recall is only 0.58 and irrelevant retrieved utterances lower all models below baseline. confidence: medium relevance: high - claim_id: opensource_voice_models_underdeveloped_longcontext role: supports claim: Open-source voice interaction models show critically underdeveloped long-context capabilities compared to closed-source systems, even on simple factual recall tasks. source: §A.1.4, Table 7 evidence: gpt-4o-mini-audio-preview scores 3.83-4.50 overall, more than 1 point above the best open-source model (Lyra, 2.83), even though the closed-source API does not support a proper multi-turn spoken dialogue context. confidence: medium relevance: high limitations: - 'ContextDialog is entirely synthetic: questions are generated by GPT-4o and corresponding audio is synthesized by Fish Speech, which may cause the benchmark to underestimate recall difficulty on real user speech with natural disfluency and acoustic variability.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.534 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA architecture: [] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [] claims: - claim_id: semantic_similarity_reliable_proxy_for_tts_data_filtering role: supports claim: Semantic similarity in embedding space can serve as a reliable proxy for linguistic equivalence when filtering TTS-synthesized speech instruction data, avoiding the over-rejection caused by WER-based criteria. source: §5.2, §5.3, Table 1 evidence: The SIM metric achieves over 98% consistency with WER when strict linguistic equivalence holds, and above 92.5% across all conditions and three TTS models on seven QA datasets. confidence: high relevance: low - claim_id: llm_continued_targets_weaker_than_human_annotated role: supports claim: Training large speech language models on LLM-continued alignment targets produces systematically weaker instruction-following capability than training on human-annotated responses, regardless of speech synthesis quality. source: §5.3, Table 3 evidence: Models trained with LLM-continued answers achieve 52.54 average ROUGE-L vs. 60.5 with human-annotated golden answers at identical data quality thresholds; the gap persists across all ablation settings. confidence: high relevance: high - claim_id: parallel_multillm_rewriting_improves_tts_data_usability role: supports claim: Parallel query rewriting with multiple structurally diverse LLMs improves TTS data usability compared to any single-LLM rewriting approach, due to complementary performance across text types. source: §5.4, Table 2 evidence: Using Llama-3, Phi-3, and Qwen2 together raises average Pass rate by 2-3 percentage points over the best single-LLM baseline across MeloTTS and ParlerTTS-Large in the single-speaker setting. confidence: medium relevance: low - claim_id: wer_filtering_causes_inappropriate_data_rejection role: complicates claim: WER-based quality filtering of TTS-synthesized speech instruction data causes inappropriate data rejection due to incidental ASR errors, distorting the resulting dataset distribution. source: §3, §5.3, Table 13 evidence: Strict WER filtering rejects samples where ASR recognition errors occur despite semantic equivalence; SIM-based filtering recovers these samples while maintaining downstream task performance, confirmed by case analysis. confidence: medium relevance: low limitations: - LSLM evaluation covers only four reading comprehension tasks measured with ROUGE-L, which may not reflect instruction-following quality in conversational or multi-turn settings, and all experiments are English-only. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.631 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL Findings task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: slam_textlm_init_accelerates_convergence role: supports claim: Text LM initialization substantially accelerates SLM convergence and improves final model quality within fixed compute budgets. source: §4.1, Figure 2, Figure 3 evidence: TWIST-style initialization from Qwen2.5-0.5B outperforms uninitialised variants across all evaluated model families, surpassing alternatives even with fewer training tokens. confidence: high relevance: low - claim_id: slam_synthetic_data_improves_semantics role: supports claim: Synthetic TTS data improves semantic modeling in SLMs when combined with real speech, but replacing real data with synthetic data alone degrades performance. source: §4.2, Table 1 evidence: Adding sTinyStories to the training mix boosts TSC from 71.14 to 78.01 and GenPPL from 145.4 to 88.3 for Qwen-0.5B; training on synthetic data exclusively yields lower sBLIMP than the mixed baseline. confidence: medium relevance: low - claim_id: slam_dpo_improves_under_tight_compute role: supports claim: Preference optimization with synthetically generated preference data substantially improves SLM generation quality even under tight compute constraints. source: §4.4, Figure 5, Table 2 evidence: 30 minutes of DPO training on SpokenSwag improves TSC from 78.01 to 82.04 and GenPPL from 88.3 to 62.8; the gain saturates quickly and additional DPO budget can degrade results. confidence: high relevance: low - claim_id: slam_scaling_laws_too_pessimistic role: contradicts claim: SLM scaling laws accurately predict compute-optimal performance boundaries for academic-scale training. source: §5, Table 2 evidence: Slam on one A5000 GPU-day achieves TSC 82.04 against the scaling-law predicted optimal of TSC 70.49 for the same compute; the authors attribute the gap partly to preference optimization (DPO) not being factored into scaling law derivations. confidence: medium relevance: low - claim_id: slam_diverse_data_hurts_low_compute role: complicates claim: Diverse multi-domain speech training data improves SLM generalization. source: §4.2, Table 1 evidence: Augmenting LibriLight and LibriSpeech with VoxPopuli, TED-LIUM, PeopleSpeech, and SWC consistently degrades TSC and SSC under the single-GPU budget. confidence: medium relevance: low limitations: - Evaluation uses only semantic modeling benchmarks (TSC, SSC, sBLIMP) and generation perplexity; acoustic and prosodic quality of the DPO-aligned outputs are not evaluated. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-acl.75 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - TTS architecture: - transformer-enc-dec relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: unitlang_textfree_guidance_matches_text_supervised role: supports claim: Discrete speech unit sequences can be compressed into text-like representations via n-gram language modelling, providing effective text-free guidance for both cross-modal and cross-lingual learning in speech translation. source: §2.1, §3.2, Table 2 evidence: Unit language (2-gram merged mHubert units, K=3) achieves average ASR-BLEU of 21.5 on VoxPopuli across four language directions, matching recognised-text training and yielding +1.2 BLEU over the baseline using raw discrete units alone. confidence: high relevance: high - claim_id: unitlang_cm_cl_objective_conflict role: complicates claim: Combining cross-modal and cross-lingual auxiliary objectives in multi-task sequence-to-sequence speech translation training can produce gradient interference that limits the gains of each individual loss. source: §2.3, §3.2, §4.3 evidence: Simultaneously applying source (L_CM) and target (L_CL) unit language losses without task prompts yields combined gains no greater than either loss alone, because CM guidance at middle encoder layers disrupts CL representation learning in upper layers. confidence: medium relevance: high - claim_id: unitlang_task_prompt_separates_objectives role: supports claim: Learnable task-specific prompt vectors can separate competing auxiliary objectives within a shared Transformer encoder by enforcing distinct intermediate representations for each task. source: §2.4, §3.2, Table 2 evidence: Two prompt vectors (b_CM, b_CL) prepended at different encoder layer boundaries, with a negative MSE diversification loss, fully recover the +1.2 BLEU combined gain from unit language guidance that is otherwise lost to task interference. confidence: medium relevance: low - claim_id: unitlang_sequence_compression_enables_alignment role: refines claim: Sequence length compression is a critical enabler of cross-lingual alignment in textless speech translation, and n-gram-merged unit sequences provide substantially better compression than BPE pseudo-languages. source: §4.1, §4.6, Figure 4, Table 6 evidence: Unit language lengths fall consistently between raw units and character-level text across all four VoxPopuli language pairs, and CL training with unit language outperforms BPE pseudo-language by +0.5 BLEU average. confidence: medium relevance: low limitations: - Evaluation uses ASR-BLEU only on synthesized speech; naturalness, fluency, and speaker consistency of the output speech are not assessed. - Validated only on four European language directions (Spanish, French, English); generalisation to typologically distant or low-resource languages is unexamined. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.findings-ijcnlp.49 published_date: "2025-07-27" entry_date: '2026-07-27' year: 2025 venue: ACL task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: a_full_duplex_speech_language_model_can_perform role: supports claim: A full-duplex speech language model can perform structured dialogue state tracking when DST is executed entirely through the model's text stream using explicit delimiter tokens. source: §3, §5.3, Table 1 evidence: J-Moshi-ext fine-tuned on synthesized JMultiWOZ data achieves JGA 65.69 and Slot F1 97.78 on the JMultiWOZ test set, with both metrics improving monotonically as training data volume increases from 25% to 100%. confidence: high relevance: high - claim_id: extending_full_duplex_speech_models_to_task_oriented role: complicates claim: Extending full-duplex speech models to task-oriented dialogue does not close the gap with text-based systems on response generation quality. source: §5.3, Table 1 evidence: The proposed method reaches BLEU 0.129 and BERTScore 0.689, against text-based upper bounds of BLEU 0.364 and BERTScore 0.830 (T5-large), with the gap attributed to insufficient training data and to the difficulty of aligning text and audio modalities during response generation. confidence: high relevance: high - claim_id: tts_data_quality_limitations_compound_into_downstream_training role: complicates claim: TTS data quality limitations compound into downstream training noise when synthesized speech is used to construct full-duplex dialogue training corpora. source: §5.3 evidence: The multilingual OuteTTS model used for Japanese speech synthesis generated speech with reduced clarity, causing recognition errors during time-annotated tokenization that hindered the model's learning of coherent text-audio token sequences for response generation. confidence: high relevance: high - claim_id: dst_timing_estimation_is_feasible_within_a_full role: supports claim: DST timing estimation is feasible within a full-duplex spoken dialogue model but introduces substantial prediction error tied to the model's speech perception accuracy. source: §5.1, §5.3 evidence: DST initiation timing (defined as the time of `` token generation) achieved a MAE of 4.9 seconds, with discrepancies attributed to the difficulty of accurately predicting user utterance boundaries from the model's audio token representations alone. confidence: high relevance: high limitations: - All experiments are conducted in Japanese using J-Moshi, which was trained on less Japanese data than the original English Moshi. The authors explicitly note that performance on English equivalents (MultiWOZ, SpokenWOZ) could be substantially higher, and cross-lingual generalisability is untested. - Evaluation is fully automatic (JGA, Slot F1, BLEU, BERTScore); no subjective listening tests or user studies are reported, leaving response naturalness, latency under real deployment conditions, and the effect of DST pauses on conversational flow unassessed. The experiment covers only the domains included in JMultiWOZ (travel planning), and extension to diverse or open-domain settings is planned but not yet evaluated. Future work includes experiments with the original Moshi on SpokenWOZ and DSTC11 datasets and application to additional dialogue domains. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.23159' published_date: "2025-07-30" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [] claims: - claim_id: fullduplex_responsiveness_floorholding_tradeoff role: supports claim: Current full-duplex spoken dialogue systems face a fundamental responsiveness-versus-floor-holding trade-off, with no single system achieving strong performance across both genuine interruptions and non-addressee overlap scenarios. source: §4, Table 2 evidence: Across five systems, GPT-4o achieves the highest interruption RESPOND rate (0.78) and fastest stop latency (0.23s) but incorrectly yields in 91-93% of non-addressee and background speech cases, while Nova Sonic holds the floor strongly (RESUME=0.98) but struggles with true interruptions (RESPOND=0.24). confidence: high relevance: high - claim_id: low_stop_latency_tradeoff_falseyield role: complicates claim: Stop latency below roughly 0.5 s under genuine user interruptions is achievable for commercial API-based systems but comes at the cost of significantly elevated false-yield rates on backchannel and background speech. source: §4.1, Table 2 evidence: GPT-4o achieves the fastest stop latency (0.23s) on genuine interruptions among evaluated systems, but the same responsiveness misfires on non-addressee and background speech, where it incorrectly yields in 91% and 93% of cases respectively. confidence: high relevance: low - claim_id: prosodic_adaptation_diverges_across_architectures role: supports claim: Full-duplex systems exhibit measurable prosodic adaptation post-interruption, but the adaptation strategies diverge across architectures, with some systems showing unstable or brittle re-entry delivery. source: §5, Table 3 evidence: GPT-4o and Gemini adopt a tempo-and-pitch-lift re-entry strategy, Freeze-Omni and Sonic reduce mean intensity while increasing variability, and Moshi's re-entry is erratic (extreme speed-up with reduced pitch and inconsistent intensity). confidence: medium relevance: high - claim_id: automated_overlap_evaluation_feasible_without_human_annotation role: supports claim: Automated evaluation of full-duplex overlap handling is feasible without human annotation by combining ASR transcription, LLM-based semantic classification, VAD-based timing measurement, and acoustic feature analysis. source: §2.3 evidence: The benchmark categorises post-overlap dialogue behavior using a GPT-4o evaluator applied to Parakeet-TDT ASR transcripts, strictly separating the evaluation modality from the generation modality, and combines this with VAD-based latency measurement and paired t-test prosodic feature analysis across five systems. confidence: high relevance: high limitations: - All evaluation stimuli are synthesized rather than drawn from real conversational corpora, and the acoustic simulations for off-axis and background speech are rule-based (fixed dB offsets, fixed filters), which may not capture the full distribution of real-world overlap conditions. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: 2025.iwslt-1.5 published_date: "2025-07-31" entry_date: '2026-07-27' year: 2025 venue: IWSLT task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: alignment_aware_compression_improves_crossmodal_understanding role: supports claim: Grounding speech compression in explicit speech-text alignment boundaries improves spoken language understanding and cross-modal reasoning in speech-augmented language models. source: §4.3, Tables 2-3 evidence: SSR-CONNECTOR (UNITY2) achieves 64.2%/68.6% (0/5-shot) on Speech-MMLU vs. 40.5%/42.75% for SPIRITLM (LLAMA3), and 74.8% on StoryCloze (S→T) vs. 61.6%, while maintaining 65.3% MMLU text accuracy. confidence: high relevance: high - claim_id: llm_finetuning_on_speech_degrades_crossmodal_understanding role: complicates claim: Fine-tuning a pre-trained LLM on speech data degrades cross-modal understanding even when catastrophic forgetting is mitigated through multitask training. source: §5.2, Table 5 evidence: All Stage 2 fine-tuning methods improve speech-only task performance (sWUGGY, sBLIMP) but consistently reduce Speech-MMLU and StoryCloze (S→T) accuracy; multitask fine-tuning limits the drop but cannot eliminate it. confidence: high relevance: high - claim_id: text_embedding_distillation_effective_for_connector_pretraining role: supports claim: Distillation from text embeddings is an effective pre-training objective for speech-text modality connectors, enabling cross-lingual transfer to ASR without explicit transcription supervision. source: §4.3, Table 3 evidence: After Stage 1 distillation alone (no ASR objective), SSR-CONNECTOR achieves 5.6%/4.0% WER on LibriSpeech clean (0/5-shot) via in-context prompting, using only the alignment-supervised distillation loss. confidence: medium relevance: low - claim_id: alignment_segmentation_reduces_lexical_task_effectiveness role: complicates claim: Alignment-aware speech segmentation reduces effectiveness on lexical tasks that depend on synthesized anomalous phoneme sequences. source: §4.3 evidence: SSR-CONNECTOR underperforms on sWUGGY because the aligner was not trained on incorrectly spoken words (non-words), causing segmentation errors, a known scope limitation of alignment-based methods. confidence: medium relevance: low limitations: - All experiments use a single language (English), a single LLM backbone (LLAMA3), and a single self-supervised speech encoder (DinoSR), which is not ablated. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.02013' published_date: "2025-08-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: srpa_quality_determined_by_text_lm_not_speech_architecture role: supports claim: The quality of spoken role-playing in both cascaded and end-to-end systems is more strongly determined by the underlying language model's text reasoning ability than by the speech synthesis or speech-to-speech architecture. source: §5.2 evidence: Cascaded systems built on strong text LMs, particularly Llama-3.1-8B, achieve the best overall English scores (0.957), outperforming GPT-4o Audio overall (0.921), while open-source end-to-end models (Qwen2.5-Omni, LLaMA-Omni) fall well below cascaded baselines across all dimensions. confidence: medium relevance: high - claim_id: finetuning_gains_larger_for_expressive_dims_than_interaction role: refines claim: End-to-end speech-language models exhibit larger gains from task-specific fine-tuning in expressive speech dimensions than in interaction or role fidelity dimensions, which remain text-LM bottlenecked. source: §5.2 evidence: 'Fine-tuning Qwen2.5-Omni on SpeechRole-Data substantially improves the resulting SpeechRole-Agent (0.801 English overall vs. 0.568 for the base model), with gains extending to unseen roles (OOD: 0.783), suggesting the benefit is not purely role memorisation.' confidence: medium relevance: high - claim_id: llm_pairwise_eval_correlates_with_human_judgement_multireference role: supports claim: LLM-based pairwise evaluation of speech quality can achieve high rank correlation with human expert judgements when using multi-reference normalisation, making automated SRPA benchmarking feasible at scale. source: §5.3, Table 3 evidence: Human-LLM agreement via Spearman's ρ ranges from 0.67 to 0.98 across metrics in English and 0.64 to 0.96 in Chinese, with Human-LLM Agreement rates exceeding 0.75 for most dimensions. confidence: high relevance: low - claim_id: opensource_e2e_lags_cascaded_on_prosody_emotion role: supports claim: Current open-source end-to-end speech models lag proprietary cascaded systems in prosody consistency and emotion appropriateness by margins large enough to constitute a capability gap, not merely a performance gap. source: §5.2, Table 2 evidence: GPT-4o Audio trails cascaded systems on prosody consistency (0.900 vs. 0.966 for Llama-3.1-8B) and emotion appropriateness (0.816 vs. 0.964), while open-source end-to-end Qwen2.5-Omni scores only 0.568 overall in English. confidence: medium relevance: low limitations: - All reference speech in SpeechRole-Eval is TTS-synthesised rather than recorded from human actors, meaning benchmark scores measure performance relative to synthetic reference quality rather than against a naturalness ceiling. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.02849' published_date: "2025-08-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec architecture: - VAE relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - variational_latent_speech_models claims: - claim_id: secousti_ssl_distillation_incomplete_disentanglement role: complicates claim: Distillation from self-supervised models such as HuBERT or WavLM does not achieve true semantic disentanglement in speech codecs, as these representations inherently retain paralinguistic content. source: §II.A, §V.B evidence: The paper motivates its cross-modal contrastive approach by noting that HuBERT/WavLM-distilled codecs (SpeechTokenizer, MimiCodec) leave incomplete separation because the distillation targets themselves carry residual paralinguistic content. confidence: medium relevance: high - claim_id: secousti_contrastive_learning_cleaner_than_classification role: supports claim: Frame-level cross-modal contrastive learning between phoneme and speech representations produces cleaner semantic-paralinguistic separation than phoneme classification loss in neural codecs. source: §V.D, Table IV evidence: In the voice conversion ablation, SecoustiCodec's frame-level contrastive approach achieves SPK-SIM 0.71 and Emo-Sim 0.86, versus SecoustiCodec-ASR-Loss (phoneme classification) at SPK-SIM 0.64 and Emo-Sim 0.69. confidence: high relevance: high - claim_id: secousti_vae_fsq_improves_utilization role: supports claim: VAE-augmented finite scalar quantization (VAE+FSQ) achieves substantially higher codebook utilisation and reduced long-tail token distribution compared to VQ-VAE in single-codebook speech codecs. source: §V.C, Figure 4, Table II evidence: The VAE+FSQ configuration achieves 98.06% codebook utilisation, substantially higher than reported rates for VQ-VAE and SimVQ under the same conditions. confidence: high relevance: low - claim_id: secousti_paralinguistic_bridge_improves_reconstruction role: supports claim: Explicit modeling of paralinguistic information as a reconstruction bridge between semantic and acoustic encodings improves both semantic completeness and reconstruction fidelity in low-bitrate streaming codecs. source: §III.B, §V.B evidence: SecoustiCodec models three coordinated representations (semantic S, acoustic A, paralinguistic G) with the approximate relationship S + G ~ A, achieving PESQ 1.77-2.58 and WER 3.99-11.58 at 0.27-1 kbps, competitive with or exceeding causal multi-codebook baselines at lower bitrate. confidence: medium relevance: low - claim_id: secousti_staged_training_necessary role: supports claim: Staged training that freezes acoustic modules before introducing semantic and KL losses is necessary for stable convergence in multi-objective codec training. source: §III.E, §V.C, Table II evidence: Ablations without the acoustic-constrained staged training strategy show significant degradation across all metrics relative to the two-stage protocol. confidence: medium relevance: low limitations: - The contrastive learning objective requires duration-aligned phoneme-level text labels during training, which the authors flag as the key limitation and identify unsupervised disentanglement as the primary future direction. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.04585' published_date: "2025-08-06" entry_date: '2026-07-27' year: 2025 venue: ACM MM task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: matching_the_token_rates_of_speech_and_facial role: supports claim: Matching the token rates of speech and facial landmark codecs enables frame-level synchronisation between synthesised speech and talking-face animations without post-hoc alignment. source: §4.1.3, §4.2 evidence: LmkCodec operates at 25 Hz with a codebook of size 1000, matching the speech tokenizer's rate and enabling frame-level alignment. confidence: high relevance: high - claim_id: llm_based_joint_prediction_of_interleaved_speech_and role: supports claim: LLM-based joint prediction of interleaved speech and visual tokens in dialogue context outperforms cascaded speech-then-video generation on both emotional accuracy and lip synchronisation. source: §6.2, §6.3, Table 2, Table 3 evidence: 'Built on Qwen2.5-0.5B, EVSLM takes the serialized multimodal context and sequentially predicts: (1) the target emotion token, (2) interleaved facial expression and speech tokens via a bimodal hard alignment strategy.' confidence: high relevance: high - claim_id: including_visual_dialogue_history_talking_face_animations_of role: supports claim: Including visual dialogue history (talking-face animations of prior turns) improves emotion accuracy and speaker consistency in conversational speech synthesis relative to audio-text-only context. source: §6.4, Table 4 evidence: 'UniTalker frames the problem as Conversational Speech-Visual Synthesis (CSVS): given the multimodal history of a dialogue (text, speech, talking-face animations, speaker identity, and emotion labels for each prior turn) and the current turn''s speaker and text, predict the target utterance''s emotion, speech, and talking-face animations jointly.' confidence: high relevance: high - claim_id: low_rate_discrete_tokenisation_of_facial_landmarks_1 role: supports claim: Low-rate discrete tokenisation of facial landmarks (1 token per frame) is more effective for LLM contextual modelling than higher-rate representations, even at a cost in geometric reconstruction fidelity. source: §6.1, §6.2, Table 1, Table 2 evidence: The LmkCodec's FSQ design with 1 token per frame also proves better than the 8-token-per-frame GRVQ variant, demonstrating that shorter sequences aid LLM contextual modeling even at some reconstruction cost. confidence: high relevance: low - claim_id: emotion_guided_conditioning_of_the_speech_renderer_including role: supports claim: Emotion-guided conditioning of the speech renderer, including predicted emotion labels as explicit conditioning, improves measured emotional expressiveness over systems that rely on implicit contextual inference alone. source: §6.4, Table 4 evidence: In speech emotion expressiveness (MOS_SE), the margin is +0.111 over Empatheia. confidence: high relevance: low limitations: - The Talking-face Animations Renderer (EchoMimic) is a pre-trained third-party module that receives no additional fine-tuning in this pipeline. Its outputs are constrained by the quality ceiling and biases of the EchoMimic base model, limiting the paper's ability to attribute animation quality gains to UniTalker specifically vs. the renderer. - 'Rendering latency is notable: speech generation takes approximately 2 seconds and animation rendering takes approximately 5 seconds per 25 frames on an RTX 4080 with 32 GB RAM, making the system unsuitable for real-time interaction in its current form. The paper acknowledges this and lists streaming optimisation as future work.' caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2507.20091' published_date: "2025-08-07" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: codec_token_speech_lms_trained_on_tens_of role: supports claim: Codec-token speech LMs trained on tens of thousands of hours of data do not develop prosodic understanding or generation as emergent capabilities through pre-training alone. source: §1, §4.2–4.4 evidence: A sentence-level [Global] token capturing aggregate prosodic extremity is optionally prepended to control expressive range during generation; the authors find it beneficial for generation tasks and detrimental for understanding tasks. confidence: high relevance: high - claim_id: replacing_codec_tokens_with_explicit_interpretable_word_level role: supports claim: Replacing codec tokens with explicit, interpretable word-level prosody annotations enables a pre-trained text LLM to acquire prosody processing capabilities — including contrastive focus, emotion recognition, and style transfer — without task-specific fine-tuning. source: §3.1, §4.2–4.4 evidence: The decoder adapts StyleTTS2's duration predictor and prosody predictor — replacing their dependence on StyleTTS2's diffusion-based style embedding with conditioning on the word-level values generated by the LLM — while leaving all other StyleTTS2 modules frozen at their pre-trained weights. confidence: high relevance: low - claim_id: disentangling_prosodic_and_linguistic_content_in_the_token role: supports claim: Disentangling prosodic and linguistic content in the token sequence trades voice quality fidelity for prosodic expressiveness and controllability. source: §3.1, §5 evidence: A sentence-level [Global] token capturing aggregate prosodic extremity is optionally prepended to control expressive range during generation; the authors find it beneficial for generation tasks and detrimental for understanding tasks. confidence: high relevance: low - claim_id: prosody_to_content_dependencies_detecting_emphasis_or_emotion role: supports claim: Prosody-to-content dependencies (detecting emphasis or emotion from speech) are more difficult for codec-token LMs than content-to-prosody generation. source: §4.3, Table 4 evidence: On prosody understanding tasks, ProsodyLM produces a statistically significant 6.6% log-probability increase for emphasised words (emphasis detection on EmphAssess), while both codec-token baselines show near-zero signal. confidence: high relevance: low - claim_id: word_level_prosody_tokens_impose_only_marginal_degradation role: supports claim: Word-level prosody tokens impose only marginal degradation on text language modelling capability when interleaved with transcription tokens in a joint sequence. source: §4.5, Table 9 evidence: Text perplexity on held-out audiobooks increases by only 2 points relative to a text-only fine-tuned Llama (11.70 vs. 13.87), indicating that prosody tokens add minimal interference to language modelling. confidence: high relevance: high limitations: - ProsodyLM is trained and evaluated exclusively on audiobooks (Librilight). Audiobook prosody is stylised and read-speech in character; generalisation to spontaneous conversation, emotional dialogue, or cross-domain settings is untested and likely requires substantial data re-collection or domain adaptation. - The decoder is frozen at StyleTTS2 pre-trained weights, which constrains output quality to that system's capability ceiling and limits the range of expressible voice characteristics. The five-dimensional prosody token cannot encode voice quality changes (breathiness, creakiness, vocal effort), which are important for fine-grained expressiveness. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.07273' published_date: "2025-08-10" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: contextual_paralinguistic_qa_data_outperforms_isolated_templates role: supports claim: Training Speech-LLMs on data that jointly encodes contextual and paralinguistic reasoning substantially outperforms training on isolated paralinguistic QA templates, even when the underlying speech encoder is frozen. source: §V.B.1, Table III evidence: Adding CPQA training data raises the human-annotated CPQA score from 41.0 (baseline) to 56.75, a 38.41% relative improvement, with the frozen Whisper encoder and Gemma2 decoder unchanged. confidence: high relevance: low - claim_id: dimensional_emotion_annotations_diversify_qa_supervision role: supports claim: Incorporating dimensional emotion annotations (valence, arousal, dominance) alongside categorical labels into LLM-generated QA data diversifies training supervision and improves generalisation to complex emotional states. source: §II.D, §V.B.1 evidence: The CPQA generation pipeline explicitly prompts GPT-4o with both categorical (Emotion2Vec) and dimensional emotion scores, and CPQA-trained models show the largest gains on contextual+emotion question types (37.56 to 53.86). confidence: medium relevance: low - claim_id: explicit_emotion_metadata_partially_compensates role: supports claim: Explicitly injecting emotion metadata into inference prompts can partially compensate for a model's limited intrinsic paralinguistic understanding, but training on contextual-paralinguistic data yields more robust generalisation across question types. source: §V.B.3, Fig. 2 evidence: Injecting explicit emotion metadata at inference (S32) raises the human-annotated CPQA score further to 59.87, a 46.02% gain, but the paper notes this technique alone does not match CPQA training's broader gains across all question types. confidence: medium relevance: low - claim_id: llm_judge_scores_correlate_with_classification_metrics role: supports claim: LLM judge scores for open-ended speech-language model evaluation correlate reliably with classification-based accuracy and F1 metrics on questions with deterministic answers, supporting their use as a proxy metric. source: §IV, §V.B.4, Table V evidence: The paper validates GPT-4o judge scores against estimated weighted accuracy and F1 derived via embedding-based label extraction, showing the ratio of correct to incorrect predictions rises monotonically with judge score. confidence: medium relevance: high limitations: - Emotion labels used both in training and inference prompts come from SER models rather than ground-truth annotations, introducing noise that may suppress performance on direct classification tasks. - 'The LLM-generated CPQA evaluation set contains evaluation confounds: direct emotion questions can benefit disproportionately from explicit metadata injection, which the authors flag but do not resolve.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.07375' published_date: "2025-08-10" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: turn_level_text_aggregation_stronger_guidance role: supports claim: Aggregating text tokens at the dialogue-turn level provides stronger semantic guidance for full-duplex speech generation than word-level token-by-token text insertion. source: §2.2, §4.2.1, Table 2 evidence: TurnGuide's turn-level interleaving achieves a GPT-score of 7.27 versus 5.91 for the Speech Chunk Interleaving (word/token-level) baseline on the Fisher test set. confidence: high relevance: high - claim_id: text_insertion_timing_more_sensitive_than_length role: supports claim: The timing of text insertion into full-duplex dialogue sequences is more sensitive to error than the length of inserted text, with mistimed insertions causing larger semantic degradation than incorrectly sized text chunks. source: §4.2.3, Table 4 evidence: Shifting text insertion 4 seconds early causes the largest degradation (GPT-score 5.61 vs. 6.76 at nominal timing), a bigger effect than the chunk-length ablations. confidence: medium relevance: high - claim_id: text_loss_weighting_improves_semantic_quality role: supports claim: Increasing the training loss weight of text tokens relative to speech tokens in a text-speech interleaved model improves semantic quality without sacrificing turn-taking naturalness. source: §4.2.1, Table 2 evidence: The 2:1 and 3:1 text-loss variants reach GPT-scores of 7.70 and 7.79 respectively, exceeding the 1:1 loss upper bound (7.39) obtained with ground-truth transcripts. confidence: medium relevance: high - claim_id: finegrained_turntaking_benchmarks_more_informative role: supports claim: Fine-grained turn-taking benchmarks are more informative than corpus-level statistical correlations for evaluating full-duplex spoken dialogue models. source: §4.2.2 evidence: On Full-Duplex-Bench, TurnGuide's behavioural differences from Moshi and dGSLM on backchanneling and interruption latency are visible in the fine-grained metrics even though the corpus-level Pearson correlation analysis (Table 7) shows parity with baselines. confidence: medium relevance: high - claim_id: gpt_based_semantic_eval_aligns_with_human_pref_above_threshold role: complicates claim: GPT-based automated semantic evaluation of spoken dialogue aligns closely with human preference judgements when score differences exceed one point, but reliability degrades for near-tied comparisons. source: Appendix A, Table 5 evidence: The paper's validation of GPT-score against a 25/30 pairwise human comparison shows agreement holds for larger score gaps but weakens for close comparisons, and the paper separately flags a circularity risk since GPT-4o is used as both semantic evaluator and implicit oracle. confidence: medium relevance: high limitations: - The evaluation uses GPT-4o as both the semantic evaluator and implicitly as an oracle for dialogue quality, introducing a potential circularity if the trained model's outputs are biased toward patterns GPT-4o scores favourably. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.08095' published_date: "2025-08-11" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: dualslm_heterogeneous_adapters_enable_perception role: supports claim: Disentangling paralinguistic and linguistic information through separate adapter architectures improves an SLM's ability to perceive both modalities independently without modifying the underlying encoder or LLM. source: §III.A, §V.A, Table I evidence: Both SLM-Qwen and SLM-Llama achieve high weighted accuracy on paralinguistic classification (gender 93.2-93.4%, pitch 82-84%, tempo 91%, energy 79-81%, emotion 90%) even when only paralinguistic embeddings are presented, demonstrating successful disentanglement while keeping the Whisper encoder and LLM frozen. confidence: high relevance: low - claim_id: dualslm_err_prevents_adapter_collapse role: supports claim: Training-time randomisation over paralinguistic embedding sources (speech, text caption, or absent) can prevent adapter collapse into task-specific vectors and preserve contextual generalisation. source: §III.C, §V.B evidence: The Equivalence Replacement Regularization (ERR) strategy randomly combines linguistic embeddings with paralinguistic embeddings drawn from different sources at equal probability during training, forcing the linguistic adapter to capture only information text cannot explain. confidence: medium relevance: low - claim_id: dualslm_adapteronly_competitive_with_finetuning role: supports claim: Parameter-efficient adapter-only training on a frozen LLM is competitive with full fine-tuning approaches for emotional dialogue tasks when the information encoding is structured by design. source: §V.A, Table II-III, §V.B, Table IV evidence: On the MELD emotion recognition benchmark, SLM-Qwen and SLM-Llama (53.3% and 54.9% weighted accuracy) are competitive with Qwen-Audio (55.7%) and Qwen2-Audio (55.3%) despite training neither the encoder nor the LLM. confidence: medium relevance: low - claim_id: dualslm_llm_judge_not_substitute_for_human role: complicates claim: Automatic LLM-as-judge evaluation of emotional conversation quality does not substitute for human subjective evaluation; score magnitudes are judge-model-dependent. source: §IV.B, §V.B evidence: The emotional conversation evaluation shows different absolute score magnitudes depending on which LLM (Qwen2.5-72B vs. Llama-3.1-70B) serves as judge, with no human listening tests reported to calibrate either judge. confidence: medium relevance: low limitations: - The system generates text responses, not speech; despite targeting emotional spoken conversation, the emotional response is encoded in linguistic content and style, not in synthesised speech prosody. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.08961' published_date: "2025-08-12" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA - VC architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: separating_the_token_used_for_llm_input_from role: supports claim: Separating the token used for LLM input from the token used for generation output can resolve the information-level conflict that makes joint optimisation of understanding and generation tasks difficult in a shared-token speech LLM. source: §DualSpeechLM, §Results and Analyses evidence: DualSpeechLM addresses the shared-token bottleneck by explicitly decoupling the input and output token spaces within a single end-to-end framework. confidence: high relevance: low - claim_id: training_a_speech_tokenizer_directly_against_a_text role: supports claim: Training a speech tokenizer directly against a text LLM's next-token prediction objective is more effective at reducing the speech-text modality gap than optimising against ASR or SSL reconstruction losses alone. source: §USTokenizer, Table 1 evidence: 'What distinguishes this tokenizer is an understanding-driven loss: the quantized vectors are projected into a frozen LLaMA-3.2-1B''s embedding space, and the tokenizer is trained to minimise next-token prediction loss on understanding tasks (ASR, emotion recognition, speech question answering).' confidence: high relevance: high - claim_id: understanding_task_supervision_produces_representations_that_transfer_to role: supports claim: Understanding-task supervision produces representations that transfer to generation quality improvements, but the converse — generation-task supervision improving understanding — is weaker and less consistent. source: §Ablation Study, §H. Discussion evidence: A semantic supervision loss trains the LLM to also predict USToken sequences for generation tasks, enforcing consistent semantic representations across pathways. confidence: high relevance: low - claim_id: stochastic_conditioning_during_training_exposing_a_generation_module role: supports claim: Stochastic conditioning during training (exposing a generation module to varied subsets of its conditioning signals) improves robustness to imperfect upstream predictions at inference time. source: §Ablation Study, Table 6 evidence: A Chain-of-Condition (CoC) strategy regularises AcousticGPT training by stochastically conditioning the module on one of three sources — prompt hidden states, predicted USTokens, or their concatenation — sampled with equal probability. confidence: high relevance: low limitations: - The entire evaluation is conducted at 4.5K hours of training data with parameter-efficient LoRA fine-tuning. The claim that USTokens reduce data requirements is plausible but untested at the scale (70K–570K hours) where competing systems are evaluated. Whether the dual-token architecture and the understanding-driven tokeniser remain advantageous at scale is an open question. - No code or demo is linked in the paper, limiting reproducibility. The 4.5K-hour training regime excludes noisy, in-the-wild, and multilingual data, so generalisation to these conditions is untested despite the paper's stated future direction of expanding to multilingual and cross-domain data. The USTokenizer's understanding-driven loss requires a frozen LLM during tokeniser training, adding a significant computational overhead at the tokenisation stage (288% memory increase) even if this overhead disappears at DualSpeechLM inference. The model size of the full system (Phi-3.5-3B + AcousticGPT) is not explicitly stated in aggregate, and the AcousticGPT's token generation speed relative to real-time is not reported. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.09600' published_date: "2025-08-13" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: cot_over_paralinguistic_cues_improves_empathy role: supports claim: Explicit chain-of-thought reasoning over paralinguistic cues (emotion, age, gender, sound events) improves empathetic response generation in speech-to-speech dialogue systems. source: §Training, Stage 3 Empathy; Table 2 evidence: Removing the dual-think chain-of-thought mechanism drops the EChat-eval multi-label score from 72.0 to 56.4 in the ablation. confidence: high relevance: low - claim_id: understanding_pretraining_reduces_paired_data_dependence role: supports claim: Pretraining on multitask speech understanding before speech-to-speech dialogue training reduces dependence on large-scale paired dialogue datasets while maintaining paralinguistic modelling quality. source: §Training, Stage 1; Table 2 evidence: Removing the understanding-driven Stage 1 training drops the multi-label EChat-eval score from 72.0 to 63.1. confidence: medium relevance: high - claim_id: synthetic_s2s_data_less_expressive_than_real role: complicates claim: Synthetic speech-to-speech data derived from TTS systems exhibits reduced emotional expressiveness compared to real human speech, creating a systematic domain gap that degrades empathetic dialogue evaluation. source: §EChat-200K Dataset evidence: The EChat-200K training corpus is synthesised via CosyVoice2 with a minority of real recordings included specifically to mitigate the acknowledged synthetic-emotion expressiveness gap. confidence: medium relevance: low - claim_id: automatic_empathy_evaluation_diverges_from_human role: complicates claim: Automatic empathy evaluation pipelines using LLM scoring and automatic emotion classifiers diverge measurably from human judgements, primarily due to emotion classifier errors and LLM hallucinations. source: §Main Results, Results of Empathetic Intelligence; Table 3 evidence: OSUM-EChat scores 72.0 (GPT-4 automatic) vs. 73.3 (human) on the multi-label metric, and the divergence is more pronounced for individual competitors, with rankings remaining consistent but absolute scores diverging. confidence: medium relevance: low - claim_id: native_multimodal_better_for_paralinguistic_nuance role: supports claim: Native multimodal models that integrate speech token prediction directly into the LLM are better suited to capturing and generating paralinguistic nuance than modularly aligned architectures that treat speech decoding separately. source: §Introduction; §Related Work, End-to-End Spoken Dialogue System evidence: OSUM-EChat, a native multimodal architecture, achieves 87.1 on sound event empathy where most modularly-aligned competitors score zero. confidence: medium relevance: high limitations: - General linguistic intelligence regresses substantially compared to the Qwen2.5-3B text base (GSM8K drops from 85 to 34), indicating the speech-capability gain comes at a significant cost to LLM reasoning that the empathy-focused evaluation does not capture. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.11224' published_date: "2025-08-15" entry_date: '2026-07-27' year: 2025 venue: ASRU task: - evaluation - TTS architecture: [] relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: prosodybench_maskedpred_relative_contours role: supports claim: SSL models using frame-wise masked prediction capture relative prosodic contours within an utterance rather than absolute acoustic magnitudes, making them insensitive to global intensity rescaling. source: §V-A, Fig. 1 evidence: HuBERT, ContentVec, and data2vec show near-zero Token Error Rate change under utterance-level uniform intensity scaling but respond strongly to word-level prosodic emphasis, attributable to waveform normalisation in their CNN front-ends. confidence: high relevance: low - claim_id: prosodybench_discrete_targets_smaller_clusters role: supports claim: Models pretrained to predict discrete targets encode phoneme-like structure effectively at small cluster sizes, while models pretrained on continuous targets require larger cluster sizes to approach the same phonemic alignment. source: §V-A, Fig. 3 evidence: HuBERT and ContentVec (discrete-target pretraining) capture phoneme-like structure even at 100 clusters, while data2vec and emotion2vec (continuous-target pretraining) show degraded prosodic sensitivity at small cluster sizes in their final layers. confidence: high relevance: low - claim_id: prosodybench_emotional_kmeans_increases_sensitivity role: supports claim: Training k-means clustering on emotionally expressive speech increases the prosodic sensitivity of resulting tokens for most SSL model and layer combinations. source: §V-B, Table I evidence: Training k-means on the MEAD emotional-speech dataset significantly increases sensitivity to prosodic variation relative to training on LibriSpeech read speech (p < 0.05) for most model/layer conditions, at the cost of moderately increased sensitivity to real speaker variation. confidence: high relevance: low - claim_id: prosodybench_moving_average_tradeoff role: refines claim: Applying a temporal moving average to SSL features before k-means clustering provides an adjustable trade-off between prosodic sensitivity and speaker invariance, with intermediate window sizes improving both simultaneously. source: §V-C, Fig. 5 evidence: For HuBERT's final layer with 2000 clusters, moving-average window sizes of 9-11 frames outperform no smoothing on prosodic sensitivity while simultaneously reducing speaker sensitivity, showing the two properties are not strictly in tension at intermediate settings. confidence: medium relevance: low - claim_id: prosodybench_layerwise_divergence role: supports claim: Differences between SSL pretraining objectives in their token-level linguistic and prosodic encoding are concentrated in the final transformer layers, while intermediate layers exhibit largely similar behaviour across model families. source: §V-A, §V-B evidence: Phone normalised mutual information (PNMI) confirms that intermediate layers (around layers 9-10) behave similarly across HuBERT, ContentVec, data2vec, and emotion2vec, with divergence concentrated in each model's final layer. confidence: medium relevance: low limitations: - The analysis measures sensitivity via Token Error Rate, a proxy for how much token sequences change in response to acoustic manipulation, rather than directly probing what information is decodable from the tokens; whether observed sensitivity differences translate to downstream task gains remains untested. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2503.04721' published_date: "2025-08-16" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: explicit_turn_state_reduces_pause_interruptions role: supports claim: Explicit turn-state prediction modules reduce unwanted interruptions during intra-turn speaker pauses more effectively than purely end-to-end architectures trained on dialogue data. source: §V, Table III evidence: Freeze-Omni, which uses an explicit turn-state classifier, reduces Takeover Rate to 0.481-0.642 during pause handling versus 0.934-0.985 for end-to-end dGSLM and Moshi. confidence: high relevance: low - claim_id: e2e_lower_latency_but_less_coherent_on_interruption role: supports claim: End-to-end full-duplex speech models respond with lower latency but produce less semantically coherent replies when confronted with user interruptions than cascaded systems with a frozen LLM backbone. source: §V, Table III evidence: Moshi responds to interruptions in 0.257s but scores only 0.765 on GPT-4o coherence, while Freeze-Omni responds in 1.409s with a coherence score of 3.615. confidence: high relevance: high - claim_id: backchanneling_largely_unsolved role: supports claim: Backchannel generation remains a largely unsolved capability in current full-duplex spoken dialogue models, with most open-source systems producing near-zero backchannel frequency. source: §V, Table III evidence: Only dGSLM produces backchannel responses at a measurable rate (Freq = 0.015); Moshi and Freeze-Omni produce near-zero backchannel frequency. confidence: high relevance: high - claim_id: commercial_systems_better_calibrated_turn_taking role: supports claim: Commercial full-duplex systems exhibit better-calibrated turn-taking behaviour on naturalistic data than open-source research models, suggesting that VAD design and deployment-scale tuning contribute substantially to perceived conversational naturalness. source: §V, Table III evidence: Gemini Live achieves the lowest pause-handling TOR (0.255-0.310) and the best backchannel-timing JSD (0.896) among all four evaluated systems. confidence: medium relevance: high - claim_id: automatic_metrics_can_be_reproducible_and_modelagnostic role: supports claim: Automatic metrics for full-duplex dialogue can be made reproducible and model-agnostic without human raters, enabling rapid cross-system comparisons that corpus-level statistics and user studies cannot efficiently provide. source: §I, §III-A evidence: Full-Duplex-Bench derives all four evaluation dimensions (pause handling, backchanneling, smooth turn-taking, interruption management) from automatic, time-aligned ASR transcripts and scenario-controlled stimuli, requiring no human annotators in the evaluation loop itself. confidence: high relevance: high limitations: - The benchmark evaluates only four models (three open-source, one commercial API) and is restricted to English, so conclusions about which architectural choices drive observed behaviours are correspondingly narrow. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0166 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: speech_encoder_transfers_paralinguistic_to_frozen_llm role: supports claim: Training a speech encoder to match LLM responses conditioned on emotion labels can transfer paralinguistic understanding to a frozen LLM without any LLM weight modification. source: §2.3, §4.1, §4.2, Table 1 evidence: SpeechEmotionLlama achieves an LLM-annotated emotion understanding score of 7.54 versus 5.83 for the best cascaded baseline, and 81.51% SER accuracy versus 80.81% for the standalone SER model, while keeping Llama 3 8B entirely frozen throughout training. confidence: high relevance: low - claim_id: cascaded_ser_provides_marginal_improvement role: complicates claim: Cascading a high-accuracy standalone emotion classifier onto a speech-LLM system provides only marginal improvement in emotional response quality. source: §3.2, §4.1, Table 1 evidence: Prepending SER model predictions (80.81% accuracy) to Baseline 2 raises the LLM-annotated emotion score from 5.59 to only 5.83, while encoder-level alignment raises it to 7.54, suggesting that discretized emotion tags do not capture the richness of paralinguistic information as effectively as continuous encoder embeddings. confidence: medium relevance: low - claim_id: proprietary_data_complicates_paralinguistic_evaluation role: complicates claim: Evaluating paralinguistic speech-LLM systems is inherently difficult when training and evaluation data are fully proprietary. source: §3.1, §3.3 evidence: All datasets (pre-training, fine-tuning, and test sets) are proprietary, precluding external replication; the LLM-based evaluation metric (Llama 3 70B as judge) also introduces a dependency on the evaluator model's own behavior, which is not systematically validated against human listeners. confidence: medium relevance: low - claim_id: ssl_pretraining_benefits_paralinguistic_understanding role: supports claim: Self-supervised speech pre-training on large-scale multilingual data provides a strong initialization for downstream paralinguistic understanding in speech encoders. source: §2.2, §3.2, §4.2 evidence: The Conformer encoder is pre-trained with BEST-RQ on approximately 15M hours before fine-tuning on emotion tasks; the system substantially outperforms baselines using a speech encoder that is not further fine-tuned on emotion-related tasks. confidence: medium relevance: low limitations: - All training and evaluation data are proprietary; results cannot be replicated externally and comparisons are restricted to in-house baselines. The role of data scale (1B-parameter encoder, 15M hours pre-training) versus the training paradigm itself is not ablated. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0246 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - codec architecture: - transformer-enc-dec relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: dcspin_speaker_invariant_dual_codebook role: supports claim: Speaker-invariant discrete speech tokens, trained with a dual-codebook objective, can improve both spoken language model performance and speech resynthesis intelligibility simultaneously. source: §3.2, §3.3, Tables 1-3 evidence: DC-Spin 100,4096 achieves 78.47 sWUGGY and 61.35 sBLIMP vs. NAST's 73.35/55.86 at the same unit count, and SpinHuBERT + DC-Spin 500,4096 + PR achieves 18.7% resynthesis WER vs. 24.4% for HuBERT + K-means 500, a roughly 23% relative intelligibility improvement. confidence: high relevance: high - claim_id: dcspin_ngram_pnmi_better_proxy_than_abx role: refines claim: N-gram predictability and phoneme/character mutual information are stronger proxies for SLM downstream performance than the widely-used ABX error rate. source: §3.5, Figure 3b evidence: Proxy analysis shows n-gram predictability, phoneme normalised mutual information (PNMI), and character normalised mutual information (CNMI) are the strongest predictors of downstream SLM performance, while ABX error rate negatively correlates with TSC and sBLIMP. confidence: medium relevance: low - claim_id: dcspin_ssl_tokens_beat_codec_at_lower_bitrate role: supports claim: SSL-based tokenizers can match or exceed neural codec tokenizers on speech intelligibility (WER) at substantially lower bitrates when the encoder is fine-tuned to suppress speaker variation. source: §3.3, Table 3 evidence: SpinHuBERT + DC-Spin + PR achieves 18.7% WER at 448 bps, surpassing EnCodec's 27.5% WER at 1.5 kbps (RVQ1:2), showing SSL tokens fine-tuned to suppress speaker variation can beat codec tokens at a fraction of the bitrate. confidence: high relevance: low - claim_id: dcspin_pr_sft_beats_asr_sft role: supports claim: Supervised fine-tuning with phoneme recognition targets provides consistent gains over ASR targets for speech resynthesis intelligibility, suggesting phoneme alignment is more directly beneficial than word-level transcription for unit-based vocoders. source: §3.3, Table 3 evidence: Across all tested unit counts and codebook sizes in Table 3, supervised fine-tuning with phoneme recognition (PR) targets consistently outperforms ASR-based SFT on resynthesis WER. confidence: medium relevance: low - claim_id: dcspin_scaling_diminishing_returns_semantic role: complicates claim: Scaling SLM model size has diminishing returns on tasks that require sentence-level semantic coherence when the tokenizer quality is the primary bottleneck. source: §3.2, Table 2 evidence: A 150M-parameter SLM trained on 6k hours exceeds sBLIMP/sWUGGY of models using up to 13B parameters and 150k+ hours, but the topic-story-cloze (TSC) gap (70.7 vs. 82.9 for SPIRIT LM) remains large, attributed to scale rather than tokenizer quality. confidence: medium relevance: low limitations: - Evaluation is English-only; the HiFi-GAN vocoder receives speaker and style IDs externally, so the speaker-invariance of the tokens is not tested in an open-vocabulary, zero-shot resynthesis scenario; Expresso WER figures are elevated by expressive speaking styles (laughter, whisper). caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0310 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA - codec architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: coarse_segmentation_large_vocab_beats_finegrained role: supports claim: Moderately coarse fixed-width segmentation combined with a large K-means vocabulary outperforms fine-grained original-resolution tokenization on zero-shot spoken language understanding benchmarks without sacrificing accuracy. source: §4, Table 3 evidence: The (N=80ms, K=16384) configuration achieves average accuracy 0.67 across five zero-shot SLU benchmarks versus 0.65 for the (N=20ms, K=128) GSLM-style baseline, while reducing training data by 50% and runtime by 70%. confidence: high relevance: high - claim_id: segment_width_vocab_size_tradeoff role: supports claim: Larger segmentation widths require proportionally larger vocabularies to preserve phonetic discriminability, analogous to the phoneme-morpheme relationship in linguistics. source: §5.1 evidence: At (N=80ms, K=128) the model cannot distinguish phonetically similar tokens (e.g. the yonder/zonder sWUGGY minimal pair), but increasing K to 16384 at the same segmentation width restores discriminability. confidence: medium relevance: low - claim_id: variable_segmentation_not_better_than_fixed role: supports claim: Variable-width segmentation based on linguistic units (phoneme, syllable, word boundaries) does not consistently outperform fixed-width segmentation of matched median duration, and incurs additional computational cost. source: §5.3 evidence: Phoneme-, syllable-, and word-level variable-width segmentation (via UnsupSeg, Sylber, GradSeg) generally matches but does not exceed fixed-width baselines of the same median segment duration across the five SLU benchmarks. confidence: medium relevance: low - claim_id: slu_benchmark_optimum_varies_by_task role: refines claim: Optimal speech tokenization settings vary across zero-shot spoken language understanding benchmarks, indicating that no single tokenization configuration is uniformly best and that multi-token ensembling may be required for broad SLU capability. source: §5.2 evidence: Different SLU benchmarks (sBLIMP, sWUGGY, pros-syntax, pros-lexical, tSC) reach their individual optimum at different (N, K) settings within the 64-configuration grid, with pros-syntax saturating even at large N due to inserted pauses while sBLIMP remains near chance across nearly all settings. confidence: medium relevance: high limitations: - Evaluation is exclusively on SLU (understanding) tasks; no speech generation quality assessment is performed. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0468 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - codec architecture: - VAE - GAN relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - adversarial_codec_speech_models - variational_latent_speech_models claims: - claim_id: directly_encoding_ssl_features_as_a_first_class role: supports claim: Directly encoding SSL features as a first-class codec stream produces stronger semantic preservation in RVQ-1 tokens than distillation from an SSL model, particularly for tonal languages where pitch fidelity is critical. source: §4.2, Table 2 evidence: The SSL encoding stream takes 16th-layer features from a frozen w2v-BERT-2.0 model (600M parameters), downsamples them via 1D average pooling to the target frame rate (factor 2 for 25 Hz, factor 4 for 12.5 Hz), and passes them through a ResNet encoder (ConvNeXt blocks, 13M parameters) before VQ quantisation. confidence: high relevance: low - claim_id: operating_a_neural_codec_at_lower_frame_rates role: supports claim: Operating a neural codec at lower frame rates with more RVQ layers at fixed token rate improves audio quality over higher-frame-rate codecs with fewer layers at the same bitrate. source: §4.3, Table 3 evidence: The waveform encoding stream follows the DAC architecture — a CNN encoder with strided convolutions reaches the same frame rate, then N-1 RVQ layers quantise the residual between the waveform feature and the RVQ-1 feature; the decoder sums both streams back before upsampling to audio. confidence: high relevance: low - claim_id: semantic_quality_of_rvq_1_tokens_is_a role: supports claim: Semantic quality of RVQ-1 tokens is a primary determinant of downstream TTS intelligibility in autoregressive codec-based systems, independent of codec audio reconstruction quality. source: §4.4, Table 4 evidence: On codec audio reconstruction quality (LibriSpeech-test-clean, normalised to 75 tokens/s, ~0.75 kbps), DualCodec 25Hz with 16384 codebook (G2) achieves MUSHRA 86.2 and UTMOS 4.08, exceeding WavTokenizer-large (MUSHRA 81.0, UTMOS 3.87) and Mimi (MUSHRA 72.8). confidence: high relevance: low - claim_id: an_ssl_based_semantic_stream_in_a_codec role: supports claim: An SSL-based semantic stream in a codec encoder can improve perceptual audio quality beyond what waveform-only codecs achieve, even when using the same decoder architecture. source: §4.3, Table 3 evidence: The waveform encoding stream follows the DAC architecture — a CNN encoder with strided convolutions reaches the same frame rate, then N-1 RVQ layers quantise the residual between the waveform feature and the RVQ-1 feature; the decoder sums both streams back before upsampling to audio. confidence: high relevance: low limitations: - The 12.5 Hz DualCodec-based TTS lags behind the 25 Hz variant in both WER and speaker similarity, indicating that the more aggressive downsampling introduces a ceiling on semantic accuracy that affects TTS quality. The paper acknowledges this gap as the primary remaining challenge. - The SSL model (w2v-BERT-2.0, 600M parameters, frozen) is required at TTS training time but not inference. This makes the training pipeline heavier than pure waveform codec approaches. It is also unclear whether the approach generalises to SSL models other than w2v-BERT-2.0, or whether the chosen 16th layer feature is optimal across languages beyond English and Mandarin. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0739 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: internal_interrupt_detection_beats_vad_based_fdsds role: supports claim: Full-duplex spoken dialogue systems that integrate interruption detection internally rather than relying on external VAD modules show systematically higher interrupt success rates under challenging conditions. source: §3.2, Table 4 evidence: Moshi, which uses a built-in full-duplex mechanism, achieves SRRate under Easy interruptions of 61.7% versus 11.3% for Freeze-Omni and 26.9% for VITA-1.5, both of which rely on external VAD. confidence: high relevance: high - claim_id: latency_and_interruption_robustness_tradeoff role: complicates claim: 'Real-time responsiveness (first-speech-emit delay) and interruption robustness trade off against each other: systems optimised for low latency may not handle mid-utterance interruptions reliably.' source: §3.2, Table 4 evidence: Freeze-Omni achieves the lowest first-speech-emit delay (449-515 ms versus 2,020-4,155 ms for Moshi) yet handles interruptions far less reliably than Moshi. confidence: medium relevance: low - claim_id: background_noise_worse_than_gap_noise_for_fdsds role: supports claim: Background noise degrades response quality in full-duplex spoken dialogue systems more severely than discrete noise events inserted into silence gaps. source: §3.2, Table 5 evidence: Continuous background noise (Noise-bg) at low SNR degrades subjective response-quality scores across all three tested systems, while gap noise inserted into silence periods has comparatively little effect, with Freeze-Omni's high VAD threshold blocking most spurious gap events. confidence: medium relevance: high - claim_id: automated_scoring_and_perplexity_are_complementary_not_sufficient role: refines claim: Automated LLM-based scoring and conditioned perplexity provide complementary signals for evaluating spoken dialogue quality, but neither alone captures interruption-handling behaviour. source: §2.3, §3.2 evidence: FD-Bench combines GPT-4o-based six-dimension subjective scoring with Llama-3.3-70B conditioned perplexity (c-PPL) alongside dedicated timing metrics (IRD, FSED) because response-quality scoring alone does not capture interruption success or timing behavior. confidence: medium relevance: high - claim_id: fdsds_far_from_human_robustness_under_interruption role: supports claim: Current full-duplex spoken dialogue systems remain far from human-level robustness under frequent interruptions, suggesting the capability remains an open engineering and research challenge. source: §3.2, §5 evidence: All three tested open-source FDSDS (Moshi, Freeze-Omni, VITA-1.5) show substantial performance degradation under frequent interruptions and noisy conditions, with no system approaching reliable interrupt handling across all tested scenarios. confidence: medium relevance: high limitations: - All reported results use speech synthesised by CosyVoice2 only; the ChatTTS and F5-TTS subsets are present in the corpus but results are not reported in the paper, and evaluation is English-only. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0756 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: affective_routing_improves_emotion_prediction_and_response role: supports claim: Structuring speech adapter routing around distinct affective dimensions improves emotion prediction and response quality in spoken dialogue systems compared to single-path adapters. source: §3.8, Table 4 evidence: A-SMiLE's task-aware top-1/top-4 expert routing for VAD versus response generation outperforms conv-based, Q-Former, and standard Transformer adapters across METEOR, ROUGE-L, and GPT-4o empathy scores on DailyTalk. confidence: medium relevance: high - claim_id: multitask_emotion_response_training_improves_both role: supports claim: Multi-task joint training of emotion regression and response generation improves both emotional alignment and text quality over response-only training in spoken dialogue. source: §3.6, Table 2 evidence: EAML fine-tuning consistently raises CCC scores and METEOR/ROUGE-L for both 0.5B and 7B backbones over text-only and cascaded baselines that receive VAD labels as auxiliary text rather than learned representations. confidence: medium relevance: high - claim_id: discrete_emotion_labels_insufficient_for_finegrained_states role: complicates claim: Discrete categorical emotion labels are insufficient for capturing fine-grained paralinguistic states such as sarcasm and depression in spoken dialogue. source: §3.1, §3.5, Table 1 evidence: The hard-case benchmark (sourced from IEMOCAP and MUStARD) specifically targets emotionally complex states where existing systems trained on categorical emotion corpora fail to capture nuance; A-SMiLE's CCC-based VAD modeling shows substantial gains on exactly these cases. confidence: medium relevance: high - claim_id: text_overlap_and_llm_empathy_metrics_may_not_reflect_human_judgment role: complicates claim: Automatic text-overlap metrics (METEOR, ROUGE-L) and LLM-based empathy scoring may not fully reflect human perceptual judgments of emotional appropriateness in dialogue responses. source: §3.2 evidence: The paper evaluates response quality exclusively with METEOR, ROUGE-L, and GPT-4o-1120 empathy scores; no human listening evaluation is reported, leaving open whether score gains translate to perceived empathy. confidence: medium relevance: low limitations: - The evaluation relies entirely on reference-text metrics (METEOR, ROUGE-L) and GPT-4o-based empathy scores, with no human perceptual study, and the DailyTalk corpus is only 20 hours of scripted dialogue between two fixed speakers. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-0874 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: separating_user_and_agent_stream_representations_using_a role: supports claim: Separating user and agent stream representations — using a pretrained speech encoder for input and a neural codec for generation — allows full-duplex S2S models to bypass LLM speech pretraining without sacrificing conversation quality. source: §3, §6.1, §6.2 evidence: Because the user stream uses a pretrained encoder rather than codec tokens, no speech-text pretraining of the LLM backbone is required — a sharp contrast with prior duplex systems such as Moshi. confidence: high relevance: high - claim_id: codec_personalisation_through_fine_tuning_on_target_speaker role: supports claim: Codec personalisation through fine-tuning on target-speaker data can recover audio quality at half the bitrate of an untuned codec, as measured by MOS, CER, and speaker similarity. source: §6.3, Table 4 evidence: Codec personalisation for the agent voice is enabled by fine-tuning NanoCodec on 21k utterances from the target speaker; this is practical precisely because agent and user modalities are handled by separate components. confidence: high relevance: low - claim_id: turn_level_alignment_between_text_and_speech_tokens role: supports claim: Turn-level alignment between text and speech tokens in duplex training is sufficient to learn barge-in behaviour; word-level alignment provides no measurable improvement. source: §3.1 evidence: 'The model is trained with multi-channel next-token prediction: text and speech targets are maintained in separate channels, aligned at the turn level (not word level), with text weighted 3× relative to speech in the loss.' confidence: high relevance: high - claim_id: full_duplex_end_to_end_models_remain_at role: supports claim: Full-duplex end-to-end models remain at a reasoning disadvantage compared to cascaded oracle systems, though the gap narrows as backbone LLM quality increases. source: §6.2, Table 3 evidence: On reasoning quality (Table 3, GPT-4o-mini scores), the proposed model outperforms Moshi across all five evaluation sets despite using a far smaller backbone; it is competitive with an oracle cascaded system on two of five sets. confidence: high relevance: high - claim_id: open_source_availability_of_training_code_and_model role: complicates claim: Open-source availability of training code and model weights is a critical bottleneck for research progress in full-duplex spoken dialogue, given the historical concentration of such systems in closed industrial labs. source: §1 evidence: Because the user stream uses a pretrained encoder rather than codec tokens, no speech-text pretraining of the LLM backbone is required — a sharp contrast with prior duplex systems such as Moshi. confidence: high relevance: high limitations: - The backbone is TinyLlama-1.1B, a relatively small LLM. The reasoning gap between the end-to-end model and the GT+LLM cascaded oracle is real and acknowledged, particularly on QA tasks. Scaling to a larger backbone remains untested and its interaction with the duplex architecture is an open question. - Training data is entirely synthetic (TTS-generated user and agent speech) except for the ASR-QA portion, which introduces a domain mismatch with natural conversation. The fixed 0.64-second silence inserted before agent turns is a hard-coded heuristic that will affect latency in practice and may not generalise to more varied conversational pacing. The evaluation does not include a listening test (MOS via human raters) for duplex conversation quality — UTMOS and GPT score are proxies. The first-response latency metric is not comparable to Moshi because Moshi's proactive interruption behaviour makes the metric inapplicable, which limits direct system comparison. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-1289 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - codec architecture: - hybrid relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: constant_frame_rate_coding_introduces_temporal_redundancy_in role: supports claim: Constant-frame-rate coding introduces temporal redundancy in neural speech codecs by allocating equal resolution to silence and phonetically dense regions alike. source: §1, §2.1 evidence: Frame rate allocation is determined by non-parametric temporal entropy computed over each resolution's receptive field window. confidence: high relevance: low - claim_id: dynamically_allocating_coarser_temporal_frames_to_low_entropy role: supports claim: Dynamically allocating coarser temporal frames to low-entropy speech regions reduces token sequence length without proportional degradation in reconstruction quality. source: §4.2, Table 1 evidence: Notably, certain mixed-granularity configurations at lower average frame rates (below 75Hz) achieve higher UTMOS scores than the 75Hz DAC+TFC configuration, indicating that forced fine-grained coding of low-entropy regions carries an unnecessary reconstruction cost. confidence: high relevance: low - claim_id: reducing_the_number_of_encoded_frames_at_equivalent role: supports claim: Reducing the number of encoded frames at equivalent bitrate can improve intelligibility, suggesting that sequence compactness benefits autoregressive downstream models independently of bitrate. source: §4.2, Table 1 evidence: At 6kbps, the WER advantage is more pronounced (4.13 → 2.85), suggesting that sequence-length reduction — rather than just bitrate — also benefits intelligibility. confidence: high relevance: low - claim_id: variable_frame_rate_allocation_and_variable_bitrate_control role: supports claim: Variable frame rate allocation and variable bitrate control are orthogonal axes in neural codec design and can be combined additively. source: §3, §5 evidence: TFC is a plug-and-play module built on top of the DAC codec backbone (24kHz, 75Hz frame rate). confidence: high relevance: low limitations: - Evaluation is restricted to a single codec backbone (DAC) and a single dataset (LibriTTS). No human listening study is reported; all quality judgements rest on UTMOS, STOI, WER, and spectral distances. Generalisability to other architectures or acoustic conditions is untested. - The paper does not report downstream TTS or speech LM experiments, so the claimed benefit of shorter token sequences for generation quality and latency is prospective rather than demonstrated. The entropy-based frame allocation heuristic uses fixed hyperparameters (bin count N, smoothing σ) without ablation; sensitivity to these choices is unknown. Training with mixed granularity ratios uses a fixed heuristic allocation that may not be optimal. The granularity ratios at inference must be set by the user; no automatic target-rate optimisation is described. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-1538 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - VC architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: conditioning_acoustic_generation_on_explicitly_predicted_text_tokens role: supports claim: Conditioning acoustic generation on explicitly predicted text tokens reduces intelligibility errors in autoregressive voice conversion relative to purely acoustic-domain approaches. source: §3.3.3, Table 1 evidence: Removing text token generation from StarVC raises WER from 6.27% to 7.30% and SECS-WavLM drops from 0.472 to 0.382; StarVC achieves the lowest WER and CER among all compared systems including diffusion-based CosyVoice (8.24%/4.27%). confidence: high relevance: low - claim_id: multi_stage_training_that_initializes_voice_conversion_with role: supports claim: Multi-stage training that initializes voice conversion with ASR pretraining improves both content preservation and speaker similarity relative to single-stage training. source: §3.3.3, Table 1 evidence: Removing multi-stage training degrades SECS-Res from 0.835 to 0.812 and raises WER from 6.27% to 7.24%; multi-stage training is the single largest contributor in the ablation study. confidence: high relevance: low - claim_id: objective_speaker_embedding_metrics_and_perceptual_speaker_similarity role: complicates claim: Objective speaker embedding metrics and perceptual speaker similarity ratings can diverge for codec-based voice conversion systems trained with strong linguistic objectives. source: §3.3.1, §3.3.2, Tables 1-2 evidence: StarVC scores marginally below CosyVoice on SECS-Res (0.835 vs. 0.839) and SECS-WavLM (0.472 vs. 0.478), yet exceeds CosyVoice on subjective SMOS (3.98 vs. 3.94), suggesting embedding-based metrics underestimate perceived similarity for this system class. confidence: high relevance: low - claim_id: autoregressive_voice_conversion_systems_can_produce_explicit_transcription role: refines claim: Autoregressive voice conversion systems can produce explicit transcription output alongside converted audio at negligible additional cost, enabling inline content verification without separate ASR inference. source: §3.3.1, Table 1 evidence: StarVC generates text tokens with WER-Text of 4.95% and CER-Text of 1.51% as a byproduct of the VC decoding process, providing word-level content verification as part of the conversion pipeline. confidence: high relevance: low limitations: - Subjective MOS evaluation involves only 20 listeners and 20 source-target pairs, making the reported SMOS and NMOS advantages over CosyVoice and OpenVoice V2 (all within overlapping confidence intervals) difficult to interpret as significant. - The evaluation covers English only on a single clean corpus (LibriTTS test-clean). Generalization to cross-lingual conversion, noisy conditions, or longer conversational utterances is untested. The three-stage training pipeline requires 180 GPU-hours on 8 H100s, representing a substantial compute cost that may limit practical adoption. Data augmentation relies on OpenVoice V2-synthesized speech, which could propagate artifacts from that system into StarVC's training distribution. Whether the text-before-speech decoding constraint generalizes to expressive or emotional speech conversion, where prosody is not captured by a pure transcription, remains an open question. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-1595 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: scheduledilt_gradual_modality_transition_improves_adaptation role: supports claim: Gradual modality transition during fine-tuning, rather than abrupt full-speech training, improves LLM adaptation from text to speech units. source: §3.2, Table 2 evidence: Scheduled interleaved speech-text training, which decays the text ratio from 0.9 to 0 over training, consistently improves ASR-BLEU over the ASR-fine-tuned baseline across seven CVSS-C language pairs without degrading UTMOS. confidence: high relevance: high - claim_id: scheduledilt_lowresource_benefit_amplified role: refines claim: The benefit of interleaved speech-text training is amplified in low-resource language settings where speech-domain supervision is scarce. source: §4, Table 2 evidence: Portuguese-to-English improves from 10.3 to 19.5 ASR-BLEU (+89%) and Russian-to-English from 6.0 to 14.1 (+135%), far larger gains than high-resource pairs like French-to-English (28.8 to 29.5). confidence: high relevance: high - claim_id: scheduledilt_twosided_interleaving_necessary role: supports claim: Applying word-aligned text interleaving to both source and target sequences simultaneously is necessary; one-sided interleaving provides substantially weaker adaptation. source: §4, Table 3 evidence: One-sided interleaving variants score only 11.1-12.6 ASR-BLEU on Portuguese-to-English versus 19.5 for two-sided interleaving. confidence: high relevance: high - claim_id: scheduledilt_modality_gap_length_and_representation role: supports claim: The modality gap between speech and text in LLM-based S2ST manifests as both a length disparity and a representation distance, and scheduled training addresses both. source: §3.1, Figure 3 evidence: Speech units are 13-21 times longer than corresponding text, and the scheduled decay from near-text to near-speech representations is designed specifically to close this length and representation gap gradually. confidence: medium relevance: low limitations: - Evaluation is limited to seven language pairs from a single corpus (CVSS-C) and a 1B-parameter LLM; scaling to larger models or more diverse corpora is untested. - The unit-based HiFi-GAN vocoder maps to a single canonical TTS voice, so speaker identity is not preserved and results do not generalise to voice-preserving S2ST. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-1641 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - TTS architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: explicit_phoneme_position_supervision_during_autoregressive_codec_training role: supports claim: Explicit phoneme position supervision during autoregressive codec training eliminates alignment errors more effectively than phoneme identity prediction or monotonic decoding constraints. source: §4.2.1, Table 2; §4.3, Table 4 evidence: VALL-E R, the closest prior baseline, reduces deletion errors well (8.37% hard CER) but its greedy decoding variant degrades severely (19.33%), showing that phoneme identity prediction alone does not solve the fundamental alignment ambiguity in repeated-character contexts. confidence: high relevance: low - claim_id: alignment_failures_in_codec_language_model_tts_including role: supports claim: Alignment failures in codec language model TTS — including phoneme skipping, repetition, and one-to-many correspondence — are fundamentally a training-objective problem rather than an inference-time problem. source: §4.2.1, §4.3, Table 4 evidence: An alignment analysis on the hard set confirms that VALL-E makes 1,656 phoneme-skipping errors and 813 repetition errors; the proposed method eliminates all three error categories entirely (zero occurrences of skipping, repetition, and one-to-many alignment errors in Table 4). confidence: high relevance: high - claim_id: jointly_predicting_phoneme_identity_and_position_introduces_conflicting role: supports claim: Jointly predicting phoneme identity and position introduces conflicting signals that degrade pronunciation accuracy compared to position-only prediction. source: §4.2.1, Table 2 evidence: The ablation VALL-E BOTH (joint identity + position prediction) achieves 6.07% hard CER but introduces more substitution errors than position-only prediction, suggesting that conflicting phoneme identity and position signals interfere with pronunciation accuracy. confidence: high relevance: low - claim_id: robustness_improvements_in_autoregressive_codec_tts_can_be role: supports claim: Robustness improvements in autoregressive codec TTS can be achieved without changes to inference-time decoding strategy or additional duration prediction stages. source: §3.3, §4.2.1, Table 1 evidence: VALL-E R, the closest prior baseline, reduces deletion errors well (8.37% hard CER) but its greedy decoding variant degrades severely (19.33%), showing that phoneme identity prediction alone does not solve the fundamental alignment ambiguity in repeated-character contexts. confidence: high relevance: low limitations: - The model is trained and evaluated exclusively on Mandarin using a proprietary G2P toolkit and character-level duration annotations from the WenetSpeech4TTS dataset. Generalisation to languages without character-aligned duration labels, or to datasets where forced-alignment quality is lower, is untested. - The approach requires phoneme duration annotations at training time to construct the position sequence, which constrains its applicability to datasets with reliable forced alignments. At inference, the enrollment speech prompt must include a duration estimate — the paper does not discuss what happens when this estimate is imprecise. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-1873 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: speech_llms_lack_robust_prosodic_comprehension role: supports claim: End-to-end Speech LLMs have not yet achieved robust prosodic comprehension, particularly for tasks that require interpretation of subtle acoustic prosodic cues without supporting contextual information. source: §3.2.2, §3.3 evidence: Across eight Mandarin prosodic tasks, all six evaluated Speech LLMs fell substantially below human-participant accuracy (human range 81.01%-95.67%; best model GPT-4o at 59.70%). confidence: high relevance: low - claim_id: aggregate_benchmark_accuracy_masks_task_dependent_disparities role: complicates claim: Aggregate benchmark accuracy scores obscure systematic task-dependent performance disparities in Speech LLMs. source: §3.3 evidence: Several models achieved human-comparable accuracy on context-rich tasks (irony, focus operator) while failing significantly on prosody-dependent tasks (focus marking, emotion without context), showing average performance masks strong variation. confidence: high relevance: low - claim_id: existing_audio_benchmarks_insufficient_for_prosodic_diagnosis role: supports claim: Existing general-purpose audio benchmarks are insufficient for diagnosing prosodic comprehension weaknesses in Speech LLMs, especially for tonal languages. source: §1, §2.1 evidence: AIR-Bench, Audiobench, SD-Eval, StyleTalk, and SALMON collectively lack tasks for prosodic disambiguation, focus marking, scalar implicature, and tone-intonation interaction, motivating MSPB's eight-task battery. confidence: high relevance: low - claim_id: safety_restrictions_can_invalidate_prosodic_benchmark_evaluation role: complicates claim: Safety and compliance restrictions in proprietary Speech LLMs can invalidate prosodic benchmark evaluations for specific task categories. source: §3.2.2 evidence: Gemini-1.5-Pro declined to respond to emotional prosody without context items, producing no evaluable output for that task category and leaving a gap in the cross-model comparison. confidence: medium relevance: low limitations: - The benchmark is limited to single-round, zero-shot evaluation in Mandarin with a single speaker's voice, and the 178-item dataset is relatively small without per-item difficulty analysis. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-2328 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - TTS - evaluation architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: statistical_watermarking_techniques_developed_for_text_language_models role: supports claim: Statistical watermarking techniques developed for text language models can be applied to autoregressive speech generation models by treating speech tokens as text tokens and applying token-distribution reweighting during generation. source: §4, Table 1 evidence: KGW and DiPmark are adapted to SpiritLM's HuBERT-based speech token vocabulary with no architectural modification to either system, achieving TPR@1%FPR up to 98.9% on C4. confidence: high relevance: high - claim_id: statistical_watermark_detection_in_autoregressive_speech_systems_is role: complicates claim: Statistical watermark detection in autoregressive speech systems is degraded by the lossy round-trip of audio decode followed by re-encode, which disrupts the token-level statistical signal. source: §3.2, §5.2, Table 6 evidence: On Dolly-CW, KGW (δ=1.0) drops from 99% to 17% TPR@1%FPR after re-encoding; DiP-reweight (α=0.4) drops from 99% to 53%, demonstrating that the mismatch is the dominant failure mode regardless of watermark method. confidence: high relevance: low - claim_id: distortion_free_watermarking_methods_preserve_generated_audio_quality role: supports claim: Distortion-free watermarking methods preserve generated audio quality at the cost of reduced detection power compared to logit-bias methods. source: §5.1, Tables 1–4 evidence: DiP-reweight maintains FAD scores near the no-watermark baseline (0.0067 vs. 0.0051 on C4 at α=0.4) while KGW (δ=2.0) raises FAD to 0.0494; correspondingly, DiP-reweight peaks at 47.5% vs. 98.9% TPR@1%FPR. confidence: high relevance: low - claim_id: the_strength_of_a_statistical_watermark_signal_before role: complicates claim: The strength of a statistical watermark signal before audio rendering is not a reliable predictor of detection accuracy in deployed conditions, because re-encoding introduces a new and substantial source of signal loss. source: §5.2, Table 5 evidence: Before re-encoding, KGW (δ=1.0) achieves 97.1% TPR@1%FPR on C4; after re-encoding this falls to 54.3%, illustrating that near-perfect pre-decoding detection does not guarantee practical detectability. confidence: high relevance: low limitations: - The framework is evaluated only on SpiritLM, which uses HuBERT-based discrete speech units. Applicability to neural codec-based AR speech models (e.g., those using EnCodec or Mimi) is untested, and re-encoded mismatch characteristics may differ substantially across codec types. - No mitigation for the re-encoded mismatch is proposed; the paper frames this as an open problem without a solution pathway. The evaluation uses Frechet Audio Distance to assess distortion but does not include subjective listening tests, leaving the perceptual impact of the KGW-reweight quality degradation unverified. Robustness against adversarial attacks on the watermark (e.g., re-synthesis, noise injection) is not evaluated. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-2564 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: large_scale_monolingual_pre_training_followed_by_stereo role: supports claim: Large-scale monolingual pre-training followed by stereo dialogue fine-tuning enables spoken dialogue models to acquire language-specific conversational behaviors. source: §5, Table 3 evidence: J-Moshi, trained on J-CHAT and stereo Japanese dialogue, exhibits more speech overlaps (5.0s/min) and more IPUs (53.2/min) than English Moshi (1.2s overlap, 35.1 IPUs), consistent with Japanese conversational norms. confidence: high relevance: high - claim_id: synthetic_spoken_dialogue_generated_by_multi_stream_tts role: supports claim: Synthetic spoken dialogue generated by multi-stream TTS improves language capability in full-duplex dialogue models when added to fine-tuning data. source: §4.3, Table 2 evidence: J-Moshi-ext (trained with 602 hours of TTS-synthesized dialogue added) achieves meaningfulness 2.30 versus J-Moshi's 2.19, a statistically distinguishable improvement, with no degradation in naturalness. confidence: high relevance: high - claim_id: neural_audio_codecs_pre_trained_on_one_language role: complicates claim: Neural audio codecs pre-trained on one language can transfer to another with minimal acoustic degradation, but the autoregressive language model component requires substantial retraining to achieve acceptable dialogue quality. source: §4.3, Table 2 evidence: Mimi re-synthesis of Japanese speech degrades by approximately 0.5 MOS from ground truth, while J-Moshi (with RQ-Transformer adapted) degrades by more than 1 MOS, identifying the language model as the primary quality bottleneck. confidence: high relevance: high - claim_id: morphological_density_differences_across_languages_affect_full_duplex role: complicates claim: 'Morphological density differences across languages affect full-duplex dialogue model training dynamics: languages with higher phoneme-to-token ratios produce sparser text-to-audio token alignments that may require adjusted training objectives.' source: §5 evidence: Japanese data preprocessing results in 88% PAD tokens in text sequences versus 65% for English in Moshi, reflecting that kanji characters encode more phonemes per token, and the authors flag this as a design consideration for future Japanese-specific training objectives. confidence: high relevance: high limitations: - Both J-Moshi and J-Moshi-ext score above 1 MOS below the Mimi re-synthesis ceiling, indicating the autoregressive RQ-Transformer component is a major quality bottleneck in Japanese. Mimi itself degrades by approximately 0.5 MOS from ground truth when applied to Japanese without adaptation, suggesting codec fine-tuning for Japanese will be necessary for production-quality systems. - The comparison with English Moshi in Table 3 is not conducted under identical experimental conditions (different test sets, possibly different prompt lengths), so turn-taking statistics should be interpreted as indicative rather than rigorously controlled. The 24.6% overall WER of the TTS-synthesized augmentation data introduces noise, and the effect of this noise on specific error categories is not analyzed. The paper does not evaluate spoken dialogue content quality beyond naturalness and meaningfulness, leaving turn-taking appropriateness and response coherence unmeasured. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-2660 published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA architecture: - transformer-enc-dec relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - transformer_encoder_decoder_speech_models claims: - claim_id: acoustic_only_vap_predicts_nextspeaker_in_triadic_conversation role: supports claim: Acoustic-only voice activity projection can predict next-speaker identity in triadic conversation with accuracy above a last-speaker baseline, though gains over baseline are modest in spontaneous overlapping speech. source: §3.2, Table 4 evidence: 'Next-speaker prediction accuracy during mutual silence outperforms a last-speaker baseline for all three model variants (SP: 62.43% vs. 55.56%; ATT: 87.50% vs. 81.25%; BOTH: 86.83% vs. 77.58%).' confidence: medium relevance: low - claim_id: multiparty_turntaking_complexity_grows_faster_than_linear role: complicates claim: The complexity of multi-party turn-taking increases faster than linearly with group size, making state-space design a fundamental constraint for VAP-style approaches beyond dyadic settings. source: §2.1, §4 evidence: The triadic model already required reducing sub-state bins from 4 to 2 per speaker to remain tractable, and the paper estimates 65,536 states for four-participant conversation under the original 4-bin scheme. confidence: medium relevance: low - claim_id: conversation_type_substantially_affects_vap_accuracy role: complicates claim: The type of conversation, spontaneous discussion versus structured attentive listening, substantially affects VAP accuracy, reflecting differences in overlapping speech prevalence. source: §3.2, Table 4 evidence: 'Performance is substantially higher on attentive listening data (ATT: 87.50%), where one speaker dominates and overlap is 21.17%, than on spontaneous discussion data (SP: 62.43%), where overlapping speech reaches 31.98%.' confidence: medium relevance: low - claim_id: multiparty_dialogue_systems_need_matched_multiparty_training_data role: supports claim: Spoken dialogue systems operating in multi-party scenarios require turn-taking models trained on matched multi-party data, as dyadic training corpora are unlikely to transfer directly. source: §2.2, §4 evidence: The paper constructs the TEIDAN triadic corpus specifically because no prior triadic corpus with per-speaker separated audio existed for training or evaluating multi-party VAP, and dyadic VAP had never been demonstrated in a triadic setting. confidence: low relevance: high limitations: - The TEIDAN corpus is small (under 5 hours), monolingual (Japanese), and not yet publicly released; reported accuracy gains are measured on the same corpus used for training and testing, with no held-out language or domain. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: interspeech-2025-bokkahallisatish25_interspeech published_date: "2025-08-17" entry_date: '2026-07-27' year: 2025 venue: Interspeech task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: interactive_voice_conversion_pairing_surfaces_speaker_dependent_bias role: supports claim: Interactive evaluation tools that pair voice conversion with side-by-side response comparison can surface speaker-dependent biases in speech foundation models that automated benchmarks do not capture. source: §2.2, §2.4 evidence: The platform lets users re-ask questions with voice-converted inputs and directly observe changes in response tone, phrasing, and behavior, positioning human interactive comparison as the primary bias discovery mechanism rather than the automated metrics. confidence: medium relevance: low - claim_id: automated_metrics_insufficient_for_fairness_assessment_alone role: complicates claim: Automated speech evaluation metrics are insufficient on their own for assessing fairness and speaker-dependent behavior in speech foundation models. source: §2.3, §2.4 evidence: Even with speech rate, pitch statistics, sentiment analysis, and semantic similarity available, the authors present human interactive comparison as the primary mode of bias discovery, arguing that quantitative measures miss subtle differences in model behavior across speaker identities. confidence: medium relevance: low - claim_id: existing_benchmarks_treat_speaker_attributes_independently role: supports claim: Existing benchmarks for speech foundation models treat speaker attributes independently, limiting the ability to detect intersectional or compounded identity-based biases. source: §1 evidence: Survey of VoxDialogue, Spoken StereoSet, and VoxEval reveals that each evaluates dimensions such as gender or age in isolation; VoxEval explicitly excludes bias and harmful content analysis as outside its scope. confidence: medium relevance: low limitations: - The platform reports no empirical benchmark results, so claims about its effectiveness at bias discovery remain qualitative and user-dependent, and the voice conversion module's perceptual fidelity is not systematically evaluated. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.15827' published_date: "2025-08-18" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: in_speech_models_reasoning_depth_and_response_latency role: supports claim: In speech models, reasoning depth and response latency are not fundamentally in conflict when the model's token generation rate substantially exceeds the real-time audio playback rate. source: §1, §3.1 evidence: 'On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model''s 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale.' confidence: high relevance: high - claim_id: interleaving_silent_reasoning_tokens_with_spoken_response_tokens role: supports claim: Interleaving silent reasoning tokens with spoken response tokens at a fixed ratio can improve accuracy on structured reasoning tasks while reducing audible output length. source: §2.3, §4.4, Table 2 evidence: 'On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model''s 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale.' confidence: high relevance: high - claim_id: the_thinking_before_speaking_paradigm_when_applied_directly role: supports claim: The "thinking-before-speaking" paradigm, when applied directly to speech, produces user-facing latency or verbosity that impairs conversational quality independently of reasoning correctness. source: §1, §2.2 evidence: 'On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model''s 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale.' confidence: high relevance: high - claim_id: multi_stage_training_separating_modality_alignment_reasoning_transfer role: supports claim: Multi-stage training — separating modality alignment, reasoning transfer, and acoustic synthesis — is an effective strategy for progressively adapting an existing speech LLM to a new generation paradigm. source: §3.3 evidence: 'Training proceeds in five stages: (1) audio adapter alignment with the language backbone; (2) mixed mathematical pretraining in thinking-before-speaking format; (3) textual thinking-in-speaking training (language model parameters only); (4) acoustic transfer — audio encoder fine-tuning with the frozen LLM; (5) Talker training with the entire Thinker frozen.' confidence: high relevance: high - claim_id: synthetic_speech_based_mathematical_reasoning_datasets_constructed_from role: supports claim: Synthetic speech-based mathematical reasoning datasets constructed from text corpora via TTS can provide sufficient training signal for spoken reasoning capabilities. source: §3.2, §4.4 evidence: To create training data for interleaved reasoning, the authors introduce SPOKEN-MATH-PROBLEMS-3M, a 3M-sample dataset constructed from public text-based maths QA corpora (GSM8K, Orca-Math, MetaMath, SimpleOP). confidence: high relevance: high limitations: - The entire evaluation uses a single benchmark (Spoken-MQA) focused on mathematics. There is no assessment of speech naturalness, intelligibility, or reasoning accuracy on open-domain conversational tasks. Reported latency claims refer to the absence of a pre-speech reasoning phase rather than to measured real-time performance metrics. - The fixed 2:8 interleaving ratio is derived from a throughput estimate for a specific GPU configuration; it is not adaptive and may be suboptimal for different deployment environments or model sizes. The training data is entirely synthetic — both the audio (produced by CosyVoice2-0.5B) and the reasoning traces (constructed algorithmically from text datasets). Whether the model generalises to naturalistic spoken queries beyond maths problems is untested. The GPT-based verification stage for dataset quality introduces a dependency on a proprietary closed model that is not reproducible. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.16188' published_date: "2025-08-22" entry_date: '2026-07-27' year: 2025 venue: EMNLP task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: avlm_visual_complementary_emotion role: supports claim: Full-face visual features encode complementary emotional information not captured by audio alone, and combining both modalities yields meaningfully higher emotion recognition accuracy than either modality individually. source: §3, Table 1 evidence: AVLM improves IEMOCAP four-class emotion recognition F1 from 61.3% (speech-only) to 66.2% (audio-visual), a +4.9 point gain, using SMIRK's disentangled facial expression parameters as the visual feature source. confidence: medium relevance: low - claim_id: avlm_qformer_prefix_fusion role: supports claim: Prefix-based visual fusion using compressed Q-Former query latents integrates more effectively into autoregressive speech LMs than direct feature concatenation, which collapses perplexity. source: §5.1, Table 3 evidence: Among three fusion strategies compared during pre-training on LRS3, Q-Former Prefix with 30% speech attention masking achieved the lowest perplexity and was adopted for all downstream fine-tuning. confidence: medium relevance: high - claim_id: avlm_visual_guidance_generation_alignment role: supports claim: Visual guidance during training improves not only classification accuracy but also the emotional alignment of generated speech in conversational settings. source: §5.3, Table 7 evidence: For expressive speech generation evaluated by Qwen2-Audio, AVLM achieves 42.49% emotion F1 versus 38.39% for the speech-only baseline, a +4.1 point gain. confidence: medium relevance: high - claim_id: avlm_prompt_only_controllability_insufficient role: complicates claim: Emotion controllability through prompt-only label manipulation is insufficient when a model is trained on data where input and response emotions are correlated; in-context demonstrations are necessary to override this bias. source: §5.3, Figure 5 evidence: Zero-shot prompt modification of the emotion label largely fails to change the emotional tone of generated speech, while in-context learning demonstrations per emotion class considerably improve responsiveness, though the model remains heavily anchored to input audio-visual context. confidence: medium relevance: low limitations: - Relies on a third-party model (Qwen2-Audio) to classify emotion in synthesised speech with no human perceptual evaluation; the fine-tuning dataset is small (4,859 synthetic pairs) and the system is evaluated on a closed four-class emotion taxonomy. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.16790' published_date: "2025-08-22" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec - TTS architecture: - diffusion - transformer-enc-dec - autoregressive-LM relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models - transformer_encoder_decoder_speech_models claims: - claim_id: text_conditioning_in_the_codec_decoder_rather_than role: supports claim: Text conditioning in the codec decoder, rather than in the language model alone, is a viable lever for achieving extreme compression rates in speech tokenization without adversarial training. source: §3.1, Table 4 evidence: In speech LM contexts — TTS, spoken dialogue — the target text is always available, so conditioning on it effectively offloads content information from the compressed token to the conditioning signal, enabling much more aggressive compression. confidence: high relevance: high - claim_id: a_single_end_to_end_training_objective_flow role: supports claim: A single end-to-end training objective (flow-matching loss) is sufficient to jointly optimise quantization and reconstruction in a speech codec, eliminating the need for multi-stage pipelines. source: §3.1, §4.2.2 evidence: TaDiCodec reframes the codec problem as an end-to-end diffusion autoencoder trained with a single flow-matching loss. confidence: high relevance: low - claim_id: the_reconstruction_generation_gap_the_degradation_in_intelligibility role: supports claim: The reconstruction-generation gap — the degradation in intelligibility when tokens are predicted by a language model rather than encoding reference speech — varies substantially across tokenizer architectures and is not captured by reconstruction metrics alone. source: §4.3, Figure 3 evidence: The low token rate (6.25 Hz vs. 25–50 Hz for typical AR TTS) means each second of speech is predicted as only 6–7 tokens, drastically shortening the generation sequences. confidence: high relevance: high - claim_id: lower_token_rates_in_speech_tokenizers_can_improve role: supports claim: Lower token rates in speech tokenizers can improve autoregressive TTS intelligibility by shortening prediction sequences and reducing error accumulation, particularly on linguistically challenging inputs. source: §4.3, Table 5 evidence: The low token rate (6.25 Hz vs. 25–50 Hz for typical AR TTS) means each second of speech is predicted as only 6–7 tokens, drastically shortening the generation sequences. confidence: high relevance: high - claim_id: binary_spherical_quantization_without_a_commitment_loss_achieves role: supports claim: Binary Spherical Quantization without a commitment loss achieves stable end-to-end training of a speech codec and produces superior representations to standard VQ under equal codebook sizes. source: §4.2.2, Table 4 evidence: The encoder is an 8-layer Llama-style bidirectional Transformer operating on mel-spectrogram frames; its output is downsampled and projected to a 14-dimensional latent, then quantized using Binary Spherical Quantization (BSQ) — an implicit-codebook scheme that projects encoder embeddings onto a unit hypersphere and binarises each dimension, yielding a codebook of size 2¹⁴ = 16,384 tokens without a learnable codebook or commitment loss. confidence: high relevance: low limitations: - 'TaDiCodec''s text-aware decoder is not a general-purpose audio codec: it requires a transcript at both training and inference time. The strong reconstruction and TTS results are conditional on text availability; performance at 6.25 Hz without text conditioning is not competitive (WER exceeds 10% at 12.5 Hz without text, per Table 4). This limits applicability to codec-transmission, speech enhancement, or any scenario where transcriptions are unavailable.' - The diffusion decoder introduces multi-step inference latency. At 32 steps, decoding speed is acceptable for generation but higher than GAN vocoders; reducing to 5 steps degrades quality noticeably. The authors propose distillation as future work but have not yet demonstrated single-step performance. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.17623' published_date: "2025-08-25" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: sdms_fail_to_sustain_emotional_coherence_across_multiturn_dialogue role: supports claim: Current spoken dialogue systems reliably fail to sustain emotionally coherent responses across multi-turn conversations, with degradation compounding across turns. source: §V, Table V evidence: Multi-turn evaluation shows Moshi's perceptual ERS drops from 0.284 to 0.271 and dGSLM's from 0.296 to 0.291, with categorical averages falling to 0.470 and 0.461 respectively, well below the human baseline of 0.925. confidence: high relevance: high - claim_id: automatic_emotion_metrics_overestimate_coherence_on_atypical_systems role: complicates claim: Automatic continuous and categorical emotion metrics correlate moderately with human perceptual judgements on most SDMs, but overestimate emotional coherence on systems with atypical output characteristics. source: §V, Figure 3, Figure 4 evidence: Moshi and dGSLM are overestimated by automatic continuous/categorical metrics relative to human ratings, showing current automatic emotion metrics do not fully capture human perception of extended emotional coherence. confidence: high relevance: low - claim_id: categorical_and_continuous_emotion_objectives_traded_off role: supports claim: Optimising for categorical emotion appropriateness and optimising for continuous affect trajectory alignment represent distinct objectives that current SDMs trade off against each other rather than jointly satisfying. source: §V, Table IV evidence: 'Freeze-Omni leads categorical-based performance (Avg: 0.738) while GLM-4-Voice leads continuous-based scores (ERS: 0.803) in single-turn settings, with the two metric types not co-ranking systems.' confidence: medium relevance: low - claim_id: synthetic_emotional_speech_sufficient_as_evaluation_stimulus role: supports claim: Synthetic emotional speech generated by controllable TTS systems provides sufficient variety and quality to serve as evaluation stimulus material when real annotated data is scarce. source: §IV-A, Table III evidence: The benchmark's synthetic dataset (500 utterances each of neutral/happy/angry/sad plus 1,251 description prompts, ~7.4 hours) is generated via GPT-4 scripting rendered through CosyVoice, and is used alongside real DailyTalk speech (21.7 hours) as the human upper-bound reference. confidence: medium relevance: low limitations: - Human perceptual evaluation covers only 20 randomly sampled dialogues per system evaluated by 20 annotators; at this sample size, inter-rater agreement statistics are not reported, limiting confidence in the perceptual ERS rankings. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.19205' published_date: "2025-08-26" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: extreme_acoustic_codec_compression_single_codebook_vae_at role: supports claim: Extreme acoustic codec compression (single-codebook VAE at 7.5 Hz) can achieve superior perceptual quality over multi-codebook discrete codecs operating at much higher frame rates. source: §3.3, Table 3 evidence: The acoustic tokenizer comparison (Table 3) shows that at 7.5 Hz — 40× fewer tokens than WavTokenizer at 40 Hz — the proposed tokenizer achieves PESQ 3.068 and UTMOS 4.181 on LibriTTS test-clean, beating all discrete codec baselines including DAC (UTMOS 3.433) and WavTokenizer 75 Hz (UTMOS 4.049). confidence: high relevance: low - claim_id: long_form_multi_speaker_tts_benefits_from_separate role: supports claim: Long-form multi-speaker TTS benefits from separate acoustic and semantic tokenizers trained with task-specific objectives rather than a single unified codec. source: §2.1 evidence: A separate semantic tokenizer mirrors the encoder architecture but is trained via an ASR proxy task, producing content-aligned representations; this component is discarded after pre-training and used only to provide hybrid acoustic-semantic context tokens during generation. confidence: high relevance: high - claim_id: scaling_the_llm_backbone_in_a_next_token role: supports claim: Scaling the LLM backbone in a next-token diffusion speech system yields consistent gains in perceptual quality, speaker similarity, and expressiveness. source: §3.1, Table 1 evidence: 'VibeVoice (paper name: VIBEVOICE) resolves the context-length bottleneck through two innovations that work in tandem: an ultra-compressed acoustic tokenizer and a next-token diffusion generation framework.' confidence: high relevance: low - claim_id: token_level_diffusion_conditioned_on_llm_hidden_states role: supports claim: Token-level diffusion conditioned on LLM hidden states enables streaming speech generation without the codebook constraints of discrete autoregressive systems. source: §2.2 evidence: The LLM processes this hybrid context and produces hidden states that condition a lightweight token-level diffusion head (4 transformer layers) adapted from the MAR image generation framework. confidence: high relevance: high - claim_id: tts_systems_optimised_for_long_form_conversational_content role: supports claim: TTS systems optimised for long-form conversational content retain competitive performance on short-utterance benchmarks without dedicated fine-tuning. source: §3.2, Table 2 evidence: On the SEED short-utterance benchmark (CommonVoice English/Chinese), VibeVoice-1.5B achieves CER of 1.16% and SIM of 0.744 for Chinese, and WER of 3.04% and SIM of 0.689 for English, despite being primarily trained for long-form content. confidence: high relevance: high limitations: - Training data is not disclosed. The paper is from Microsoft Research but does not specify the data composition, size, or any cleaning procedures, making it impossible to assess whether the reported gains are attributable to architecture or data advantage. - The model is limited to English and Chinese; other languages produce unpredictable outputs. The system does not model overlapping speech — a significant gap for realistic conversational audio. The subjective evaluation used only 8 long-form test conversations, which is a narrow sample; standard benchmark evaluations (SEED) are short-utterance only and do not capture the long-form quality the paper targets. Speaker similarity at 7.5 Hz remains below the best short-utterance systems (e.g., Seed-TTS at 0.762 SIM for English), suggesting the compressed representation sacrifices some speaker identity fidelity. The maximum of 4 speakers is a hard constraint imposed by the context design. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.20660' published_date: "2025-08-28" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec - evaluation architecture: [] relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: acoustic_semantic_retention_orthogonal_in_codec_design role: supports claim: Acoustic reconstruction quality and semantic information retention are largely orthogonal objectives in neural audio codec design, and optimising for one does not reliably improve the other. source: §4.3, Figure 2 evidence: DAC and BigCodec rank highest on acoustic metrics (PESQ, STOI) but show substantially worse WER in the ASR probing task than Mimi, X-Codec-2.0, and the Baichuan-Audio tokenizer, whose training objectives incorporate semantic information. confidence: high relevance: high - claim_id: codecs_without_semantic_objectives_have_higher_wer role: supports claim: Codecs trained without explicit semantic objectives achieve state-of-the-art signal reconstruction but exhibit substantially higher word error rates than semantically-trained codecs of comparable bitrate. source: §4.3.1 evidence: DAC-24k and BigCodec's acoustic-first training objectives yield state-of-the-art PESQ/STOI but higher ASR-probing WER than semantically-trained codecs of comparable bitrate. confidence: high relevance: low - claim_id: single_codebook_codecs_sacrifice_paralinguistic_content role: supports claim: Single-codebook, low-bitrate codecs that preserve textual semantic content tend to sacrifice paralinguistic information such as speaker emotion and music characteristics. source: §4.3.2, Table 5 evidence: X-Codec-2.0's single-codebook, low-bitrate design yields strong WER but poor classification accuracy across music, emotion, and sound datasets in Table 5. confidence: high relevance: low - claim_id: clean_speech_codec_benchmarks_dont_generalise role: complicates claim: Codec performance on clean speech benchmarks does not generalise to noisier, more expressive, or musically rich audio domains, with performance gaps widening at low bitrates. source: §4.2, Table 4 evidence: Performance gaps between codecs widen on the self-collected noisy/expressive dataset and non-speech domains relative to clean LibriSpeech-based evaluation, with all audio downsampled to 16 kHz for cross-model comparison. confidence: medium relevance: low - claim_id: flow_matching_codecs_competitive_on_nonspeech_audio role: supports claim: Flow-matching-based audio codecs achieve competitive perceptual quality on non-speech audio relative to traditional RVQ-GAN-based codecs at equivalent bitrate and codebook size. source: §4.2 evidence: FlowDec slightly outperforms DAC-24k-8nq on Music datasets despite identical codebook and bitrate configuration. confidence: medium relevance: low limitations: - The benchmark's semantic evaluation relies on two embedding-based proxy tasks (ASR probing via a benchmark-specific frozen LSTM, and linear-probe classification) that do not directly measure how codec representations affect downstream generation quality or dialogue coherence in a deployed SLM; WER values are not comparable to codec-vendor-reported ASR metrics. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.02020' published_date: "2025-09-02" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: reducing_speech_tokenizer_frame_rate_to_12_5hz role: supports claim: Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection. source: §2.1, §4.1, Table 1 evidence: Crucially, the quantised features are passed through a semantic decoder that predicts the original Whisper features (explicit semantic supervision), stabilising text-to-token modelling over long sequences. confidence: high relevance: high - claim_id: a_dual_transformer_architecture_for_multi_layer_rvq role: supports claim: A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns. source: §2.2 evidence: This dual-transformer avoids the delay-pattern's drawbacks (weak contextual conditioning, high first-packet latency) and achieves under 100ms first-packet latency. confidence: high relevance: low - claim_id: autoregressive_tts_systems_trained_on_multi_speaker_dialogue role: supports claim: Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels. source: §3.2, §4.3, Table 3 evidence: 'Emotion inference (interactive chat): 76.7–93.3% accuracy across 6 emotions (surprise, sadness, happiness, concern, apology, anger) inferred from implicit contextual cues without explicit emotion labels.' confidence: high relevance: high - claim_id: sentence_by_sentence_multi_speaker_dialogue_tts_systems role: supports claim: Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track. source: §4.4, Table 4 evidence: 'Three-stage curriculum training: pretraining on 1.1M hours monologue, post-training on 300k hours 2–5 speaker dialogue data, then task-specific SFT (eg. 15h emotion corpus for chat, 50h podcast hosts for podcast fine-tuning).' confidence: high relevance: low - claim_id: fine_tuning_a_post_trained_dialogue_tts_model role: supports claim: Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials. source: §4.4, Figure 4 evidence: 'Three-stage curriculum training: pretraining on 1.1M hours monologue, post-training on 300k hours 2–5 speaker dialogue data, then task-specific SFT (eg. 15h emotion corpus for chat, 50h podcast hosts for podcast fine-tuning).' confidence: high relevance: low limitations: - '- Currently limited to 3-minute dialogues with up to 4 speakers; scaling requires extending training corpus. - English speaker similarity (SIM 0.665) lags Mandarin (0.736), attributed to limited English voice diversity in training data — a data rather than architectural limitation. - Trails Mimi on PESQ metrics, likely because Mimi was trained on a massive English-only corpus closely matching LibriSpeech. - Emotion fine-tuning is demonstrated for a single distinctive female voice; generalisation to arbitrary voices and more nuanced emotional transitions is not evaluated. - No ablation of the semantic supervision contribution vs. the lower frame rate independently.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.05359' published_date: "2025-09-03" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: medium evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: vocabulary_size_dominant_design_variable_speech_lm_pretraining role: supports claim: Discrete vocabulary size is the dominant design variable in speech LM continual pre-training, with compact cluster sets substantially outperforming fine-grained discretizations regardless of encoder choice. source: §3.1, Table 1 evidence: WavLM NLL degrades from 2.05 (k=500) to 4.01 (k=2500) and 4.21 (k=5000) for SmolLM-135M at step 300; the same degradation pattern holds for HuBERT, XLS-R, and Wav2Vec 2.0 across all model sizes. confidence: high relevance: high - claim_id: optimal_vocabulary_granularity_varies_by_model_capacity role: complicates claim: 'Optimal discrete vocabulary granularity for speech LM pre-training is not fixed across model capacities: larger models tolerate higher cluster counts where smaller models fail.' source: §3.2, Table 2 evidence: SmolLM-135M degrades sharply at k=1,000 (NLL=2.19) versus k=500 (NLL=2.05), while SmolLM-1.7B maintains stable NLL of 1.83 to 1.94 across k=125 to k=1,000, suggesting capacity-dependent discretization regimes. confidence: high relevance: high - claim_id: domain_matching_critical_for_discrete_unit_quality_and_robustness role: supports claim: Domain matching between the clustering training corpus and the target speech domain is a critical factor for both discrete unit quality and robustness to acoustic perturbations. source: §3.3, Table 3 evidence: LibriHeavy-trained k-means (read English, in-domain with LibriSpeech) achieves NLL=2.62 clean and 2.70 under heavy noise; GigaSpeech-trained k-means degrades from 3.07 to 3.09 and CommonVoice from 2.85 to 3.11, showing that exposure to noisier audio during k-means construction does not improve robustness. confidence: high relevance: low - claim_id: ssl_discrete_units_encode_phoneme_distinctions_unsupervised role: supports claim: Self-supervised discrete speech units encode phoneme-level distinctions without explicit phonetic supervision, as evidenced by their systematic alignment with forced-alignment phoneme sequences. source: §3.5, Figure 2 evidence: Phoneme confusion matrices on LibriSpeech test-clean (WavLM, k=125) show a strong diagonal structure where vowels and consonants each specialize into distinct unit subsets; the pattern persists across all four k-means data sources and holds at k=250. confidence: high relevance: low limitations: - Evaluation uses only NLL on LibriSpeech, a proxy metric; no downstream task evaluation (spoken QA, spoken language understanding, ASR) is included, so the practical implications of the observed NLL differences for real applications remain untested. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.03940' published_date: "2025-09-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: paralinguistic_appropriateness_weakest_capability_in_spoken_dialogue_models role: supports claim: Acoustic quality (paralinguistic appropriateness) is the most consistently underperformed capability across current spoken dialogue models, even for proprietary systems, lagging well behind text generation and reasoning dimensions. source: '§Experiments: LLM-based Result, Table 3' evidence: 'Across the LLM-based dimensions, Paralinguistic Appropriateness is universally the weakest dimension (GPT-4o: 3.82, best open-source Step-Audio: 3.43), well below Contextual Coherence (GPT-4o: 4.48).' confidence: medium relevance: high - claim_id: parameter_count_does_not_predict_speech_synthesis_quality_dialogue role: supports claim: 'Model parameter count does not reliably predict speech synthesis quality in spoken dialogue systems: smaller, better-trained models can match or exceed much larger counterparts on naturalness metrics.' source: '§Experiments: Metric-based Results, Table 2' evidence: Step-Audio at 132B achieves the highest BertScore F1 (84.16) but scores poorly on speech naturalness (UTMOS 2.42), while Qwen2.5-Omni at 7B achieves UTMOS 3.57, nearly matching GPT-4o's 3.66. confidence: medium relevance: high - claim_id: acoustically_aware_llm_judge_aligns_with_human_judgment_roleplay role: supports claim: LLM-based evaluation augmented with discrete acoustic features achieves strong alignment with human judgment on multi-dimensional spoken role-playing tasks. source: '§Evaluation Framework, §Experiments: Subjective experimental results' evidence: Human evaluation on 20 sampled instances yields a Pearson correlation of 0.762 between the acoustically-aware LLM judge scores (which augment transcribed text with pitch, energy, speaking-rate bins, and Emotion2Vec labels) and human annotator scores. confidence: medium relevance: low - claim_id: optimal_context_window_length_exists_for_spoken_roleplay role: supports claim: There is an optimal context window length for spoken role-playing models beyond which additional conversational history degrades rather than improves performance. source: '§Experiments: Ablation study on context length, Table 4' evidence: An ablation over context window length (4-10 turns) shows a non-monotonic trend, with a 6-turn context achieving the best overall score. confidence: medium relevance: high limitations: - Human evaluation for judge validation covers only 20 dialogue instances, and character-quality validation covers only 20 characters rated by 5 annotators; the benchmark is sourced exclusively from movie dialogue, which is scripted and stylistically distinct from spontaneous conversation. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.04093' published_date: "2025-09-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM - flow-matching relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - autoregressive_speech_language_models - flow_matching_speech_decoders claims: - claim_id: fullduplex_conversational_data_improves_tts_prosodic_naturalness role: supports claim: Incorporating spontaneous full-duplex conversational data into TTS training improves the prosodic naturalness of synthesized speech, as measured by both acoustic distance metrics and human preference. source: §4.1, §4.2, Table 2 evidence: Fine-tuning CosyVoice-300M on 15 hours of dual-track conversational speech reduced F0 Wasserstein distance by 7.08% (Chinese) and 3.67% (English), and the fine-tuned model was preferred by a majority in A/B preference tests in both languages (45.0% vs. 33.9% Chinese; 46.4% vs. 40.7% English). confidence: medium relevance: high - claim_id: existing_corpora_lack_overlap_interruption_annotations role: supports claim: Existing TTS and dialogue corpora lack the annotated overlap and interruption dynamics needed for full-duplex conversational modelling. source: §1 evidence: Turn-based datasets like Switchboard constrain to sequential exchanges with limited overlap, and DailyTalk rarely includes spontaneous overlapping speech or explicit interruption annotations, leaving a data gap for full-duplex conversational TTS. confidence: medium relevance: high - claim_id: small_corpus_finetuning_gains_inconsistent_across_acoustic_dims role: complicates claim: Fine-tuning TTS models on small conversational corpora yields perceptible naturalness gains but leaves spectral fidelity inconsistent across languages and acoustic feature dimensions. source: §4.1, §4.2, Table 2, Table 3 evidence: MOS naturalness improved by only 0.03-0.04 points in both Chinese and English, and English spectrum l2 distance worsened by 0.85% despite improvements in F0 (3.67%) and ZCR (7.44%), suggesting that small-scale fine-tuning does not uniformly improve all acoustic dimensions. confidence: medium relevance: high limitations: - The datasets are small (10h Chinese / 5h English, 14 unique speakers total), and the validation experiment tests only fine-tuning an existing TTS model rather than end-to-end full-duplex spoken dialogue system improvement. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.04685' published_date: "2025-09-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - adversarial_codec_speech_models claims: - claim_id: content_adaptive_token_allocation_in_acoustic_tokenisers_achieves role: supports claim: Content-adaptive token allocation in acoustic tokenisers achieves better reconstruction quality than fixed-rate designs at equal or lower token budgets. source: §4.2, Table 1 evidence: VARSTok at 30.95 Hz achieves UTMOS 3.8949 on LibriTTS test-clean, surpassing the 40 Hz WavTokenizer (3.6107) while using 23% fewer tokens; the 36.81 Hz configuration (UTMOS 4.000) nearly matches the 75 Hz WavTokenizer (4.025) with fewer than half the tokens. confidence: high relevance: low - claim_id: dynamically_segmented_speech_tokens_carry_more_semantically_discriminative role: supports claim: Dynamically segmented speech tokens carry more semantically discriminative information than uniformly sampled tokens at the same average rate. source: §4.3, Table 2 evidence: All VARSTok configurations outperform the 40 Hz WavTokenizer on all four ARCH benchmark classification tasks (emotion, digit recognition, intent), despite operating at lower average frame rates. confidence: high relevance: high - claim_id: encoding_token_duration_implicitly_in_the_vq_codebook role: supports claim: Encoding token duration implicitly in the VQ codebook index eliminates the need for auxiliary duration predictors and preserves compatibility with autoregressive speech language models. source: §3.4, §4.4, Table 3 evidence: The implicit duration coding scheme maps each cluster's content index k and duration d to a single token ID D = (d-1)*K + k, enabling a standard cross-entropy autoregressive model to generate variable-rate token sequences without modification; MOS and WER improve over the fixed-rate baseline in zero-shot TTS. confidence: high relevance: high - claim_id: more_aggressive_temporal_compression_in_variable_rate_tokenisers role: complicates claim: More aggressive temporal compression in variable-rate tokenisers trades reconstruction quality for token efficiency beyond a practical compression threshold. source: §4.2, Table 1 evidence: Increasing S_max from 2 to 8 reduces the average frame rate from 46.5 Hz to 22.38 Hz but degrades UTMOS from 4.038 to 3.647 and PESQ from 2.069 to 1.453 on LibriTTS test-clean; the optimal configuration (tau=0.7, S_max=4 at 30.95 Hz) sits at the knee of this trade-off curve. confidence: high relevance: low - claim_id: inference_speed_in_autoregressive_speech_lms_depends_primarily role: refines claim: Inference speed in autoregressive speech LMs depends primarily on sequence length rather than vocabulary size, so variable-rate tokenisers with expanded vocabularies still accelerate decoding. source: §J, Table 5 evidence: VARSTok (tau=0.6) achieves RTF 0.487 versus 0.766 for the 40 Hz WavTokenizer baseline (36% speedup) despite expanding the token vocabulary from K to K*S_max = 16,384 entries, because shorter sequences reduce the dominant cost of attention computation over more function evaluations. confidence: high relevance: high limitations: - Evaluation is restricted to English (LibriTTS). The clustering algorithm relies on cosine similarity in a WavTokenizer embedding space trained on English read speech; whether the density-peak boundaries remain meaningful for other languages, spontaneous speech, or emotionally expressive styles is untested. - Speaker similarity under more aggressive compression (tau=0.6, 26.29 Hz) does show a statistically modest decline in objective SIM (0.880 vs 0.918 for the baseline), and while subjective SMOS remains comparable, the long-tail impact on voices far from the training distribution is unknown. Codebook collapse becomes severe for K above 4096 in the expanded index space, suggesting that very large vocabulary configurations require dedicated regularisation strategies not addressed here. The clustering algorithm is not differentiable, so joint end-to-end training with a downstream TTS model is not straightforward. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.04702' published_date: "2025-09-04" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: [] relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: olespeech_conversational_corpus_alignment role: supports claim: Specialized alignment pipelines can turn public conversational broadcasts into large speech corpora with speaker diarization and word-level timing suitable for spoken-language-model training. source: §2.4, §3 evidence: OleSpeech-IV reports more than 5,000 hours with human-sourced speaker labels and Olign-generated word timestamps and confidence scores. confidence: medium relevance: high - claim_id: olespeech_e2e_alignment_errors role: complicates claim: General-purpose end-to-end speech aligners can introduce boundary errors and overconfident scores that weaken automatically annotated conversational training data. source: §3.1.1.2, §3.1.2 evidence: The reported Whisper examples show boundary smearing and high confidence for misrecognized words, whereas Olign assigns zero duration to hallucinated insertions. confidence: medium relevance: high - claim_id: olespeech_overlap_alignment_unvalidated role: complicates claim: Overlapping multi-party speech remains an incompletely validated challenge for conversational-corpus alignment. source: §3.1.1.3 evidence: Olign exposes per-speaker timings for overlapping speech, but the paper reports no quantitative accuracy benchmark and states evaluation is ongoing. confidence: high relevance: high - claim_id: olespeech_calibrated_confidence_review role: supports claim: Calibrated word-level confidence scores can focus human review on ambiguous conversational segments instead of requiring complete re-annotation. source: §3.1.2 evidence: Olign flags mid-confidence words for targeted inspection through per-word and sentence-level confidence outputs. confidence: medium relevance: high limitations: - The full dataset and Olign aligner are proprietary, while the open subset contains only 100 hours. - No quantitative alignment benchmark or downstream spoken-language-model evaluation is reported. - Full alignment support is currently limited to English. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.06502' published_date: "2025-09-08" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: speaker_conditioned_vad_reduces_false_barge_in role: supports claim: Speaker-conditioned VAD substantially reduces false barge-in rates in full-duplex spoken conversational agents compared to standard VAD, at the cost of slightly higher target-speaker response latency. source: §3.1, Table 2 evidence: pVAD achieves a 10.2% false barge-in rate versus 33.4% (LiveKit SileroVAD) and 78.1% (Ten VAD), with T90 of 170 ms versus 140 ms and 90 ms respectively. confidence: high relevance: high - claim_id: compact_classifiers_match_large_models_for_eot_detection role: supports claim: Compact task-specific classifiers for semantic end-of-turn detection can match parameter-heavy general-purpose language models while imposing negligible latency overhead. source: §3.2, Table 3 evidence: A 170M-parameter BERT-based EoT model achieves 96.0%/94.9% average accuracy on Chinese/English, on par with a 7B-parameter model (95.8%/94.4%), while using far fewer parameters and thus adding minimal latency to the pipeline. confidence: high relevance: high - claim_id: semi_cascaded_architectures_offer_paralinguistic_awareness role: supports claim: Semi-cascaded architectures that replace the ASR+LLM cascade with a single AudioLLM offer paralinguistic awareness and reduced error propagation while maintaining competitive end-to-end latency. source: §2.3.2 evidence: The semi-cascaded variant conditioned on user speech emotion achieves a simpler pipeline with potential latency benefits, mitigates ASR error propagation, and enables paralinguistic-consistent TTS synthesis, without sacrificing the modularity needed for independent component optimisation. confidence: medium relevance: low - claim_id: e2e_s2s_models_face_training_challenges_for_fullduplex role: complicates claim: End-to-end speech-to-speech models face training and architectural challenges that limit their use as a foundation for full-duplex interaction systems. source: §1 evidence: The authors identify catastrophic forgetting in LLM backbones and token length mismatch between text and speech sequences as key drawbacks of end-to-end approaches, motivating the pluggable modular controller design. confidence: medium relevance: high - claim_id: fullduplex_evaluation_requires_system_level_metrics role: supports claim: System-level evaluation of full-duplex voice interaction requires metrics beyond component-level quality scores, covering barge-in accuracy, semantic end-of-turn detection, and end-to-end latency jointly. source: §3 evidence: FireRedChat introduces three interaction-level metrics (barge-in T90, false barge-in rate, and end-to-first-response P50/P95) that capture naturalness, control accuracy, and efficiency independently of individual ASR, LLM, or TTS scores. confidence: high relevance: high limitations: - All evaluation uses proprietary test data (1,000 utterances per language for barge-in, 25 synthetic utterances for latency) constructed by the authors; external reproducibility is not possible, and the semi-cascaded pipeline's paralinguistic advantages are described only qualitatively, with no objective metric (MOS, emotion accuracy) quantifying the improvement. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.09716' published_date: "2025-09-09" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: lalm_judges_approximate_human_style_evaluation role: supports claim: Large audio-language model judges can approximate human evaluation of speaking style adherence with near-human-level reliability. source: §3.3, Table 2 evidence: LALM-as-a-Judge (Gemini-2.5-pro) achieves Spearman correlations of 77.01% (English) and 73.03% (Chinese) with consensus human scores, comparable to inter-human agreement of 78.58% and 70.54%. confidence: high relevance: high - claim_id: opensource_slms_deficit_in_expressive_style_vs_commercial role: complicates claim: Current open-source spoken language models exhibit substantial deficits in controllable expressive speech generation compared to commercial systems. source: §3.2, Table 1 evidence: Open-source models (Step-Audio, Kimi-Audio, Qwen2.5-Omni, Baichuan-Audio) score 2.46-3.11 overall on VStyle, while commercial systems (GPT-4o, Doubao) reach 3.74-4.10, with the gap attributed to weak acoustic feature modeling and lower instruction-following robustness. confidence: high relevance: high - claim_id: style_control_uneven_across_languages role: complicates claim: Speaking style control does not transfer evenly across languages even within the same system, indicating language-specific rather than language-general style representations in current models. source: §3.2 evidence: 'Doubao and Kimi-Audio score substantially higher in Chinese than English (e.g., Doubao: 4.10 vs. 3.88), while the GPT-4o series shows the reverse, attributed to imbalanced training data distributions and differences in pronunciation habits.' confidence: medium relevance: low - claim_id: staged_hierarchical_evaluation_captures_finer_distinctions role: supports claim: Staged hierarchical evaluation frameworks capture finer distinctions in speech generation quality than flat or single-dimension scoring approaches. source: §2.3, Figure 2, §3.3 evidence: VStyle's LALM-as-a-Judge separates content correctness, style adherence, and naturalness into sequential assessment gates, enabling discrimination between systems that achieve content but fail on style versus those achieving both; this structure is validated against human evaluation at near-human-level correlation. confidence: high relevance: high limitations: - The instruction dataset is built from manually designed seeds expanded by LLMs, meaning its distribution reflects annotator preferences rather than natural user interaction needs; audio instructions are synthesized via commercial voice cloning, narrowing the style range in input prompts relative to naturally produced speech. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.09174' published_date: "2025-09-11" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: decoupling_semantic_training_objectives_from_acoustic_token_prediction role: supports claim: Decoupling semantic training objectives from acoustic token prediction substantially reduces knowledge degradation in speech-to-speech LLMs. source: §5.1, Table 4 evidence: EchoX's Echo training, which generates speech targets from the model's own semantic hidden states, raises average QA accuracy from 24.3 (T2C without Echo) to 37.1 (EchoX-3B) on the same data, compared to 12.8 for direct interleaved training. confidence: high relevance: high - claim_id: unit_based_speech_token_compression_via_language_model role: supports claim: Unit-based speech token compression via language-model segmentation improves downstream accuracy and reduces sequence length without sacrificing audio quality. source: §5.3, Table 5, Figure 7 evidence: Unit language achieves 4.57 length ratio vs. 9.31 for raw units while improving accuracy on all three QA benchmarks and maintaining comparable audio quality in spectral comparison. confidence: high relevance: high - claim_id: streaming_inference_in_speech_llms_can_be_achieved role: supports claim: Streaming inference in speech LLMs can be achieved with minimal accuracy degradation when the segmentation boundary is determined by semantic similarity rather than fixed length. source: §5.4, Table 6 evidence: EchoX's cosine-similarity trigger reduces first-token latency from 138 to 27 tokens at 3B scale with less than 1.5 percentage points of accuracy drop on any benchmark. confidence: high relevance: low - claim_id: training_data_efficiency_in_speech_llms_may_depend role: complicates claim: Training data efficiency in speech LLMs may depend more on the training paradigm than on data volume. source: §4.2, Table 2 evidence: EchoX achieves competitive performance on spoken QA against models trained on millions of hours using only approximately 6,200 hours, but this result holds specifically for factual QA and has not been tested on broader spoken dialogue tasks. confidence: high relevance: high - claim_id: speech_naturalness_and_response_helpfulness_are_not_jointly role: complicates claim: Speech naturalness and response helpfulness are not jointly optimised by the same training signal in speech-to-speech LLMs. source: §Appendix C, Figure 8 evidence: Human evaluation shows EchoX wins clearly on helpfulness but performs only competitively on naturalness, reflecting a training objective focused on semantic correctness rather than prosodic quality. confidence: high relevance: low limitations: - Speech quality is assessed only via brief spectral comparison (Figure 7) and a 5-rater human study. No perceptual quality metric (MOS, DNSMOS) or automatic speech recognition accuracy on generated audio is reported as a primary evaluation result, making it difficult to characterise the system's output quality independently of QA accuracy. - The evaluation benchmarks are limited to factual knowledge QA (Llama Questions, Web Questions, TriviaQA). It is unclear whether Echo training retains its advantage on open-ended dialogue, instruction following, or longer-form conversational tasks. The human evaluation was conducted with only five raters on one dataset (AlpacaEval), which limits statistical confidence in the naturalness comparison. The model trains on synthesised assistant audio from GPT-SoVITS, which may introduce a fixed timbre bias and limit voice diversity. The streaming threshold and window size are fixed hyperparameters with no ablation reported on their sensitivity. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2508.18240' published_date: "2025-09-15" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: semantic_nonverbal_capability_gap_in_s2s_llms role: supports claim: Current speech-to-speech language models exhibit a systematic capability gap between semantic understanding and non-verbal speech dimensions such as paralinguistic and ambient sound processing. source: §6.3.1, Table 2 evidence: On MTalk-Bench, all evaluated models outperform the human baseline on semantic rubric scores but underperform on paralinguistic and ambient sound dimensions, with no model exceeding 80/100 overall. confidence: high relevance: high - claim_id: multiturn_dialogue_verbosity_compensates_for_context_bottleneck role: complicates claim: Multi-turn spoken dialogue systems compensate for context-accumulation bottlenecks by increasing response verbosity rather than informational density. source: §6.3.2, Table 4 evidence: Turn-level rubric scores on MTalk-Bench show a non-linear dip from Turn 1 to Turn 2 with partial recovery at Turn 3, while content density declines linearly across turns even as token count increases. confidence: medium relevance: high - claim_id: modality_aware_design_outperforms_scale_in_multiturn_dialogue role: supports claim: Task-specific modality-aware design provides greater performance gains in multi-turn spoken dialogue than parameter scaling alone. source: §6.3.3, Table 2 evidence: Step-Audio-Chat (130B, with explicit text transcription of prior turns) achieves the highest overall Arena Elo (1049) among open-source models, while parameter count shows no reliable correlation with rubric or arena scores within the evaluated set. confidence: medium relevance: high - claim_id: llm_judge_unreliable_on_raw_audio_nonverbal_cues role: complicates claim: LLM-as-judge automated evaluation of speech quality is unreliable for non-verbal audio cues when models are given raw waveforms. source: §7.2.2 evidence: LLM evaluators achieve near-zero Spearman correlation with human evaluators on ambient sound and paralinguistic dimensions under raw-audio evaluation, while the same dimensions yield Spearman rho above 0.85 when non-verbal cues are provided as text annotations. confidence: high relevance: low - claim_id: arena_rubrics_agreement_requires_large_performance_gaps role: refines claim: Arena-style and rubrics-based evaluation methods yield consistent model rankings for speech systems, but only when performance gaps between models are sufficiently large. source: §7.2.1, §7.2.3, Table 2 evidence: Bootstrap rubric-exclusion analysis shows Spearman rho above 0.95 alignment between Arena and Rubrics rankings, but small ELO and rubric score differences correspond to statistically insignificant win-rate gaps. confidence: high relevance: low limitations: - The benchmark contains only 270 audio samples, and the paper's own analysis shows that Arena and Rubrics evaluations yield reliable distinctions only when performance gaps are large, a material constraint on discriminating between competitive models. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.12171' published_date: "2025-09-15" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: - flow-matching relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - flow_matching_speech_decoders claims: - claim_id: raw_accuracy_masks_demographic_fairness_failures role: supports claim: Raw accuracy metrics are insufficient to surface demographic fairness failures in speech-aware LLMs. source: §7, Table 1 evidence: Models achieve overall fairness scores of 0.43-0.83, but when constrained to tasks they can solve for at least one speaker, constrained fairness drops below 5% across all models, a discrepancy invisible to accuracy-only evaluation. confidence: high relevance: low - claim_id: population_level_robustness_masks_speaker_group_inconsistency role: complicates claim: Speech-aware LLMs that perform robustly at the population level may still exhibit systematic speaker-group inconsistencies. source: §7, Table 1 evidence: Audio Flamingo 3 achieves an overall fairness of 0.83 but a constrained fairness of 0.002, and its cross-modal robustness trails overall fairness by 13 percentage points, indicating modality inconsistency even when all speaker groups receive the same answer. confidence: high relevance: high - claim_id: voice_cloning_scales_diverse_speaker_benchmark_construction role: supports claim: Voice cloning provides a scalable alternative to human recording for constructing diverse-speaker evaluation benchmarks. source: §3, §6 evidence: C3T uses a Voicebox-based conditional flow matching TTS to synthesize 14-29 speaker variants per task instance, covering age, gender, and accent groups from ESLTTS and GLOBE, replacing a recording effort that would otherwise be infeasible at scale. confidence: high relevance: low - claim_id: crossmodal_degradation_reflects_speaker_dependent_response_variation role: refines claim: Cross-modal degradation in speech-aware LLMs reflects not only ASR errors but also speaker-characteristic-dependent response variation that survives correct transcription. source: §1, §5 evidence: The benchmark explicitly tests scenarios where transcription is correct but model output differs across speakers, isolating demographic sensitivity from recognition error as distinct failure modes. confidence: medium relevance: high limitations: - The benchmark is English-only and the speaker pool for age and gender evaluation is limited to three accent regions, and the custom fairness and robustness metrics are not directly comparable to existing benchmark scores, limiting cross-study synthesis. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.14882' published_date: "2025-09-18" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: a_single_transformer_flattened_architecture_can_match_or role: supports claim: A single-Transformer flattened architecture can match or surpass hierarchical speech LM designs on acoustic consistency tasks when both use identical data and comparable parameter budgets. source: §4.3, Table 1 evidence: Llama-Mimi-1.3B outperforms CSM-1.3B on all SALMon acoustic consistency dimensions and speaker similarity (0.346 vs. 0.320) under controlled training conditions. confidence: high relevance: high - claim_id: flattening_rvq_tokens_into_a_single_autoregressive_sequence role: complicates claim: 'Flattening RVQ tokens into a single autoregressive sequence creates an inherent acoustic-linguistic trade-off: strong acoustic performance comes at the cost of weaker linguistic benchmarks relative to SSL-based phonetic-token approaches.' source: §4.3, Table 1 evidence: Llama-Mimi-1.3B achieves best acoustic consistency but underperforms TWIST-1.3B on sWUGGY (68.7 vs. 71.7) and T-Story Cloze (64.0 vs. 69.9), attributed to the Q-fold sequence length increase from RVQ token flattening. confidence: high relevance: low - claim_id: increasing_the_number_of_rvq_quantizers_in_a role: complicates claim: Increasing the number of RVQ quantizers in a flattened speech LM improves audio quality but degrades spoken content coherence, because longer token sequences shift modeling capacity toward acoustic reconstruction. source: §4.4, Table 5 evidence: Ablation with Q∈{2,4,8} shows Q=8 achieves best Audiobox-Aesthetics scores and speaker similarity (0.474) but worst content quality (2.54), while Q=2 yields content quality (3.53) comparable to TWIST-1.3B. confidence: high relevance: high - claim_id: applying_a_higher_loss_weight_to_semantic_tokens role: refines claim: Applying a higher loss weight to semantic tokens in speech LM training shifts the acoustic-linguistic balance toward linguistic accuracy, but causes measurable degradation in acoustic consistency and speaker similarity. source: §4.4, Table 3 evidence: Llama-Mimi-1.3B with semantic weight λ=100 gains on sBLIMP (55.4 vs. 54.3) and T-Story Cloze (68.4 vs. 64.0), but loses substantially on room consistency (74 vs. 92) and speaker similarity (0.196 vs. 0.346). confidence: high relevance: high - claim_id: scaling_model_size_in_flattened_speech_lms_consistently role: supports claim: Scaling model size in flattened speech LMs consistently improves performance across both acoustic and linguistic tasks, with the largest gains in spoken content quality. source: §4.4, Table 4 evidence: Llama-Mimi-8B improves content quality (4.03 vs. 3.01) and T-Story Cloze (67.6 vs. 64.0) over the 1.3B model, with qualitative analysis showing more semantically coherent long-form continuations. confidence: high relevance: high limitations: - Evaluations are conducted exclusively on English speech using LibriSpeech prompts; generalisation to other languages, speakers, or acoustic conditions is untested. No human listening tests are reported; all acoustic and linguistic evaluations rely on automated metrics. - The paper evaluates only speech continuation, not text-conditioned generation or dialogue. Whether the acoustic-linguistic trade-off in flattened designs persists when text tokens are added to the sequence (as in Moshi's inner monologue approach) is left as an open question. The 8B model is evaluated only on a subset of tasks, and the cost of the Q-fold sequence length increase at large scale is not fully characterised. Training Mimi weights end-to-end jointly with the LM is not explored; the frozen codec is an architectural constraint, not a systematic choice. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.16589' published_date: "2025-09-20" entry_date: '2026-07-27' year: 2025 venue: EMNLP task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: content_understanding_does_not_guarantee_direct_paralinguistic_recognition role: supports claim: Speech-LLMs that excel at content understanding do not reliably excel at direct paralinguistic recognition, revealing a systematic capability gap across model families. source: §4.2.1, Table 3 evidence: GPT-4o achieves 69.68 UAS on Content and 67.48 on Contextual Paralinguistic (long set) but only 30.34 on Direct Paralinguistic, while Qwen2-Audio reverses this pattern with 62.29 on Direct Paralinguistic-long but substantially lower Content and Contextual Paralinguistic scores. confidence: high relevance: low - claim_id: cascaded_pipelines_lose_paralinguistic_signal_vs_endtoend role: supports claim: Cascaded ASR-LLM pipelines lose critical paralinguistic signal compared to end-to-end speech-LLMs on tasks requiring direct or contextual paralinguistic reasoning. source: §4.2.1, Table 3 evidence: Cascade models score 33.3-44.18 on Direct Paralinguistic-long versus 60+ for end-to-end speech-LLMs, with the gap narrowing on direct-emotion-type questions where emotion cues are partly recoverable from transcripts. confidence: high relevance: low - claim_id: llm_judge_needs_multirun_averaging_for_paralinguistic_stability role: complicates claim: LLM-as-judge evaluation of paralinguistic reasoning requires multi-run averaging to achieve stable estimates, with single-run scores varying by up to 2.57 points per question type. source: §A.1.2, Table 5 evidence: LLaMA-3-70B judge scores showed standard deviations of 0.44-1.73 (long set) and 0.35-2.57 (short set) across five independent runs, with human-LLM correlation of 0.71 on a 27-pair subset of contextual paralinguistic questions. confidence: high relevance: low - claim_id: temperature_effects_on_speechllm_performance_are_task_and_length_dependent role: complicates claim: Inference temperature has task-dependent and input-length-dependent effects on speech-LLM performance, with no single optimal value across question types. source: §4.2.3, Figure 2 evidence: GPT-4o on contextual-gender-type questions peaks at mid-range temperatures, MERaLiON consistently favors lower values, and MERaLiON on contextual-speaker-count questions shows the largest gains with higher temperature on the short set. confidence: medium relevance: low - claim_id: shorter_inputs_generally_easier_except_for_direct_gender role: refines claim: Shorter speech inputs are generally easier for speech-LLMs across most question types, but direct gender inference does not follow this trend. source: §4.2.2 evidence: Models consistently score higher on the short set (10-30s) than the long set (30-60s) for Content, Contextual Paralinguistic, Contextual Emotion, Contextual Speaker-count, Direct Emotion, and Direct Speaker-count types, but direct-gender-type performance does not show the same length-dependent improvement. confidence: medium relevance: low limitations: - The benchmark is small in scale (under 500 samples per set), reducing statistical reliability for low-frequency question types, and QA pairs were generated by non-overlapping annotators without a cross-verification round, introducing potential individual annotation bias. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.17765' published_date: "2025-09-22" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: qwen3omni_causal_convnet_vocoder_latency role: supports claim: Replacing a block-wise diffusion vocoder with a lightweight causal convolutional decoder, driven by a multi-codebook autoregressive token predictor, can substantially reduce first-packet latency in streaming speech generation without sacrificing competitiveness on content-consistency metrics. source: §2.4, §2.5, Table 1, Table 13 evidence: The Talker's multi-codebook AR scheme plus a 200M-parameter causal ConvNet Code2Wav stage achieves a 234ms end-to-end first-packet latency at 1x concurrency and the lowest reported content-consistency error on SEED test-en (1.39) among all compared zero-shot TTS systems. confidence: high relevance: high - claim_id: qwen3omni_multimodal_no_degradation role: supports claim: Mixing unimodal and cross-modal training data from the earliest stage of pretraining allows a language model to add new input/output modalities without degrading its original text, vision, or audio-specific capabilities relative to matched unimodal baselines. source: §6, Table 16 evidence: A controlled comparison of parameter-matched text-only, vision-only, and Omni models trained on identical corpora, schedules, and compute shows the Omni model matches or exceeds the unimodal baselines on text, vision, and video benchmarks. confidence: high relevance: high - claim_id: qwen3omni_voice_cloning_language_unevenness role: complicates claim: Strong zero-shot voice-cloning performance in one or two conditioning languages does not guarantee comparable speaker-similarity performance uniformly across all supported languages. source: §5.2.2, Table 14 evidence: Against MiniMax-Speech and ElevenLabs Multilingual v2 on a 10-language test set, the system leads by a substantial margin on Chinese, English, and French but reports only competitive, non-leading speaker-similarity or content-consistency scores on several other languages. confidence: medium relevance: low - claim_id: qwen3omni_purpose_built_audio_encoder role: supports claim: A large-scale, purpose-built supervised audio encoder trained from scratch for a multimodal LLM's audio pathway can outperform reusing a general pretrained ASR encoder (e.g., Whisper) as the perceptual front-end for both speech understanding and downstream speech generation. source: §1, §2.2, Table 6, Table 7 evidence: Replacing the Whisper-based audio encoder from the predecessor system with AuT, trained from scratch on 20 million hours of supervised audio at a 12.5 Hz token rate, is cited as a key driver of gains across ASR, lyric-ASR, and voice-interaction benchmarks. confidence: medium relevance: high limitations: - Speech generation quality is evaluated exclusively with automatic metrics (WER/CER, embedding-based speaker similarity); no human MOS or listening-test results are reported for the Talker's synthesized speech. - The reported 234ms first-packet latency is described as theoretical, measured under a specific deployment configuration, and degrades substantially under higher concurrency (up to 1172ms at 6-way concurrency). caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2501.04561' published_date: "2025-09-23" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models claims: - claim_id: openomni_text_pivot_alignment role: supports claim: Progressive, text-pivoted alignment across modality pairs can substitute for paired tri-modal training data without sacrificing downstream omnimodal task performance. source: §4.2, Table 1 evidence: OpenOmni trains only on speech-text and image-text pairs (no image-speech-text triples) yet outperforms VITA, which is trained on 5M tri-modal samples, by 4 points on OmniBench while using a 7B rather than 7x8B language model and roughly 5x fewer training samples. confidence: high relevance: high - claim_id: openomni_nar_latency_quality_tradeoff role: complicates claim: Non-autoregressive discrete-unit speech decoding trades generation quality for latency relative to autoregressive decoding. source: §4.2, §D 'AR mode' evidence: The AR mode (16K-unit vocabulary) yields higher speech generation quality but slower streaming, while the NAR mode (CTC loss, 6K-unit vocabulary) achieves under-1-second latency for up to 30 seconds of speech (5x faster) at the cost of slightly worse generation quality. confidence: medium relevance: high - claim_id: openomni_ctc_dpo_emotional_coherence role: supports claim: Direct preference optimization can be adapted to discrete-unit, CTC-trained speech generators to improve emotional coherence without an auxiliary emotion-control module. source: §4.2, Table 4 evidence: CTC-DPO training on the 9K-pair EO2S-9K preference dataset (Plutchik-based emotion categories, CosyVoice-synthesized positive/negative pairs) raises Emotion2Vec-classified accuracy from 57.9% to 70.4% on Chinese and 62.6% to 65.4% on English test speech. confidence: high relevance: low - claim_id: openomni_moe_necessary_for_ctc_stability role: complicates claim: Mixture-of-experts capacity is necessary, not merely beneficial, for stabilizing CTC-loss training of multilingual non-autoregressive speech decoders. source: §3.4, §C, Table 6 evidence: A single feed-forward decoder layer (1 expert) fails to converge on bilingual WeNetSpeech/LibriSpeech data (CER/WER of 113.6/129.7/87.8/96.5), while a 4-expert MoE layer brings these down to single digits (8.5/8.4/4.2/4.7). confidence: high relevance: low limitations: - The system is trained and validated only on Chinese and English; generalization of the alignment and speech-generation strategy to other languages is untested. - Evaluation of emotional and omnimodal quality relies on automated classifiers (Emotion2Vec, Whisper-based WER) and the authors' own benchmarks rather than independent human listening tests. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.19186' published_date: "2025-09-23" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - codec architecture: - GAN relevance: medium evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: - adversarial_codec_speech_models claims: - claim_id: greedy_per_layer_code_selection_in_residual_vector role: supports claim: Greedy per-layer code selection in residual vector quantization leaves quantization error on the table that can be recovered purely at test time, without retraining the codec. source: §5.2, Table 3 evidence: Beam-search encoding (B=16) lowers average L2 quantization error on LibriTTS from 5.096 to 4.625 for EnCodec and from 22.29 to 21.81 for HiFi-Codec relative to greedy (B=1) encoding of the same pre-trained checkpoints. confidence: high relevance: low - claim_id: reductions_in_rvq_quantization_error_obtained_by_a role: supports claim: Reductions in RVQ quantization error obtained by a better search strategy translate into measurable gains on standard reconstruction-quality metrics, not just the raw error term. source: §5.3, Table 1 evidence: Increasing beam size from 1 to 16 improves PESQ, STOI, NISQA, and SI-SNR simultaneously for both EnCodec and HiFi-Codec on LibriTTS test-clean. confidence: high relevance: low - claim_id: adding_more_rvq_codebooks_higher_bit_rate_does role: refines claim: Adding more RVQ codebooks (higher bit-rate) does not by itself eliminate the suboptimality of greedy encoding; the search-strategy gap persists even at high codebook counts. source: §5.5, Table 4 evidence: At 24 kbps with 32 codebooks, EnCodec's greedy encoding still trails beam-search encoding (PESQ 3.670 vs. 3.691, NISQA 3.992 vs. 4.001), a gap of comparable relative size to the one seen at 6 kbps with 8 codebooks. confidence: high relevance: low - claim_id: a_naive_implementation_of_wider_search_codec_encoding role: complicates claim: A naive implementation of wider-search codec encoding introduces a latency cost that scales with search width, and only a parallel hardware-aware implementation avoids this trade-off. source: §5.6, Table 5 evidence: Sequential CPU beam-search encoding increases inference time by 285% from B=1 to B=16 (43.53 ms to 167.6 ms per 5-second clip), while a GPU-parallelized implementation of the same algorithm increases by only about 9% (6.780 ms to 7.366 ms) over the same range. confidence: high relevance: low limitations: - The evaluation is confined to two pre-trained codec checkpoints (EnCodec, HiFi-Codec); the paper does not test the algorithm on more recent RVQ-GAN codec designs (e.g. DAC-style codecs) or on codecs with substantially larger codebook sizes, where the exponential O(S^L) exhaustive-search cost the method is designed to avoid is even more pronounced. Beam size B and the per-step candidate width k are always set equal in the experiments, so their individual contributions to the error reduction are not disentangled. All reported gains are measured on intrinsic reconstruction-quality metrics (PESQ, STOI, NISQA, SI-SNR, mel distance); the paper does not evaluate whether the improved codec reconstructions change downstream outcomes for systems that consume the codes, such as word error rate in a codec-token TTS or speech language model pipeline. The non-verbal vocalization test set is an internal, non-public dataset, limiting reproducibility of that portion of the evaluation. caveats: - For spoken language models, this paper provides medium-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.20410' published_date: "2025-09-24" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: llm_endpoint_detector_avoids_asr_pass_while_streaming role: supports claim: A dedicated LLM-based endpoint detector can perform semantic turn-taking judgments directly from raw audio without a separate ASR pass, while still supporting real-time, chunk-wise streaming inference. source: §3.2 evidence: Phoenix-VAD encodes audio directly through a Zipformer encoder and adapter into an LLM, avoiding an explicit ASR module, and generates each chunk prediction in approximately 50 ms on a single NVIDIA A6000 GPU. confidence: medium relevance: low - claim_id: detecting_stop_harder_than_detecting_continue_in_endpoint_detection role: complicates claim: In semantic endpoint detection, identifying that a speaker has stopped is substantially harder than identifying that they are continuing, because the 'stop' class is both rarer and more ambiguous at pause boundaries. source: §3.2, Table 2, Table 3 evidence: Across both test conditions, Stop Speaking recall is 0.888-0.892 versus 0.994-0.995 for Continue Speaking, and Stop Speaking F1 (0.905-0.918) consistently trails Continue Speaking F1 (0.992-0.993). confidence: high relevance: low - claim_id: smaller_chunk_size_amplifies_timestamp_label_noise role: complicates claim: Reducing the temporal chunk size in streaming, window-based speech classification does not straightforwardly improve responsiveness, because finer granularity can amplify sensitivity to label-timestamp noise near decision boundaries. source: §3.3, Table 5 evidence: Shrinking the chunk size from 320 ms to 160 ms lowers Stop Speaking F1 from 0.905 to 0.819, attributed to increased prediction uncertainty near boundaries and magnified Paraformer timestamp error. confidence: medium relevance: low - claim_id: adapters_should_be_jointly_trained_not_pretrained_on_asr_alone role: refines claim: Adapters that bridge audio encoders to an LLM backbone should be trained jointly with the target task rather than pretrained solely on an auxiliary ASR objective, since ASR-only supervision optimizes for lexical alignment at the expense of the fine-grained timing cues the downstream task needs. source: §3.3, Table 5 evidence: Freezing an ASR-pretrained adapter and tuning only LoRA weights collapses Stop Speaking F1 from 0.905 (jointly trained) to 0.325, while Continue Speaking F1 degrades far less (0.993 to 0.914). confidence: high relevance: low limitations: - All training and evaluation data is synthetically generated via Index-TTS with inserted silences simulating hesitation and interruption; the system has not been evaluated on real spontaneous conversational recordings or genuine user interruptions. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2506.21875' published_date: "2025-09-26" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: s2t_performance_does_not_predict_s2s_quality role: supports claim: Strong performance on speech-to-text dialogue benchmarks does not reliably predict a model's speech-to-speech generation quality. source: §3.2, Appendix G evidence: Kimi-Audio's text-response quality on VoiceBench's CommonEval subset drops from 3.97 (text-only mode) to 3.00 (dual-modal speech+text mode), and its overall WildSpeech-Bench score (4.54) trails several models it is known to outperform on S2T-only benchmarks. confidence: high relevance: high - claim_id: e2e_speech_llms_remain_vulnerable_to_background_noise role: complicates claim: End-to-end speech LLMs remain broadly vulnerable to background acoustic interference despite strong performance in clean conditions. source: §3.4, Figure 5 evidence: Adding either environmental noise (ESC-50) or human background speech (LibriTTS) causes a significant performance decline across every evaluated model, including the top-performing GPT-4o-Audio, with human background speech producing the larger drop. confidence: high relevance: low - claim_id: cascaded_pipelines_match_e2e_except_on_paralinguistic_tasks role: refines claim: Cascaded ASR-LLM-TTS pipelines can match dedicated end-to-end speech LLMs on general conversational quality, but the gap re-emerges specifically on tasks requiring paralinguistic understanding. source: §3.2, Table 2 evidence: The naive Whisper + Qwen2.5-7B + MeloTTS pipeline scores comparably to GLM-4-Voice and MiniCPM overall (5.24 vs. 5.03/5.08) but scores markedly worse on the stress (2.95) and tone (4.12) paralinguistic subcategories, consistent with prosodic information being lost during ASR transcription. confidence: medium relevance: high - claim_id: query_aware_checklist_evaluation_improves_judge_alignment_with_humans role: supports claim: Query-aware, checklist-based evaluation improves an LLM judge's alignment with human judgment over generic scoring rubrics for speech-to-speech systems. source: §3.3, Figure 4 evidence: An ablation removing either the query-aware checklist or the multi-round ASR-plus-scoring procedure reduces the Pearson correlation between automatic and human expert scores relative to the full evaluation method. confidence: high relevance: low limitations: - Automatic scoring depends on a proprietary judge (GPT-4o-mini) and a proprietary ASR model (Whisper-large-v3) in the evaluation loop; the benchmark is deliberately restricted to English, non-toxic, single-turn queries. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.22243' published_date: "2025-09-26" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: full_duplex_systems_trade_off_jumpin_versus_premature_takeover role: complicates claim: Full-duplex speech systems face a trade-off between avoiding mid-turn jump-ins and avoiding premature takeover during a user's pause; tuning to reduce one failure mode appears to increase the other. source: §3.2, Table 2 evidence: Moshi shows the lowest turn-taking latency but the highest jump-in rate (JIR = 0.785) and the lowest (best) pause-handling takeover rate (0.55), while Freeze-Omni, Vita-1.5, and Gemini show near-zero JIR but ~85-94% pause-handling takeover rate. confidence: high relevance: high - claim_id: commercial_systems_more_topically_coherent_under_interruption role: supports claim: Commercial full-duplex speech systems currently produce more topically coherent responses to user interruptions than open-source full-duplex systems. source: §3.2, Table 3 evidence: Gemini reaches Turn-Taking Rate 0.935 and GPT-4o-rated Topic Shift Score 4.51/5 on the user-interrupt scenario, versus 0.226-0.539 TTR and 0.75-2.9 TSS for Moshi, Freeze-Omni, and Vita-1.5. confidence: medium relevance: high - claim_id: emergency_detection_does_not_imply_proactive_interrupt role: complicates claim: A full-duplex system's ability to recognize an emergency in its generated response content does not imply it will proactively interrupt the user to act on it. source: §3.2, Table 3 evidence: Gemini attains the highest Emergency Detection Score (2.19) but a model-interrupt takeover rate of 0, while open-source models with higher takeover rates (up to 0.535) score lower on emergency detection, indicating their interruptions are driven by generally aggressive turn-taking rather than emergency recognition. confidence: medium relevance: high - claim_id: current_models_generate_backchannels_rarely_with_poor_timing role: complicates claim: Current full-duplex speech language models generate listener backchannels rarely and with poor timing alignment to human-judged reference positions. source: §3.2, Table 4; §4 evidence: All four evaluated systems show near-zero Backchannel Rate (0.002-0.031) and Jensen-Shannon Divergence near the 1.0 maximum (0.947-0.986) against GPT-4o-annotated reference backchannel timing, though isolated strong backchannel cases were observed in the model-interrupt scenario. confidence: medium relevance: high limitations: - Benchmark data is fully synthetic and only one commercial system is evaluated due to budget constraints; GPT-4o-based automated scoring is not independently validated by human raters. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2510.02352' published_date: "2025-09-27" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: closed_source_sdms_show_lower_decision_bias_than_open_source role: supports claim: Closed-source spoken dialogue systems can exhibit substantially lower paralinguistic decision bias than open-source alternatives evaluated under the same protocol. source: §3.3.1, Table 2 evidence: Gemini-2.5-Flash achieves Group Unfairness Scores of 0.12-0.14 across award, interview, and assignment decision tasks, compared to 0.17-0.20 for Qwen2.5-Omni and 0.19-0.21 for GLM-4-Voice on the same FairDialogue samples. confidence: high relevance: high - claim_id: fairness_ranking_does_not_transfer_across_task_types role: complicates claim: A model's relative fairness ranking on one conversational task type does not transfer to another task type. source: §3.3.1, Table 2 evidence: GPT-4o Audio and GLM-4-Voice, which are not the most biased models on decision-making, show the largest cross-group disparities (SNSR 0.63-0.72 and 0.66-0.68) on recommendation tasks, while Gemini-2.5 and Qwen2.5-Omni, weaker on recommendation-related accent bias in decisions, perform comparatively better here. confidence: high relevance: high - claim_id: single_turn_bias_eval_understates_multi_turn_bias role: complicates claim: Single-turn bias evaluation can understate a spoken dialogue system's paralinguistic bias relative to multi-turn interaction. source: §3.3.2, Table 3 evidence: 'On samples where all four models initially gave identical negative decisions in single-turn evaluation (no apparent bias), applying four rounds of corrective feedback revealed attribute-dependent revision success rates: Elder Male speakers reached 88-95% revision success versus 69-91% for Young Female speakers, with the effect attributed to age in Qwen2.5-Omni and to gender in GLM-4-Voice.' confidence: high relevance: high - claim_id: voice_only_paralinguistic_attributes_sufficient_to_bias_outputs role: supports claim: Paralinguistic attributes carried only through voice (not through text content) are sufficient to bias spoken dialogue LLM outputs in real-world-relevant decision and recommendation tasks. source: §2, §3.3.1 evidence: FairDialogue holds text content demographically neutral while varying only synthesized speech attributes (age, gender, accent), and all four evaluated models still show measurable disparities in decision acceptance rates and recommendation rankings across these attribute groups. confidence: high relevance: high limitations: - The evaluation covers only four spoken dialogue systems and paralinguistic variation is synthetic (TTS-controlled) rather than drawn from real speakers with those attributes; the multi-turn protocol is restricted to a single negate-and-reassert feedback pattern. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.23938' published_date: "2025-09-28" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - hybrid_multistage_speech_models claims: - claim_id: bimodal_fusion_substantially_improves_turntaking_accuracy_over_single_modality role: supports claim: Fusing acoustic and linguistic modality signals in a single turn-taking or dialogue-state classifier substantially improves accuracy over either modality used alone. source: §4.3, Table 3 evidence: Ablation shows the bimodal model reaches 95.75% average accuracy across four turn states, versus 85.5% for an acoustic-only classifier (fine-tuned Whisper + linear layer) and 86.25% for a linguistic-only classifier (fine-tuned Qwen2.5-0.5B on transcripts alone). confidence: high relevance: low - claim_id: intermediate_asr_transcription_improves_turn_state_detection role: supports claim: Generating an explicit intermediate ASR transcription before predicting a dialogue-turn label, rather than predicting the label directly from acoustic features, improves detection accuracy in a speech-conditioned LLM. source: §4.3, Table 3 evidence: Removing the ASR+Turn-Detection paradigm while keeping the same architecture (Easy Turn only-state) drops average accuracy from 95.75% to 87.88%. confidence: high relevance: low - claim_id: lightweight_turntaking_classifiers_trade_accuracy_for_efficiency_widening_on_incomplete role: complicates claim: Lightweight, latency-optimized turn-taking classifiers trade off accuracy against efficiency, and the gap widens specifically for detecting incomplete (as opposed to complete) utterances. source: §4.2, Table 2 evidence: Smart Turn V2 (95 MB, 27 ms latency) reaches 78.67% accuracy on complete but only 62% on incomplete, and provides no backchannel or wait detection at all, versus 96.33%/97.67% for the proposed model. confidence: high relevance: low - claim_id: semantic_only_detectors_need_separate_asr_without_surpassing_compact_bimodal_alternative role: complicates claim: Semantic-only turn-taking detectors built on large LLM backbones require a separate upstream ASR module, adding parameter count and memory overhead without necessarily surpassing the accuracy of a compact, natively bimodal alternative. source: §4.2, Table 2 evidence: TEN Turn Detection (7B-parameter LLM plus a Paraformer ASR frontend, 7,220 MB parameters, 15,419 MB memory, 204 ms latency) is outperformed on every measured state by an 850 MB, 2,559 MB bimodal model, despite the latter's slightly higher latency (263 ms). confidence: high relevance: low limitations: - All reported accuracy numbers come from the authors' own Easy Turn testset, built and annotated by the same team; the training data is heavily imbalanced across states (backchannel and wait contribute only 10 and 23 of 1,145 hours). caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.25131' published_date: "2025-09-29" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - TTS architecture: - autoregressive-LM - flow-matching - hybrid relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models - hybrid_multistage_speech_models - flow_matching_speech_decoders claims: - claim_id: chunking_text_into_aligned_segments_with_a_short role: supports claim: Chunking text into aligned segments with a short token-delay before speech decoding reduces error accumulation in long-form autoregressive speech generation. source: §4.2, Table 6 evidence: Removing chunk-based decoding raises Long-TTS-Eval error rates above those of concurrent long-form TTS baselines, and with it enabled MGM-Omni-TTS-2B achieves EN-hard WER 26.26 versus 42.48-98.61 for CosyVoice2, MOSS-TTSD-v0.5, and Higgs-Audio-v2. confidence: high relevance: high - claim_id: multi_token_parallel_decoding_is_not_restricted_to role: supports claim: Multi-token parallel decoding is not restricted to RVQ speech tokenizers and can be applied effectively to finite scalar quantization (FSQ) tokenizers. source: §4.2, Table 6 evidence: Increasing parallel decoding size on the CosyVoice2 FSQ tokenizer maintains TTS quality on Seed-TTS-Eval while cutting inference RTF by roughly 3x at parallel size 4. confidence: high relevance: high - claim_id: increasing_the_parallel_decoding_size_trades_off_synthesis role: complicates claim: Increasing the parallel decoding size trades off synthesis error rate against inference speed rather than improving both simultaneously. source: §4.2 evidence: Larger parallel sizes in the ablation slightly raise audio error rate even as they substantially accelerate inference, leading the authors to select a parallel size of 4 as a balance point. confidence: high relevance: low - claim_id: separating_multimodal_reasoning_from_speech_synthesis_into_distinct role: refines claim: Separating multimodal reasoning from speech synthesis into distinct model components can improve long-form audio understanding without sacrificing speech generation efficiency. source: §4.1.1, §4.1.3, Figure 5, Table 5b evidence: The dual-track brain-mouth design lets the MLLM handle needle-in-the-haystack audio inputs up to 4,500 seconds while the SpeechLM independently achieves the lowest RTF among compared long-form TTS systems. confidence: high relevance: high limitations: - 'The long-form evaluation itself is partly self-authored: Long-TTS-Eval is introduced by this paper, and while its construction and normalized-text scoring procedure are documented, results on it cannot yet be cross-checked against independent replications. The comparison in Table 5b is limited to three baseline systems, and the qualitative long-speech examples in the appendix (a classical Chinese poem and a code-switched English-Chinese poem) are illustrative rather than a systematic error analysis. The paper does not report results on emotion or prosody control, nor does it evaluate robustness to reference audio recorded in noisy or far-field conditions. The 32B MLLM variant''s long-form and vision-speech results are mixed relative to the 7B variant (e.g., lower TextVQA-Speech and EN-hard performance context is not directly reported for 32B TTS), suggesting scaling benefits are not uniform across all sub-tasks.' caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.26276' published_date: "2025-09-30" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - TTS - SCA architecture: - autoregressive-LM relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: - autoregressive_speech_language_models claims: - claim_id: acoustic_consistency_improved_by_lm_side_recipe_independent_of_scale role: supports claim: Acoustic consistency in autoregressive speech language models can be improved through LM-side embedding initialization and auxiliary training objectives, independent of parameter count. source: §4, Table 1 evidence: The 0.7B speech-only CAST model scores 90.8 speaker and 90.0 gender consistency on SALMON, exceeding the 7B SpiritLM (81.0/85.0) and 7B Twist (71.0/70.0), despite using an order of magnitude fewer parameters and the same evaluation protocol. confidence: high relevance: high - claim_id: text_speech_interleaving_trades_consistency_for_semantic_grounding role: complicates claim: Interleaving text and speech tokens in a shared-vocabulary language model improves semantic and lexical grounding at the cost of acoustic consistency. source: §4, Tables 1-2 evidence: The 1.0B interleaved model drops SALMON consistency by 5-9 points across every measured factor relative to the 1.0B speech-only model (e.g., speaker 83.5 vs. 90.0) while sWUGGY rises from 67.0 to 73.7 and semantic-acoustic alignment scores improve by 6-7.5 points. confidence: high relevance: high - claim_id: coarse_then_fine_auxiliary_objectives_improve_identity_stability role: supports claim: Auxiliary objectives that require the model to plan coarse content structure before predicting fine acoustic detail materially contribute to speaker- and gender-identity stability beyond what embedding initialization alone provides. source: §4, Table 4 evidence: Removing the delayed coarse-label and next-code auxiliary losses drops speaker consistency from 90.8 to 83.5 in the 0.7B model and from 90.0 to 83.0 in the 1.0B model, with background and room consistency declining more moderately. confidence: high relevance: low - claim_id: ssl_initialization_biases_toward_content_over_prosody role: refines claim: Initializing speech-token embeddings from self-supervised acoustic features biases the resulting representation toward content structure at the expense of fine-grained prosodic detail. source: §4, Table 3 evidence: Linear probes on frozen LM features show the SSL-initialized variant gains on average +16% relative accuracy on content-leaning tasks (ESC-50, US8K, RAVDESS) while losing about 7% relative accuracy on prosody-leaning tasks (VIVAE, EMOVO), compared to a randomly initialized baseline. confidence: medium relevance: low limitations: - Consistency is evaluated exclusively through likelihood-based pairwise preference scoring (SALMON) rather than generated-sample quality or human listening tests; comparisons against baseline speech LMs are confounded by differences in tokenizer and training data beyond scale. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. - id: '2509.26542' published_date: "2025-09-30" entry_date: '2026-07-27' year: 2025 venue: arXiv task: - SCA - evaluation architecture: [] relevance: high evidence_role: - core_evidence - infrastructure current_role: active_evidence method_family: [] claims: - claim_id: substantial_text_voice_reasoning_gap_widens_with_reasoning_steps role: supports claim: A substantial reasoning-accuracy gap exists between text and voice modalities within the same model family, and the gap widens with the number of reasoning steps a task requires. source: §5.1, Table 3 evidence: GPT-5 text reaches 74.8% accuracy on competition math versus 6.1% for GPT-realtime voice (a 68.7-point gap), while the same model pair differs by only ~21 points on factual recall (48.3% vs. 27.4%); macro-averaged across five tracks, text models reach 54.0% vs. 11.3% for voice. confidence: high relevance: low - claim_id: more_thinking_time_insufficient_to_close_voice_reasoning_gap role: complicates claim: Increasing inference-time deliberation ('thinking time') before a voice system speaks is not, by itself, sufficient to close a voice-native reasoning gap. source: §5.2 evidence: Enabling Audio Flamingo 3's thinking mode raises time-to-first-response by 530% (2.40s to 15.14s) but overall accuracy decreases from 1.7% to 1.5%, with Context-track accuracy dropping from 3.8% to 1.8%. confidence: high relevance: low - claim_id: reasoning_narration_decoupling_narrows_but_does_not_close_gap role: complicates claim: Decoupling reasoning from speech narration in a cascade architecture narrows a voice-native reasoning gap but introduces new grounding and logical-consistency failures that prevent full parity with text. source: §5.2, Figure 5 evidence: A cascade pairing a strong text reasoner with a separate narration model still trails the text-only upper bound by 15.7 points on Math (59.1% vs. 74.8%) and collapses to 0.2% accuracy on the Context track, with elevated rates of UNSUPPORTED_FACT and LOGICAL_CONTRADICTION errors attributed to the narration stage. confidence: high relevance: low - claim_id: speech_intelligibility_not_primary_bottleneck_for_voice_reasoning role: supports claim: Speech output intelligibility is not the primary bottleneck limiting reasoning performance in voice-interactive systems. source: §5.2, Table 3 evidence: Word Error Rate across evaluated voice systems ranges from 7.9% (Gemini-2.5-Flash-audio) to 19.8% (Freeze-Omni) with no corresponding difference in reasoning accuracy; all systems in this WER range perform uniformly poorly on VERA's reasoning tracks. confidence: high relevance: low - claim_id: voice_architectures_fail_reasoning_in_systematically_different_ways role: refines claim: Different voice system architectures fail on reasoning tasks in systematically different, architecture-linked ways rather than through a single uniform error pattern. source: §5.3, Figure 6 evidence: Native streaming models (GPT-realtime, Gemini-2.5-Flash-Audio) under-produce NO_FINAL_ANSWER and OFF_TARGET errors, favoring fluent-but-incorrect completions, while the LiveAnswer cascade over-produces UNSUPPORTED_FACT, OFF_TARGET, and LOGICAL_CONTRADICTION deviations, and end-to-end models (Moshi, Qwen2.5-Omni) show yet another bimodal pattern of extreme over- or under-completion. confidence: medium relevance: low limitations: - The study is an observational, cross-architecture diagnostic rather than a controlled experiment, since the 12 evaluated voice systems differ in training data, objectives, and proprietary implementation details, preventing isolation of modality as the sole causal factor. caveats: - For spoken language models, this paper provides high-relevance evidence; claims primarily concerning other concepts remain included but are marked low relevance. claim_clusters: - id: discrete_tokens_enable_scalable_speech_language_modeling claim: Discrete speech tokens make large-scale language modeling of audio tractable by converting waveforms into compact prediction sequences. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2301.02111' - '2305.15255' - '2306.12925' - '2402.05755' - '2402.13236' - '2408.16532' - '2410.00037' - '2410.03751' - '2410.11190' - '2411.17607' - '2411.18803' - '2412.02612' - '2412.15649' - '2501.06282' - '2502.06490' - '2502.11946' - '2502.17239' - '2504.08528' - '2504.10344' - 2025.naacl-demo.12 - '2505.02625' - '2506.10274' - '2507.16632' - '2507.18119' - 2025.acl-long.65 - 2025.acl-long.682 - 2025.acl-long.817 - 2025.findings-acl.101 - 2025.findings-acl.1051 - 2025.findings-acl.115 - 2025.findings-ijcnlp.49 - '2508.04585' - '2508.07375' - '2508.08961' - '2508.09600' - interspeech-2025-0246 - interspeech-2025-0310 - interspeech-2025-0874 - interspeech-2025-1641 - interspeech-2025-2328 - '2508.16790' - '2508.19205' - '2508.20660' - '2509.02020' - '2509.04685' - '2509.09174' - '2509.25131' contradicting_papers: [] refining_papers: - iclr-2025-868masI331 - 2025.findings-ijcnlp.49 - '2508.02849' - interspeech-2025-0310 - interspeech-2025-2564 - '2509.04685' - '2509.26276' caveats: - Token compactness alone does not guarantee preservation of linguistic content, speaker identity, or prosody. last_reviewed: '2026-07-27' - id: semantic_acoustic_token_tradeoff claim: Speech tokenizers face a persistent trade-off between semantic intelligibility and fine acoustic, speaker, and prosodic fidelity. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2410.00037' - '2410.03751' - '2411.18803' - '2412.02612' - '2502.06490' - '2502.11946' - '2412.18603' - 2025.acl-long.681 - 2025.acl-long.817 - 2025.findings-acl.101 - 2025.findings-acl.115 - '2507.23159' - '2507.20091' - interspeech-2025-0246 - '2508.19205' - '2508.20660' - '2509.09174' - '2509.14882' - '2509.26276' contradicting_papers: [] refining_papers: - '2509.14882' - '2509.26276' caveats: - The trade-off depends on frame rate, codebook structure, training objective, and downstream decoder capacity. last_reviewed: '2026-07-27' - id: low_frame_rate_and_variable_rate_tokens_reduce_sequence_cost claim: Low- and variable-frame-rate speech tokenization reduces language-model sequence cost while preserving useful content when allocation follows acoustic information density. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2411.17607' - '2502.06490' - '2504.10344' - iclr-2025-dGSOn7sdWg - '2508.04585' - '2508.16790' - '2509.02020' - '2509.04685' - '2509.17765' contradicting_papers: [] refining_papers: - iclr-2025-868masI331 caveats: - Aggressive compression eventually harms intelligibility, timing precision, or speaker and prosodic detail. last_reviewed: '2026-07-27' - id: unified_models_support_joint_understanding_and_generation claim: A shared language-model backbone can support both speech understanding and speech generation across conversational tasks. status: strongly_supported confidence: high supporting_papers: - '2408.16725' - '2409.06666' - '2410.03751' - '2410.11190' - '2410.17799' - '2411.13577' - '2412.15649' - 2025.naacl-demo.21 - '2505.02625' - 2025.acl-long.912 - 2025.findings-acl.115 - '2507.23159' - '2508.02013' - '2503.04721' - interspeech-2025-0739 - '2508.15827' - '2509.03940' - '2509.17765' - '2501.04561' - '2509.22243' - '2510.02352' - '2509.25131' contradicting_papers: [] refining_papers: - 2025.naacl-demo.21 - '2507.16632' - 2025.findings-acl.115 - '2508.02013' - '2509.09716' - '2509.22243' - '2510.02352' caveats: - Unified capability does not imply equal competence across semantic, acoustic, and interaction dimensions. last_reviewed: '2026-07-27' - id: speech_text_alignment_preserves_language_model_capabilities claim: Explicit speech–text alignment and modality connectors allow pretrained text language models to acquire speech capabilities while preserving much of their linguistic reasoning. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2305.11000' - '2402.05755' - '2408.16725' - '2410.03751' - '2410.11190' - '2410.17799' - '2411.17607' - '2412.02612' - '2412.15649' - '2501.06282' - '2409.20007' - '2502.17239' - iclr-2025-dGSOn7sdWg - '2505.02625' - '2507.16632' - '2505.15670' - '2507.18119' - 2025.acl-long.388 - 2025.acl-long.682 - 2025.acl-long.912 - 2025.findings-acl.115 - 2025.findings-acl.534 - 2025.findings-acl.75 - 2025.iwslt-1.5 - '2508.02849' - '2508.04585' - '2508.07375' - '2508.08961' - interspeech-2025-0756 - interspeech-2025-0874 - interspeech-2025-1595 - interspeech-2025-1641 - '2508.15827' - '2508.16188' - '2509.02020' - '2509.04702' - '2509.17765' - '2501.04561' - '2506.21875' contradicting_papers: [] refining_papers: - '2411.13577' - 2025.findings-acl.115 - 2025.findings-acl.75 - 2025.iwslt-1.5 - '2508.02849' - interspeech-2025-0874 - interspeech-2025-1595 - interspeech-2025-2564 - '2509.04702' - '2509.12171' - '2509.22243' - '2509.26276' caveats: - Alignment can transfer transcript-accessible knowledge but may underrepresent information carried only by acoustics. last_reviewed: '2026-07-27' - id: interleaved_training_bridges_modalities claim: Interleaving speech and text during training improves cross-modal transfer and supports models that reason in text while communicating in speech. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - '2306.12925' - '2402.05755' - '2408.16725' - '2410.03751' - '2410.17799' - '2411.17607' - '2412.02612' - '2412.15649' - '2501.06282' - '2502.11946' - '2503.14345' - '2507.16632' - '2505.15670' - '2507.18119' - 2025.acl-long.682 - 2025.findings-acl.1051 - 2025.findings-acl.75 - 2025.iwslt-1.5 - '2508.02013' - '2508.04585' - '2507.20091' - '2508.07375' - interspeech-2025-0874 - interspeech-2025-1595 - '2508.15827' - '2508.16790' - '2509.02020' - '2509.09174' - '2501.04561' contradicting_papers: [] refining_papers: - '2411.13577' - '2505.15670' - interspeech-2025-1595 - '2509.25131' - '2509.26276' caveats: - Synthetic interleaving can inherit teacher errors and may not reproduce natural conversational timing or paralinguistic behavior. last_reviewed: '2026-07-27' - id: streaming_spoken_models_can_reach_interactive_latency claim: Incremental tokenization, generation, and decoding enable spoken language models to operate at interactive latency. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2408.16725' - '2409.06666' - '2410.03751' - '2410.11190' - '2411.13577' - '2411.18803' - '2412.02612' - '2501.06282' - '2502.11946' - '2505.02625' - '2505.15670' - 2025.acl-long.682 - 2025.acl-long.912 - 2025.findings-acl.1051 - '2508.15827' - '2508.19205' - '2509.17765' contradicting_papers: [] refining_papers: - '2411.13577' - 2025.acl-long.817 - 2025.findings-acl.101 - '2501.04561' caveats: - Reported latency is often hardware- and implementation-specific and may exclude endpointing or upstream processing. last_reviewed: '2026-07-27' - id: full_duplex_requires_joint_listening_speaking_and_turn_taking claim: Natural full-duplex speech interaction requires models to listen while speaking and explicitly handle overlap, interruption, and turn transitions. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2408.02622' - '2410.00037' - '2410.03751' - '2410.11190' - '2410.17799' - '2411.00774' - '2411.13577' - '2501.06282' - 2025.naacl-long.484 - '2505.15670' - 2025.findings-ijcnlp.49 - '2507.23159' - '2508.07375' - '2503.04721' - interspeech-2025-0739 - interspeech-2025-0874 - interspeech-2025-2564 - interspeech-2025-2660 - '2509.04093' - '2509.06502' - '2509.22243' contradicting_papers: [] refining_papers: - '2504.08528' - 2025.naacl-demo.21 - '2505.15670' - 2025.acl-long.682 - 2025.findings-ijcnlp.49 - interspeech-2025-0874 - interspeech-2025-2564 - '2509.04702' - '2509.06502' - '2509.22243' caveats: - Many demonstrations use constrained scenarios, and benchmark definitions of successful interruption handling remain inconsistent. last_reviewed: '2026-07-27' - id: paralinguistic_reasoning_remains_weaker_than_semantics claim: Spoken language models remain less reliable on prosody, emotion, speaker traits, and non-verbal cues than on transcript-level semantics. status: strongly_supported confidence: high supporting_papers: - '2402.13236' - '2409.06666' - '2410.00037' - '2411.13577' - '2501.06282' - '2409.20007' - '2502.06490' - '2504.08528' - '2506.10274' - '2507.16632' - '2507.18119' - 2025.acl-long.388 - 2025.findings-acl.101 - '2508.02849' - '2508.04585' - '2507.20091' - '2508.08961' - '2508.09600' - interspeech-2025-0756 - '2508.16188' - '2508.17623' - '2509.02020' - '2509.03940' - '2509.04685' - '2508.18240' - '2510.02352' contradicting_papers: [] refining_papers: - '2409.20007' - '2504.08528' - '2507.16632' - '2507.18119' - '2508.02849' - interspeech-2025-0756 - '2506.21875' - '2510.02352' caveats: - Performance varies with prompting, speech duration, evaluator, and whether the model receives raw audio or derived labels. last_reviewed: '2026-07-27' - id: long_context_and_memory_are_required_for_conversation claim: Multi-turn spoken interaction requires explicit mechanisms for conversational context, memory, and long-horizon state. status: strongly_supported confidence: high supporting_papers: - '2412.15649' - 2025.coling-industry.29 - '2503.14345' - '2505.02625' - '2507.18119' - 2025.acl-long.912 - 2025.findings-acl.470 - 2025.findings-ijcnlp.49 - '2508.04585' - '2508.15827' - '2508.17623' - '2509.02020' - '2509.03940' - '2508.18240' contradicting_papers: [] refining_papers: - 2025.coling-industry.29 - 2025.naacl-demo.21 - 2025.naacl-long.484 - 2025.findings-acl.470 - '2508.18240' - '2510.02352' caveats: - Longer context windows do not ensure that models retrieve or apply the relevant conversational state. last_reviewed: '2026-07-27' - id: long_form_speech_generation_needs_hierarchical_coherence claim: Long-form speech generation benefits from hierarchical or multiscale planning that separates linguistic continuity from acoustic rendering. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - 2025.coling-industry.29 - '2503.14345' - '2412.18603' - '2508.19205' - '2509.09716' - '2509.14882' - '2509.25131' contradicting_papers: [] refining_papers: - '2412.18603' - '2509.25131' caveats: - Long-form evaluations often emphasize local quality and under-measure discourse coherence and error accumulation. last_reviewed: '2026-07-27' - id: instruction_and_conversational_data_scale_slm_capability claim: Large, diverse speech-instruction and conversational datasets expand spoken-language-model task coverage and generalization. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - '2409.06666' - '2501.06282' - '2409.20007' - 2025.acl-long.912 - 2025.findings-acl.534 - '2508.09600' - '2509.04093' contradicting_papers: [] refining_papers: - 2025.coling-industry.29 caveats: - Dataset scale can conceal demographic imbalance, synthetic-label noise, and narrow conversational structure. last_reviewed: '2026-07-27' - id: reward_and_preference_alignment_improve_spoken_behavior claim: Reward modeling and preference alignment can improve spoken-language-model helpfulness, naturalness, and behavioral control beyond supervised training alone. status: strongly_supported confidence: high supporting_papers: - 2025.coling-industry.29 - '2502.11946' - '2505.09558' - '2507.16632' - 2025.acl-long.682 - 2025.acl-long.997 - '2509.04093' contradicting_papers: [] refining_papers: - 2025.coling-industry.29 - 2025.acl-long.997 - '2508.07375' caveats: - Reward proxies can optimize measured traits while degrading unmeasured acoustic or interaction qualities. last_reviewed: '2026-07-27' - id: cascaded_and_end_to_end_systems_have_complementary_strengths claim: Cascaded and end-to-end spoken systems retain complementary advantages in reasoning quality, acoustic expressiveness, controllability, and latency. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2305.11000' - '2402.05755' - '2408.16725' - '2409.06666' - '2410.03751' - '2410.17196' - '2411.13577' - '2411.17607' - '2412.02612' - '2501.06282' - '2409.20007' - '2502.17239' - '2504.08528' - 2025.naacl-demo.21 - '2507.16632' - '2505.15670' - 2025.acl-long.65 - 2025.findings-acl.101 - '2508.02013' - '2508.04585' - '2503.04721' - interspeech-2025-0756 - interspeech-2025-0874 - '2509.04702' - '2509.06502' - '2509.17765' contradicting_papers: [] refining_papers: - '2411.13577' - '2504.08528' - 2025.naacl-demo.21 - 2025.naacl-long.484 - '2505.15670' - 2025.acl-long.997 - '2508.02013' - '2509.04702' - '2509.06502' - '2506.21875' caveats: - Comparisons often use different base language models, data, and speech decoders, preventing clean architectural attribution. last_reviewed: '2026-07-27' - id: multidimensional_benchmarks_expose_spoken_model_gaps claim: Multidimensional benchmarks reveal spoken-language-model failures that text-only accuracy and aggregate speech-quality scores miss. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2402.05755' - '2411.17607' - 2025.coling-industry.29 - '2502.11946' - '2502.17239' - '2504.08528' - iclr-2025-dGSOn7sdWg - 2025.naacl-demo.12 - '2506.10274' - 2025.acl-long.912 - '2507.23159' - '2508.02013' - '2508.07375' - interspeech-2025-0310 - '2508.15827' - '2508.19205' - '2509.03940' - '2509.04685' - '2509.09174' - '2508.18240' - '2509.17765' - '2506.21875' contradicting_papers: [] refining_papers: - '2411.13577' - 2025.coling-industry.29 - '2409.20007' - 2025.naacl-demo.21 - '2507.16632' - interspeech-2025-0310 - interspeech-2025-0756 - '2509.04702' - '2508.18240' - '2509.12171' caveats: - Benchmark conclusions depend on judge reliability, prompt coverage, language, and whether interaction is live or replayed. last_reviewed: '2026-07-27' - id: speech_modality_introduces_reasoning_and_robustness_gaps claim: Replacing text with speech input introduces measurable reasoning, robustness, and instruction-following gaps even when the underlying language model is capable. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - '2402.05755' - '2408.16725' - '2412.02612' - '2409.20007' - '2502.17239' - '2505.02625' - '2505.15670' - 2025.acl-long.388 - 2025.acl-long.912 - 2025.findings-acl.1051 - 2025.findings-acl.534 - 2025.iwslt-1.5 - '2508.08961' - '2508.15827' contradicting_papers: [] refining_papers: - '2505.15670' - 2025.findings-acl.115 - '2509.09716' - '2509.12171' caveats: - The observed gap combines recognition, acoustic representation, alignment, and generation errors rather than one isolated failure. last_reviewed: '2026-07-27' - id: spoken_models_require_bias_privacy_and_security_evaluation claim: Deployment-ready spoken language models require explicit evaluation of demographic bias, privacy, misuse, and generated-speech provenance. status: emerging confidence: medium supporting_papers: - interspeech-2025-2328 - '2510.02352' contradicting_papers: [] refining_papers: - '2509.12171' - '2510.02352' caveats: - Current studies cover limited populations and threat models, so passing one benchmark does not establish broad safety. last_reviewed: '2026-07-27' method_families: - id: autoregressive_speech_language_models name: Autoregressive speech language models summary: Decoder-only and codec-language-model systems that generate speech tokens or continuous frames sequentially, often sharing a text-LLM backbone. papers: - '2209.03143' - '2301.02111' - '2301.11325' - '2305.09636' - '2305.11000' - '2305.15255' - '2306.12925' - '2310.00704' - '2402.05755' - '2408.02622' - '2408.16725' - '2410.00037' - '2410.11190' - '2410.17799' - '2411.00774' - '2411.13577' - '2411.17607' - '2412.02612' - '2412.15649' - '2501.06282' - '2502.11946' - '2502.17239' - '2503.14345' - '2504.10344' - iclr-2025-868masI331 - iclr-2025-dGSOn7sdWg - 2025.naacl-demo.12 - 2025.naacl-long.484 - '2505.02625' - '2505.09558' - '2507.02380' - '2412.18603' - '2507.16632' - '2505.15670' - '2507.18119' - 2025.acl-long.65 - 2025.acl-long.681 - 2025.acl-long.682 - 2025.acl-long.817 - 2025.acl-long.912 - 2025.acl-long.997 - 2025.findings-acl.101 - 2025.findings-acl.1051 - 2025.findings-acl.115 - 2025.findings-acl.631 - 2025.findings-ijcnlp.49 - 2025.iwslt-1.5 - '2508.04585' - '2507.20091' - '2508.07375' - '2508.08961' - '2508.09600' - interspeech-2025-0310 - interspeech-2025-0874 - interspeech-2025-1538 - interspeech-2025-1595 - interspeech-2025-1641 - interspeech-2025-2328 - interspeech-2025-2564 - '2508.15827' - '2508.16188' - '2508.16790' - '2509.02020' - '2509.05359' - '2509.04093' - '2509.09174' - '2509.14882' - '2509.17765' - '2501.04561' - '2509.20410' - '2509.25131' - '2509.26276' open_questions: - How can autoregressive speech models reduce latency and exposure bias without sacrificing long-context coherence? - id: hybrid_multistage_speech_models name: Hybrid and multistage speech models summary: Systems combining text reasoning, speech tokenization, acoustic rendering, or cascaded modules while retaining language-model control. papers: - '2310.00704' - '2409.06666' - '2410.00037' - '2411.00774' - '2411.13577' - '2501.06282' - '2502.11946' - '2502.17239' - '2505.02625' - '2506.23325' - '2507.16632' - 2025.acl-long.1252 - 2025.acl-long.65 - 2025.acl-long.682 - 2025.acl-long.912 - interspeech-2025-0756 - interspeech-2025-0874 - interspeech-2025-1289 - '2508.19205' - '2509.06502' - '2509.17765' - '2501.04561' - '2509.23938' - '2509.25131' open_questions: - Which hybrid boundaries preserve text intelligence while minimizing latency and cross-stage error propagation? - id: flow_matching_speech_decoders name: Flow-matching speech decoders summary: Spoken language models that use flow matching for acoustic realization, waveform generation, or low-step conditional decoding. papers: - '2411.17607' - '2412.02612' - '2412.15649' - '2501.06282' - '2502.11946' - '2502.17239' - '2503.14345' - iclr-2025-tQ1PmLfPBL - '2505.02625' - '2507.02380' - '2507.16632' - '2507.18119' - 2025.acl-long.912 - 2025.findings-acl.101 - '2508.04585' - '2508.09600' - '2509.04093' - '2509.12171' - '2509.25131' open_questions: - Can flow-matching decoders provide stable streaming synthesis under incremental language-model conditioning? - id: transformer_encoder_decoder_speech_models name: Transformer encoder–decoder speech models summary: Speech-aware encoder–decoder systems and modality connectors that map acoustic inputs into language-model representations and outputs. papers: - '2411.13577' - '2411.19842' - '2409.20007' - 2025.acl-long.388 - 2025.findings-acl.75 - '2508.07273' - '2508.08095' - interspeech-2025-0166 - interspeech-2025-0246 - interspeech-2025-2660 - '2508.16790' open_questions: - When do dedicated speech encoders outperform direct token interleaving under matched data and model scale? - id: adversarial_codec_speech_models name: Adversarial codec front ends summary: Speech-language-model pipelines using GAN-trained codecs or waveform decoders to compress acoustics and reconstruct perceptually plausible speech. papers: - '2408.16532' - '2411.18803' - '2506.23325' - '2507.18897' - 2025.acl-long.654 - 2025.acl-long.682 - interspeech-2025-0468 - '2509.04685' - '2509.19186' open_questions: - How should adversarial codec quality be balanced against semantic token usefulness and language-model sequence length? - id: variational_latent_speech_models name: Variational and latent speech models summary: Systems using variational or continuous latent representations to preserve acoustic detail, identity, or diversity alongside language modeling. papers: - '2408.16532' - '2411.19842' - '2504.10344' - 2025.acl-long.65 - 2025.acl-long.654 - '2508.02849' - interspeech-2025-0468 open_questions: - Can continuous latent speech representations retain language-model scalability without reintroducing difficult density modeling? reassessment_queue: [] open_questions: - Which speech representation best balances semantic content, acoustics, sequence length, and streaming latency for a shared language-model backbone? - How can full-duplex systems learn interruption, backchannel, overlap, and endpoint behavior from natural conversations rather than scripted simulations? - What alignment objectives preserve text-LLM reasoning while adding paralinguistic and speaker-aware competence unavailable from transcripts? - When should deployed spoken assistants use end-to-end generation, cascaded components, or a dynamically routed hybrid? - Which live, multilingual evaluation protocol can jointly measure reasoning, acoustic quality, turn-taking, memory, safety, and user experience? trend_notes: - The field progressed from offline codec language modeling in 2022–2023 to streaming, duplex, and omnimodal interaction systems in 2024–2025. - Token research increasingly targets low or variable frame rates, single-codebook streams, and explicit semantic–acoustic separation to reduce language-model sequence cost. - Speech–text interleaving and modality connectors are widely used to preserve pretrained text intelligence while adding speech input and output. - Evaluation shifted toward full-duplex, multi-turn, in-the-wild, role-playing, prosody, and reasoning benchmarks as aggregate quality metrics proved insufficient. - Recent systems increasingly combine supervised instruction tuning with reward or preference alignment, while exposing new cross-dimension and safety trade-offs.