concept: autoregressive-codec-tts last_updated: "2026-07-25" paper_count: 165 papers: - id: "1609.03499" published_date: "2016-09-12" entry_date: "2026-07-25" year: 2016 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: medium evidence_role: - historical_context current_role: foundational method_family: [historical_autoregressive_precursors] claims: - claim_id: direct_generation_of_raw_audio_waveforms_without_intermediate role: supports claim: "Direct generation of raw audio waveforms, without intermediate vocoder parameters, produces substantially higher naturalness than parametric or concatenative synthesis pipelines as judged by human listeners." source: "§3.2, Table 1" evidence: "On the MOS naturalness test using proprietary Google speech databases, WaveNet (conditioned on linguistic features and log-F0) scored 4.21 on North American English and 4.08 on Mandarin Chinese, compared to 3.86 / 3.47 for HMM-driven concatenative and 3.67 / 3.79 for LSTM-RNN parametric baselines (Table 1)." confidence: high relevance: low - claim_id: dilated_causal_convolutions_enable_autoregressive_audio_models_to role: supports claim: "Dilated causal convolutions enable autoregressive audio models to achieve receptive fields orders of magnitude larger than standard causal convolutions with comparable computational cost." source: "§2.1, Figure 3" evidence: "The core component is a stack of dilated causal convolutions: dilation doubles with each layer (1, 2, 4, ..., 512), and the pattern is repeated across multiple blocks, yielding a receptive field that grows exponentially with depth rather than linearly." confidence: high relevance: high - claim_id: a_single_autoregressive_model_conditioned_on_speaker_identity role: supports claim: "A single autoregressive model conditioned on speaker identity can represent many voices with shared internal structure, and multi-speaker training improves per-speaker quality relative to single-speaker training." source: "§3.1" evidence: "In the multi-speaker experiment on VCTK (109 speakers, 44 hours), a single global-conditioning model captured all speakers' characteristics, with validation performance improving over single-speaker training, suggesting that speaker-level representations are shared and mutually reinforcing in the model's internal structure." confidence: high relevance: high - claim_id: receptive_field_size_is_a_binding_constraint_for role: supports claim: "Receptive field size is a binding constraint for prosodic naturalness: when the receptive field is insufficient to cover phrase-level F0 contours, prosody degrades even when segmental quality remains high." source: "§3.2" evidence: "Adding the external F0 conditioning resolved this, showing that receptive field size is a binding constraint for prosodic naturalness independent of model capacity." confidence: high relevance: low - claim_id: autoregressive_raw_waveform_generation_achieves_high_naturalness_at role: supports claim: "Autoregressive raw-waveform generation achieves high naturalness at the cost of sequential sample-level inference, creating a fundamental speed-quality trade-off that constrains deployment in real-time applications." source: "§4, §3.2" evidence: "All timestep predictions are computed in parallel during training (ground-truth samples available at all positions), but inference is strictly sequential: one sample at a time, each fed back into the network to predict the next." confidence: high relevance: high limitations: - "Inference is strictly sequential at the sample level, requiring approximately one computation step per generated sample." - "At the reported generation rates (roughly 1.5× real-time compute), WaveNet is not suitable for real-time TTS deployment without hardware-specific optimisation or a parallel decoding approximation." caveats: - "WaveNet predates codec-token TTS; it is included as the foundational autoregressive speech-generation precursor." - id: "1703.10135" published_date: "2017-03-29" entry_date: "2026-07-25" year: 2017 venue: "arXiv" task: [TTS] architecture: [transformer-enc-dec] relevance: low evidence_role: - historical_context current_role: foundational method_family: [historical_autoregressive_precursors] claims: - claim_id: end_to_end_tts_models_trained_from_characters role: supports claim: "End-to-end TTS models trained from characters with seq2seq attention can match or exceed production statistical parametric systems in subjective naturalness without hand-engineered linguistic features." source: "§5.2, Table 2" evidence: "The encoder thus produces robust sequence representations without requiring phoneme-level alignment or a hand-crafted linguistic frontend." confidence: high relevance: low - claim_id: training_stability_and_alignment_quality_in_character_level role: supports claim: "Training stability and alignment quality in character-level seq2seq TTS improve substantially when the decoder emits multiple output frames per attention step rather than one." source: "§3.3" evidence: "A key design choice is the reduction factor r: the decoder predicts r non-overlapping mel frames per step rather than one, dividing decoding steps by r, accelerating training convergence, and encouraging the attention to advance more readily through the input sequence (r=2 in the main experiments)." confidence: high relevance: low - claim_id: cbhg_style_encoders_combining_multi_scale_convolution_highway role: supports claim: "CBHG-style encoders combining multi-scale convolution, highway networks, and bidirectional recurrence yield more robust text representations than standard RNN encoders, reducing mispronunciation rates." source: "§3.2, §5.1" evidence: "The encoder centres on a novel CBHG module (1-D Convolution Bank, Highway network, and bidirectional GRU)." confidence: high relevance: low - claim_id: post_processing_networks_with_access_to_the_full role: supports claim: "Post-processing networks with access to the full decoded sequence improve harmonic structure in predicted spectrograms compared to frame-level-only decoding." source: "§3.4, §5.1" evidence: "Removing the post-processing net eliminates fine harmonic structure in the output." confidence: high relevance: low limitations: - "Evaluated on a single internal speaker in a controlled studio environment." - "No multi-speaker, out-of-domain, or noisy-data experiments are reported." caveats: - "Tacotron is an autoregressive acoustic model rather than a codec-token model, so its relevance is historical." - id: "2106.15561" published_date: "2021-06-29" entry_date: "2026-07-25" year: 2021 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, diffusion, GAN, VAE, transformer-enc-dec] relevance: medium evidence_role: - historical_context current_role: historical_context method_family: [historical_autoregressive_precursors, hybrid_ar_token_continuous_renderer] claims: - claim_id: the_transition_from_attention_based_alignment_to_explicit role: supports claim: "The transition from attention-based alignment to explicit duration prediction in non-autoregressive acoustic models eliminates word skipping and repeating artefacts while enabling parallel mel-spectrogram generation." source: "§3.4, Table 11" evidence: "The five advanced topic chapters cover: (1) fast TTS, contrasting parallel non-autoregressive generation with lightweight model design and domain-knowledge speedup techniques; (2) low-resource TTS, covering self-supervised pre-training, cross-lingual transfer, and data mining; (3) robust TTS, analysing alignment learning failures and the shift from attention mechanisms to explicit duration prediction; (4) expressive TTS, cataloguing explicit label conditioning, reference encoder approaches, VAE and flow-based variation modeling, and disentanglement methods across information granularities from frame to utterance level; and (5) adaptive TTS, covering few-shot, zero-shot, and untranscribed data adaptation scenarios." confidence: high relevance: high - claim_id: modelling_variation_information_prosody_style_speaker_at_multiple role: supports claim: "Modelling variation information (prosody, style, speaker) at multiple granularities from frame to utterance level improves expressiveness while fine-grained control remains difficult when relying on implicit latent representations." source: "§3.5.1, §3.5.3" evidence: "The five advanced topic chapters cover: (1) fast TTS, contrasting parallel non-autoregressive generation with lightweight model design and domain-knowledge speedup techniques; (2) low-resource TTS, covering self-supervised pre-training, cross-lingual transfer, and data mining; (3) robust TTS, analysing alignment learning failures and the shift from attention mechanisms to explicit duration prediction; (4) expressive TTS, cataloguing explicit label conditioning, reference encoder approaches, VAE and flow-based variation modeling, and disentanglement methods across information granularities from frame to utterance level; and (5) adaptive TTS, covering few-shot, zero-shot, and untranscribed data adaptation scenarios." confidence: high relevance: low - claim_id: diffusion_based_vocoders_achieve_high_perceptual_quality_but role: supports claim: "Diffusion-based vocoders achieve high perceptual quality but suffer from slow inference due to iterative reverse diffusion, motivating step-reduction methods." source: "§2.4, Table 6" evidence: "GAN-based vocoders are non-autoregressive with O(1) inference; flow and diffusion models require O(T) iterations proportional to the number of sampling steps." confidence: high relevance: low - claim_id: zero_shot_speaker_adaptation_that_relies_solely_on role: supports claim: "Zero-shot speaker adaptation that relies solely on a speaker encoder without any fine-tuning produces noticeably lower voice quality than few-shot adaptation, particularly for speakers distant from the training distribution." source: "§3.6.2" evidence: "The five advanced topic chapters cover: (1) fast TTS, contrasting parallel non-autoregressive generation with lightweight model design and domain-knowledge speedup techniques; (2) low-resource TTS, covering self-supervised pre-training, cross-lingual transfer, and data mining; (3) robust TTS, analysing alignment learning failures and the shift from attention mechanisms to explicit duration prediction; (4) expressive TTS, cataloguing explicit label conditioning, reference encoder approaches, VAE and flow-based variation modeling, and disentanglement methods across information granularities from frame to utterance level; and (5) adaptive TTS, covering few-shot, zero-shot, and untranscribed data adaptation scenarios." confidence: high relevance: low - claim_id: low_resource_tts_benefits_from_cross_lingual_transfer role: supports claim: "Low-resource TTS benefits from cross-lingual transfer because human languages share vocal organs and phonetic structures, though phoneme set mismatches require explicit bridging." source: "§3.3" evidence: "The five advanced topic chapters cover: (1) fast TTS, contrasting parallel non-autoregressive generation with lightweight model design and domain-knowledge speedup techniques; (2) low-resource TTS, covering self-supervised pre-training, cross-lingual transfer, and data mining; (3) robust TTS, analysing alignment learning failures and the shift from attention mechanisms to explicit duration prediction; (4) expressive TTS, cataloguing explicit label conditioning, reference encoder approaches, VAE and flow-based variation modeling, and disentanglement methods across information granularities from frame to utterance level; and (5) adaptive TTS, covering few-shot, zero-shot, and untranscribed data adaptation scenarios." confidence: high relevance: low limitations: - "The coverage horizon is mid-2021, which means the survey predates several paradigm shifts: neural codec-based discrete speech tokenisation, large language model-based TTS (VALL-E, NaturalSpeech 2), flow-matching acoustic models, and the instruction-conditioned and zero-shot capabilities that came with LLM-scale training." - "The survey's taxonomy (acoustic model + vocoder pipeline) is also less applicable to fully codec-native systems where this separation dissolves." caveats: - "The survey predates the neural-codec LM paradigm and provides taxonomy rather than direct codec-TTS evidence." - id: "2209.03143" published_date: "2022-09-07" entry_date: "2026-07-25" year: 2022 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - historical_context current_role: foundational method_family: [semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: combining_self_supervised_semantic_tokens_with_codec_acoustic role: supports claim: "Combining self-supervised semantic tokens with codec acoustic tokens in a hierarchical language model resolves the quality-versus-coherence tension that affects single-tokenizer audio language models." source: "§III-B, §III-C, Table I" evidence: "AudioLM treats audio generation as a language modeling problem over a hybrid vocabulary of discrete tokens, and solves the quality-versus-coherence tension by separating the representation into two complementary token streams. **Semantic tokens** are extracted by applying k-means clustering (K=1024) to the intermediate representations of w2v-BERT XL (0.6B parameters), a self-supervised model trained with contrastive and masked language modeling objectives." confidence: high relevance: high - claim_id: semantic_and_acoustic_tokens_in_speech_carry_complementary role: supports claim: "Semantic and acoustic tokens in speech carry complementary information: semantic tokens primarily encode linguistic content and prosody, while acoustic tokens primarily encode speaker identity and recording conditions." source: "§IV-C, §IV-D, Tables II–III" evidence: "This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)*" confidence: high relevance: high - claim_id: autoregressive_language_modeling_over_discrete_audio_tokens_can role: supports claim: "Autoregressive language modeling over discrete audio tokens can produce speech continuations indistinguishable from real speech to human listeners in an unpaired forced-choice test." source: "§IV-G" evidence: "This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)*" confidence: high relevance: high - claim_id: the_semantic_to_acoustic_hierarchical_generation_pattern_transfers role: supports claim: "The semantic-to-acoustic hierarchical generation pattern transfers across audio domains: a model trained on piano music without symbolic notation also benefits from the two-tier tokenization." source: "§IV-I" evidence: "This shows semantic tokens faithfully carry linguistic content through the generation pipeline. *(§IV-C, Table II)* **Disentanglement of content and speaker:** Speaker classification accuracy on acoustic generations conditioned on the same semantic tokens is 3.2% (versus 100% for SoundStream reconstructions), confirming that speaker identity is encoded in the acoustic tokens and not the semantic tokens. *(§IV-D, Table III)* **Speaker-consistent continuation:** When continuing 3-second prompts from unseen speakers in LibriSpeech test-clean, the speaker classifier identifies the original speaker in 92.6% of generated continuations, demonstrating that prompting with both semantic and acoustic tokens preserves voice identity. *(§IV-F, Table III)* **Human perception test:** In a forced-choice listening test where raters judged whether a 7-second continuation was real or synthesized (first 3 seconds known-real), raters achieved only 51.2% accuracy, which a binomial test (p=0.23) confirms is not statistically distinguishable from chance. *(§IV-G)* **Linguistic knowledge:** On the ZeroResource Challenge 2021 benchmarks, AudioLM achieves 71.5% sWUGGY (all) / 83.7% (in-vocab) and 64.7% sBLIMP, improving over GSLM (68.7% in-vocab, 57.1% sBLIMP) and surpassing non-causal models and even a supervised forced-alignment topline on sBLIMP. *(§IV-E, Table IV)* **Piano continuation:** A model retrained on 40k hours of internal piano data generates continuations preferred over acoustic-token-only baselines in 83.3% of pairs in a subjective test, demonstrating the framework generalises beyond speech. *(§IV-I)* **Anti-spoofing:** A companion classifier trained to detect AudioLM-generated speech achieves 98.6% accuracy, despite human listeners being at chance. *(§IV-H)*" confidence: high relevance: high - claim_id: self_supervised_speech_representations_trained_with_masked_language role: supports claim: "Self-supervised speech representations trained with masked language modeling objectives encode sufficient lexical and syntactic information to outperform earlier causal spoken language models on zero-resource linguistic benchmarks." source: "§IV-E, Table IV" evidence: "AudioLM treats audio generation as a language modeling problem over a hybrid vocabulary of discrete tokens, and solves the quality-versus-coherence tension by separating the representation into two complementary token streams. **Semantic tokens** are extracted by applying k-means clustering (K=1024) to the intermediate representations of w2v-BERT XL (0.6B parameters), a self-supervised model trained with contrastive and masked language modeling objectives." confidence: high relevance: high limitations: - "AudioLM is a continuation model only: it generates continuations of an audio prompt but cannot synthesise speech from a specified transcript." - "The paper explicitly frames TTS integration (encoder-decoder with text conditioning) as future work, which means the WER/CER results reflect acoustic fidelity to a given semantic token sequence, not instruction-following capability." caveats: - "AudioLM is an audio-continuation model rather than text-conditioned TTS, but establishes hierarchical semantic/acoustic token generation." - id: "2210.13438" published_date: "2022-10-24" entry_date: "2026-07-25" year: 2022 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure] claims: - claim_id: a_multi_scale_stft_discriminator_alone_is_sufficient role: supports claim: "A multi-scale STFT discriminator alone is sufficient for perceptual quality in neural audio codecs, removing the need for waveform-domain discriminators." source: "§4.5.1, Table 2" evidence: "A key training contribution is the multi-scale STFT (MS-STFT) discriminator, replacing the MSD+Mono-STFT combination from SoundStream." confidence: high relevance: high - claim_id: gradient_balancers_that_normalise_loss_contributions_by_expected role: supports claim: "Gradient balancers that normalise loss contributions by expected gradient magnitude substantially stabilise training when combining reconstruction, adversarial, and commitment losses with widely varying natural scales." source: "§3.4, Table A.4" evidence: "The training objective combines L1 time-domain loss, multi-scale mel-spectrogram loss, adversarial loss, feature matching loss, and RVQ commitment loss." confidence: high relevance: low - claim_id: residual_vector_quantization_supports_variable_bitrate_operation_from role: supports claim: "Residual vector quantization supports variable-bitrate operation from a single model by varying the number of active codebooks at inference, with each additional codebook yielding diminishing quality returns." source: "§3.2, Table 1" evidence: "The RVQ layer uses up to 32 codebooks of 1024 entries each (10 bits per codebook), supporting multiple target bandwidths (1.5, 3, 6, 12, 24 kbps at 24 kHz) by selecting a variable number of codebooks during training." confidence: high relevance: low - claim_id: auxiliary_transformer_language_models_over_rvq_codes_can role: supports claim: "Auxiliary Transformer language models over RVQ codes can reduce effective bitrate by 25-40% through entropy coding without perceptual quality degradation, at the cost of increased latency." source: "§3.3, §4.5" evidence: "Optionally, a 5-layer Transformer language model trained over RVQ codes enables arithmetic entropy coding, reducing effective bitrate by 25-40% without perceptual quality degradation." confidence: high relevance: high - claim_id: neural_audio_codecs_outperform_traditional_dsp_codecs_at role: supports claim: "Neural audio codecs outperform traditional DSP codecs at low bitrates across both speech and music domains, with the quality gap widening as bitrate decreases." source: "§4.5, Table 1, Figure 3" evidence: "For stereophonic 48 kHz music (Table 4), EnCodec at 6 kbps matches MP3 at 64 kbps (both ~82.9 MUSHRA), achieving comparable quality at 10x lower bitrate." confidence: high relevance: high limitations: - "The 48 kHz model in non-streamable configuration operates slower than real time on CPU, limiting deployment without GPU acceleration or hardware-specific optimisation." - "Arithmetic coding precision issues (floating-point non-determinism across architectures) required a probability rounding workaround that the authors note may be insufficient for practical deployment." caveats: - "EnCodec contributes the discrete acoustic representation used by later codec language models rather than a TTS generator." - id: "2301.02111" published_date: "2023-01-05" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS, codec] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: foundational method_family: [codec_tokenizer_infrastructure, hierarchical_ar_nar_codec_tts] claims: - claim_id: treating_tts_as_conditional_language_modeling_over_discrete role: supports claim: "Treating TTS as conditional language modeling over discrete codec tokens enables zero-shot speaker generalisation as in-context learning, without speaker-specific fine-tuning or engineered speaker encoders." source: "§4.1, §5.2" evidence: "VALL-E treats TTS as a conditional language modeling problem over discrete acoustic tokens." confidence: high relevance: high - claim_id: training_on_large_scale_semi_supervised_speech_data role: supports claim: "Training on large-scale semi-supervised speech data, even with noisy transcriptions and diverse acoustic conditions, yields stronger generalisation to unseen speakers than training on smaller clean corpora." source: "§1, §5.2" evidence: "On LibriSpeech test-clean (zero-shot, 3-second prompt, 40 unseen speakers), VALL-E achieves SMOS 4.38 ± 0.10 versus YourTTS's 3.45 ± 0.09, a +0.93 gap that closes most of the distance to ground truth (4.50 ± 0.10)." confidence: high relevance: medium - claim_id: the_hierarchical_structure_of_residual_vector_quantization_supports role: supports claim: "The hierarchical structure of residual vector quantization supports a two-stage AR+NAR generation pipeline in which first-codebook tokens carry speaker identity and subsequent codebooks refine fine acoustic detail." source: "§4.2" evidence: "Audio is tokenised using EnCodec, a convolutional encoder-decoder with residual vector quantization (RVQ): the encoder produces embeddings at 75 Hz, and each frame is modelled by eight hierarchical quantizers with 1024 entries each, corresponding to a 6K bitrate." confidence: high relevance: high - claim_id: speaker_similarity_in_zero_shot_codec_tts_improves role: supports claim: "Speaker similarity in zero-shot codec TTS improves monotonically with acoustic prompt length, suggesting that speaker identity modelling does not saturate within a few seconds." source: "§5.3, Table 6" evidence: "Speaker similarity improves monotonically with longer prompts, and the gap between systems widens when YourTTS is evaluated on its truly unseen 11 speakers (Table 6)." confidence: high relevance: high - claim_id: stochastic_sampling_in_autoregressive_codec_generation_introduces_output role: supports claim: "Stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation." source: "§4.3, §5.4" evidence: "The paper presents evidence that stochastic sampling in autoregressive codec generation introduces output diversity — varying speech rate, prosody, and accent realisation — that is both a feature for data augmentation and a complication for deterministic evaluation." confidence: high relevance: high limitations: - "Synthesis robustness is a material constraint: the autoregressive first-stage LM exhibits attention alignment failures that cause word deletions, insertions, and repetitions." - "WER on LibriSpeech test-clean is 5.9%, nearly three times the ground-truth rate of 2.2%." caveats: - "VALL-E directly establishes autoregressive neural-codec language modeling for zero-shot TTS." - id: "2301.11325" published_date: "2023-01-26" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [] architecture: [autoregressive-LM] relevance: medium evidence_role: - architecture_variant current_role: influential method_family: [semantic_acoustic_token_cascade] claims: - claim_id: hierarchical_autoregressive_modeling_over_semantic_and_acoustic_tokens role: supports claim: "Hierarchical autoregressive modeling over semantic and acoustic tokens enables long-form music generation (several minutes) with temporal coherence at 24 kHz." source: "§3.2, §6" evidence: "An ablation removing the semantic modeling stage shows that generating acoustic tokens directly from MuLan tokens preserves FAD scores but degrades KLD (1.01 to 1.05) and MCC (0.51 to 0.49), confirming that the semantic intermediary improves text adherence and long-term structure." confidence: high relevance: high - claim_id: a_joint_audio_text_embedding_space_can_substitute role: supports claim: "A joint audio-text embedding space can substitute for paired text-audio supervision at training time, allowing generative models to be trained on audio-only corpora and conditioned on text at inference." source: "§3.1, §4.2" evidence: "The system uses three independently pretrained and frozen models: SoundStream (a neural audio codec operating at 24 kHz, 6 kbps, with 12-level RVQ yielding 600 acoustic tokens per second), w2v-BERT (a 600M-parameter self-supervised model whose intermediate representations are discretized into 25 semantic tokens per second via k-means clustering), and MuLan (a contrastive music-text joint embedding model trained on weakly paired music-text data)." confidence: high relevance: low - claim_id: semantic_token_intermediaries_improve_adherence_to_text_descriptions role: supports claim: "Semantic token intermediaries improve adherence to text descriptions in hierarchical audio generation beyond what direct acoustic token prediction achieves." source: "§5, Table 1" evidence: "An ablation removing the semantic modeling stage shows that generating acoustic tokens directly from MuLan tokens preserves FAD scores but degrades KLD (1.01 to 1.05) and MCC (0.51 to 0.49), confirming that the semantic intermediary improves text adherence and long-term structure." confidence: high relevance: high - claim_id: for_text_conditioned_music_generation_perceptual_audio_quality role: supports claim: "For text-conditioned music generation, perceptual audio quality (FAD) and semantic text alignment (MCC, KLD) are complementary evaluation axes that do not always correlate with each other." source: "§4.4, Table 1" evidence: "The system uses three independently pretrained and frozen models: SoundStream (a neural audio codec operating at 24 kHz, 6 kbps, with 12-level RVQ yielding 600 acoustic tokens per second), w2v-BERT (a 600M-parameter self-supervised model whose intermediate representations are discretized into 25 semantic tokens per second via k-means clustering), and MuLan (a contrastive music-text joint embedding model trained on weakly paired music-text data)." confidence: high relevance: low - claim_id: large_autoregressive_audio_lms_trained_on_extensive_unlabeled role: supports claim: "Large autoregressive audio LMs trained on extensive unlabeled corpora memorize only a small fraction of training sequences exactly, but approximate semantic matches affect a higher proportion of generated outputs under targeted prompting." source: "§5, Figure 3" evidence: "Memorization analysis shows exact token matches below 0.2% even when prompting with 10 seconds of training audio, and approximate matches (based on optimal transport between token histograms) below 1%, with detected approximate matches largely corresponding to low-entropy, repetitive token sequences." confidence: high relevance: high limitations: - "MCC, one of the two primary text-adherence metrics, is computed using MuLan itself, the same model used for conditioning." - "This circularity biases the metric in MusicLM's favour relative to baselines that do not use MuLan representations." caveats: - "MusicLM validates hierarchical codec-token autoregression in music, providing adjacent rather than speech-specific evidence." - id: "2303.03926" published_date: "2023-03-07" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: [scaled_general_ar_codec_tts] claims: - claim_id: large_scale_multilingual_codec_language_models_can_transfer role: supports claim: "Large-scale multilingual codec language models can transfer speaker identity, emotion, and acoustic environment across languages from a single source utterance without paired bilingual data." source: "§3, §5.3, Table 3" evidence: "A multilingual autoregressive codec language model (φ_MAR) is a 12-layer Transformer decoder (attention dimension 1024, FFN dimension 4096) that generates first-layer acoustic tokens from a concatenated prompt of source-language phonemes, target-language phonemes, and source first-layer acoustic tokens." confidence: high relevance: high - claim_id: language_id_conditioning_is_essential_for_native_sounding role: supports claim: "Language ID conditioning is essential for native-sounding accent in cross-lingual codec TTS: removing it significantly degrades accent scores even while modestly improving speaker similarity." source: "§5.5, Table 6" evidence: "Ablation of the language ID module confirms its importance: removing it drops accent scores from 4.10 to 2.98 (scale 1-5, higher is more native-sounding) and degrades translation quality by roughly 1.5 BLEU (Table 6)." confidence: high relevance: high - claim_id: in_context_learning_with_acoustic_token_prompts_provides role: supports claim: "In-context learning with acoustic token prompts provides stronger cross-lingual voice preservation than speaker embedding approaches across both TTS and speech-to-speech translation tasks." source: "§5.3, §5.4, Tables 3, 5" evidence: "On zero-shot cross-lingual TTS (English generation prompted by Chinese speech), VALL-E X substantially outperforms YourTTS across all metrics: speaker similarity improves from 0.30 to 0.36 ASV-Score, WER drops from 8.53 to 4.07, and NISQA naturalness rises from 3.36 to 3.54 (Table 2)." confidence: high relevance: high - claim_id: the_ar_nar_two_stage_codec_language_model role: supports claim: "The AR/NAR two-stage codec language model architecture extends naturally to cross-lingual generation by treating bilingual phoneme sequences as concatenated prompts." source: "§3.2, §3.4" evidence: "VALL-E X treats cross-lingual speech synthesis as a conditional multilingual language modelling task over discrete acoustic tokens, extending the VALL-E architecture to handle bilingual prompts." confidence: high relevance: high limitations: - "Evaluations cover only English and Chinese, and test sets are small (40 speakers, 1373 samples for TTS; 14 speakers, 350 utterances for S2ST on EMIME)." - "Generalisation to typologically distant language pairs, more than two languages, or lower-resource settings is untested." caveats: - "The system directly extends codec language-model TTS to cross-lingual speaker and accent transfer." - id: "2305.02765" published_date: "2023-05-04" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [codec] architecture: [GAN, VAE] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: grouping_residual_vector_quantization_into_parallel_chains_rather role: supports claim: "Grouping residual vector quantization into parallel chains rather than a single sequential chain improves reconstruction quality per codebook, enabling competitive fidelity with fewer total quantizers." source: "§3.3, Table 1" evidence: "The key innovation is the group-residual vector quantization (GRVQ) module: rather than applying a single RVQ chain to z, GRVQ first splits z into two groups (z1, z2), then applies independent RVQ chains to each group in parallel." confidence: high relevance: low - claim_id: the_burden_that_codec_codebook_count_imposes_on role: supports claim: "The burden that codec codebook count imposes on downstream generation models is a practical constraint that drives codec architecture choices independently of raw reconstruction quality." source: "§1 Introduction, §3.3" evidence: "The 4-codebook HiFi-Codec thus outperforms the 12-codebook EnCodec on PESQ while matching on STOI, representing a 3x reduction in codebook count at equivalent or better reconstruction quality." confidence: high relevance: high - claim_id: objective_speech_quality_metrics_such_as_pesq_and role: supports claim: "Objective speech quality metrics such as PESQ and STOI are insufficient alone to characterise codec reconstruction quality, and subjective evaluation is necessary but often omitted in codec research." source: "§6 Limitations" evidence: "Evaluated on objective speech quality metrics (PESQ and STOI), HiFi-Codec with 4 codebooks at 24kHz (downsample factor 240) achieves PESQ 3.63 / STOI 0.95, compared to Facebook's EnCodec at 8 codebooks (PESQ 3.01 / STOI 0.94) and 12 codebooks (PESQ 3.21 / STOI 0.95)." confidence: high relevance: high - claim_id: publicly_available_training_code_and_pre_trained_baselines role: supports claim: "Publicly available training code and pre-trained baselines for neural audio codecs are necessary for reproducible research, as previously these were unavailable for EnCodec and SoundStream." source: "§5 Conclusion, §6 Limitations" evidence: "The authors also release AcademiCodec, an open-source toolkit providing training code and pre-trained models for EnCodec, SoundStream, and HiFi-Codec." confidence: high relevance: high limitations: - "No subjective evaluation is included." - "The paper's own §6 acknowledges this as a limitation: \"Subjective evaluation is always the best choice, but this part is missed in this study.\" All quality comparisons rest solely on PESQ and STOI, which the authors themselves note may not accurately reflect perceptual quality." caveats: - "HiFi-Codec targets the codec representation and downstream token burden rather than performing TTS itself." - id: "2305.07243" published_date: "2023-05-12" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, diffusion, VAE] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, continuous_token_autoregression] claims: - claim_id: conditioning_a_diffusion_decoder_on_the_continuous_latent role: supports claim: "Conditioning a diffusion decoder on the continuous latent activations of an autoregressive model rather than its discrete token outputs substantially improves output quality in a cascaded AR-diffusion TTS pipeline." source: "§2.2.2, Appendix B.4" evidence: "The \"TorToise Trick\" is the most consequential design choice: after training the diffusion decoder on discrete VQVAE tokens to convergence, it is fine-tuned to decode the final hidden-state activations of the autoregressive model rather than the discrete tokens." confidence: high relevance: high - claim_id: contrastive_re_ranking_of_multiple_autoregressive_candidates_using role: supports claim: "Contrastive re-ranking of multiple autoregressive candidates using a text-speech discriminator measurably improves the final output quality of a TTS system without requiring the expensive decoder to process every candidate." source: "§2.3, §4" evidence: "Third, CLVP (Contrastive Language-Voice Pretrained Transformer) is a CLIP-style dual encoder trained contrastively on text/speech pairs; it scores autoregressive output candidates during re-ranking before the expensive diffusion step, allowing many candidates to be filtered cheaply." confidence: high relevance: high - claim_id: applying_image_generation_scaling_techniques_large_scale_self role: supports claim: "Applying image-generation scaling techniques (large-scale self-supervised data, generalist transformer architectures, multi-stage AR-then-diffusion generation) to speech synthesis yields high-expressiveness multi-speaker TTS even when trained by a single researcher on commodity hardware." source: "§7" evidence: "Second, a GPT-2-style autoregressive decoder (30 transformer layers, dim 1024, 16 heads) is trained to predict sequences of these speech tokens conditioned on text and a speaker conditioning vector." confidence: high relevance: medium - claim_id: building_a_large_scale_tts_training_corpus_by role: supports claim: "Building a large-scale TTS training corpus by scraping and filtering internet audio (audiobooks, podcasts) with automatic transcription is a viable path to tens-of-thousands-of-hours datasets without manual labelling." source: "§5, Appendix A" evidence: "The extended dataset of 49,000 hours was self-assembled from audiobooks and podcasts, transcribed with a fine-tuned wav2vec2-large model that was adapted to predict punctuation." confidence: high relevance: medium limitations: - "No formal listening test or MOS table is reported." - "The primary quality claim rests on informal sample comparisons; the paper's own evaluation suite (CLVP-FID) is not a standard benchmark." caveats: - "The paper directly studies scale in autoregressive speech-token generation." - id: "2305.09636" published_date: "2023-05-16" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - acceleration_evidence current_role: influential method_family: [hierarchical_ar_nar_codec_tts, semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: parallel_iterative_masked_decoding_adapted_to_rvq_structure role: supports claim: "Parallel, iterative masked decoding adapted to RVQ structure enables acoustic token generation two orders of magnitude faster than autoregressive generation at matched perceptual quality." source: "§4.3, Figure 3" evidence: "The runtime gain is two orders of magnitude: SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4 versus AudioLM's acoustic stages which scale prohibitively with sequence length." confidence: high relevance: high - claim_id: non_autoregressive_rvq_level_by_level_decoding_maintains role: supports claim: "Non-autoregressive RVQ-level-by-level decoding maintains better voice and acoustic consistency over long sequences than autoregressive chunk-and-prompt approaches." source: "§4.2, Table 1, Figure 2" evidence: "On long continuations, SoundStorm maintains higher prompt-to-generation acoustic consistency than AudioLM's slide-and-prompt chunking strategy." confidence: high relevance: high - claim_id: fine_level_rvq_tokens_are_conditionally_independent_given role: supports claim: "Fine-level RVQ tokens are conditionally independent given coarser tokens and can be decoded greedily in a single pass without measurable quality loss." source: "§3.3, §4.3" evidence: "For subsequent levels, greedy decoding in a single pass suffices, exploiting the conditional independence of fine-level tokens given coarser ones." confidence: high relevance: high - claim_id: confidence_based_iterative_decoding_provides_a_meaningful_quality role: supports claim: "Confidence-based iterative decoding provides a meaningful quality gain over greedy decoding at the coarsest RVQ level, but additional iterations at finer levels yield no significant improvement for speech." source: "§4.3, Figure 4" evidence: "An ablation on decoding iterations (Figure 4) shows that 16 iterations in the first RVQ level provides approximately 0.1-0.2 quality gain over greedy decoding; additional iterations on finer levels yield no statistically significant improvement, validating the greedy-at-fine-levels strategy." confidence: high relevance: medium - claim_id: coupling_a_text_to_semantic_token_model_with role: supports claim: "Coupling a text-to-semantic token model with an efficient acoustic generator enables real-time synthesis of controllable multi-speaker dialogue at 30-second horizons." source: "§5" evidence: "SoundStorm is a 350M-parameter Conformer trained on the acoustic generation task, taking as conditioning the time-aligned semantic tokens from AudioLM (originating from w2v-BERT quantised with k-means) and producing SoundStream codec tokens (50 frames/second, 12 RVQ levels, codebook size 1024 per level)." confidence: high relevance: high limitations: - "Evaluation uses a DNSMOS-style estimator rather than human listening tests for audio quality comparisons, and the subjective baseline is carried over from earlier AudioLM papers rather than re-run." - "Direct perceptual comparisons between SoundStorm and AudioLM on the same conditions by human raters are not reported." caveats: - "SoundStorm replaces slow acoustic-token autoregression with parallel masked generation while retaining an autoregressive semantic stage." - id: "2305.11000" published_date: "2023-05-18" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [unified_speech_text_language_model] claims: - claim_id: expanding_an_llm_s_token_vocabulary_with_discretised role: supports claim: "Expanding an LLM's token vocabulary with discretised speech units enables a single model to perform both speech comprehension and speech generation without a cascade pipeline." source: "§4.1" evidence: "Stage 1 (modality-adaptation pre-training) trains LLaMA-13B on LibriLight (60K hours) with a next-token prediction objective over discrete unit sequences, teaching the model to handle the speech modality at scale (96 A100 GPUs, 900 steps)." confidence: high relevance: high - claim_id: a_multi_stage_training_curriculum_separating_modality_adaptation role: supports claim: "A multi-stage training curriculum, separating modality adaptation, cross-modal instruction tuning, and chain-of-modality alignment, is necessary to acquire reliable cross-modal instruction-following from an LLM backbone." source: "§4.2" evidence: "Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability." confidence: high relevance: medium - claim_id: the_chain_of_modality_pattern_generating_a_text role: supports claim: "The chain-of-modality pattern, generating a text intermediate before the speech response, is a practical mechanism for transferring LLM reasoning capability to speech output." source: "§3.2, §4.2" evidence: "This chain-of-modality pattern serialises cross-modal reasoning as an explicit text intermediate, mitigating the need for direct speech-to-speech reasoning." confidence: high relevance: medium - claim_id: large_scale_instruction_dataset_construction_via_gpt_4 role: supports claim: "Large-scale instruction dataset construction via GPT-4-generated task descriptions applied to existing ASR corpora is a scalable approach to bootstrapping cross-modal training data." source: "§3.1" evidence: "Stage 2 (cross-modal instruction fine-tuning) trains on SpeechInstruct, a dataset the authors construct by discretising ASR corpora into unit-text pairs and generating 100 diverse task instructions per task type using GPT-4; text instruction data from moss-002-sft is mixed in to preserve language capability." confidence: high relevance: medium limitations: - "The paper provides no quantitative evaluation: no MOS, WER, or speaker similarity scores are reported, and no comparison to cascade baselines is made." - "All results are case studies." caveats: - "SpeechGPT applies discrete speech-unit language modeling to spoken conversational generation." - id: "2306.00814" published_date: "2023-06-01" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: maintaining_constant_temporal_resolution_throughout_a_gan_vocoder role: supports claim: "Maintaining constant temporal resolution throughout a GAN vocoder, with ISTFT as the sole upsampling step, eliminates aliasing artefacts and dramatically reduces inference cost without sacrificing perceptual quality." source: "§3.1, §4.3, Table 6" evidence: "Vocos is a GAN vocoder with an isotropic architecture: the temporal resolution of the feature sequence remains constant throughout the network, matching the hop-size resolution of the STFT." confidence: high relevance: low - claim_id: implicit_phase_wrapping_via_a_unit_circle_activation role: supports claim: "Implicit phase wrapping via a unit-circle activation is essential for stable GAN training of complex-valued spectrogram generators; alternatives that clamp or clip phase angles substantially degrade output quality." source: "§3.2, §4.1.1, Table 1" evidence: "Phase is handled by a unit-circle activation: the raw phase output p is mapped via cosine and sine to obtain a complex coefficient STFT = M * (cos(p) + j*sin(p)), implicitly enforcing phase wrapping into (-pi, pi] without requiring a clipped nonlinearity." confidence: high relevance: low - claim_id: fourier_domain_vocoders_reduce_periodicity_errors_more_effectively role: supports claim: "Fourier-domain vocoders reduce periodicity errors more effectively than time-domain GANs, suggesting that modelling harmonics in the frequency domain provides a stronger inductive bias for voiced speech." source: "§4.1.1, Table 1" evidence: "In objective metrics, Vocos leads all baselines on VISQOL (4.66) and PESQ (3.70), and achieves the best periodicity score (0.101), indicating fewer harmonic artefacts than time-domain GANs (Table 1)." confidence: high relevance: low - claim_id: convnext_blocks_with_isotropic_architecture_outperform_dilated_resblocks role: supports claim: "ConvNeXt blocks with isotropic architecture outperform dilated ResBlocks in the Fourier-domain vocoder setting, even though dilated convolutions were motivated by the need to expand receptive fields in time-domain models." source: "§4.1.1, Table 1" evidence: "Ablations show that the unit-circle phase activation is essential (removing it reduces PESQ from 3.70 to 3.57), ConvNeXt blocks outperform standard ResBlocks (PESQ 3.70 vs 3.53), and Snake activations (used in BigVGAN) do not help in the Fourier-domain setting (Table 1)." confidence: high relevance: low - claim_id: a_fourier_domain_gan_vocoder_trained_as_a role: supports claim: "A Fourier-domain GAN vocoder trained as a neural codec decoder can substantially improve perceptual quality over the original codec decoder across all bitrates without architectural changes to the upstream codec." source: "§4.2, Table 5" evidence: "For the EnCodec decoder variant, Vocos substantially outperforms the original EnCodec decoder in perceptual quality: at 12 kbps, Vocos scores MOS 4.00 vs EnCodec's 3.08; at 1.5 kbps the gap is even wider (2.73 vs 1.09) (Table 5)." confidence: high relevance: high limitations: - "Vocos's mel-spectrogram MOS scores are reported on LibriTTS using crowd-sourced listeners; the ground-truth MOS (3.81) is noticeably below what might be expected for studio speech, suggesting the evaluation pool or headphone compliance filtering may limit the discriminative power of the subjective test." - "Statistical equivalence with BigVGAN is shown, but the test may be underpowered for detecting small differences." caveats: - "Vocos is a codec/vocoder decoder contribution rather than an autoregressive token generator." - id: "2306.12925" published_date: "2023-06-22" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: initializing_a_speech_text_llm_from_a_pretrained role: supports claim: "Initializing a speech-text LLM from a pretrained text-only checkpoint substantially outperforms training from scratch at equivalent model scale." source: "§5.4.2, Table 6" evidence: "At the same 8B model scale, fine-tuning from a pretrained PaLM-2 checkpoint substantially outperforms training the speech-text model from scratch on the reported translation evaluation." confidence: high relevance: medium - claim_id: audio_tokenizer_quality_is_a_primary_bottleneck_in role: supports claim: "Audio tokenizer quality is a primary bottleneck in LLM-based speech generation and understanding: stronger semantic tokenizers yield large downstream gains independent of LM scale." source: "§5.4.3, Table 7" evidence: "Ablation over tokenizers shows a large gap: USM-v2 tokens reach 30.5 BLEU on CoVoST2 AST while w2v-BERT tokens yield only 15.2 BLEU at the same model scale, establishing audio tokenizer quality as the primary performance bottleneck (§5.4.3, Table 7)." confidence: high relevance: high - claim_id: a_unified_multimodal_vocabulary_that_interleaves_text_and role: supports claim: "A unified multimodal vocabulary that interleaves text and audio tokens enables zero-shot speech translation to language pairs not seen during speech training, by inheriting translation capability from text pretraining." source: "§5.2, Table 3" evidence: "Zero-shot AST on FLEURS shows AudioPaLM-2 achieving 20.7 BLEU on language pairs where only ASR data was seen during training, outperforming Whisper (19.6) despite Whisper having seen speech-to-text translation data for all those languages." confidence: high relevance: high - claim_id: training_on_combined_tasks_that_decompose_complex_speech role: supports claim: "Training on combined tasks that decompose complex speech operations into intermediate text steps improves performance over direct end-to-end decoding." source: "§5.4.4, Table 8" evidence: "For complex tasks, the model can also be trained on combined tasks, emitting intermediate outputs (transcription, then translation, then translated audio) in a single autoregressive pass, which mirrors chain-of-thought prompting and improves performance relative to direct decoding." confidence: high relevance: medium - claim_id: voice_identity_preservation_in_cross_lingual_speech_synthesis role: supports claim: "Voice identity preservation in cross-lingual speech synthesis can exceed high-quality TTS-based references when an audio LM is conditioned on a short spoken prompt." source: "§5.3, Table 4" evidence: "Voice conditioning is provided as a 3-second reference clip encoded as both audio tokens and SoundStream tokens, enabling cross-lingual voice transfer." confidence: high relevance: medium limitations: - "The entire system depends on the quality of the audio tokenizer, which is not released and requires access to Google-internal USM models." - "The best-performing configuration (AudioPaLM-2 with USM-v2 tokens) is not reproducible externally; the published ablations use the multilingual w2v-BERT tokenizer as the weakest condition, suggesting that reported performance at USM-v2 quality cannot be independently verified." caveats: - "AudioPaLM directly unifies text and audio token prediction for speech generation and translation." - id: "2308.16692" published_date: "2023-08-31" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS, codec] architecture: [GAN] relevance: high evidence_role: - infrastructure - core_evidence current_role: influential method_family: [codec_tokenizer_infrastructure, hierarchical_ar_nar_codec_tts, semantic_acoustic_token_cascade] claims: - claim_id: separating_semantic_content_from_paralinguistic_information_across_rvq role: supports claim: "Separating semantic content from paralinguistic information across RVQ layers within a single codec improves both reconstruction quality and speech language model coherence compared to undifferentiated acoustic tokenisation." source: "§4.4, Tables 2 and 4" evidence: "The paper then demonstrates utility by building a Unified Speech Language Model (USLM) on top of SpeechTokenizer: a 12-layer transformer autoregressive model produces first-layer (semantic) tokens from phoneme input, while a 12-layer non-autoregressive model generates layers 2-8 (paralinguistic) conditioned on the first-layer tokens and an acoustic speaker prompt." confidence: high relevance: high - claim_id: acoustic_tokens_from_standard_neural_codecs_encode_content role: supports claim: "Acoustic tokens from standard neural codecs encode content and speaker identity in an entangled form that causes systematic word errors in autoregressive language model generation." source: "§2.3, Table 3" evidence: "The paper then demonstrates utility by building a Unified Speech Language Model (USLM) on top of SpeechTokenizer: a 12-layer transformer autoregressive model produces first-layer (semantic) tokens from phoneme input, while a 12-layer non-autoregressive model generates layers 2-8 (paralinguistic) conditioned on the first-layer tokens and an acoustic speaker prompt." confidence: high relevance: high - claim_id: a_distillation_objective_computed_per_feature_dimension_d role: supports claim: "A distillation objective computed per feature dimension (D-axis) produces stronger semantic guidance to a codec's first quantizer than the conventional per-timestep (T-axis) formulation." source: "Appendix C, Table 7" evidence: "During training, a HuBERT model acts as a frozen semantic teacher guiding the first RVQ quantizer via two distillation objectives: a continuous cosine-similarity loss computed dimension-wise (not timestep-wise) between the first-layer quantized output and HuBERT layer-9 or average-layer representations, and a pseudo-label prediction loss over HuBERT unit targets." confidence: medium relevance: high - claim_id: the_first_layer_tokens_of_a_hierarchically_disentangled role: supports claim: "The first-layer tokens of a hierarchically disentangled codec can serve as a zero-shot voice conversion mechanism by swapping higher-layer tokens from a reference speaker, without requiring a separate conversion model." source: "§5.2, Table 5" evidence: "The paper then demonstrates utility by building a Unified Speech Language Model (USLM) on top of SpeechTokenizer: a 12-layer transformer autoregressive model produces first-layer (semantic) tokens from phoneme input, while a 12-layer non-autoregressive model generates layers 2-8 (paralinguistic) conditioned on the first-layer tokens and an acoustic speaker prompt." confidence: high relevance: high - claim_id: codec_tokens_trained_without_explicit_content_supervision_exhibit role: supports claim: "Codec tokens trained without explicit content supervision exhibit poor codebook utilisation and weak phoneme-code correspondence, increasing the modelling burden on downstream language models." source: "Appendix F, Table 8" evidence: "On SLMTokBench (Table 3), SpeechTokenizer RVQ-1 tokens achieve mutual information of 31.6-32.9 (depending on teacher) versus 16.5 for EnCodec RVQ-1, confirming the content-focused design." confidence: medium relevance: high limitations: - "SpeechTokenizer is trained solely on English LibriSpeech." - "While preliminary results in Appendix G suggest cross-lingual token transfer is plausible, the codec is not validated for multilingual speech language models and the text-alignment properties of RVQ-1 may not hold for typologically distant languages." caveats: - "SpeechTokenizer directly studies how codec-token structure affects downstream autoregressive speech language modeling." - id: "2310.00704" published_date: "2023-10-01" entry_date: "2026-07-25" year: 2023 venue: "arXiv" task: [TTS, VC, singing] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [hierarchical_ar_nar_codec_tts, fully_autoregressive_multicodebook, hybrid_ar_token_continuous_renderer] claims: - claim_id: training_a_single_audio_language_model_across_diverse role: supports claim: "Training a single audio language model across diverse generation tasks (TTS, voice conversion, sound synthesis, music, singing) produces consistent performance improvements over task-specific models trained on the same data." source: "§3.4.1, Appendix C.1, Table 17" evidence: "Multi-task training consistently outperforms single-task training across all 11 tasks (Appendix C.1, Table 17), confirming that cross-task sharing is beneficial rather than neutral." confidence: high relevance: high - claim_id: the_autoregressive_property_is_critical_for_audio_generation role: supports claim: "The autoregressive property is critical for audio generation quality: parallel and delay-based codec prediction approaches yield measurably lower naturalness than fully autoregressive methods when codec quantization levels are held constant." source: "§3.4.2, Tables 4–5" evidence: "With codec quantization levels held constant, fully autoregressive prediction receives higher naturalness ratings than the parallel and delayed prediction variants in the architecture ablation." confidence: high relevance: high - claim_id: hierarchical_factorisation_of_rvq_codec_token_sequences_into role: supports claim: "Hierarchical factorisation of RVQ codec token sequences into inter-frame and intra-frame modeling substantially reduces training memory and time relative to flat-sequence autoregressive prediction, with comparable generation quality." source: "§2.3, §3.4.2, Table 4" evidence: "The multi-scale Transformer matches flat-sequence autoregressive prediction in TTS quality while reducing the memory and training-time burden of modeling RVQ sequences." confidence: high relevance: high - claim_id: pre_training_on_a_broad_multi_task_audio role: supports claim: "Pre-training on a broad multi-task audio corpus enables strong adaptation to unseen audio generation tasks via fine-tuning on small datasets, outperforming task-specific models trained from scratch on those tasks." source: "§3.3, Appendix B.5–B.8, Table 17" evidence: "Training proceeds in two stages: joint pretraining on 7 tasks (165K hours), followed by fine-tuning to add 4 new tasks (audio editing, speech dereverberation, instructed TTS, speech editing)." confidence: high relevance: medium - claim_id: signal_level_metrics_such_as_pesq_are_poorly role: supports claim: "Signal-level metrics such as PESQ are poorly suited for evaluating generative audio models: systems achieving higher perceptual MOS scores routinely score lower on PESQ than discriminative baselines." source: "§3.2, §3.4.2, Table 11" evidence: "On speech enhancement, UniAudio obtains higher DNSMOS and human MOS than the compared systems while its PESQ remains below discriminative baselines." confidence: high relevance: medium limitations: - "Model checkpoints are not released due to misuse concerns, limiting reproducibility." - "Only code and demos are public." caveats: - "UniAudio directly evaluates autoregressive codec modeling across TTS and other audio-generation tasks." - id: "2401.07333" published_date: "2024-01-14" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: active_evidence method_family: [alignment_constrained_autoregression] claims: - claim_id: interleaving_phoneme_tokens_with_their_corresponding_acoustic_frames role: supports claim: "Interleaving phoneme tokens with their corresponding acoustic frames in the training sequence substantially reduces phoneme-level alignment errors in autoregressive codec LM TTS, including repetitions, transpositions, and omissions." source: "§3.2, §4.2, Table 3" evidence: "ELLA-V reorders the hybrid token sequence so that phonemes are interleaved with their corresponding acoustic tokens rather than prepended en bloc." confidence: high relevance: high - claim_id: autoregressive_codec_language_models_that_concatenate_all_phoneme role: supports claim: "Autoregressive codec language models that concatenate all phoneme tokens ahead of all acoustic tokens are prone to infinite-silence generation, with failure rates exceeding 80% under greedy decoding." source: "§1, Table 1" evidence: "The stability advantage is especially pronounced under near-greedy decoding: VALL-E's infinite-silence rate (INF%) reaches 87% at greedy decoding, while ELLA-V's forced-truncation rate (CUT%) remains low across the entire top-p range." confidence: high relevance: high - claim_id: explicit_forced_alignment_supervision_at_training_time_enables role: supports claim: "Explicit forced-alignment supervision at training time enables fine-grained phoneme-level control at inference, allowing deterministic truncation of abnormal synthesis and making greedy decoding viable." source: "§3.2.1, §3.3, Figure 5" evidence: "The stability advantage is especially pronounced under near-greedy decoding: VALL-E's infinite-silence rate (INF%) reaches 87% at greedy decoding, while ELLA-V's forced-truncation rate (CUT%) remains low across the entire top-p range." confidence: high relevance: medium - claim_id: structural_alignment_constraints_in_the_token_sequence_provide role: supports claim: "Structural alignment constraints in the token sequence provide larger accuracy gains than naturalness or speaker similarity improvements, suggesting that intelligibility and speaker identity are relatively easy to preserve while alignment robustness remains the primary challenge." source: "§4.2, Table 2" evidence: "The alignment-guided sequence reordering produces its largest improvement in recognition accuracy, while speaker-similarity and naturalness changes are comparatively smaller." confidence: high relevance: high limitations: - "Results are compared only against a VALL-E baseline reproduced on LibriSpeech 960h, not against the original VALL-E trained on 60k hours of LibriLight." - "The restricted training data limits direct comparison to the published VALL-E numbers and leaves open whether the gains hold at scale." caveats: - "ELLA-V directly addresses alignment failures in autoregressive codec-language-model TTS." - id: "2402.01912" published_date: "2024-02-02" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - control_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: automatic_acoustic_labeling_can_substitute_for_human_annotations role: supports claim: "Automatic acoustic labeling can substitute for human annotations in training large-scale instruction-conditioned speech language models without a loss in attribute control accuracy relative to human-labeled systems." source: "§3.1, §3.2, §4.1" evidence: "Objective attribute control is validated using the same automatic classifiers that labeled the training data." confidence: high relevance: high - claim_id: including_a_small_proportion_of_high_fidelity_audio role: supports claim: "Including a small proportion of high-fidelity audio (approximately 1%) in a predominantly noisy training corpus, combined with explicit recording-quality labels, enables a speech LM to generate professional-sounding speech on demand from text prompts alone." source: "§3.1.2, §4.2, Table 1" evidence: "High audio fidelity is achieved through a combination of: (1) using DAC rather than EnCodec, which the authors report provides subjectively and objectively superior audio quality, and (2) including LibriTTS-R in training, which provides approximately 500 hours (roughly 1%) of clean, professionally enhanced speech alongside the crowd-sourced data." confidence: high relevance: high - claim_id: the_choice_of_neural_audio_codec_has_a role: supports claim: "The choice of neural audio codec has a measurable effect on perceptual audio quality in autoregressive TTS; higher-fidelity codecs translate directly to higher MOS and objective quality scores." source: "§3.3, §4.2, Table 1–2" evidence: "On samples prompted for excellent recording quality, the higher-fidelity codec system outperforms Audiobox on PESQ, STOI, and SI-SDR and approaches the ground-truth values." confidence: high relevance: high - claim_id: natural_language_conditioning_on_accent_can_be_achieved role: supports claim: "Natural language conditioning on accent can be achieved in a single TTS model covering dozens of accents, though classifier accuracy reflects the noise and imbalance inherent in automatic accent labeling of crowd-sourced data." source: "§3.1.1, §4.1" evidence: "A large language model (Stable Beluga 2) then converts keyword tuples into natural language sentences, e.g. \"a woman with a deep voice speaking slowly and somewhat monotonously with a Hungarian accent in an echoey room with background noise.\" This approach is applied to the 45k-hour English Multilingual LibriSpeech corpus and the smaller 585-hour LibriTTS-R corpus (included for its higher audio fidelity from the Miipher speech-enhancement model)." confidence: high relevance: medium limitations: - "The evaluation compares only against Audiobox." - "No standard TTS baselines (reference-based zero-shot systems, encoder-decoder models) are included, making it impossible to assess whether the MOS gains arise from the conditioning approach, the codec choice, or the training data mix." caveats: - "The system directly uses autoregressive codec modeling for language-guided expressive TTS." - id: "2402.05755" published_date: "2024-02-08" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [unified_speech_text_language_model] claims: - claim_id: continuous_pretraining_of_a_text_llm_on_interleaved role: supports claim: "Continuous pretraining of a text LLM on interleaved speech and text tokens transfers the text model's few-shot learning and semantic reasoning abilities to the speech modality." source: "§4.2, §4.3, Table 4" evidence: "Crucially, Spirit LM BASE achieves 36.9% accuracy on MMLU (5-shot, text-only), demonstrating that the text LLM's capabilities are largely preserved despite pretraining on 30B+ speech tokens." confidence: high relevance: high - claim_id: word_level_interleaving_of_speech_and_text_during role: supports claim: "Word-level interleaving of speech and text during training is more effective than parallel ASR/TTS training or speech-only fine-tuning for cross-modal semantic understanding." source: "§4.2, Table 6" evidence: "Ablations confirm that interleaving is the critical training ingredient: removing interleaved data while keeping speech and text separately drops cross-modal StoryCloze performance by more than 10 points." confidence: high relevance: medium - claim_id: expressive_speech_properties_sentiment_pitch_style_can_be role: supports claim: "Expressive speech properties (sentiment, pitch, style) can be modeled in a language model through discrete token streams that supplement phonetic tokens, enabling cross-modal sentiment preservation." source: "§5, Table 3" evidence: "The Expressive version supplements HuBERT tokens with two additional token streams: pitch tokens from a VQ-VAE trained on F0 (64-unit codebook at 12.5 Hz), and style tokens from speaker-normalized SpeechProp features clustered into 100 units at 1 Hz." confidence: high relevance: high - claim_id: adding_expressive_speech_tokens_to_a_speech_lm role: supports claim: "Adding expressive speech tokens to a speech LM improves expressivity at the cost of moderate degradation in lexical and grammatical speech understanding." source: "§4.2, Table 4" evidence: "On standard speech comprehension benchmarks, Spirit LM Base outperforms prior speech-only LMs (GSLM, TWIST, AudioLM) on StoryCloze, which requires high-level semantic understanding, while remaining competitive on WUGGY (lexical) and BLIMP (grammatical)." confidence: high relevance: high - claim_id: cascade_speech_pipelines_remain_substantially_stronger_than_end role: supports claim: "Cascade speech pipelines remain substantially stronger than end-to-end unified models on task-specific metrics such as ASR WER and TTS intelligibility at equivalent model scale." source: "§4.3, Table 5" evidence: "Adding explicit ASR+TTS parallel training data to the recipe dramatically improves these numbers (6.0% WER, 6.7% CER), showing the base interleaved model is not optimized for task-specific generation." confidence: high relevance: medium limitations: - "Spirit LM's vocoder is conditioned on only 4 speaker voices from the Expresso dataset, which severely constrains the diversity and quality of synthesized speech; the model cannot generalize to arbitrary target speakers at inference without retraining the vocoder." - "The STSP benchmark is evaluated using fine-tuned automatic classifiers rather than human listeners, which may not capture perceptual sentiment fidelity accurately." caveats: - "Spirit LM uses interleaved discrete speech and text tokens for unified generation, with TTS as one evaluated direction." - id: "2402.08093" published_date: "2024-02-12" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - acceleration_evidence current_role: influential method_family: [semantic_acoustic_token_cascade] claims: - claim_id: scaling_autoregressive_codec_tts_to_500m_parameters_and role: supports claim: "Scaling autoregressive codec TTS to 500M+ parameters and 10K+ hours of training data produces qualitatively different prosody rendering on linguistically complex inputs compared to smaller models trained on less data." source: "§4.3, Figure 4, Table 5" evidence: "The WavLM-based speechcodes match VQ-VAE on English (MUSHRA 74.7 vs 74.8, non-significant) but outperform it on Spanish despite Spanish comprising only 2% of training data (MUSHRA 74.7 vs 73.3, significant for some speakers), supporting the hypothesis that better representations matter more in lower-resource languages (Table 3)." confidence: high relevance: high - claim_id: ssl_based_speech_representations_with_explicit_speaker_disentanglement role: supports claim: "SSL-based speech representations with explicit speaker disentanglement outperform purely acoustic codec representations for zero-shot TTS, particularly in lower-resource languages." source: "§4.1, Table 3" evidence: "The WavLM-based speechcodes match VQ-VAE on English (MUSHRA 74.7 vs 74.8, non-significant) but outperform it on Spanish despite Spanish comprising only 2% of training data (MUSHRA 74.7 vs 73.3, significant for some speakers), supporting the hypothesis that better representations matter more in lower-resource languages (Table 3)." confidence: high relevance: high - claim_id: a_streamable_convolutional_decoder_can_match_or_exceed role: supports claim: "A streamable convolutional decoder can match or exceed a diffusion-based spectrogram decoder in subjective naturalness while reducing synthesis compute by approximately 3x and enabling low-latency streaming." source: "§4.2, §4.5, Table 4" evidence: "The speechcode decoder matches or improves over the diffusion decoder on 4 of 6 evaluated speakers, while offering a 3x compute reduction and enabling streaming (Table 4)." confidence: high relevance: medium - claim_id: applying_bpe_to_discrete_speech_tokens_reduces_autoregressive role: supports claim: "Applying BPE to discrete speech tokens reduces autoregressive sequence length by approximately 40% without degrading downstream synthesis quality, enabling longer-context training." source: "§2.2.3" evidence: "A speech tokenizer converts waveforms into a compact discrete sequence; an autoregressive Transformer (\"SpeechGPT\") predicts those tokens from text and a reference speech clip; and a convolutional decoder renders the predicted tokens to audio." confidence: high relevance: high - claim_id: autoregressive_tts_trained_at_scale_generalises_to_a role: supports claim: "Autoregressive TTS trained at scale generalises to a wide range of textual phenomena without any explicit prosody annotation or task-specific supervision." source: "§4.3, §6" evidence: "Baseline comparisons (Tortoise, Bark, YourTTS) use publicly available checkpoints not trained on comparable data volumes; the gains partly reflect data scale rather than architecture alone." confidence: high relevance: high limitations: - "Model weights are not released, and evaluation uses proprietary test speakers." - "The MUSHRA baselines (YourTTS, Bark, TortoiseTTS) are not trained on comparable data or compute, making architecture-level conclusions difficult to separate from scale effects." caveats: - "BASE TTS directly studies scaled autoregressive speech-token prediction and streamable decoding." - id: "2402.13236" published_date: "2024-02-20" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS, SCA, codec] architecture: [] relevance: medium evidence_role: - historical_context current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: residual_vector_quantisation_is_the_dominant_quantisation_strategy role: supports claim: "Residual vector quantisation is the dominant quantisation strategy across neural audio codec models, with variation concentrated in discriminator design, bitrate, and semantic token integration rather than in the core compression mechanism." source: "§II.A, Table II" evidence: "Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion)." confidence: high relevance: high - claim_id: codec_based_audio_language_models_increasingly_target_multi role: supports claim: "Codec-based audio language models increasingly target multi-task coverage rather than single-task specialisation, with several systems spanning TTS, voice conversion, speech editing, speech enhancement, and translation in a single framework." source: "§III.B, Table III" evidence: "Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion)." confidence: high relevance: high - claim_id: integrating_semantic_tokens_from_self_supervised_speech_representations role: supports claim: "Integrating semantic tokens from self-supervised speech representations into the codec quantisation process improves audio quality at low bitrates, with HuBERT-guided RVQ being the most common approach." source: "§II.B" evidence: "Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion)." confidence: high relevance: high - claim_id: discrete_units_derived_from_self_supervised_representations_enable role: supports claim: "Discrete units derived from self-supervised representations enable textless speech language modelling but sacrifice speaker and paralinguistic information relative to codec-based approaches." source: "§III.A" evidence: "Key observations surfaced through comparison include: most codec-based LMs depend on Encodec or SoundStream as their tokeniser backbone; task coverage is expanding beyond TTS and music generation toward speech editing, enhancement, translation, and instruction-conditioned generation; and the integration of semantic tokens into codecs is an active design axis with differing approaches (distillation from HuBERT vs. additive fusion)." confidence: high relevance: high limitations: - "The survey covers only open-source codec models and does not include proprietary codecs used in industry systems." - "Evaluation methodology is not addressed: the paper does not compare codecs on shared benchmarks or report reproduction numbers, making it difficult to assess quality claims from the original papers in a unified way." caveats: - "This overview supplies taxonomy and comparative context rather than a new codec-TTS system." - id: "2403.16973" published_date: "2024-03-25" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - architecture_variant current_role: influential method_family: [full_duplex_multistream_generation] claims: - claim_id: autoregressive_codec_language_models_can_perform_speech_infilling role: supports claim: "Autoregressive codec language models can perform speech infilling with naturalness approaching that of the original unedited recording when masked spans are relocated to the end of the sequence, enabling bidirectional context conditioning within a causal framework." source: "§3.1, §5.3, Table 5" evidence: "The core mechanism is a two-step token rearrangement procedure applied before autoregressive modeling: In the first step (causal masking), the spans to be generated are relocated from their original position in the codec matrix to the end of the sequence." confidence: high relevance: high - claim_id: zero_shot_tts_and_speech_editing_can_be role: supports claim: "Zero-shot TTS and speech editing can be unified as a single autoregressive infilling operation without task-specific architectural components, at no cost to performance on either task." source: "§3.4, §5.4, Table 6" evidence: "VoiceCraft reframes both speech editing (infilling) and zero-shot TTS (continuation) as left-to-right language modeling over rearranged neural codec token sequences." confidence: high relevance: high - claim_id: wer_measured_by_asr_systems_is_an_unreliable role: supports claim: "WER measured by ASR systems is an unreliable proxy for perceptual intelligibility when evaluating speech synthesis quality: systems can achieve lower WER than ground truth recordings while receiving substantially lower intelligibility ratings from human listeners." source: "§5.3, §5.4" evidence: "A recurrent observation across both tasks: FluentSpeech achieves lower WER than the ground truth (3.5% vs 3.8% for TTS; 4.5% vs 5.4% for editing), yet receives substantially lower intelligibility MOS from human listeners." confidence: high relevance: medium - claim_id: evaluation_of_speech_synthesis_exclusively_on_audiobook_data role: supports claim: "Evaluation of speech synthesis exclusively on audiobook data underestimates the performance gap between systems when applied to in-the-wild recordings with diverse accents, noise, and speaking styles." source: "§5.3, §5.4" evidence: "This suggests that ASR-based WER can be a misleading proxy for perceptual intelligibility when systems produce dull or robotic speech that ASR models handle well." confidence: high relevance: medium - claim_id: scaling_autoregressive_codec_lm_parameters_consistently_improves_objective role: supports claim: "Scaling autoregressive codec LM parameters consistently improves objective metrics across intelligibility and acoustic fidelity measures, with larger gaps between larger model sizes suggesting further gains from continued scaling." source: "§5.2, Table 3" evidence: "Ablation over model sizes (120M, 430M, 830M) shows consistent gains from scaling across WER, MCD, F0, and energy distance metrics." confidence: high relevance: high limitations: - "The inference-time artifact mitigation (generating 10 candidates and discarding the 4 longest) adds significant latency and compute cost." - "The strategy is acknowledged as inelegant, and the underlying cause (repetitive loop generation) is unresolved." caveats: - "VoiceCraft directly advances autoregressive codec-token generation for both TTS and speech editing." - id: "2404.03204" published_date: "2024-04-04" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - architecture_variant current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, alignment_constrained_autoregression] claims: - claim_id: autoregressive_codec_language_model_tts_can_be_substantially role: supports claim: "Autoregressive codec language model TTS can be substantially stabilised by predicting prosody tokens as explicit intermediate targets before speech token generation, without requiring reranking or a separate alignment model at inference." source: "§3.2, Table 2" evidence: "Rather than having the AR Transformer jump directly from phoneme input to speech token generation, RALL-E inserts an explicit intermediate prediction stage: the model first generates a phoneme-level prosody token sequence encoding discretised pitch (quantised to 256 buckets) and duration (capped at 32 frames) for the full input utterance, then uses those prosody tokens as an additional conditioning signal when generating the first-layer RVQ speech tokens autoregressively." confidence: high relevance: high - claim_id: duration_guided_attention_masking_which_restricts_each_speech role: supports claim: "Duration-guided attention masking, which restricts each speech token to attend only to a local phoneme window based on predicted alignment, provides significant robustness improvements beyond prosody conditioning alone." source: "§3.3, Table 4" evidence: "RALL-E uses the predicted duration to restrict each speech token's attention to a local phoneme window of width 2k+1 centered on the corresponding phoneme (k=1 in the final system), masking all other phoneme and prosody positions." confidence: high relevance: high - claim_id: the_robustness_deficit_of_ar_codec_tts_relative role: supports claim: "The robustness deficit of AR codec TTS relative to non-autoregressive methods is most pronounced on structurally unusual inputs (repetitive patterns, numeric sequences, code strings) where learned implicit alignment is most likely to fail." source: "§4.4, Table 1" evidence: "On 50 hard sentences containing repetitive numbers, file paths, and code strings, RALL-E reduces the error rate from 68% to 4%, closely approaching non-autoregressive NaturalSpeech 2 (0%) (Table 1, §4.4)." confidence: high relevance: high - claim_id: reranking_over_multiple_samples_and_explicit_intermediate_prosody role: supports claim: "Reranking over multiple samples and explicit intermediate prosody prediction address the same underlying alignment problem and can be combined for additive gain, but prosody CoT prompting reduces the dependency on reranking by improving single-sample quality." source: "§4.2, Table 2" evidence: "Rather than having the AR Transformer jump directly from phoneme input to speech token generation, RALL-E inserts an explicit intermediate prediction stage: the model first generates a phoneme-level prosody token sequence encoding discretised pitch (quantised to 256 buckets) and duration (capped at 32 frames) for the full input utterance, then uses those prosody tokens as an additional conditioning signal when generating the first-layer RVQ speech tokens autoregressively." confidence: high relevance: medium limitations: - "The paper relies on an internal proprietary alignment tool for extracting phoneme-speech alignments during training." - "Alignment quality directly affects duration-guided masking effectiveness (§3.3 notes that alignment errors required loosening the masking window from k=0 to k=1)." caveats: - "RALL-E directly targets robustness and alignment in autoregressive codec-language-model TTS." - id: "2406.00654" published_date: "2024-06-02" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - evaluation_caution current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: standard_supervised_training_objectives_for_tts_produce_a role: supports claim: "Standard supervised training objectives for TTS produce a systematic mismatch with human perceptual evaluation metrics such as MOS and WER, and correcting this mismatch through preference-aware fine-tuning yields large performance gains." source: "§1, §4.2, Table 1" evidence: "The framework is applied as a fine-tuning pass on a pre-trained VoiceCraft model (330M or 830M parameters), which uses SpeechTokenizer (EnCodec-based, 4 RVQ codebooks, vocabulary of 2048) for discrete acoustic tokenisation." confidence: high relevance: medium - claim_id: existing_rlhf_methods_requiring_pairwise_preference_data_from role: supports claim: "Existing RLHF methods requiring pairwise preference data from the same input (DPO) are difficult to apply directly to autoregressive codec TTS because these models lack sufficient output diversity to form meaningful preference pairs from a fixed transcript-prompt combination." source: "§4.2, Appendix B" evidence: "In the sampling stage, a fixed target transcript is paired with a batch of distinct speech prompts drawn from an unseen speaker pool, yielding a diverse set of synthesised outputs; varying the prompt (rather than stochastic decoding) provides much richer quality variation across samples." confidence: high relevance: high - claim_id: uncertainty_in_human_speech_quality_annotations_is_not role: supports claim: "Uncertainty in human speech quality annotations is not noise to be discarded but an informative signal that, when incorporated into the optimization objective, improves the consistency of generated speech across listeners." source: "§4.2, §6.3, Table 3" evidence: "An ablation without uncertainty weighting (UNO-null) achieves similar mean MOS but higher output variance, confirming that uncertainty modelling specifically improves generation consistency rather than average quality." confidence: high relevance: medium - claim_id: rlhf_style_alignment_for_tts_can_be_achieved role: supports claim: "RLHF-style alignment for TTS can be achieved with a small number of self-generated samples (hundreds) without access to ground truth speech, making it practical for post-training fine-tuning at low computational cost." source: "§4.1, §5, Appendix D" evidence: "Only 400 self-generated samples (balanced 200 positive / 200 negative) are needed, requiring approximately 10 minutes of fine-tuning on a single A100 GPU." confidence: high relevance: medium - claim_id: alignment_objectives_designed_for_naturalness_mos_transfer_to role: supports claim: "Alignment objectives designed for naturalness MOS transfer to other perceptual dimensions such as emotion by substituting the selection criterion, demonstrating that preference-based fine-tuning generalises beyond a single quality axis." source: "§6.4, Table 4" evidence: "The method extends naturally to emotion TTS by substituting the selection criterion in the sampling stage: using emotional ESD dataset prompts and sorting by valence or arousal score instead of MOS drives the model to generate speech with target affective properties." confidence: high relevance: medium limitations: - "The comparison with SpeechAlign is acknowledged by the authors to be partially unfair: SpeechAlign-DPO requires ground truth speech as positive samples during optimization, which is additional supervision not available to UNO." - "Presenting both as baselines without fully separating this distinction may understate SpeechAlign's performance under matched conditions." caveats: - "The work directly applies human-preference optimization to an autoregressive codec-TTS model." - id: "2406.02430" published_date: "2024-06-04" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS, VC] architecture: [autoregressive-LM, diffusion, hybrid] relevance: high evidence_role: - core_evidence - architecture_variant current_role: influential method_family: [hybrid_ar_token_continuous_renderer, alignment_constrained_autoregression, preference_optimized_autoregression] claims: - claim_id: autoregressive_codec_language_models_trained_at_sufficient_scale role: supports claim: "Autoregressive codec language models trained at sufficient scale can generate zero-shot speech with naturalness and speaker similarity indistinguishable from human speech as measured by CMOS." source: "§3.1, Table 1" evidence: "Seed-TTS ICL receives CMOS scores of -0.07 in English and -0.08 in Chinese relative to human speech while retaining strong speaker-similarity scores." confidence: high relevance: high - claim_id: self_distillation_on_perturbation_generated_pairs_is_an role: supports claim: "Self-distillation on perturbation-generated pairs is an effective method for timbre disentanglement in TTS systems, achieving large speaker similarity gains in voice conversion without modifying model architecture or loss functions." source: "§4.1, Table 6" evidence: "ICL: +0.14 preference win - RL-SER improves emotion control from (0.46, 0.44, 0.53, 0.13) to (0.91, 0.80, 0.78, 0.82) across angry/happy/sad/surprise **Voice conversion (Table 6, self-distillation):** - Without self-distillation: SIM 0.491 EN, 0.636 ZH - With self-distillation: SIM 0.753 EN, 0.791 ZH — far exceeding DiffVC (0.311) and HierSpeech++ (0.387) **Speaker fine-tuning (Table 3):** - SFT CMOS +0.37 over ICL baseline **Deployed model (Table 5):** - Latency 0.028× offline, RTF 0.132×, with CMOS only -0.02 vs. offline model **ASR data utility (Table 2):** ASR model trained on synthetic Seed-TTS LibriSpeech data achieves WER within 0.31% absolute of real-data baseline on clean sets." confidence: high relevance: medium - claim_id: reinforcement_learning_post_training_with_objective_speech_quality role: supports claim: "Reinforcement learning post-training with objective speech quality rewards improves robustness and speaker similarity in autoregressive TTS while introducing reward hacking that must be carefully managed." source: "§4.2, Tables 7, 8" evidence: "For post-training, REINFORCE is used with two reward configurations: Seed-TTS-RL-SIM-WER (WER + speaker similarity as rewards) and Seed-TTS-RL-SER (speech emotion recognition accuracy as reward)." confidence: high relevance: high - claim_id: end_to_end_diffusion_models_that_predict_total role: supports claim: "End-to-end diffusion models that predict total duration and learn local text-audio alignment implicitly can match or exceed the quality of autoregressive codec LM systems without per-phoneme duration supervision." source: "§4.3, Table 10" evidence: "This enables zero-shot voice conversion without architectural changes. **Seed-TTSDiT (NAR variant):** A fully diffusion-based architecture (Diffusion Transformer / DiT) that directly maps Gaussian noise to vocoder latents conditioned on text, audio prompt, and a total duration signal." confidence: high relevance: high - claim_id: synthetic_speech_generated_by_a_high_quality_zero role: supports claim: "Synthetic speech generated by a high-quality zero-shot TTS system can train ASR models that approach real-data performance on clean test sets, though a gap persists on noisy sets due to background noise smoothing." source: "§3.1, Table 2" evidence: "ICL: +0.14 preference win - RL-SER improves emotion control from (0.46, 0.44, 0.53, 0.13) to (0.91, 0.80, 0.78, 0.82) across angry/happy/sad/surprise **Voice conversion (Table 6, self-distillation):** - Without self-distillation: SIM 0.491 EN, 0.636 ZH - With self-distillation: SIM 0.753 EN, 0.791 ZH — far exceeding DiffVC (0.311) and HierSpeech++ (0.387) **Speaker fine-tuning (Table 3):** - SFT CMOS +0.37 over ICL baseline **Deployed model (Table 5):** - Latency 0.028× offline, RTF 0.132×, with CMOS only -0.02 vs. offline model **ASR data utility (Table 2):** ASR model trained on synthetic Seed-TTS LibriSpeech data achieves WER within 0.31% absolute of real-data baseline on clean sets." confidence: high relevance: medium limitations: - "- Model weights not released; no reported parameter count. - Zero-shot ICL struggles with speakers having strong accents or unique styles not captured in a short (15s) prompt — the fine-tuned model is still preferred for 'hard' speakers. - Synthesized speech slightly over-smooths background noise (ASR degrades on noisy test sets). - Does not handle singing or speech with background music well. - Reward hacking (over-slow, over-articulated speech) in RL post-training requires careful tuning. - Multi-shot ICL for long-form generation not yet demonstrated." caveats: - "Seed-TTS directly combines autoregressive token generation with a diffusion renderer in a scaled TTS family." - id: "2406.04904" published_date: "2024-06-07" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, GAN] relevance: high evidence_role: - core_evidence - control_evidence current_role: influential method_family: [scaled_general_ar_codec_tts] claims: - claim_id: multilingual_zero_shot_tts_training_degrades_speaker_similarity role: supports claim: "Multilingual zero-shot TTS training degrades speaker similarity compared to monolingual training on the same data, reflecting a fundamental trade-off in cross-lingual speaker conditioning." source: "§4.1, Table 2, Table 3" evidence: "The multilingual training penalty on speaker similarity is acknowledged by the authors as an expected trade-off." confidence: high relevance: medium - claim_id: a_perceiver_resampler_based_speaker_conditioning_encoder_producing role: supports claim: "A Perceiver Resampler-based speaker conditioning encoder, producing multiple fixed-length embeddings from variable-length reference audio, improves voice cloning robustness in massively multilingual autoregressive TTS over single-embedding approaches." source: "§2" evidence: "The key architectural departure from Tortoise is the Conditioning Encoder: rather than compressing a reference audio clip into a single 1,024-dim embedding, XTTS uses six scaled dot-product attention layers followed by a Perceiver Resampler to produce 32 fixed-size 1,024-dim embeddings per reference clip, independent of clip length." confidence: high relevance: high - claim_id: evaluating_multilingual_tts_models_against_monolingual_baselines_on role: supports claim: "Evaluating multilingual TTS models against monolingual baselines on the same language produces misleading comparisons, because the multilingual model's per-language training data is substantially reduced." source: "§3.2, §4.1" evidence: "English comparisons against StyleTTS 2 and HierSpeech++ are against monolingual models, whereas XTTS is trained on 16 languages simultaneously." confidence: high relevance: medium - claim_id: a_small_amount_of_target_speaker_fine_tuning role: supports claim: "A small amount of target-speaker fine-tuning data (approximately 10 minutes) substantially improves speaker similarity in cross-lingual zero-shot synthesis, including extreme prosody styles such as whispering." source: "§5" evidence: "The paper also demonstrates that ten minutes of adaptation data for a target speaker improves SECS from 0.585 to 0.717 in cross-lingual transfer, including a whispered voice style transferred to all 16 languages." confidence: high relevance: medium - claim_id: low_frequency_codec_codebook_entries_can_be_pruned role: supports claim: "Low-frequency codec codebook entries can be pruned without quality loss and improve expressiveness in multilingual discrete-token TTS." source: "§2" evidence: "The first component is a VQ-VAE (13M parameters) that encodes mel-spectrograms into discrete codes using a single codebook of 8,192 entries at 21.53 Hz." confidence: high relevance: high limitations: - "Speaker similarity lags behind monolingual specialists in English, and the multilingual evaluation uses cross-lingual prompting (English speaker references for non-English languages), which may understate true within-language similarity." - "No human listening test was conducted for multilingual outputs beyond subjective English comparisons." caveats: - "XTTS directly extends autoregressive codec-language-model TTS to multilingual zero-shot synthesis." - id: "2406.05370" published_date: "2024-06-08" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - architecture_variant current_role: influential method_family: [scaled_general_ar_codec_tts] claims: - claim_id: adaptive_sampling_that_detects_and_breaks_token_repetition role: supports claim: "Adaptive sampling that detects and breaks token repetition loops can stabilise autoregressive codec LM decoding without requiring forced-alignment auxiliary data." source: "§3.4.1, Table 1" evidence: "The system is trained on Libriheavy (50k hours of labelled English audiobooks) using only utterance-level speech-transcription pairs, with no forced alignment or per-speaker auxiliary data." confidence: high relevance: high - claim_id: grouping_codec_codes_into_multi_token_ar_steps role: supports claim: "Grouping codec codes into multi-token AR steps reduces effective sequence length and simultaneously improves long-context modelling quality at moderate group sizes." source: "§3.1, §4.2.1, Table 1" evidence: "This reduces sequence length by a factor of G, cutting both inference latency and the context length that the Transformer must attend over, which the paper shows improves long-context modelling." confidence: high relevance: high - claim_id: autoregressive_codec_tts_can_match_or_exceed_ground role: supports claim: "Autoregressive codec TTS can match or exceed ground-truth speech on robustness and speaker similarity metrics when evaluated on clean English audiobook benchmarks." source: "§4.2.2, Table 2; §4.3.2, Table 5" evidence: "On LibriSpeech test-clean with a reference utterance as prompt, VALL-E 2 (G=1) achieves SMOS of 4.61 vs. ground truth SMOS of 4.13, and CMOS of +0.033 relative to ground truth (Table 2)." confidence: high relevance: high - claim_id: prompt_availability_in_both_the_ar_and_nar role: supports claim: "Prompt availability in both the AR and NAR stages is independently necessary for preserving speaker identity; removing either prompt degrades speaker similarity substantially." source: "§4.2.3, Table 3; §4.3.3, Table 6" evidence: "Ablations confirm that prompt input to both AR and NAR models is critical for speaker similarity (removing it collapses SIM by 50–80%), and that the NAR model's explicit acoustic condition splitting substantially improves speaker identity capture." confidence: high relevance: medium - claim_id: inference_time_multiple_sampling_followed_by_metric_based role: supports claim: "Inference-time multiple sampling followed by metric-based selection can substantially close the single-sample robustness gap, but at proportional computational cost." source: "§4.1.3, Table 1" evidence: "On the same dataset with a 3s prefix prompt, single-sampling WER reaches 1.6% against ground truth WER of 1.6%, matching it with no multiple-sampling overhead (Table 1)." confidence: high relevance: medium limitations: - "Human parity is claimed solely from results on LibriSpeech test-clean and VCTK; both benchmarks are read speech from controlled or semi-controlled recording conditions." - "Generalisation to spontaneous, noisy, or low-resource speech is undemonstrated and the authors explicitly flag this caveat." caveats: - "VALL-E 2 directly improves the robustness and sampling strategy of autoregressive codec TTS." - id: "2406.07855" published_date: "2024-06-12" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - architecture_variant current_role: active_evidence method_family: [alignment_constrained_autoregression] claims: - claim_id: phoneme_monotonic_alignment_in_decoder_only_autoregressive_tts role: supports claim: "Phoneme monotonic alignment in decoder-only autoregressive TTS can close most of the robustness gap caused by unconstrained attention, achieving near-ground-truth WER without encoder-decoder architectural changes." source: "§3.2, Table 1" evidence: "Unlike encoder-decoder monotonic attention methods, this approach is compatible with the decoder-only Transformer architecture." confidence: high relevance: high - claim_id: downsampling_only_the_first_rvq_layer_of_a role: supports claim: "Downsampling only the first RVQ layer of a neural codec at inference time reduces autoregressive steps and latency by more than half, with negligible impact on PESQ and STOI." source: "§3.1, Table 5" evidence: "Applying merging only to the first RVQ layer (rather than multiple layers) preserves audio quality: PESQ and STOI drop negligibly when only layer 1 is merged at 2x, while merging four or eight layers causes significant degradation." confidence: high relevance: high - claim_id: robustness_improvements_that_route_additional_phoneme_tokens_through role: supports claim: "Robustness improvements that route additional phoneme tokens through the autoregressive stream (as in ELLA-V) improve WER but increase inference time, illustrating a robustness-efficiency trade-off in codec LM TTS." source: "§5.3, Table 4" evidence: "For inference efficiency (Table 4), VALL-E R (2x merging) generates 10 seconds of speech in 3.67 seconds (375 AR steps), compared to VALL-E's 10.27 seconds (750 AR steps) — a 64% reduction." confidence: high relevance: high - claim_id: explicit_phoneme_level_alignment_in_a_codec_lm role: supports claim: "Explicit phoneme-level alignment in a codec LM enables independent control of prosody and timbre by substituting preset phoneme sequences at inference, enabling a form of voice conversion." source: "§3.2.3, Table 3" evidence: "Because VALL-E R explicitly tracks phoneme progress during inference, it also supports prosody control: replacing the self-predicted phoneme sequence with a preset aligned phoneme sequence from a reference utterance allows timbre and prosody to be controlled independently, enabling a form of voice conversion." confidence: high relevance: high limitations: - "All evaluations use LibriSpeech (clean English read speech)." - "Robustness gains from monotonic alignment and codec-merging quality preservation have not been tested on noisy, expressive, or multilingual speech." caveats: - "VALL-E R directly introduces monotonic alignment constraints for autoregressive codec TTS." - id: "2406.18009" published_date: "2024-06-26" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [flow-matching] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: influential method_family: [hierarchical_ar_nar_codec_tts] claims: - claim_id: explicit_phoneme_alignment_supervision_in_non_autoregressive_tts role: supports claim: "Explicit phoneme alignment supervision in non-autoregressive TTS, while useful for early convergence, imposes a naturalness ceiling that joint character-level training can exceed." source: "§3.4, Table 2" evidence: "4) shows E2 TTS takes longer to converge than Voicebox (which benefits from explicit alignment supervision early in training) but surpasses it by end of training, consistent with the hypothesis that joint learning of duration and acoustic modeling yields a better end state." confidence: high relevance: high - claim_id: flow_matching_mel_spectrogram_generators_can_learn_alignment role: supports claim: "Flow-matching mel spectrogram generators can learn alignment implicitly from raw characters and filler tokens, eliminating the need for grapheme-to-phoneme converters, phoneme aligners, and duration models." source: "§2.1, §2.3" evidence: "The CMOS comparison directly identifies phoneme alignment as the primary bottleneck to naturalness: replacing it with joint character-and-filler modeling closes the gap." confidence: high relevance: high - claim_id: unsupervised_pretraining_on_unlabeled_speech_improves_downstream_zero role: supports claim: "Unsupervised pretraining on unlabeled speech improves downstream zero-shot TTS performance in both intelligibility and speaker similarity." source: "§3.4, Table 1" evidence: "On LibriSpeech-PC test-clean (1,132 samples, 39 speakers), E2 TTS trained on Libriheavy achieves WER 2.0% and speaker similarity (SIM-o) 0.675 from random initialization, improving to WER 1.9% and SIM-o 0.708 with unsupervised pretraining." confidence: high relevance: low - claim_id: zero_shot_tts_speaker_similarity_scores_at_inference role: supports claim: "Zero-shot TTS speaker similarity scores at inference can exceed those of ground-truth recordings on standard speaker verification metrics, suggesting the metrics reward consistency within a generation rather than perceptual identity." source: "§3.4, Table 2" evidence: "On LibriSpeech-PC test-clean (1,132 samples, 39 speakers), E2 TTS trained on Libriheavy achieves WER 2.0% and speaker similarity (SIM-o) 0.675 from random initialization, improving to WER 1.9% and SIM-o 0.708 with unsupervised pretraining." confidence: high relevance: low - claim_id: non_autoregressive_tts_systems_trained_jointly_on_duration role: supports claim: "Non-autoregressive TTS systems trained jointly on duration and acoustics scale predictably with training data volume without requiring architectural changes." source: "§3.4, Table 1" evidence: "Training on 200,000 hours of proprietary data reaches WER 1.9% and SIM-o 0.707 from random initialization." confidence: high relevance: high limitations: - "The CMOS evaluation covers only 39 samples from 39 LibriSpeech speakers read English." - "The naturalness finding (\"indistinguishable from ground truth\") is thus narrow in scope: spontaneous speech, non-native speakers, expressive or prosodically complex content, and cross-lingual settings are untested." caveats: - "E2 TTS is a fully non-autoregressive alternative and contributes comparative evidence against codec-LM complexity." - id: "2407.05407" published_date: "2024-07-07" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence - architecture_variant current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: inserting_a_vector_quantizer_into_a_supervised_asr role: supports claim: "Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens." source: "§2.1, §5.1, Table 7" evidence: "The supervised VQ-inserted ASR encoder obtains 3.18% and 7.56% WER on LibriSpeech test-clean and test-other, outperforming the compared unsupervised token representations." confidence: high relevance: high - claim_id: in_autoregressive_codec_lm_tts_both_the_text role: supports claim: "In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning." source: "§5.2, Table 7" evidence: "2.89%/6.57% for the vanilla ASR model (only a minor degradation). **LLM:** An autoregressive transformer models the sequence [SOS, speaker-embedding, text-encodings, speech-turn-token, speech-tokens, EOS]." confidence: high relevance: high - claim_id: asr_re_ranking_is_an_effective_post_hoc role: supports claim: "ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute." source: "§5.3, Tables 8, 9" evidence: "Generating multiple candidates and selecting them with ASR scoring improves content consistency without retraining the speech generator, while increasing inference cost." confidence: high relevance: high - claim_id: instruction_fine_tuning_on_a_modest_amount_of role: supports claim: "Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model." source: "§2.4, §5.4, Table 10" evidence: "For cross-lingual cloning, the prompt text and tokens are omitted to prevent source-language prosodic leakage; only speaker embedding and prompt mel are passed to the flow model. **CosyVoice-instruct:** Built on CosyVoice-base with instruction fine-tuning on 556h of curated data covering speaker identity descriptions, speaking style (emotion, gender, rate, pitch), and fine-grained paralinguistics (laughter, breath, emphasis)." confidence: high relevance: medium - claim_id: high_quality_tts_synthesized_speech_can_serve_as role: supports claim: "High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus." source: "§5.5, Table 11" evidence: "2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement." confidence: high relevance: medium limitations: - "- Only a single VQ codebook (4096 codes) is used; multi-level RVQ and its effect on quality vs. compression is left for future work. - The choice of VQ insertion layer (after layer 6 of 12) is not ablated — optimal placement is unresolved. - Cross-lingual cloning omits prompt prosody to prevent leakage, which may reduce naturalness in target language. - Instruction fine-tuning data amounts (556h) are modest; broader paralinguistic coverage remains open. - No subjective (MOS) evaluation in the main paper; relies entirely on objective WER/CER/SS metrics." caveats: - "CosyVoice directly combines autoregressive supervised semantic-token generation with flow-based acoustic rendering." - id: "2407.08551" published_date: "2024-07-11" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence - architecture_variant current_role: influential method_family: [continuous_token_autoregression] claims: - claim_id: continuous_mel_spectrogram_representations_preserve_more_speaker_relevant role: supports claim: "Continuous mel-spectrogram representations preserve more speaker-relevant acoustic information than vector-quantized codec codes at standard compression rates." source: "§5.1, Table 1" evidence: "This replaces the top-p discrete sampling of codec language models with a learned continuous sampling mechanism, providing output diversity without requiring manual sampling configuration." confidence: high relevance: high - claim_id: autoregressive_tts_models_trained_to_predict_continuous_frames role: supports claim: "Autoregressive TTS models trained to predict continuous frames can achieve naturalness comparable to human speech while avoiding the silence and repetition failures endemic to discrete codec language models." source: "§5.2, Table 3" evidence: "This replaces the top-p discrete sampling of codec language models with a learned continuous sampling mechanism, providing output diversity without requiring manual sampling configuration." confidence: high relevance: high - claim_id: variational_sampling_in_the_continuous_latent_space_is role: supports claim: "Variational sampling in the continuous latent space is more effective than top-p discrete sampling for improving output diversity and speaker similarity in autoregressive TTS." source: "§5.3, Table 4" evidence: "This replaces the top-p discrete sampling of codec language models with a learned continuous sampling mechanism, providing output diversity without requiring manual sampling configuration." confidence: high relevance: high - claim_id: a_reduction_factor_that_predicts_multiple_frames_per role: supports claim: "A reduction factor that predicts multiple frames per autoregressive step can substantially reduce inference time with only modest degradation in speaker similarity." source: "§5.4, Table 5" evidence: "The reduction factor r provides a throughput knob: r=2 halves AR steps and inference time with modest speaker similarity loss, while r=4 reduces inference time to one quarter of the baseline and still outperforms most VALL-E variants on WER." confidence: high relevance: high limitations: - "The subjective evaluation rests on only 40 samples from a single English corpus (LibriSpeech test-clean)." - "The naturalness and speaker similarity advantages may not generalize to noisier prompts, non-native accents, or other languages." caveats: - "The paper directly tests autoregressive speech generation without vector-quantized codec tokens." - id: "2408.02622" published_date: "2024-08-05" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: full_duplex_speech_generation_requires_that_the_listening role: supports claim: "Full-duplex speech generation requires that the listening channel information be injected at intermediate representation layers, not at the input or output level, to avoid degrading speech generation quality." source: "§6.2, Table 2" evidence: "Early fusion degrades severely (WER 33.56%), likely because fusing at the input embedding level conflates speaking and listening signals." confidence: high relevance: high - claim_id: a_single_layer_discrete_token_autoregressive_tts_backbone role: supports claim: "A single-layer discrete token autoregressive TTS backbone can integrate real-time audio input streams with minimal WER degradation relative to a no-listening baseline in controlled conditions." source: "§6.2, Table 2" evidence: "At each autoregressive step, the model conditions its next-token prediction on both the speaking channel history and the real-time listening channel embeddings up to that point." confidence: high relevance: high - claim_id: robustness_to_noise_and_sensitivity_to_unseen_speaker role: supports claim: "Robustness to noise and sensitivity to unseen speaker interruptions are distinct challenges in full-duplex SLMs, and voice-based generalisation introduces significantly higher error rates than command-based triggering." source: "§6.2, Table 3" evidence: "The higher error rates reflect the diversity of interruption commands and the speaker independence test condition." confidence: high relevance: medium - claim_id: joint_fine_tuning_of_both_the_speech_generation role: supports claim: "Joint fine-tuning of both the speech generation backbone and the streaming SSL encoder is necessary for full-duplex models to reach peak interactive capability; freezing either component degrades turn-taking recall." source: "§6.3, Table 4" evidence: "Ablation results show that fine-tuning both the TTS backbone and SSL encoder jointly (rather than freezing either) is essential for peak performance." confidence: high relevance: high limitations: - "The system produces speech tokens but not semantic speech responses: it stops speaking when interrupted but does not generate a contextually appropriate spoken reply." - "The paper evaluates TTS output and turn-taking accuracy, not full dialogue capability." caveats: - "The system extends autoregressive speech-token generation to simultaneous listening and speaking." - id: "2408.16532" published_date: "2024-08-29" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [codec] architecture: [GAN, VAE] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure] claims: - claim_id: a_single_large_codebook_quantizer_can_achieve_higher role: supports claim: "A single large-codebook quantizer can achieve higher perceptual reconstruction quality than multi-quantizer residual VQ systems at substantially higher bitrates, when supported by a strong decoder." source: "§4.2, Table 1, Table 2" evidence: "When comparing models constrained to a single quantizer, WavTokenizer (40 or 75 tokens) substantially outperforms DAC's 1-quantizer variant (100 tokens) across all metrics, where DAC at 1 kbps achieves only UTMOS 1.49." confidence: high relevance: low - claim_id: codec_tokens_produced_by_a_single_quantizer_enable role: supports claim: "Codec tokens produced by a single quantizer enable better downstream autoregressive speech generation than tokens from multi-quantizer codecs, as measured by intelligibility and speaker similarity." source: "§4.2, Appendix I, Table 12" evidence: "In downstream TTS evaluation (using an autoregressive language model backbone following the MusicGen/ParlerTTS paradigm on LibriTTS), WavTokenizer-based synthesis achieves WER 5.1% and speaker similarity 0.61, versus 6.9% WER and 0.59 SPK-SIM for DAC at 9 kbps (Table 12), demonstrating that the single-quantizer representations transfer better to generation tasks despite lower bitrate." confidence: high relevance: high - claim_id: incorporating_attention_mechanisms_in_the_codec_decoder_and role: supports claim: "Incorporating attention mechanisms in the codec decoder and extending the training context window improves the semantic richness of discrete audio tokens without requiring distillation from a semantic model." source: "§3.3, §4.3, Table 9" evidence: "This improvement is achieved without distillation from a semantic model, relying instead on the attention decoder and extended context window." confidence: high relevance: high - claim_id: expanding_the_vq_codebook_space_beyond_the_conventional role: supports claim: "Expanding the VQ codebook space beyond the conventional 1024 entries improves reconstruction quality under extreme compression, but excessively large codebooks reduce codebook utilisation and yield diminishing returns." source: "§3.2, §4.3, Table 5" evidence: "Standard codebooks of size 2^10 are insufficient to represent the full speech space under single-quantizer constraints; WavTokenizer uses 4096 entries (2^12) as the default after empirical analysis showing that codebook usage concentrates below 2^12 even at large training scale." confidence: high relevance: low - claim_id: inverse_fourier_transform_decoding_significantly_outperforms_mirrored_transposed role: supports claim: "Inverse Fourier transform decoding significantly outperforms mirrored transposed-convolution upsampling in high-compression codec settings." source: "§3.3, §4.3, Table 7" evidence: "The decoder replaces the conventional transposed-convolution upsampling stack with an inverse Fourier transform structure borrowed from Vocos." confidence: high relevance: high limitations: - "The downstream TTS evaluation uses only LibriTTS (~960 hours) and a single model configuration; the claimed advantages of WavTokenizer over multi-quantizer codecs in generative modelling have not been validated at the scale of systems like VALL-E or Voicebox, where the codec is a fixed component in a much larger pipeline." - "Acoustic codecs including WavTokenizer lack ASR-level speech understanding capabilities; the authors note this constrains use in unified multimodal understanding-and-generation frameworks (GPT-4o paradigm)." caveats: - "WavTokenizer contributes an efficient codec representation for downstream audio language models rather than a TTS model." - id: "2408.16725" published_date: "2024-08-29" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: influential method_family: [unified_speech_text_language_model, masked_parallel_codec_alternatives] claims: - claim_id: simultaneous_text_and_audio_generation_conditioned_on_text role: supports claim: "Simultaneous text and audio generation, conditioned on text tokens generated in parallel, enables streaming speech output without the latency penalty of sequential text-then-audio decoding." source: "§3.2" evidence: "This avoids the latency penalty of sequential text-then-audio generation while retaining the reasoning structure of the text modality." confidence: high relevance: high - claim_id: audio_reasoning_quality_in_end_to_end_speech role: supports claim: "Audio reasoning quality in end-to-end speech LMs lags behind text reasoning quality when trained on similar data volumes, and batch inference strategies can partially bridge this gap." source: "§3.2, §4.4" evidence: "A second inference strategy, batch parallel decoding, addresses the observed gap between text and audio reasoning quality." confidence: high relevance: high - claim_id: a_three_stage_adapter_based_training_curriculum_can role: supports claim: "A three-stage adapter-based training curriculum can integrate speech input and output into a frozen language model backbone with minimal degradation to text capabilities." source: "§3.3" evidence: "Mini-Omni builds on Qwen2-0.5B as its backbone language model and extends it with two lightweight adapters: a Whisper-small encoder adapter for speech input and a TTS adapter consisting of six additional transformer blocks for speech output." confidence: high relevance: high - claim_id: multi_codebook_audio_codecs_with_high_token_rates role: supports claim: "Multi-codebook audio codecs with high token rates require parallel decoding schemes to maintain practical streaming throughput in autoregressive speech LMs." source: "§3.1, §3.2" evidence: "The high codebook depth makes flattened token sequences impractically long, so the model instead uses parallel decoding across all eight heads (one text head plus seven SNAC codebook heads) with a one-step delay between adjacent layers." confidence: high relevance: high limitations: - "The paper reports no MOS or naturalness metrics for speech output, making it impossible to quantitatively compare audio quality against TTS or SCA baselines." - "The claim that quality is \"on par with common TTS systems\" is unsupported." caveats: - "Mini-Omni directly uses autoregressive speech-token generation for streaming spoken interaction." - id: "2409.00750" published_date: "2024-09-01" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS, VC] architecture: [autoregressive-LM] relevance: high evidence_role: - negative_evidence - architecture_variant current_role: influential method_family: [semantic_acoustic_token_cascade, masked_parallel_codec_alternatives] claims: - claim_id: non_autoregressive_masked_generative_transformers_can_achieve_human role: supports claim: "Non-autoregressive masked generative transformers can achieve human-level speaker similarity in zero-shot TTS without requiring explicit text-speech alignment or phone-level duration supervision." source: "§4.2.1, Table 2" evidence: "A flow-matching duration predictor estimates total speech length from text and prompt duration, but phone-level durations are only used as intermediates to derive the total, not as direct conditioning." confidence: high relevance: high - claim_id: replacing_k_means_quantisation_of_ssl_features_with role: supports claim: "Replacing k-means quantisation of SSL features with VQ-VAE vector quantisation reduces information loss in tonal languages and improves downstream acoustic token prediction." source: "§3.2.1" evidence: "Semantic tokens are not extracted via k-means (which loses tonal information, problematic for Chinese) but from a custom VQ-VAE trained on W2v-BERT 2.0 hidden states (17th layer)." confidence: high relevance: high - claim_id: masked_generative_tts_substantially_outperforms_autoregressive_tts_on role: supports claim: "Masked generative TTS substantially outperforms autoregressive TTS on hard-text robustness (tongue twisters, repeating phrases) while maintaining competitive naturalness on standard benchmarks." source: "§4.2.2, Appendix J, Table 13" evidence: "In multilingual settings (Japanese, Korean, French, German), MaskGCT substantially outperforms XTTS-v2 and Emilia baseline systems on SIM-O across all four languages, with lower WER in three of four." confidence: high relevance: high - claim_id: parallel_iterative_decoding_in_masked_generative_models_yields role: supports claim: "Parallel iterative decoding in masked generative models yields constant inference cost regardless of output length, in contrast to autoregressive decoding whose cost scales linearly with utterance duration." source: "§4.2.2" evidence: "During inference, the T2S model decodes in 25-50 parallel iterative steps (constant regardless of output length)." confidence: high relevance: high - claim_id: zero_shot_style_cloning_via_in_context_learning role: supports claim: "Zero-shot style cloning via in-context learning extends to accent and emotion transfer without task-specific architectural changes." source: "§4.3, Tables 4–5" evidence: "Text and prompt semantic tokens are prepended as a prefix, enabling in-context learning without alignment supervision." confidence: high relevance: medium limitations: - "Speech content editing is acknowledged as \"not very robust\" by the authors, who attribute this to a training objective mismatch (mask-and-predict vs. fill-in-mask)." - "The editing capability is demonstrated qualitatively only, with no quantitative evaluation reported." caveats: - "MaskGCT replaces autoregressive codec decoding with masked generation and therefore provides a direct competing architecture." - id: "2409.03283" published_date: "2024-09-05" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence - architecture_variant current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: separating_the_waveform_generation_stage_into_a_low role: supports claim: "Separating the waveform generation stage into a low-sampling-rate Mel decoder and a super-resolution vocoder allows a system trained predominantly on low-sampling-rate data to produce high-fidelity 48kHz output." source: "§3.3" evidence: "In both cases, a BigVGAN-V2-based super-resolution vocoder upsamples the 16kHz Mel spectrogram to 48kHz waveform, trained on a curated 294-hour high-sampling-rate subset." confidence: high relevance: medium - claim_id: few_shot_fine_tuning_of_a_large_foundation role: supports claim: "Few-shot fine-tuning of a large foundation TTS model substantially outperforms zero-shot in-context learning for highly expressive, distinctive target voices, even with only one hour of data." source: "§5.2.1, Table 5" evidence: "Two downstream applications are demonstrated: (1) zero-shot and few-shot voice cloning for video dubbing, where UGC scenarios use in-context learning and PUGC scenarios use 1-hour supervised fine-tuning of the LM and flow-matching decoder; (2) instruction-tuned human-like speech generation adding four emotion categories via a dedicated embedding layer and 13 paralinguistic behaviour types via token insertion or embedding injection." confidence: high relevance: medium - claim_id: prompt_audio_enhancement_improves_voice_cloning_quality_for role: supports claim: "Prompt audio enhancement improves voice cloning quality for noisy prompts but can slightly degrade performance when prompts are already clean." source: "§5.2.2, Table 6" evidence: "Prompt enhancement before voice cloning is beneficial at SNR 0dB and 10dB but can slightly degrade performance at SNR 20dB (Table 6), recommending selective application based on noise level." confidence: high relevance: medium - claim_id: instruction_tuning_with_a_small_domain_specific_dataset role: supports claim: "Instruction tuning with a small domain-specific dataset dramatically improves emotion controllability in a pre-trained TTS language model, raising accuracy from near-chance to near-ceiling." source: "§5.3, Table 7" evidence: "Instruction tuning raises emotion-classification accuracy from 45–87% before tuning to 97–100% across the four evaluated emotion categories." confidence: high relevance: high - claim_id: autoregressive_tts_systems_trained_on_predominantly_one_language role: supports claim: "Autoregressive TTS systems trained on predominantly one language show markedly higher pronunciation error rates on under-represented languages, even at large data scales." source: "§5.1.2, Table 3" evidence: "CosyVoice's 5.68% in Chinese, but both systems show substantially higher rates in English and code-switch conditions (12% and 8.5% for FireRedTTS), attributed to the limited English proportion in training data (Table 3)." confidence: high relevance: high limitations: - "All evaluations are conducted on proprietary internal test sets with no publicly released benchmarks, data, or model weights." - "This makes direct comparison with other systems difficult to reproduce and limits the generalisability of the reported numbers." caveats: - "FireRedTTS directly evaluates autoregressive and flow-based branches within an industrial TTS framework." - id: "2409.05377" published_date: "2024-09-09" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [codec] architecture: [GAN, VAE] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure] claims: - claim_id: at_very_low_bitrates_around_1_kbps_model role: supports claim: "At very low bitrates (around 1 kbps), model capacity is a decisive factor in reconstruction quality, with larger models substantially outperforming architecturally sophisticated but smaller codecs." source: "§III-C, §IV-B, Table I" evidence: "The inference RTF of BigCodec on CPU is 1.1x, just barely real-time, compared to 3.1x for BigCodec-base, reflecting the compute cost of the larger model." confidence: high relevance: high - claim_id: low_dimensional_vector_quantisation_before_codebook_lookup_substantially role: supports claim: "Low-dimensional vector quantisation before codebook lookup substantially improves codebook utilisation in single-codebook, single-quantisation-step codec designs." source: "§III-A, §IV-B" evidence: "Third, low-dimensional vector quantization: latent variables are projected into an 8-dimensional space before quantization against a single codebook of 8192 codes, avoiding the sparse high-dimensional quantization problem that leads to poor codebook utilisation." confidence: high relevance: high - claim_id: adding_sequential_lstm_modelling_to_a_convolutional_codec role: supports claim: "Adding sequential (LSTM) modelling to a convolutional codec encoder improves both perceptual quality and speaker similarity at low bitrates, independently of the parameter count effect." source: "§III-A, §IV-D, Table III" evidence: "The LSTM contributes independently of the parameter count increase it brings." confidence: high relevance: high - claim_id: scaling_codec_model_size_beyond_a_saturation_point role: supports claim: "Scaling codec model size beyond a saturation point (approximately 159M parameters in this setting) yields no further reconstruction benefit, analogous to scale-up behaviour observed in neural vocoders." source: "§IV-D, Table III" evidence: "Scaling beyond 159M (BigCodec-300M) and scaling training data to 60k hours (LibriLight) both yield no measurable improvement, indicating a capacity saturation point." confidence: high relevance: high - claim_id: increasing_training_data_volume_from_960_hours_to role: supports claim: "Increasing training data volume from 960 hours to 60k hours does not improve codec reconstruction quality, suggesting that model capacity rather than data quantity is the binding constraint at this bitrate." source: "§IV-D, Table III" evidence: "Scaling beyond 159M (BigCodec-300M) and scaling training data to 60k hours (LibriLight) both yield no measurable improvement, indicating a capacity saturation point." confidence: high relevance: high limitations: - "BigCodec is trained exclusively on clean English speech (LibriSpeech 960h), while competing codecs such as EnCodec and DAC train on diverse multilingual datasets including music and environmental sounds." - "The multilingual generalisation result is encouraging, but the clean-speech-only training domain limits applicability to noisy or music-heavy audio without fine-tuning." caveats: - "BigCodec advances the low-bitrate codec substrate used by speech language models rather than generating TTS itself." - id: "2410.00037" published_date: "2024-09-17" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence - architecture_variant current_role: foundational method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: eliminating_the_text_bottleneck_in_spoken_dialogue_requires role: supports claim: "Eliminating the text bottleneck in spoken dialogue requires modeling acoustic tokens jointly with semantic tokens in a single generative model, as purely semantic approaches cannot capture paralinguistic information or generate in arbitrary voices." source: "§3.4, §5.4" evidence: "Moshi's RQ-Transformer jointly predicts text, semantic speech, and acoustic speech tokens for both dialogue participants at each timestep." confidence: high relevance: high - claim_id: predicting_time_aligned_text_tokens_as_a_per role: supports claim: "Predicting time-aligned text tokens as a per-frame prefix to audio tokens substantially improves the linguistic quality and factual accuracy of speech generated by audio language models, with minimal inference overhead." source: "§3.4.4, §5.3, Table 6" evidence: "Inner Monologue inserts time-aligned text tokens as a prefix to the semantic token at each frame." confidence: high relevance: high - claim_id: modeling_conversation_as_parallel_autoregressive_streams_for_each role: supports claim: "Modeling conversation as parallel autoregressive streams for each speaker, without explicit turn boundaries, enables full-duplex spoken interaction and allows training on naturally overlapping speech." source: "§3.4.3, §5.6, Table 9" evidence: "The multi-stream design models both sides of the conversation as parallel autoregressive token streams with no explicit turn boundary; Moshi generates silence tokens when the user is speaking." confidence: high relevance: high - claim_id: adversarial_only_training_of_neural_audio_codecs_substantially role: supports claim: "Adversarial-only training of neural audio codecs substantially improves subjectively rated audio quality relative to mixed reconstruction-adversarial objectives, despite degrading objective metrics such as VisQOL." source: "§3.3, §5.2, Table 4" evidence: "Mimi's adversarial-only training raises MUSHRA from 58.8 with mixed losses to 81.0, despite worsening objective reconstruction metrics." confidence: high relevance: high - claim_id: standard_objective_audio_quality_metrics_visqol_mosnet_are role: supports claim: "Standard objective audio quality metrics (VisQOL, MOSNet) are unreliable proxies for perceived quality when the training objective changes, making human evaluation indispensable for codec comparison." source: "§5.2, §5.8" evidence: "Mimi's adversarial-only configuration receives substantially higher MUSHRA despite objective metrics such as VisQOL moving in the opposite direction." confidence: high relevance: high limitations: - "Moshi's spoken factual question answering lags substantially behind its Helium text baseline, including 22.8 versus 56.4 on TriviaQA." - "Audio training causes factual-knowledge degradation that the available instruction-tuning data does not fully recover." caveats: - "Moshi establishes a real-time full-duplex speech language model using hierarchical autoregressive audio tokens." - id: "2410.03751" published_date: "2024-10-01" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA, TTS] architecture: [] relevance: medium evidence_role: - historical_context current_role: active_evidence method_family: [semantic_acoustic_token_cascade, unified_speech_text_language_model, preference_optimized_autoregression] claims: - claim_id: end_to_end_speech_generation_models_avoid_the role: supports claim: "End-to-end speech generation models avoid the information loss, latency, and cumulative error introduced by cascaded ASR-LLM-TTS pipelines, but require integrating speech tokenisation and synthesis into a unified training regime." source: "§I, §II" evidence: "Text-speech representation alignment is identified as a key training strategy: interleaved token sequences (text and speech tokens mixed in a single stream), multi-sequence parallel generation (simultaneous text and speech token decoding), and cross-modal supervision all appear in the literature with different latency/quality trade-offs." confidence: high relevance: high - claim_id: semantic_tokenizers_and_acoustic_tokenizers_impose_an_inherent role: supports claim: "Semantic tokenizers and acoustic tokenizers impose an inherent trade-off: semantic tokens produce coherent content but poor acoustic quality, while acoustic tokens enable high-fidelity reconstruction but risk content inaccuracies." source: "§III-A, §IV-A1" evidence: "Semantic tokenizers (HuBERT, wav2vec 2.0, W2v-BERT) excel at content fidelity but produce speech that lacks fine acoustic detail." confidence: high relevance: high - claim_id: initialising_a_speech_lm_from_a_text_pretrained role: supports claim: "Initialising a speech LM from a text-pretrained checkpoint accelerates convergence and improves speech understanding, whereas initialisation from image-pretrained checkpoints yields worse results than random initialisation." source: "§IV-B1" evidence: "Key findings from the literature reviewed include: - HuBERT consistently outperforms CPC and wav2vec 2.0 as a semantic tokenizer in speech generation quality (GSLM). - Text-pretrained LM checkpoints yield faster convergence and better speech understanding than cold initialisation; image-pretrained checkpoints perform worse than cold initialisation, confirming that cross-modal transfer requires semantic proximity. - Interleaved speech-text pre-training (as in SPIRIT-LM) demonstrably improves alignment between text and speech representations, measurable through feature similarity visualisations. - Mixed-objective tokenizers (SpeechTokenizer, Mimi) show promise in balancing semantic and acoustic trade-offs, but the area is described as early-stage. - Post-alignment for SpeechLMs (Align-SLM, SpeechAlign) remains underexplored relative to its counterpart in text LMs. - Full-duplex modelling is identified as an open frontier; most evaluated systems still use streaming architectures without true simultaneous bidirectional capability." confidence: high relevance: high - claim_id: interleaving_speech_and_text_tokens_during_pre_training role: supports claim: "Interleaving speech and text tokens during pre-training measurably improves cross-modal representation alignment compared to training on speech tokens alone." source: "§IV-B1" evidence: "Text-speech representation alignment is identified as a key training strategy: interleaved token sequences (text and speech tokens mixed in a single stream), multi-sequence parallel generation (simultaneous text and speech token decoding), and cross-modal supervision all appear in the literature with different latency/quality trade-offs." confidence: high relevance: high - claim_id: post_alignment_techniques_rlhf_dpo_for_speech_lms role: supports claim: "Post-alignment techniques (RLHF, DPO) for speech LMs remain substantially underexplored relative to their established role in text LM development, leaving semantic consistency and acoustic quality gaps in deployed systems." source: "§IV-B3, §VII" evidence: "Key findings from the literature reviewed include: - HuBERT consistently outperforms CPC and wav2vec 2.0 as a semantic tokenizer in speech generation quality (GSLM). - Text-pretrained LM checkpoints yield faster convergence and better speech understanding than cold initialisation; image-pretrained checkpoints perform worse than cold initialisation, confirming that cross-modal transfer requires semantic proximity. - Interleaved speech-text pre-training (as in SPIRIT-LM) demonstrably improves alignment between text and speech representations, measurable through feature similarity visualisations. - Mixed-objective tokenizers (SpeechTokenizer, Mimi) show promise in balancing semantic and acoustic trade-offs, but the area is described as early-stage. - Post-alignment for SpeechLMs (Align-SLM, SpeechAlign) remains underexplored relative to its counterpart in text LMs. - Full-duplex modelling is identified as an open frontier; most evaluated systems still use streaming architectures without true simultaneous bidirectional capability." confidence: high relevance: high limitations: - "The survey's arXiv version was submitted in October 2024 and the rapidly evolving SpeechLM landscape means several systems surveyed (notably Moshi, Mini-Omni, Llama-Omni) were still very recent preprints without peer-reviewed evaluation." - "The coverage of full-duplex systems and post-alignment techniques is acknowledged by the authors as incomplete." caveats: - "The survey provides taxonomy and trend evidence rather than a new autoregressive codec-TTS system." - id: "2410.11190" published_date: "2024-10-15" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant - core_evidence current_role: active_evidence method_family: [unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: staged_adapter_training_encoder_alignment_before_language_model role: supports claim: "Staged adapter training (encoder alignment before language model fine-tuning) enables tri-modal extension of a compact language model with minimal data without catastrophic forgetting of the base model's text capabilities." source: "§3.3" evidence: "Stage 1 trains only the adapter weights on speech recognition and image captioning data, aligning multimodal feature distributions to the text embedding space without modifying the language model." confidence: high relevance: high - claim_id: using_continuous_encoder_features_whisper_rather_than_discrete role: supports claim: "Using continuous encoder features (Whisper) rather than discrete audio tokens for speech input yields more stable and semantically consistent representations, reducing ASR loss instability during training." source: "§3.1, \"Audio Encoder\"" evidence: "The backbone is Qwen2-0.5B, augmented with two pretrained encoders: CLIP ViT-B/32 for vision (producing a 50-element feature sequence) and Whisper-small for audio input (operating as a continuous feature extractor rather than a discrete tokenizer)." confidence: high relevance: high - claim_id: adding_a_third_modality_vision_to_an_audio role: supports claim: "Adding a third modality (vision) to an audio-text spoken conversational agent modestly degrades ASR performance, likely due to diluted training data proportion rather than architectural interference." source: "§4.4, Table 2" evidence: "Mini-Omni2 builds on the Mini-Omni framework by adding a visual modality to an existing audio-text spoken conversational agent." confidence: high relevance: medium - claim_id: command_based_semantic_interruption_intent_token_classification_provides role: supports claim: "Command-based semantic interruption (intent token classification) provides a viable alternative to VAD-based full-duplex detection, with the advantage of robustness to noise and unrelated background sounds." source: "§3.4" evidence: "The duplex interaction mechanism uses a command-based approach rather than full-duplex VAD: the model receives streaming SNAC-encoded tokens of the user's audio in real time and emits frame-level `irq` (interrupt) or `n-irq` (no interrupt) state tokens." confidence: high relevance: high limitations: - "Evaluation coverage is incomplete: no naturalness MOS, SMOS, or intelligibility metrics for speech output are reported in this version, and vision benchmark results are explicitly deferred." - "Claims about speech quality and vision understanding capability cannot be independently verified from this paper alone." caveats: - "Mini-Omni2 extends autoregressive speech-token generation to multimodal and duplex interaction." - id: "2410.17799" published_date: "2024-10-23" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: full_duplex_spoken_dialogue_can_be_achieved_by role: supports claim: "Full-duplex spoken dialogue can be achieved by flattening interleaved speech and text token streams into a single autoregressive sequence, without modifying the backbone LLM architecture." source: "§3, §3.3.2" evidence: "OmniFlatten builds on Qwen2-0.5B as a backbone text LLM and extends it to full-duplex speech conversation through three progressive post-training stages, without modifying the backbone architecture." confidence: high relevance: high - claim_id: progressive_curriculum_training_modality_alignment_followed_by_half role: supports claim: "Progressive curriculum training (modality alignment followed by half-duplex, then full-duplex) improves final full-duplex dialogue quality compared to training directly on full-duplex data." source: "§4.3, Table 3" evidence: "Ablations confirm each training stage contributes cumulatively: omitting modality alignment and half-duplex stages both hurt final full-duplex scores." confidence: high relevance: medium - claim_id: eliminating_intermediate_text_output_from_dialogue_models_substantially role: complicates claim: "Eliminating intermediate text output from dialogue models substantially reduces response latency but causes a significant drop in semantic coherence, indicating a fundamental trade-off between speed and content quality in speech-to-speech generation." source: "§3.3.2, §4.3, Table 3" evidence: "The 2-stream model (speech-only output) drops significantly to 2.19/3.06, indicating that eliminating intermediate text harms semantic coherence." confidence: high relevance: high - claim_id: turn_taking_response_latency_in_full_duplex_speech role: supports claim: "Turn-taking response latency in full-duplex speech models can be reduced by chunked interleaved sequence training, with practical response times under 200 ms achievable at 0.5B parameter scale." source: "§4.3, Table 4" evidence: "OmniFlatten builds on Qwen2-0.5B as a backbone text LLM and extends it to full-duplex speech conversation through three progressive post-training stages, without modifying the backbone architecture." confidence: high relevance: medium limitations: - "The 0.5B backbone is substantially smaller than comparators (LLaMA-Omni 8B, GLM-Voice 9B, Moshi 7B), making LLM-score comparisons in Table 3 not directly attributable to the method alone. The paper acknowledges GLM-Voice results may reflect test-set leakage. Dialogue quality scores remain well below the ground-truth ceiling." - "Training data is entirely synthesised from text dialogues via a TTS pipeline; real conversational dynamics (natural prosody, disfluencies, real interruption patterns) are not represented. The model does not handle backchannels from either speaker, a basic feature of natural human conversation. User turn-taking accuracy at 25 tokens remains below 55% for both models evaluated, leaving interruption handling far from reliable. The paper does not report naturalness MOS, making direct quality comparison to TTS-oriented systems difficult. All evaluation uses simulated test data matching the training distribution, raising questions about real-world robustness." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2411.00774" published_date: "2024-11-01" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - architecture_variant current_role: influential method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: freezing_the_llm_backbone_during_speech_modality_alignment role: supports claim: "Freezing the LLM backbone during speech-modality alignment reduces the intelligence gap between spoken and text question-answering performance compared to fine-tuned approaches." source: "§3.4, Table 3" evidence: "For speech output: (1) training a single-codebook TiCodec on speech data alone; (2) training the NAR and AR speech decoders on ~3k hours of text-speech pairs using LLM token embeddings as semantic input (LLM embedding layer frozen); (3) prefix kv-cache fine-tuning of the NAR prefix decoder on Q&A data to close the distribution gap between LLM output text tokens and the training text used in stage 2." confidence: high relevance: medium - claim_id: a_three_stage_training_curriculum_using_large_asr role: supports claim: "A three-stage training curriculum, using large ASR corpora for encoder pretraining followed by small-scale multi-modal Q&A fine-tuning, is sufficient to achieve competitive spoken dialogue quality without updating backbone LLM parameters." source: "§2.2.2, §2.3.2" evidence: "For speech input: (1) CTC-based ASR pretraining of the speech encoder; (2) connecting the encoder to the frozen LLM via the adapter, trained with ASR-style supervised objectives on 110k hours of Chinese/English speech; (3) prompt-embedding fine-tuning on 60k synthesised multi-round Q&A pairs with all components except trainable prompt embeddings frozen." confidence: high relevance: medium - claim_id: chunk_level_state_classification_integrated_into_the_llm role: supports claim: "Chunk-level state classification integrated into the LLM's prefill stage enables duplex interruption detection without requiring a separate monitoring model or additional LLM context." source: "§2.4" evidence: "Duplex capability is achieved by adding a chunk-level state-prediction classification head to the LLM's output layer during stage-3 training." confidence: high relevance: medium - claim_id: decoupling_encoder_and_llm_kv_cache_per_user role: supports claim: "Decoupling encoder and LLM KV-cache per user session allows a server-side pool of model replicas to handle concurrent users with chunk-granular scheduling." source: "§2.4" evidence: "A server-side scheduling strategy keeps encoder and LLM KV-caches isolated per user, allowing any model replica to serve any chunk of any active session without coordination overhead." confidence: high relevance: medium limitations: - "The spoken Q&A benchmarks used for intelligence comparison (Web Questions, LlaMA Questions, Trivia QA) were synthesised from text using edge-tts rather than collected from real speakers. Results on naturally spoken or noisy input are not reported, limiting generalisability claims about real-world speech understanding." - "Speech output quality is evaluated primarily through CER on 1000 utterances using a single speaker, not through subjective MOS ratings or speaker naturalness benchmarks. It is therefore difficult to assess voice quality relative to other systems. The system supports a limited number of output speakers and does not yet support style or voice instruct-following, which the authors flag as future work. Emotion understanding and audio captioning are also deferred to a planned encoder upgrade. The duplex state classifier operates at chunk boundaries (approximately 160-320 ms non-statistical latency), which may be perceptible in fast-paced dialogue." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2411.01156" published_date: "2024-11-02" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade] claims: - claim_id: eliminating_grapheme_to_phoneme_conversion_by_directly_feeding role: supports claim: "Eliminating grapheme-to-phoneme conversion by directly feeding raw text to an LLM backbone is viable for multilingual TTS and can improve handling of context-dependent polyphonic words." source: "§1, §3" evidence: "The first component is the Dual Autoregressive (Dual-AR) language model, which replaces G2P preprocessing by leveraging an LLM backbone for direct linguistic feature extraction from raw text." confidence: high relevance: medium - claim_id: hierarchical_decomposition_of_autoregressive_token_generation_into_semantic role: supports claim: "Hierarchical decomposition of autoregressive token generation into semantic-level and acoustic-level stages improves codebook stability in grouped scalar quantization." source: "§3.1, §3.1.1" evidence: "This hierarchical decomposition is intended to improve the stability of the GFSQ codebook during autoregressive generation, which the authors report as a persistent challenge in token-based TTS when codebook dimensions are large." confidence: high relevance: high - claim_id: grouped_finite_scalar_vector_quantization_achieves_higher_codebook role: supports claim: "Grouped Finite Scalar Vector Quantization achieves higher codebook utilisation than residual vector quantization alternatives, mitigating dead-code collapse." source: "§3.2.2, §3.2.3" evidence: "The quantization scheme, GFSQ (Grouped Finite Scalar Vector Quantization), combines Finite Scalar Quantization (FSQ) with Group Vector Quantization (GVQ), partitioning the feature space into groups and applying scalar quantization per group." confidence: high relevance: medium - claim_id: real_time_tts_inference_with_low_first_packet role: supports claim: "Real-time TTS inference with low first-packet latency is achievable on consumer GPU hardware through standard inference optimisations without architectural compromise." source: "§4.2" evidence: "Inference uses KV caching, torch compile, and other optimisations, achieving a first-packet latency of 150ms and a real-time factor of approximately 1:5 on an RTX 4060 mobile GPU and 1:15 on an RTX 4090." confidence: high relevance: medium limitations: - "The entire experimental evaluation is conducted on a proprietary test set with undisclosed composition and size. No public benchmark is used, making it impossible to independently verify the claimed superiority over CosyVoice and F5-TTS or to compare against the broader literature." - "The MOS evaluation uses \"naive listeners\" rather than trained raters or crowdsourced panels following standard listening test protocols (e.g. ITU-T P.800), which may inflate scores relative to conventional evaluations. The paper does not report model size, training compute, or inference memory requirements in full, limiting reproducibility. DPO training details are omitted from the main training description. The paper does not evaluate cross-lingual transfer or accent preservation, which are claimed motivations for the non-G2P design. It is also unclear how the system handles low-resource languages beyond the eight listed in the training data." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2411.17607" published_date: "2024-11-26" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - architecture_variant current_role: influential method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: synthetic_speech_text_interleaved_data_generated_by_converting role: supports claim: "Synthetic speech-text interleaved data, generated by converting text spans to speech tokens using a learned text-to-token model, enables effective cross-modal knowledge transfer from pre-trained LLMs to the speech domain." source: "§2.2, §3.3.1, Table 5" evidence: "To generate 600B tokens of synthetic interleaved data without requiring any parallel speech-text corpus, a 1.5B text-to-token model is trained on TTS datasets to predict speech token sequences from text." confidence: high relevance: high - claim_id: lower_speech_tokenizer_frame_rates_improve_speech_language role: supports claim: "Lower speech tokenizer frame rates improve speech language modelling performance within a fixed token budget, with gains plateauing around 12.5 Hz where information loss begins to outweigh efficiency benefits." source: "§3.3.2, Figure 3a" evidence: "On pre-training benchmarks (Table 3), the 9B model outperforms all baselines on speech language modelling tasks, matching Spirit-LM and Moshi on Spoken TopicStoryCloze (S setting: 82.9) while substantially exceeding them on cross-modal settings (T→S: 85.0, S→T: 93.6)." confidence: high relevance: high - claim_id: supervised_speech_tokenization_derived_from_asr_model_fine role: supports claim: "Supervised speech tokenization derived from ASR model fine-tuning achieves stronger semantic preservation at low frame rates than unsupervised codec-based tokenisers, while maintaining competitive speech reconstruction quality." source: "§2.1, Table 1" evidence: "Training uses ASR supervision rather than waveform reconstruction, yielding discrete tokens with strong semantic preservation even at low frame rates." confidence: high relevance: high - claim_id: speech_text_pre_training_with_interleaved_data_substantially role: supports claim: "Speech-text pre-training with interleaved data substantially narrows the gap between speech-only and speech-to-text performance on spoken question answering, suggesting that cross-modal alignment transfers factual knowledge from text representations to speech decoding." source: "§3.2, Table 3" evidence: "On pre-training benchmarks (Table 3), the 9B model outperforms all baselines on speech language modelling tasks, matching Spirit-LM and Moshi on Spoken TopicStoryCloze (S setting: 82.9) while substantially exceeding them on cross-modal settings (T→S: 85.0, S→T: 93.6)." confidence: high relevance: medium - claim_id: an_intermediate_text_response_in_speech_generation_text role: supports claim: "An intermediate text response in speech generation (text-guided mode) provides meaningful accuracy gains for knowledge-intensive tasks, but a well-pre-trained model operating purely in the speech domain can still match text-guided baselines from prior work." source: "§3.2, Table 4" evidence: "Even in direct speech-to-speech mode (without text guidance), the model (scores 3.18 / 3.2) still performs comparably to text-guided baselines from prior work." confidence: high relevance: high limitations: - "No code or model weights are publicly released with this paper, and the proprietary Chinese ASR dataset (10k hours) used in tokenizer training cannot be replicated by third parties, limiting reproducibility of the full pipeline." - "The fine-tuning dataset (SpeechDialog-90K) is synthesised using MeloTTS for speech responses, so the chatbot's output speech may inherit MeloTTS quality characteristics rather than reflecting the model's own generative capacity. The evaluation of spoken chatbots relies on GPT-4 scoring, which is a reasonable but proxy measure that may not correlate perfectly with human judgements. The paper does not evaluate multilingual spoken QA or chatbot performance despite training on English and Chinese data. Full-duplex conversation capability, demonstrated by Moshi, is not explored. Scaling beyond 9B parameters is left as future work." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2411.19842" published_date: "2024-11-29" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [codec] architecture: [transformer-enc-dec, VAE] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure] claims: - claim_id: scaling_transformer_architecture_parameter_count_in_neural_audio role: supports claim: "Scaling transformer architecture parameter count in neural audio codecs produces consistent quality improvements across objective and subjective metrics." source: "§4.6, Table 4" evidence: "Scaling ablations (Table 4) confirm consistent quality improvement as parameter count grows from 240M to 540M to 950M, with all metrics improving monotonically." confidence: high relevance: high - claim_id: finite_scalar_quantization_achieves_near_perfect_codebook_utilization role: supports claim: "Finite scalar quantization achieves near-perfect codebook utilization without explicit utilization regularization, simplifying downstream generative modeling compared to RVQ." source: "§3.2, §A.8, Table 9" evidence: "Near-perfect codebook utilization follows from this formulation without any explicit utilization regularization." confidence: high relevance: low - claim_id: a_neural_codec_trained_exclusively_on_english_speech role: supports claim: "A neural codec trained exclusively on English speech can generalize effectively to unseen languages, outperforming multilingual-trained baselines of similar scale on most objective metrics." source: "§A.5, Table 7" evidence: "On multilingual LibriSpeech (Table 7), TAAE trained on English only outperforms all baselines across all seven evaluated languages on most metrics, including Mimi which was trained on 7M hours (roughly 70 times more data), though Mimi leads slightly on PESQ in several languages." confidence: high relevance: high - claim_id: perceptual_losses_derived_from_self_supervised_speech_models role: supports claim: "Perceptual losses derived from self-supervised speech models (WavLM-Large features) are critical for achieving intelligible reconstruction at low bitrates, beyond what adversarial and spectral reconstruction losses alone provide." source: "§3.4, §A.1, Table 3" evidence: "Finetuning replaces the reconstruction losses with a WavLM-Large perceptual feature-matching loss, which the paper shows is essential for achieving intelligible, high-quality reconstruction (Table 3, Appendix A.1)." confidence: high relevance: low - claim_id: systematic_spectral_bias_in_multi_resolution_stft_discriminators role: supports claim: "Systematic spectral bias in multi-resolution STFT discriminators, arising from power-of-two FFT configurations, causes periodic reconstruction artifacts that disproportionately affect large-capacity codec architectures." source: "§3.3, §B.5" evidence: "Pretraining uses an adversarial feature-matching loss from a multi-resolution complex STFT discriminator, augmented with exponentially decaying L1 waveform and STFT reconstruction losses." confidence: high relevance: high limitations: - "Training data is 16 kHz English audiobook speech only (105k hours). Multilingual generalization results are promising but the model was not trained or optimized for non-English data; claims about multilingual capability should be interpreted cautiously relative to models with dedicated multilingual training at scale." - "The model has not been evaluated on noisy speech, overlapping speakers, or environmental audio, which are common real-world conditions. The large parameter count (950M) requires substantially more compute than lighter baselines (DAC at 76M, Mimi at 80M); while the RTF is acceptable on H100 GPUs for longer utterances, latency for short clips is roughly 3x that of smaller models, which matters for streaming applications. The post-hoc Residual FSQ decomposition is restricted to specific level configurations (L = 2^n + 1); arbitrary bitrate targets are not directly achievable without retraining. The systematic bias analysis in the discriminator (§B.5) raises open questions about whether similar biases appear in other convolutional discriminator architectures and how to address them in the general case." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2412.02612" published_date: "2024-12-03" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: speech_text_interleaved_pre_training_at_trillion_token role: supports claim: "Speech-text interleaved pre-training at trillion-token scale enables an LLM to acquire speech understanding and generation capabilities that substantially close the gap between spoken and textual reasoning quality." source: "§4.1, Table 4" evidence: "The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607)." confidence: high relevance: high - claim_id: single_codebook_supervised_speech_tokenizers_derived_from_asr role: supports claim: "Single-codebook supervised speech tokenizers derived from ASR models can achieve compact bitrates (below 200bps) while retaining sufficient semantic fidelity for both downstream language modeling and speech synthesis." source: "§3.1, Table 1" evidence: "The design deliberately sacrifices full acoustic fidelity for semantic compactness: a single codebook at 12.5Hz avoids the multi-codebook complexity of RVQ-based codecs while preserving enough semantic content for accurate ASR (2.10 WER on LibriSpeech clean) and high-quality reconstruction (MOSNet 3.39)." confidence: high relevance: high - claim_id: streaming_interleaved_generation_templates_alternating_text_and_speech role: supports claim: "Streaming interleaved generation templates, alternating text and speech token output, enable low-latency spoken responses without sacrificing content coherence by ensuring text generation consistently precedes its corresponding speech." source: "§3.3" evidence: "The base LLM is continued pre-training on 1 trillion tokens: 30% text, one epoch of 700k-hour unsupervised speech, one epoch of supervised ASR/TTS data, and the remainder comprising interleaved speech-text data synthesized from text corpora using a text-to-token model (described in 2411.17607)." confidence: high relevance: high - claim_id: end_to_end_speech_language_models_that_include role: supports claim: "End-to-end speech language models that include dedicated speech pre-training produce measurably higher-quality and more stylistically controllable speech responses than LLMs fine-tuned solely on speech question-answering data." source: "§5.2, Table 6" evidence: "Training uses a mix of supervised ASR data (LibriSpeech, GigaSpeech, CommonVoice, AISHELL-1, MLS-Eng, Wenet) and 700k hours of pseudo-labelled unsupervised speech." confidence: high relevance: high - claim_id: decoupling_the_text_and_speech_output_subtasks_during role: supports claim: "Decoupling the text and speech output subtasks during fine-tuning, through separate loss-masking passes at different epoch rates, addresses the discrepancy in learning dynamics between the two modalities." source: "§4.2.2" evidence: "Separate loss-masking passes train the text-output and speech-output subtasks at different rates (4 vs. 20 epochs) to account for their different learning speeds." confidence: high relevance: medium limitations: - "The paper reports no subjective listening test (MOS/MUSHRA) on the chat model output; UTMOS is used as a proxy for speech naturalness, and the chat evaluation relies on GPT-4o scoring of ASR transcriptions, introducing cascaded error from both the vocoder quality and the Whisper transcription step." - "The 175bps tokenizer trades acoustic fidelity for compactness: VisQOL at 12.5Hz (2.52) is lower than SpeechTokenizer variants and the 50Hz variant of the same system. This may limit voice cloning quality and the fidelity of paralinguistic feature reproduction (accent, fine-grained emotion), though the paper does not directly evaluate these." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2412.10117" published_date: "2024-12-13" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: finite_scalar_quantization_achieves_full_codebook_utilization_in role: supports claim: "Finite scalar quantization achieves full codebook utilization in supervised speech tokenizers, capturing substantially more semantic content than vector quantization at equivalent bitrates." source: "§2.2, §4.1, Table 4" evidence: "FSQ achieves 100% codebook utilization vs. 23% for VQ, and reduces ASR error rate from 18.26% to 10.67% on CommonVoice EN — indicating substantially more semantic content is preserved." confidence: high relevance: high - claim_id: replacing_a_randomly_initialized_custom_lm_with_a role: supports claim: "Replacing a randomly initialized custom LM with a pre-trained LLM backbone improves content consistency in hybrid TTS systems without requiring a separate text encoder." source: "§2.3, §4.3, Table 7" evidence: "Unified Text-Speech LM: Uses Qwen2.5-0.5B as the backbone, removing the text encoder and speaker embedding from the sequence entirely." confidence: high relevance: medium - claim_id: streaming_and_non_streaming_synthesis_can_be_unified role: supports claim: "Streaming and non-streaming synthesis can be unified in a single autoregressive model through interleaved text-speech token sequences, with virtually lossless quality on typical inputs relative to offline mode." source: "§2.3, §4.2, Table 8" evidence: "Two training modes share one model: (1) Non-streaming: [S, text tokens, T, speech tokens, E]; (2) Streaming: text and speech tokens are interleaved at ratio N:M=5:15 — every 5 text tokens are followed by 15 speech tokens, with \"filling tokens\" inserted when text tokens are not yet available." confidence: high relevance: high - claim_id: training_a_flow_matching_model_simultaneously_on_multiple role: complicates claim: "Training a flow matching model simultaneously on multiple causal mask types — from non-causal to full-causal — enables a single model to span the latency-quality trade-off continuum at inference time, with masks providing implicit self-distillation." source: "§2.4, §4.3, Table 8" evidence: "Chunk-aware Causal Flow Matching: The flow matching model is trained with four mask types simultaneously: non-causal (offline, best quality), full-causal (lowest latency), Chunk-M (past + M future frames), and Chunk-2M (near-offline quality with moderate latency)." confidence: high relevance: medium - claim_id: differentiable_asr_reward_optimization_generalizes_better_to_out role: supports claim: "Differentiable ASR reward optimization generalizes better to out-of-domain and hard-case inputs than preference-based DPO in TTS speaker fine-tuning." source: "§2.8, §4.7, Table 11" evidence: "A differentiable ASR reward (Gumbel-softmax sampling through FSQ indices → frozen ASR decoder) avoids the four-forward-pass overhead of standard DPO and generalizes better to out-of-domain/hard cases." confidence: high relevance: medium limitations: - "- EN quality still lags CosyVoice 2 behind Seed-TTS and F5-TTS on SEED test-en (WER 2.57% vs. 2.25% and 1.83%), reflecting data imbalance toward Chinese. - Japanese synthesis degrades due to character set overlap with Chinese (CER 18.79% test-ja vs. 7.98% test-ko). - Cannot control timbre through text instructions. - Singing not supported. - Streaming still incurs a hard degradation on test-hard, suggesting that contextual information from future text is important for difficult patterns." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2412.15649" published_date: "2024-12-20" entry_date: "2026-07-25" year: 2024 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: decoupling_speaker_identity_from_semantic_content_in_spoken role: supports claim: "Decoupling speaker identity from semantic content in spoken dialogue systems enables zero-shot timbre control without modifying the language model or adding speaker-conditioning layers." source: "§3.4" evidence: "Because the semantic tokens encode only linguistic content (not speaker identity), a conditional flow-matching vocoder conditioned on a speaker audio prompt converts semantic tokens to mel spectrograms, which HiFi-GAN then synthesizes to waveform." confidence: high relevance: high - claim_id: grouping_audio_tokens_to_reduce_the_frequency_mismatch role: supports claim: "Grouping audio tokens to reduce the frequency mismatch between text and speech representations substantially improves speech-text alignment and training efficiency in parallel audio-text dialogue models." source: "§3.3, Table 5" evidence: "With G = 3, the audio sequence length is reduced by 3x, enabling the model to converge with less than half the GPU hours of the G = 1 baseline while achieving substantially better speech-text alignment (4.54% ASR-WER vs. 18.23% at G = 1)." confidence: high relevance: high - claim_id: single_stage_fine_tuning_on_dialogue_data_can role: supports claim: "Single-stage fine-tuning on dialogue data can match or outperform multi-stage pipelines that include ASR or TTS pre-training, because modality-specific pre-training degrades instruction-following and general knowledge retention." source: "§5.3.2, Table 6" evidence: "The entire system is trained in a single stage on dialogue data (primarily VoiceAssistant-400K, ~664 hours of instruction audio), with full fine-tuning of all components except the Whisper encoder, which is kept frozen." confidence: high relevance: medium - claim_id: replacing_audio_token_history_with_text_only_history role: supports claim: "Replacing audio-token history with text-only history in multi-turn spoken dialogue models improves the system's ability to handle longer conversation contexts while leveraging pre-trained LLM in-context learning." source: "§3.5" evidence: "For multi-turn dialogue, Historical Text Prompting replaces the typical approach of maintaining interleaved audio-text token histories." confidence: high relevance: high - claim_id: current_spoken_dialogue_models_consistently_underperform_text_only role: supports claim: "Current spoken dialogue models consistently underperform text-only LLMs of similar scale on semantic content quality, even after dialogue fine-tuning." source: "§5.1, Table 3" evidence: "At 0.5B scale, SLAM-Omni substantially outperforms Mini-Omni and Mini-Omni2 on ChatGPT-based content scoring (39.32 vs. 22.58 and 26.56 overall), achieves the highest UTMOS across all SDMs including the larger 7-9B models (4.45 vs. 4.37 for Freeze-Omni), and the lowest ASR-WER (4.54% vs. 6.05% for Mini-Omni and 10.42% for LLaMA-Omni)." confidence: high relevance: medium limitations: - "Historical text prompting discards all non-verbal information from prior dialogue turns (prosody, emotion, paralinguistic cues). In scenarios where voice-level context matters for dialogue coherence, this strategy may reduce response quality in ways not captured by the text-based evaluation metrics used." - "The system is evaluated exclusively at 0.5B scale. The single-stage training advantage may not hold for larger LLMs, where the data volume required for joint audio-text modeling grows substantially." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2502.04128" published_date: "2025-02-06" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [preference_optimized_autoregression] claims: - claim_id: single_stage_autoregressive_tts_trained_with_next_token role: supports claim: "Single-stage autoregressive TTS trained with next-token prediction over discrete speech tokens is competitive with multi-stage AR+NAR pipelines on intelligibility and speaker similarity in continuation mode, though SIM-o gaps remain due to codec acoustic reconstruction limits." source: "§3.2.4, Table 3" evidence: "However, speaker similarity (SIM-o) for direct inference is notably weaker than these baselines, reflecting the acoustic reconstruction ceiling of a single-VQ codec versus multi-layer RVQ or mel-based vocoders." confidence: high relevance: high - claim_id: both_model_scale_and_training_data_volume_independently role: supports claim: "Both model scale and training data volume independently improve TTS quality across naturalness, prosody, and text comprehension, consistent with scaling laws observed in text LLMs." source: "§2.3, §3.2.2, Tables 2, 4" evidence: "Scaling both model size and training data consistently improves all metrics: WER on test-en drops from 3.71% (1B-80k) to 2.97% (8B-250k), and emotion similarity on ESD rises from 0.753 to 0.778 (English) across model sizes (Table 4)." confidence: high relevance: medium - claim_id: inference_time_compute_scaling_via_speech_understanding_verifiers role: complicates claim: "Inference-time compute scaling via speech understanding verifiers can substantially improve speaker similarity and emotional expressiveness beyond what train-time scaling alone achieves, at the cost of additional inference compute." source: "§2.4, §3.2.3, Figure 2, Table 2" evidence: "Inference-time scaling uses off-the-shelf speech understanding models as verifiers." confidence: high relevance: medium - claim_id: pure_process_reward_model_beam_search_for_tts role: supports claim: "Pure process reward model beam search for TTS is prone to mode collapse that degrades content accuracy (WER), and a hybrid partial-PRM strategy is needed to preserve both speaker similarity and intelligibility." source: "§3.2.3, Figure 2" evidence: "Two search strategies are compared: Best-of-N (output reward model, ORM) and beam search with a process reward model (PRM) that evaluates speaker similarity at each 0.5-second step." confidence: high relevance: medium - claim_id: single_vq_codecs_can_achieve_intelligibility_and_naturalness role: complicates claim: "Single-VQ codecs can achieve intelligibility and naturalness competitive with multi-layer RVQ codecs at the same token rate, but acoustic fidelity (speaker similarity) remains the limiting factor for single-VQ reconstruction." source: "§3.1.2, Table 1" evidence: "However, speaker similarity (SIM-o) for direct inference is notably weaker than these baselines, reflecting the acoustic reconstruction ceiling of a single-VQ codec versus multi-layer RVQ or mel-based vocoders." confidence: high relevance: high limitations: - "The SIM-o gap between Llasa and RVQ-based baselines is intrinsic to the single-VQ design: acoustic reconstruction from a 65,536-entry single codebook at 50 Hz is weaker than 8-layer RVQ codecs, and this gap is only partially recovered by inference-time search. Systems requiring high timbre fidelity in a single inference pass would need a different codec design." - "Inference-time compute scaling requires running multiple candidates (beam search or Best-of-N) with auxiliary verifier models, which increases latency and compute cost substantially and makes the approach unsuitable for real-time or low-resource applications. The paper does not characterize latency or wall-clock overhead of the search strategies." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2502.05512" published_date: "2025-02-08" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: speaker_conditioning_via_a_multi_reference_conformer_perceiver role: supports claim: "Speaker conditioning via a multi-reference Conformer Perceiver improves zero-shot voice cloning stability and timbre consistency over single-vector speaker embeddings." source: "§2.3, Table 4" evidence: "Speaker conditioning uses a Conformer-based Perceiver encoder with a subsampling rate of 2, replacing the single-vector speaker embeddings used by Tortoise and CosyVoice." confidence: high relevance: medium - claim_id: direct_waveform_decoding_from_lm_hidden_states_via role: supports claim: "Direct waveform decoding from LM hidden states via a GAN vocoder achieves competitive audio quality with faster inference than diffusion-based intermediate representation decoding." source: "§2.4, Table 5" evidence: "The speech decoder directly converts the LM's last hidden state to waveform via BigVGAN2, with a 4x interpolation step from 25 Hz to 100 Hz before decoding." confidence: high relevance: medium - claim_id: fsq_reaches_near_100_codebook_utilisation_with_less role: supports claim: "FSQ reaches near-100% codebook utilisation with less training data than VQ, though VQ converges to similar utilisation with sufficient data scale." source: "§3.3.2, Figure 2" evidence: "The paper compares Vector Quantization (VQ) against Finite Scalar Quantization (FSQ) with levels [8, 8, 8, 6, 5], finding that VQ collapses to around 55% codebook utilisation at 6k hours of training data whereas FSQ reaches near-100% utilisation; at 34k hours both converge." confidence: high relevance: medium - claim_id: hybrid_character_pinyin_tokenisation_enables_reliable_correction_of role: supports claim: "Hybrid character-pinyin tokenisation enables reliable correction of Chinese polyphonic character mispronunciations at inference time without requiring a separate grapheme-to-phoneme module." source: "§3.3.1, Table 2" evidence: "To handle polyphonic characters, a character-pinyin hybrid modelling approach is used at training time: 50% of training samples have 20% of their non-polyphonic Chinese characters randomly replaced with pinyin, training the model to accept pinyin corrections at inference." confidence: high relevance: high limitations: - "Model size is not reported, and the training data pipeline uses proprietary internet-sourced audio with pseudo-labels from commercial ASR; neither the data nor the code is released, limiting reproducibility." - "The system is limited to Chinese and English, with acknowledged weak emotional expression replication. Instruction-based voice generation is explicitly unsupported. The MOS evaluation relies on 100 samples from an unspecified test set distribution, and the SPK-SIM metric uses ERes2Net rather than a standardised model, making direct comparison with published baselines difficult. The paper does not report streaming latency or real-time factor, despite positioning the hybrid architecture as streaming-capable." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2502.06490" published_date: "2025-02-10" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, VC, codec] architecture: [] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: acoustic_tokens_and_semantic_tokens_occupy_fundamentally_distinct role: complicates claim: "Acoustic tokens and semantic tokens occupy fundamentally distinct points in a reconstruction-versus-semantics trade-off space, and no single tokenization strategy currently achieves strong performance on both axes simultaneously." source: "§VI-C, Table I" evidence: "General-purpose acoustic tokens achieve the strongest reconstruction fidelity (EnCodec Q=8: PESQ 2.83, WER 1.53%) but carry high bitrates (6 kbps) and relatively weak semantic probe performance (IC accuracy 34.8%)." confidence: high relevance: high - claim_id: speaker_disentanglement_in_acoustic_tokens_enables_voice_conversion role: supports claim: "Speaker disentanglement in acoustic tokens enables voice conversion capability but consistently reduces reconstruction quality metrics at equivalent bitrates." source: "§VI-D, Table I" evidence: "Semantic SSL tokens (HuBERT Large, WavLM Large) achieve the best voice conversion speaker similarity (SECS ~0.87–0.88) and superior intent classification (67–74%) at very low bitrates (0.55 kbps), but suffer severe prosody loss (GPE ~15–18%, versus ~1% for acoustic tokens) and poor signal quality (PESQ ~1.15)." confidence: high relevance: high - claim_id: k_means_clustering_on_ssl_model_embeddings_discards role: supports claim: "K-means clustering on SSL model embeddings discards prosody information more severely than supervised or internally-quantized semantic tokenizers, making offline clustering ill-suited for tasks requiring prosody fidelity." source: "§VI-C, §VI-D, Table I" evidence: "The supervised S3 Tokenizer offers an intermediate profile: good VC speaker similarity (0.868) with better prosody retention than clustering-based SSL tokens, suggesting its ASR objective preserves some paralinguistic structure that k-means discards." confidence: high relevance: high - claim_id: acoustic_byte_pair_encoding_achieves_greater_length_reduction role: supports claim: "Acoustic byte-pair encoding achieves greater length reduction on tokens with lower information density, such as speaker-decoupled and semantic tokens, than on general-purpose acoustic tokens." source: "§V-A, Figure 8" evidence: "Acoustic BPE length reduction disproportionately compresses tokens with low information density: semantic tokens and speaker-decoupled LSCodec tokens compress more than general-purpose acoustic tokens, confirming that BPE gain is inversely related to information density." confidence: high relevance: high - claim_id: single_codebook_tokens_at_very_low_frame_rates role: supports claim: "Single-codebook tokens at very low frame rates improve compatibility with language model generation but currently exhibit measurable quality and intelligibility degradation relative to multi-codebook or higher frame-rate alternatives." source: "§VIII.1, §VI-C" evidence: "Semantic SSL tokens (HuBERT Large, WavLM Large) achieve the best voice conversion speaker similarity (SECS ~0.87–0.88) and superior intent classification (67–74%) at very low bitrates (0.55 kbps), but suffer severe prosody loss (GPE ~15–18%, versus ~1% for acoustic tokens) and poor signal quality (PESQ ~1.15)." confidence: high relevance: high limitations: - "The experimental comparisons are conducted on English data only (LibriTTS, LibriSpeech), leaving multilingual tokenization trade-offs unexplored. The unified vocoder (CTX-vec2wav) is specifically designed for semantic tokens, which may introduce a systematic advantage for semantic token types in reconstruction experiments. Not all acoustic tokens support voice conversion in the paper's framework, so the VC comparison covers only a subset of systems." - "Open questions identified by the survey include: the bitrate lower bound for single-codebook tokens with acceptable intelligibility; whether causal SSL architectures can match non-causal models for semantic token quality; how VFR tokens perform on generative tasks beyond ASR; and whether token vocoders trained at scale can match flow matching-based alternatives for timbre controllability." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2502.07243" published_date: "2025-02-11" entry_date: "2026-07-25" year: 2025 venue: "ICLR" task: [TTS, VC] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hierarchical_ar_nar_codec_tts, semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: the_vq_vae_codebook_vocabulary_size_can_function role: complicates claim: "The VQ-VAE codebook vocabulary size can function as a self-supervised information bottleneck for progressive disentanglement of timbre, style, and linguistic content in self-supervised speech representations." source: "§3.1, Table 2" evidence: "At its core, the approach treats the VQ-VAE codebook vocabulary size as a controllable information bottleneck applied to HuBERT-Large 18th-layer features: a large vocabulary (4,096 entries) produces content-style tokens that preserve style and content but strip out timbre; a small vocabulary (32 entries) produces content tokens that strip out both timbre and most style, retaining primarily linguistic content." confidence: high relevance: medium - claim_id: zero_shot_style_imitation_accent_and_emotion_conversion role: supports claim: "Zero-shot style imitation (accent and emotion conversion) without annotation can match or exceed supervised baselines that rely on parallel corpora and style labels." source: "§4.3, Table 4" evidence: "For zero-shot style imitation (Table 4), Vevo-Style outperforms accent conversion baselines (ASR-AC, VoiceShop, Conv-Speak) and emotion conversion baseline (Emovox) in N-CMOS and style accuracy metrics (A-ACC, A-CMOS, E-ACC, E-CMOS) despite using no parallel corpus, no style labels, and no textual transcriptions." confidence: high relevance: medium - claim_id: hybrid_two_stage_pipelines_combining_autoregressive_style_modeling role: supports claim: "Hybrid two-stage pipelines combining autoregressive style modeling with flow-matching acoustic generation can decouple style and timbre control more effectively than single-stage approaches that use in-context learning to mimic all speech attributes jointly." source: "§3.4, Tables 3–5" evidence: "The second stage, acoustic modeling, uses a 334M-parameter flow-matching transformer to generate mel spectrograms from content-style tokens, conditioned on a timbre reference mel spectrogram via a span-masking strategy inspired by Voicebox." confidence: high relevance: high - claim_id: autoregressive_models_in_zero_shot_tts_consistently_trade role: supports claim: "Autoregressive models in zero-shot TTS consistently trade intelligibility (higher WER) for stronger style imitation compared to non-autoregressive alternatives trained on the same data." source: "§4.4, Tables 5, 9" evidence: "For zero-shot TTS (Table 5, ACCENT+EMOTION samples), Vevo-TTS achieves emotion similarity MOS of 4.03 and accent similarity MOS of 4.12, surpassing CosyVoice (trained on 171K hours of in-the-wild data) and MaskGCT (trained on 100K hours) on these style dimensions, while being trained only on 60K hours of audiobooks." confidence: high relevance: high - claim_id: duration_reduction_on_content_tokens_improves_style_transfer role: supports claim: "Duration reduction on content tokens improves style transfer fidelity by removing unit-level duration patterns that encode source speaking style." source: "§4.5, Table 6" evidence: "A duration reduction step collapses consecutive duplicate content tokens before feeding them to the model, which further suppresses unit-level duration patterns that encode style, and also shortens inference sequence length." confidence: high relevance: high limitations: - "Style imitation evaluations (Table 4) use demo website samples from baseline systems as the test set, meaning evaluation conditions (recording environment, speaker demographics, utterance content) differ between Vevo and baselines. These comparisons are suggestive but not controlled, and the reported improvements should be treated as approximate." - "Training is restricted to English audiobook speech (clean, single-domain), and no multilingual or expressive speech experiments are reported. The content-style token vocabulary size (K_s = 4096) and content token vocabulary size (K_c = 32) are empirically selected; the authors note these may not be globally optimal. The AR content-style model has 463M parameters and requires sequential decoding, introducing latency that could be problematic for streaming applications. The self-supervised disentanglement quality depends on HuBERT-Large features, requiring a large pre-trained SSL model as a prerequisite. Style controllability through a single reference utterance may be brittle for rare or highly expressive speaking styles not represented in the audiobook training distribution." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2502.11946" published_date: "2025-02-17" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, preference_optimized_autoregression] claims: - claim_id: a_dual_codebook_interleaved_tokenizer_that_combines_linguistic role: supports claim: "A dual-codebook interleaved tokenizer that combines linguistic and semantic representations can achieve lower ASR error rates than either codebook alone, without sacrificing acoustic reconstruction quality." source: "§4.4, §6.2.1" evidence: "This interleaving is the core tokenizer innovation: semantic tokens maintain semantic coherence in generated continuations, while linguistic tokens preserve the acoustic quality needed for intelligible reconstruction." confidence: high relevance: high - claim_id: scaling_autoregressive_llm_backbone_size_from_3b_to role: supports claim: "Scaling autoregressive LLM backbone size from 3B to 130B parameters produces substantial gains in speech synthesis intelligibility on standard TTS benchmarks, suggesting speech generation quality is LLM-scale-sensitive." source: "§6.2.2, Table 3" evidence: "On the SEED-TTS benchmark for TTS quality, Step-Audio-TTS-3B achieves 1.31% CER (Chinese) and 2.31% WER (English), outperforming CosyVoice, MaskGCT, and FireRedTTS on intelligibility while remaining competitive on speaker similarity." confidence: high relevance: high - claim_id: rlhf_reward_models_trained_on_speech_interaction_data role: supports claim: "RLHF reward models trained on speech interaction data can exhibit systematic failure modes (such as rewarding evasive non-answers to unclear audio) unless explicit counter-examples are constructed during reward model training." source: "§5.2.6" evidence: "The AQTA system is post-trained with PPO-based RLHF using a two-stage reward model, where a known failure mode (\"deaf hacking\") was identified and mitigated by constructing explicit counter-examples in the reward training data." confidence: high relevance: medium - claim_id: speculative_response_generation_triggered_by_voice_activity_detection role: supports claim: "Speculative response generation triggered by voice activity detection can reduce per-response latency by approximately 500 ms, with roughly 40% of pre-generated responses being usable, enabling practical real-time conversational systems." source: "§3.4" evidence: "Real-time inference uses speculative response generation triggered by voice activity detection pauses." confidence: high relevance: medium - claim_id: synthetic_speech_data_generated_by_a_large_multi role: supports claim: "Synthetic speech data generated by a large multi-modal model can substitute for manually curated recordings in training TTS systems for low-resource dialects, emotions, and singing styles." source: "§5.1.1" evidence: "The same decoder architecture underlies the distilled standalone Step-Audio-TTS-3B model, which is trained on synthetic data produced by the 130B model rather than manually curated recordings." confidence: high relevance: medium limitations: - "The StepEval-Audio-360 benchmark is proprietary and created by the same team; human evaluation results on it cannot be independently reproduced. Open-source benchmark comparisons mix locally re-run models with results taken from original publications, complicating direct numerical comparison." - "Speaker similarity scores for the distilled Step-Audio-TTS-3B are noticeably lower than CosyVoice 2 on both Chinese and English SEED-TTS tests, suggesting that the dual-codebook approach trades some acoustic identity preservation for intelligibility gains. The AQTA+TTS design still relies on a cascade: errors in ASR transcription of history or in text generation propagate to the TTS stage. The paper's future work section acknowledges that purely end-to-end audio-in/audio-out (AQAA) remains unsolved. Evaluation for singing, RAP, and dialect control is limited to instruction following scores without reference audio; absolute quality in these dimensions is difficult to assess from the reported numbers alone." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2502.17239" published_date: "2025-02-24" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: multi_codebook_rvq_tokenizers_with_semantic_alignment_objectives role: supports claim: "Multi-codebook RVQ tokenizers with semantic alignment objectives better preserve both acoustic and linguistic content than single-codebook designs, with each additional layer reducing ASR error substantially up to 8 layers." source: "§3.1, Table 1" evidence: "Layerwise dropout concentrates semantic content in the top codebook layers, analogous to the design principle in SpeechTokenizer." confidence: high relevance: high - claim_id: staged_pretraining_frozen_llm_first_then_joint_training role: supports claim: "Staged pretraining (frozen LLM first, then joint training) measurably reduces intelligence degradation in end-to-end speech LMs relative to single-stage joint training." source: "§3.3.1, Table 5" evidence: "This staged approach is designed to prevent audio modality training from overwriting textual knowledge; ablations (Table 5) show it closes roughly 2 percentage points of the S→T accuracy gap on sStoryCloze relative to single-stage training." confidence: high relevance: high - claim_id: text_guided_aligned_generation_where_the_model_completes role: supports claim: "Text-guided aligned generation, where the model completes text tokens before emitting the corresponding audio tokens, mitigates semantic incoherence in speech LM outputs." source: "§3.3, §4.1" evidence: "During inference the model alternates text and audio token generation, with modality-switching special tokens enforcing alignment, so text completion always precedes audio generation for the corresponding utterance." confidence: high relevance: high - claim_id: flow_matching_decoders_trained_as_post_vq_refinement role: supports claim: "Flow-matching decoders trained as post-VQ refinement stages recover significant audio quality lost during quantisation, with UTMOS improvements of 0.6 points possible without retraining the LLM." source: "§3.2, Table 3" evidence: "Baichuan-Audio builds on a 7B-parameter LLM backbone (Baichuan) extended with three interconnected components: a custom audio tokenizer, an independent audio head, and a flow-matching audio decoder." confidence: high relevance: medium - claim_id: end_to_end_speech_lms_evaluated_in_s role: supports claim: "End-to-end speech LMs evaluated in S→S mode exhibit notably lower benchmark performance than the same model evaluated in S→T mode, indicating that audio token generation itself introduces a quality penalty beyond the comprehension step." source: "§4.3, Table 8" evidence: "During inference the model alternates text and audio token generation, with modality-switching special tokens enforcing alignment, so text completion always precedes audio generation for the corresponding utterance." confidence: high relevance: high limitations: - "The TTS quality evaluation is limited to an in-house test set (MED-TTS); no comparison against standard TTS benchmarks (VCTK, LJSpeech, LibriTTS test-clean) or against dedicated TTS systems is reported. The naturalness and speaker similarity of the generated speech relative to state-of-the-art TTS systems is therefore unknown." - "The intelligence gap to GPT-4o-Audio remains large (roughly 15-20 percentage points on QA tasks), and the paper does not explain what architectural or data factors account for this difference. The OpenAudioBench evaluation uses GPT-4o as judge, which may introduce evaluation bias. The full-duplex and interruption-handling capabilities common in deployed spoken conversational agents are not evaluated. Training data mix decisions (e.g. dropping INTLV audio loss, ITTS loss design) are motivated empirically but without systematic ablation." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2503.01710" published_date: "2025-03-03" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hierarchical_ar_nar_codec_tts] claims: - claim_id: disentangling_speech_tokens_into_linguistic_content_and_speaker role: supports claim: "Disentangling speech tokens into linguistic content and speaker attributes within a single-stream codec enables a standard LLM to perform zero-shot TTS without a multi-stage pipeline." source: "§3, §4.1" evidence: "Speaker similarity in zero-shot TTS is Spark-TTS's relative weakness: it scores 0.672 SIM on test-zh and 0.584 SIM on test-en, outperforming Llasa (0.684, 0.574) but falling behind multi-stage models such as MaskGCT (0.774, 0.714) and Seed-TTS (0.796, 0.762)." confidence: high relevance: high - claim_id: small_llm_backbones_can_achieve_competitive_zero_shot role: supports claim: "Small LLM backbones can achieve competitive zero-shot TTS intelligibility when the codec reduces per-token modeling complexity through semantic alignment." source: "§6.4, Table 4" evidence: "The Spark-TTS language model fine-tunes Qwen2.5-0.5B in a decoder-only configuration, directly predicting the concatenated token sequence: for zero-shot TTS, the model conditions on text and global tokens extracted from a reference audio, then predicts the semantic token sequence." confidence: high relevance: high - claim_id: single_stage_autoregressive_tts_consistently_trails_multi_stage role: supports claim: "Single-stage autoregressive TTS consistently trails multi-stage or non-autoregressive methods on speaker similarity metrics, even when intelligibility is comparable." source: "§6.4, Table 4, Limitation" evidence: "Speaker similarity in zero-shot TTS is Spark-TTS's relative weakness: it scores 0.672 SIM on test-zh and 0.584 SIM on test-en, outperforming Llasa (0.684, 0.574) but falling behind multi-stage models such as MaskGCT (0.774, 0.714) and Seed-TTS (0.796, 0.762)." confidence: high relevance: high - claim_id: fsq_based_global_token_quantization_with_learnable_cross role: supports claim: "FSQ-based global token quantization with learnable cross-attention queries produces better speaker attribute representation than group-VQ at equivalent token lengths." source: "§6.2, Table 2" evidence: "The global tokenizer processes the Mel spectrogram through an ECAPA-TDNN encoder followed by a cross-attention module with learnable queries, producing a fixed-length sequence of 32 tokens that represent time-invariant speaker attributes." confidence: high relevance: high - claim_id: attribute_controllable_tts_benefits_from_hierarchical_coarse_to role: supports claim: "Attribute-controllable TTS benefits from hierarchical coarse-to-fine prediction within the LM inference loop rather than requiring separate conditioning modules." source: "§4.1, §6.3" evidence: "For attribute-controlled voice creation, a chain-of-thought (CoT) mechanism is employed: the model first predicts fine-grained attribute values (pitch in Hz rounded to integer, speaking rate in syllables-per-second) from coarse-grained labels (gender, 5-level pitch, 5-level speed), then generates global tokens, then semantic tokens." confidence: high relevance: medium limitations: - "Speaker similarity in zero-shot cloning is meaningfully lower than multi-stage methods (SIM 0.672 vs. 0.774 for MaskGCT on test-zh). The paper attributes this to AR variability without explicit disentanglement constraints between semantic and global tokens, and no solution is evaluated in this work." - "The VoxBox training data and BiCodec codec are trained on separate, relatively limited datasets (3k hours for BiCodec; 102.5k hours for the LM). The BiCodec training data is English-only (LibriSpeech + Emilia EN/CN), which may limit acoustic reconstruction quality for languages outside this distribution." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2503.14345" published_date: "2025-03-18" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: spontaneous_scripting_from_an_llm_is_roughly_as role: supports claim: "Spontaneous scripting from an LLM is roughly as important as the acoustic modeling choice for perceived spontaneity in long-form dialogue TTS." source: "§4.2.2, Table 3" evidence: "The paper demonstrates empirically that spontaneous scripting alone accounts for a large fraction of the final audio spontaneity." confidence: high relevance: medium - claim_id: full_sequence_interleaving_of_text_and_speech_codes role: supports claim: "Full-sequence interleaving of text and speech codes, extended to 40,000-token context windows with speaker-change tokens, enables coherent long-form zero-shot multi-speaker synthesis that turn-level concatenation cannot match." source: "§3.2.1, Tables 1–2" evidence: "The sequence design interleaves full podcast text with full speech codes (rather than per-turn interleaving), using speaker-change tokens to signal speaker transitions." confidence: high relevance: high - claim_id: curriculum_learning_progressively_exposing_a_codec_lm_to role: supports claim: "Curriculum learning, progressively exposing a codec LM to increasing dialogue complexity, is an effective strategy for developing long-context and spontaneous generation capability without requiring matched long-context data from the outset." source: "§3.2.1" evidence: "Training uses curriculum learning across three stages: single-turn single-speaker zero-shot TTS; long-context two-speaker non-conversational data (audiobooks, up to 40,000 tokens / 800 seconds); and long-context two-speaker conversational data (podcasts)." confidence: high relevance: high - claim_id: automatic_speaker_similarity_metrics_cosine_embedding_similarity_can role: supports claim: "Automatic speaker similarity metrics (cosine embedding similarity) can disagree with subjective speaker similarity ratings in long-form generation settings, particularly when the acoustic model attends to prosodic rather than purely timbral features." source: "§4.2.1" evidence: "However, SIM-O (automatic speaker cosine similarity) is notably lower for MoonCast on English (0.53 vs. 0.75 for the concatenation baseline), attributed to insufficient audiobook data in the English training curriculum, which the authors acknowledge as a limitation." confidence: high relevance: medium - claim_id: chunk_wise_autoregressive_decoding_with_a_causal_chunk role: supports claim: "Chunk-wise autoregressive decoding with a causal chunk mask provides a practical solution to the continuity and memory constraints of mel-spectrogram reconstruction from long semantic code sequences." source: "§3.2.2" evidence: "A chunk-wise causal attention mask allows each chunk to attend to all previously generated chunks, preserving continuity across boundaries without loading the full sequence." confidence: high relevance: high limitations: - "All evaluation is conducted on a small internal test set (4 knowledge sources for podcast, 7 podcasts for the script ablation), and training data is entirely proprietary (~515K hours). Results cannot be independently reproduced, and generalisability to other domains or languages beyond Chinese and English is unverified." - "The notably lower SIM-O score for English (0.53 vs. 0.75 baseline) reveals unresolved speaker consistency challenges in the English long-context setting, which the authors attribute to an imbalanced training curriculum favouring audiobook over conversational English data. Hallucinations in speaker attribution (utterances assigned to the wrong speaker) emerge from the interplay of semantic token timbre leakage, diarization errors in training data, and ambiguous filler-word interpretations; no mitigation is proposed beyond discussion. The two-speaker restriction (host + guest) is a deliberate scope limitation; extension to three or more speakers is left as future work. Evaluation is subjective-only for multi-speaker interactions; no standardised zero-shot TTS benchmark (LibriSpeech, VCTK) is used, limiting direct comparison to single-speaker zero-shot systems." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2504.02407" published_date: "2025-04-03" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [flow-matching] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: reformulating_the_output_of_a_flow_matching_tts role: supports claim: "Reformulating the output of a flow-matching TTS model as a Gaussian distribution is sufficient to enable standard policy-gradient RL algorithms without architectural redesign." source: "§2.2" evidence: "The model is then trained under a modified flow-matching objective that maximises the log-likelihood of the target velocity under the predicted Gaussian, making the pretraining phase functionally equivalent to the original flow-matching loss while rendering the outputs probabilistic." confidence: high relevance: low - claim_id: grpo_with_wer_and_speaker_similarity_rewards_consistently role: supports claim: "GRPO with WER and speaker-similarity rewards consistently reduces word error rate and increases speaker similarity in non-autoregressive TTS post-training, across diverse Mandarin datasets." source: "§3.2.2, Table 1, Table 2" evidence: "Speaker similarity rises from 0.698 to 0.730 (4.6% relative)." confidence: high relevance: high - claim_id: rl_post_training_that_optimises_speaker_similarity_reward role: supports claim: "RL post-training that optimises speaker-similarity reward also improves robustness to noisy reference audio, not only clean-reference speaker cloning." source: "§3.2.2, Table 1" evidence: "A noise-robustness subset using noisy reference audio shows an even larger WER benefit (33.6% relative reduction: 2.32% to 1.54%), suggesting GRPO also improves robustness to reference audio quality." confidence: high relevance: low - claim_id: hard_text_tongue_twisters_and_repetitive_phrasing_degrades role: supports claim: "Hard text (tongue twisters and repetitive phrasing) degrades all flow-matching TTS variants substantially, and RL training with WER rewards provides smaller relative gains in this regime than on plain text." source: "§3.2.2, Table 1" evidence: "On the hard set (tongue twisters and repetitive text), WER drops from 11.30% to 10.63% (6.1% relative) and SIM improves from 0.673 to 0.711 (5.6% relative)." confidence: high relevance: low - claim_id: the_grpo_phase_requires_only_a_small_fraction role: supports claim: "The GRPO phase requires only a small fraction of the pretraining data (100 h out of 7,226 h) and update steps to achieve measurable objective improvements, suggesting that RL fine-tuning is data-efficient for this task." source: "§3.1" evidence: "The entire GRPO phase uses only 100 hours of data from the same Mandarin corpus and runs for 1,100 update steps on 8 A100 GPUs, compared to 1 million steps in pretraining." confidence: high relevance: low limitations: - "No subjective evaluation (MOS or MUSHRA) is reported. All claimed improvements are based on automatic WER and speaker cosine similarity, which are proxy metrics. Whether GRPO post-training preserves or alters naturalness remains untested." - "Evaluation is restricted to Mandarin (Seed-TTS-eval test-cn). Generalisation to English, multilingual, or cross-lingual settings is not studied. The model builds directly on F5-TTS architecture and the WenetSpeech4TTS corpus; adapting to other flow-matching backbones or languages would require separate experiments. The hard-set WER (10.63% for F5R-TTS vs. 11.30% for F5) remains high in absolute terms, indicating that difficult phonetic sequences are not resolved by RL post-training alone. Future directions identified by the authors include PPO and DDPO integration, improved reward design for challenging scenarios, and larger-scale training experiments." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2504.10344" published_date: "2025-04-14" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec, TTS] architecture: [autoregressive-LM, VAE] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: frame_level_quantization_without_cross_frame_context_limits role: supports claim: "Frame-level quantization without cross-frame context limits codec semantic richness and increases the difficulty of autoregressive LM training on the resulting tokens." source: "§1, Figure 1" evidence: "Semantic retention, measured by ASR WER on LibriSpeech features using the codec's quantized representations, is 18.3% for ALMTokenizer versus 44.6% for WavTokenizer and 44.1% for DAC; emotion classification accuracy (29%) approaches WavLM (29%) and HuBERT (31%)." confidence: high relevance: high - claim_id: initialising_vq_codebook_entries_from_semantic_priors_derived role: supports claim: "Initialising VQ codebook entries from semantic priors derived from self-supervised models improves downstream recognition accuracy without additional distillation overhead." source: "§3.2, Table 6" evidence: "Semantic priors from k-means clustering of wav2vec 2.0 (for speech) and BEATs (for general audio) features initialise and fix the codebook of the first VQ layer, injecting structured semantic content without requiring online distillation." confidence: high relevance: medium - claim_id: lower_token_sequence_length_lower_hz_frame_rate role: supports claim: "Lower token-sequence length (lower Hz frame rate) improves both training and inference efficiency for audio language models, independent of bitrate." source: "Appendix B, Table 8" evidence: "Comparison with SemantiCodec shows ALMTokenizer is 30x faster at inference (RTF 0.031 vs. 0.92) while achieving better reconstruction and semantic scores at lower bitrate (Table 16)." confidence: high relevance: high - claim_id: reconstruction_quality_alone_is_an_insufficient_predictor_of role: supports claim: "Reconstruction quality alone is an insufficient predictor of a codec's suitability for autoregressive language modelling: codecs with stronger semantic content produce more robust LM-based TTS even at higher bitrates." source: "§4.6, Table 4" evidence: "An AR prediction loss trains a lightweight autoregressive transformer to predict the quantized features of deeper RVQ layers from shallower ones, making the latent space more amenable to sequential LM modelling." confidence: high relevance: high - claim_id: a_masked_autoencoder_auxiliary_loss_during_codec_training role: supports claim: "A masked autoencoder auxiliary loss during codec training increases semantic information in learned representations at a modest reconstruction cost." source: "§3.3, Table 6" evidence: "Semantic retention, measured by ASR WER on LibriSpeech features using the codec's quantized representations, is 18.3% for ALMTokenizer versus 44.6% for WavTokenizer and 44.1% for DAC; emotion classification accuracy (29%) approaches WavLM (29%) and HuBERT (31%)." confidence: high relevance: high limitations: - "All downstream LM experiments use the same 1B-parameter LLaMA backbone with limited training data (2000 hours speech, ~500 hours each sound/music). Gains observed may not transfer to larger-scale audio LM systems, where baseline tokenizers may reach ceiling performance." - "The two-stage training adds complexity and discards large sub-networks (MAE encoder/decoder, AR prediction transformer) after training, increasing resource cost without reuse. The authors note this explicitly and flag it as future work. Sound and music reconstruction remains challenging at 0.41 kbps, with VISQOL scores substantially lower than speech. Semantic retention still lags SSL models for ASR (18.3% WER vs. 6.2% for WavLM), indicating the codec does not fully substitute for purpose-built SSL representations. Code and model weights were not released at submission time, limiting reproducibility." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "iclr-2025-868masI331" published_date: "2025-04-24" entry_date: "2026-07-25" year: 2025 venue: "ICLR" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts] claims: - claim_id: reducing_the_first_layer_rvq_frame_rate_of role: supports claim: "Reducing the first-layer RVQ frame rate of a neural audio codec through hierarchical multi-resolution distillation enables autoregressive TTS to generate minute-long speech with stable intelligibility." source: "§4, §7.3, Table 3, Table 4" evidence: "MReQ-Encodec at 8 Hz achieves WER 9.79% on MinutesSpeech test-90s where standard Encodec at 8 Hz produces 100% WER and naive VALL-E at 48 Hz with long training data yields 16.14%." confidence: high relevance: high - claim_id: lowering_the_codec_frame_rate_in_autoregressive_tts role: complicates claim: "Lowering the codec frame rate in autoregressive TTS improves temporal coherence and intelligibility for long utterances but degrades speaker similarity." source: "§7.3, Table 4, Table 6" evidence: "HALL-E consistently lags VALL-E by 0.025-0.042 SIM points on MinutesSpeech tests; the paper attributes this to acoustic information loss when compressing from 48 Hz to 8 Hz in the first RVQ layer." confidence: high relevance: high - claim_id: post_training_from_a_pre_trained_lm_checkpoint role: supports claim: "Post-training from a pre-trained LM checkpoint is essential for hierarchical codec TTS: training from scratch without pre-trained weights collapses quality." source: "§7.4, Table 10" evidence: "Removing VALL-E pre-training from HALL-E increases WER from 9.79% to 49.8% on MinutesSpeech test-90s, and removing MReQ pre-training similarly degrades codec reconstruction." confidence: high relevance: high - claim_id: the_frame_rate_at_which_autoregressive_speech_token role: refines claim: "The frame rate at which autoregressive speech token generation becomes unstable is approximately 8 Hz, consistent with average phoneme durations of around 100 ms." source: "Appendix C.1, Table 18" evidence: "Ablation at 4 Hz first-layer rate raises WER to 20.07%, while 8 Hz yields 9.79%; the paper notes that phoneme duration averaging ~100 ms corresponds to ~10 Hz, making 4 Hz fundamentally insufficient." confidence: high relevance: high - claim_id: length_balanced_training_data_covering_the_target_synthesis role: supports claim: "Length-balanced training data covering the target synthesis duration is necessary for autoregressive models to generalize to long-form speech." source: "§7.3, Table 4" evidence: "VALL-E trained only on segments up to 28 seconds achieves WER 39.77% on test-90s, and training on longer data decreases but does not eliminate the gap; HALL-E's 8 Hz formulation resolves instability that remains even with longer VALL-E training data." confidence: high relevance: high limitations: - "Speaker similarity consistently degrades with frame rate reduction: HALL-E's SIM scores are 0.025-0.042 lower than VALL-E across test conditions. This reflects a fundamental trade-off between temporal compression and acoustic fidelity preservation that the paper does not resolve." - "The current NAR model processes audio at full 48 Hz resolution, which limits NAR input length to around 28-54 seconds during training and requires a sliding window at inference. Integrating cross-attention text conditioning addresses alignment but does not eliminate the NAR length bottleneck. The paper also does not compare against streaming or chunked autoregressive synthesis approaches, which represent a practical alternative. MinutesSpeech training data consists entirely of English podcast speech, and generalization to other languages, domains, or reading styles is untested." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "iclr-2025-cuFzE8Jlvb" published_date: "2025-04-24" entry_date: "2026-07-25" year: 2025 venue: "ICLR" task: [TTS] architecture: [autoregressive-LM, VAE] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [alignment_constrained_autoregression, continuous_token_autoregression] claims: - claim_id: continuous_latent_representations_can_replace_discrete_vector_quantization role: supports claim: "Continuous latent representations can replace discrete vector quantization in autoregressive TTS without sacrificing generation quality." source: "§5.1, Table 1, Table 2" evidence: "GMM-LM trained on continuous GMM-VAE encoder features outperforms VALL-E (RVQ-based) on WER, speaker similarity, Q-MOS, and S-MOS on LibriSpeech test-clean across all prompt lengths, while using 10.3% of VALL-E's parameter count." confidence: high relevance: high - claim_id: longer_audio_prompts_do_not_uniformly_improve_zero role: complicates claim: "Longer audio prompts do not uniformly improve zero-shot speaker cloning across AR architectures." source: "§5.1, Table 2" evidence: "VALL-E's WER increases monotonically with prompt length (6.04% at 3s, 7.54% at 8s, 9.68% at 15s), suggesting that simple cross-attention cannot leverage extended speaker context in AR decoding; the proposed GMM-LM shows the opposite trend, consistently benefiting from longer prompts." confidence: high relevance: medium - claim_id: strict_monotonic_alignment_substantially_reduces_word_error_rate role: supports claim: "Strict monotonic alignment substantially reduces word error rate in autoregressive TTS compared to standard cross-attention and soft monotonic variants." source: "Appendix A.1, Table 6" evidence: "Among alignment strategies tested on the same GMM-LM architecture, stochastic monotonic alignment with ST-Gumbel achieves WER 2.72% vs. 6.6% for cross-attention alone; even monotonic attention with Gumbel (without the stochastic binary forward pass) scores 3.34%." confidence: high relevance: high - claim_id: continuous_speech_representations_improve_downstream_autoregressive_model_perfor role: supports claim: "Continuous speech representations improve downstream autoregressive model performance relative to discrete counterparts, independent of the alignment mechanism." source: "Appendix A.6, Table 10" evidence: "A head-to-head ablation comparing GMM-LM (continuous) against discrete AR models (VQ-VAE single codebook and DAC multi-codebook with delayed prediction), all using the proposed monotonic alignment, shows GMM-LM achieves WER 2.72% vs. 5.35% and 5.87% for the discrete variants." confidence: high relevance: high - claim_id: increasing_the_number_of_gaussian_components_in_continuous role: complicates claim: "Increasing the number of Gaussian components in continuous AR modeling yields diminishing returns and can reduce quality through overfitting." source: "§5.4, Table 4, Table 5, Appendix A.3" evidence: "GMM-LM with 6 diagonal-covariance Gaussians (WER 2.72%, SIM 0.91) outperforms 3-Gaussian (WER 2.89%, SIM 0.85), but 10-Gaussian degrades to WER 5.21%, SIM 0.71; the 6-mixture GMM-VAE also shows worse evaluation-set reconstruction than the 3-mixture model despite better training-set fit." confidence: high relevance: medium limitations: - "The evaluation compares against VALL-E (2023), a dated AR baseline. Stronger AR systems published by the time of this paper's submission are not included, limiting the strength of the superiority claim for continuous AR over discrete AR in general." - "The model is evaluated exclusively on English audiobook speech (LibriLight training, LibriSpeech evaluation). Generalisation to conversational speech, noisy in-the-wild data, or other languages is not demonstrated, though noise robustness experiments (Appendix A.4) show the method degrades gracefully under additive noise in prompts." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "iclr-2025-dGSOn7sdWg" published_date: "2025-04-24" entry_date: "2026-07-25" year: 2025 venue: "ICLR" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: influential method_family: [unified_speech_text_language_model] claims: - claim_id: reducing_the_token_rate_of_speech_representations_below role: supports claim: "Reducing the token rate of speech representations below 10 Hz is sufficient to preserve semantic content adequate for spoken language modelling, while substantially improving training and inference efficiency." source: "§5.6, Table 6" evidence: "SyllableLM at 6.25 Hz, 90M parameters, matches or exceeds TWIST models up to 13B parameters on sBLIMP semantic understanding benchmarks, with 30x less training compute and 4.5x faster inference than an equal-sized TWIST baseline." confidence: high relevance: high - claim_id: the_loss_surface_of_a_masked_prediction_ssl role: supports claim: "The loss surface of a masked-prediction SSL model encodes latent syllabic segmentation boundaries discoverable without additional supervised signal or cross-modal supervision." source: "§3.1, Table 1" evidence: "LossPred, applied to a frozen HuBERT student-teacher pair without any training, achieves F1-50 of 59.6 on syllabic boundary detection, outperforming the feature-similarity-based baseline (47.3) while requiring no fine-tuning." confidence: high relevance: medium - claim_id: iterative_student_teacher_distillation_over_pseudo_syllabic_boundaries role: supports claim: "Iterative student-teacher distillation over pseudo-syllabic boundaries progressively sharpens SSL encoder representations toward syllable-level organisation." source: "§5.3, Table 2" evidence: "SylBoost applied to HuBERT improves boundary detection F1 from 60.1 (LossPred initialisation) to 70.2 after two iterations; applying it to Data2Vec2 reaches 73.2, each iteration producing a measurable gain over the previous." confidence: high relevance: medium - claim_id: low_frequency_speech_units_that_improve_semantic_modelling role: complicates claim: "Low-frequency speech units that improve semantic modelling efficiency may sacrifice robustness to speaker rate variation." source: "Appendix A.5, Table 11" evidence: "SylBoost unit counts collapse under audio speedups of 0.5x and 0.6x relative to original length, performing comparably to SD-HuBERT only at mild speedups (0.8x–0.9x range), while showing greater robustness to slowdowns." confidence: high relevance: medium - claim_id: units_optimised_for_semantic_modelling_in_audiobook_speech role: complicates claim: "Units optimised for semantic modelling in audiobook speech may lose paralinguistic information, limiting applicability to domains requiring prosodic or tonal fidelity." source: "§6" evidence: "The authors note that low-frequency SylBoost units may lose paralinguistic features such as tone, and the entire evaluation is conducted on audiobook data (LibriSpeech / LibriLight); performance on spontaneous or multi-speaker speech is not reported." confidence: high relevance: medium limitations: - "All training and evaluation uses English audiobook data (LibriSpeech, LibriLight). Generalisation to spontaneous conversational speech, other languages, or multi-speaker settings is not demonstrated." - "The interleaved vocoder decoding pipeline introduces a dependency on the TWIST tokeniser and vocoder for resynthesis, meaning the system is not fully end-to-end and unit bitrate for the final waveform is partially bounded by TWIST's own quality ceiling (WER 6.3%). The efficiency gains in the SpeechLM do not fully apply to the decoding pipeline, which involves an additional language model. Scaling beyond 300M parameters was not attempted due to compute constraints, leaving open whether the efficiency advantage persists at very large model scales. The base encoder quality (Data2Vec2 vs newer models like w2v-BERT 2.0) is acknowledged as a confounding factor in cross-model comparisons." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2025.findings-naacl.184" published_date: "2025-04-29" entry_date: "2026-07-25" year: 2025 venue: "NAACL" task: [TTS, codec] architecture: [autoregressive-LM, VAE, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure, hybrid_ar_token_continuous_renderer, continuous_token_autoregression] claims: - claim_id: continuous_speech_token_representations_preserve_high_frequency_information role: supports claim: "Continuous speech token representations preserve high-frequency information more effectively than discrete RVQ tokens, with the gap largest in the 8kHz band." source: "§3.3, Table 3" evidence: "The information retention of continuous vs. discrete tokenizers is measured as average attention rate (transfer function analogue) across frequency bands: - 2kHz: continuous 0.94 vs discrete 0.95 (comparable) - 5kHz: continuous 0.81 vs discrete 0.78 - 8kHz: continuous 0.55 vs discrete 0.34 (large gap) The continuous tokenizer maintains substantially better high-frequency information." confidence: high relevance: high - claim_id: replacing_rvq_with_a_continuous_tokenizer_in_an role: supports claim: "Replacing RVQ with a continuous tokenizer in an autoregressive TTS framework reduces word error rate and improves speaker similarity compared to a discrete-token baseline." source: "§3.2, Table 1" evidence: "The information retention of continuous vs. discrete tokenizers is measured as average attention rate (transfer function analogue) across frequency bands: - 2kHz: continuous 0.94 vs discrete 0.95 (comparable) - 5kHz: continuous 0.81 vs discrete 0.78 - 8kHz: continuous 0.55 vs discrete 0.34 (large gap) The continuous tokenizer maintains substantially better high-frequency information." confidence: high relevance: high - claim_id: continuous_speech_tokenizers_exhibit_greater_robustness_to_sampling role: supports claim: "Continuous speech tokenizers exhibit greater robustness to sampling rate variation than discrete tokenizers, particularly when window length ratios are modified." source: "§3.4" evidence: "Continuous tokenizer is more robust than discrete when window length ratios are modified (simulating mixed narrow/wideband data)." confidence: high relevance: high - claim_id: two_stage_training_tokenizer_pre_training_followed_by role: supports claim: "Two-stage training (tokenizer pre-training followed by joint LM-tokenizer training) is necessary for continuous-token TTS; removing either stage degrades both intelligibility and speaker similarity." source: "§2.4.3, Appendix A.1, Table 4" evidence: "Ablation confirms that both pre-training the tokenizer and joint training are necessary; removing either significantly hurts WER and SIM." confidence: high relevance: high - claim_id: autoregressive_generation_over_continuous_speech_tokens_reduces_temporal role: supports claim: "Autoregressive generation over continuous speech tokens reduces temporal discontinuity artefacts relative to RVQ-based generation, as measured by NISQA continuity scores." source: "§3.2, Table 2" evidence: "The continuity quality score (NISQA-based) nearly doubles VALL-E (3.61 vs 1.80), consistent with the claim that continuous tokens reduce temporal discontinuities." confidence: high relevance: high limitations: - "- Evaluation is English-only on LibriSpeech test-clean, a relatively clean/read-speech benchmark; generalization to spontaneous or multilingual speech is untested. - Model size is not reported, making cost comparisons with VALL-E/MELLE impossible. - The continuous LM output requires MSE regression rather than cross-entropy classification; training stability and mode averaging in the continuous space are not thoroughly discussed. - No subjective MOS evaluation from human raters; EMoS (automated MOS) is used instead. - The relationship to MELLE (which also drops discrete tokens, via mel-spectrogram prediction) deserves more analysis — they represent two distinct approaches to the same problem. - How the continuous tokenizer interacts with multi-speaker generalization is not tested." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.findings-naacl.279" published_date: "2025-04-29" entry_date: "2026-07-25" year: 2025 venue: "NAACL" task: [TTS] architecture: [autoregressive-LM, GAN, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: few_shot_fine_tuning_on_a_small_amount role: supports claim: "Few-shot fine-tuning on a small amount of target-speaker studio audio substantially improves naturalness, clarity, and speaker similarity over a zero-shot pretrained multilingual TTS model, even when the zero-shot model was pretrained on thousands of hours of in-language speech." source: "§4, Table 1" evidence: "Fine-tuning on 20 minutes per speaker (4 speakers, 10 epochs) raises SMOS from 4.456 (BnTTS-0, zero-shot) to 4.624 (BnTTS-n, few-shot) and SECS (reference) from 0.529 to 0.548 on BnStudioEval." confidence: high relevance: medium - claim_id: expressive_conversationally_prosodic_tts_output_can_score_worse role: complicates claim: "Expressive, conversationally-prosodic TTS output can score worse on ASR-based intelligibility metrics than flatter-sounding commercial systems, even when subjective listening scores favor the expressive system." source: "§4, \"High CER in Text Generation\", Tables 1–2" evidence: "BnTTS-n and BnTTS-0 show higher CER than Google Cloud TTS and Azure TTS on both BnStudioEval and BnTTSTextEval despite BnTTS-n achieving the highest SMOS, Naturalness, and Clarity scores overall; the authors attribute this to ASR systems being better suited to standardized, less expressive speech patterns." confidence: high relevance: medium - claim_id: autoregressive_codec_based_tts_models_trained_primarily_on role: complicates claim: "Autoregressive codec-based TTS models trained primarily on longer utterances can fail to generate short sequences reliably under default sampling settings." source: "§4, \"Effect of Sampling and Prompt Length on Short Speech Generation\", Table 3" evidence: "Under default generation settings (T=0.85, TopK=50), the model achieves only 0.699 DurationEquality and 0.081 CER on a 23-sample short-text subset (under 30 characters); adjusting prompt length and sampling temperature/top-K improves this to 0.827 DurationEquality and 0.015 CER." confidence: high relevance: high - claim_id: adapting_a_multilingual_zero_shot_tts_architecture_to role: refines claim: "Adapting a multilingual zero-shot TTS architecture to a new low-resource language via continual pretraining can match or exceed existing language-specific commercial and open-source systems without requiring training from scratch." source: "§3 \"BnTTS model\", §7 \"Limitations\", Tables 1–2" evidence: "BnTTS-n, initialized from the pretrained XTTS checkpoint and continually pretrained on 3.85k hours of Bengali data, outperforms IndicTTS, Google Cloud TTS, and Azure TTS on SMOS, Naturalness, and Clarity across both evaluation datasets, while the authors note training from scratch was not attempted due to resource constraints." confidence: high relevance: medium limitations: - "The system struggles to adapt to speakers with unique vocal traits absent from training, and fails to generate short sequences (under two words or 20 characters) in some cases even after mitigation; these failures stem from limitations inherited from the underlying XTTS foundation model rather than being fully resolved by the Bangla adaptation." - "The authors did not train XTTS from scratch for Bangla, relying instead on continual pretraining due to resource constraints, leaving the ceiling of a from-scratch Bangla model untested. A substantial portion of the pretraining corpus (3636 of 3856 hours) is pseudo-labeled and not manually reviewed, which may introduce label noise not fully captured by the reported metrics. The evaluation is limited to four in-house speakers for the few-shot speaker-adaptation setting, and dialectal variation within Bangla is not addressed." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.naacl-demo.12" published_date: "2025-04-29" entry_date: "2026-07-25" year: 2025 venue: "NAACL" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: a_speech_language_model_can_be_initialized_from role: supports claim: "A speech language model can be initialized from a pre-trained text LLM and jointly trained on speech recognition, speech synthesis, text continuation, and audio continuation without substantially degrading the text-only capability of the base model." source: "§4.3, Table 5" evidence: "The 1.7B multi-task model trained on ASR, TTS, TextLM, and AudioLM objectives scores MMLU 30.5, ARC-C 41.3, and HellaSwag 50.4, close to the text-only LLaMA-3.2-1B baseline (32.2, 32.8, 41.2) despite carrying three additional speech tasks." confidence: high relevance: high - claim_id: concatenating_neural_codec_tokens_with_self_supervised_speech role: supports claim: "Concatenating neural codec tokens with self-supervised speech representations frame-by-frame is a viable tokenization strategy for both speech understanding and generation tasks within a single sequential model." source: "§3.3" evidence: "The \"Codec_SSL\" scheme (ESPnet-Codec combined with XEUS SSL tokens) is used for the headline ASR and TTS experiments and the paper reports it \"behaves well in both speech understanding and generation\" *(§3.3)*." confidence: medium relevance: high - claim_id: a_decoder_only_autoregressive_speech_language_model_can role: supports claim: "A decoder-only autoregressive speech language model can match or exceed dedicated, larger ASR-only systems on English benchmarks while using substantially fewer parameters." source: "§4.2, Table 3" evidence: "A 442M-parameter ESPnet-SpeechLM ASR model reaches average WER 5.4% across six English test sets, matching OWSM v3.1-medium (1.02B, 5.4%) and beating Whisper-small (244M, 6.4%) and Whisper-medium (769M, 5.7%)." confidence: high relevance: high - claim_id: cross_system_comparisons_of_speech_language_models_reported role: complicates claim: "Cross-system comparisons of speech language models reported in the literature are frequently not run under matched conditions, limiting how much can be concluded from any single performance table." source: "§4.3, Table 5" evidence: "In the multi-task comparison (Table 5), competitor numbers for Moshi, VITA, GLM-4-Voice, and others are taken directly from their own published reports rather than reproduced by the authors, and the paper explicitly flags this with footnote markers." confidence: high relevance: high - claim_id: combining_a_codec_tokenizer_with_a_self_supervised role: complicates claim: "Combining a codec tokenizer with a self-supervised tokenizer frame-by-frame is reported as an effective design choice but is not validated against single-tokenizer ablations in the same controlled setting." source: "§3.3" evidence: "The claim that Codec_SSL tokenization \"behaves well\" rests on a single line of justification without a paired ablation against codec-only or SSL-only tokenization on the same task and dataset." confidence: high relevance: high limitations: - "The multi-task model's TTS quality (Proxy MOS 3.99, WER 6.0%) is noticeably weaker than the single-task TTS model trained on the same architecture (Proxy MOS 4.03, WER 3.1%), indicating a capacity or interference cost to joint multi-task training that the paper reports but does not analyze further. Most training and evaluation data is English-only (the multilingual text corpus is used only for the TextLM objective, not for speech tasks), so the demonstrated speech capabilities are not evidence of multilingual generalization. Competitor numbers in the multi-task comparison table are drawn from third-party reports under unmatched training data and conditions rather than reproduced by the authors, which the paper itself notes. As a system/demo paper, the contribution is the toolkit and its reference recipes rather than a novel architecture or training method; the headline numbers serve to validate functionality rather than push state of the art on any individual benchmark." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.naacl-long.591" published_date: "2025-04-29" entry_date: "2026-07-25" year: 2025 venue: "NAACL" task: [TTS] architecture: [transformer-enc-dec] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: monotonic_alignment_can_be_learned_end_to_end role: supports claim: "Monotonic alignment can be learned end-to-end as a latent property of an encoder-decoder TTS model via backpropagation, without requiring forced alignments or dynamic programming during training." source: "§3.3, §3.4" evidence: "VAT's alignment layer learns continuous alignment positions through interpolated relative position biases (IRPBs); alignment trajectories emerge from joint training with no external supervision, and generalise to utterances far longer than the training distribution." confidence: high relevance: low - claim_id: augmenting_cross_attention_with_a_learned_monotonic_alignment role: supports claim: "Augmenting cross-attention with a learned monotonic alignment position enables unbounded length generalisation in encoder-decoder TTS without degrading naturalness relative to an unmodified Transformer baseline." source: "§5.1, §5.2, §5.3, Table 1" evidence: "VAT achieves near-zero CER on inputs up to 1500 characters (~90 seconds) despite training only on utterances up to 9.6 seconds, while matching the T5 baseline in side-by-side naturalness evaluations (SxS -0.06 ± 0.14 on Lessac, 0.01 ± 0.14 on LibriTTS)." confidence: high relevance: low - claim_id: standard_mos_evaluations_are_insufficient_to_surface_robustness role: complicates claim: "Standard MOS evaluations are insufficient to surface robustness failures in autoregressive TTS because raters cannot detect dropped or repeated words without access to target transcripts." source: "§5.1, §5.2, Table 1" evidence: "The T5 baseline achieves overlapping MOS with VAT (3.75 vs. 3.68 on Lessac) while producing a CER of 10.2 versus VAT's 3.3; the perceptual quality rating is statistically indistinguishable despite systematic robustness failures." confidence: high relevance: high - claim_id: autoregressive_transformer_tts_without_explicit_alignment_guidance_fails role: complicates claim: "Autoregressive Transformer TTS without explicit alignment guidance fails on repeated words even within training sequence length limits." source: "§5.4" evidence: "The T5 baseline makes errors on 14 of 27 (52%) repeated-word test phrases, including phrases with as few as 2 repetitions of a single word, while VAT makes zero errors across all 27 templates." confidence: high relevance: high - claim_id: duration_based_tts_achieves_better_asr_measured_character role: refines claim: "Duration-based TTS achieves better ASR-measured character error rates than expressive autoregressive models, but the gain is attributable to hyper-intelligible, monotone synthesis rather than superior text coverage." source: "§5.3, Table 1" evidence: "NAT achieves CER 3.3 on LibriTTS, below both VAT (4.6) and ground truth (3.6), yet VAT is preferred over NAT in naturalness side-by-sides because NAT's unsupervised duration predictor produces robotic, monotonous prosody." confidence: high relevance: high limitations: - "Training speed is affected by the need to compute alignment positions serially during training, imposing a 12-20% slowdown relative to the T5 baseline depending on model scale. All experiments use English and a speaker-conditioned (non-zero-shot) setting; generalisation to other languages and to audio-prompted zero-shot scenarios is untested. Evaluation compares against T5, Tacotron-GMMA, and NAT; no direct comparison with codec LM systems (VALL-E, SPEAR-TTS, MQTTS) is provided, which the paper attributes to incompatible dataset scales and evaluation protocols. Hyper-parameter choices for the alignment layer, IRPB initialization, and maximum distance penalty are reported but not systematically ablated." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2025.naacl-srw.6" published_date: "2025-04-29" entry_date: "2026-07-25" year: 2025 venue: "NAACL" task: [TTS, codec] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure, hybrid_ar_token_continuous_renderer] claims: - claim_id: non_overlapping_encoder_receptive_fields_in_rvq_codecs role: supports claim: "Non-overlapping encoder receptive fields in RVQ codecs improve downstream language model likelihood and end-to-end TTS metrics relative to the standard causal overlapping setup." source: "§3.1, Table 1" evidence: "Replacing the overlapping causal encoder with a framewise encoder on a DAC-based codec reduces LM NLL by more than 8% and improves WER, NISQA, and speaker similarity on LibriTTS-R test-clean, despite slightly worsening Mel-L1 reconstruction." confidence: high relevance: high - claim_id: better_codec_audio_reconstruction_quality_does_not_reliably role: complicates claim: "Better codec audio reconstruction quality does not reliably predict better end-to-end speech generation quality in codec-LM systems." source: "§4, Table 1" evidence: "The framewise encoder achieves higher NLL and better TTS metrics than the causal baseline while scoring slightly worse on Mel-spectral L1 reconstruction distance, demonstrating that reconstruction-optimised codecs can be suboptimal for downstream LM training." confidence: high relevance: high - claim_id: increasing_rvq_codec_frame_duration_can_substantially_reduce role: supports claim: "Increasing RVQ codec frame duration can substantially reduce codec-LM inference latency with little or no degradation in TTS intelligibility and speaker similarity, provided the bitrate is held approximately constant by adjusting codebook depth." source: "§3.3, §4, Table 2" evidence: "Doubling frame duration from 11ms to 22ms yields a 1.94x inference speedup with WER 4.21%, NISQA 4.42, and speaker similarity 81.0%, matching or improving on the 11ms framewise baseline. Quadrupling to 44ms further accelerates inference (3.2-3.8x) but substantially degrades WER and speaker similarity." confidence: high relevance: high - claim_id: a_single_lm_trained_with_codebook_level_dropout role: supports claim: "A single LM trained with codebook level dropout can efficiently approximate the performance profile of training one LM per candidate RVQ level count." source: "§3.2, §4, Figure 2" evidence: "Training with 90%-full CL drop on a 12-level codec produces per-level performance curves for WER, NISQA, and speaker similarity that closely track those of 12 independently trained LMs across all Q' values 1-12." confidence: high relevance: medium - claim_id: the_optimal_number_of_rvq_codebook_levels_for role: refines claim: "The optimal number of RVQ codebook levels for end-to-end codec-LM TTS differs across evaluation dimensions, and more levels are not universally better for end-to-end performance even when they monotonically improve codec reconstruction." source: "Appendix B, Figure 3" evidence: "End-to-end FAD reaches a global minimum at 9 levels before degrading, while WER reaches its best at 3-4 levels and NISQA and speaker similarity peak at approximately 9 levels, in contrast to codec Mel-L1 which improves monotonically." confidence: high relevance: high limitations: - "The evaluation is conducted on a single English TTS corpus (LibriTTS-R test-clean) using automatic metrics only (WER via Whisper, NISQA, cosine speaker similarity). No human listening test is reported, so perceptual quality gains are estimated rather than directly validated." - "The codec is trained on proprietary in-house podcast data (1.7K hours), which limits reproducibility for the codec training stage specifically. The LM training does use the public LibriTTS-R dataset. The codebook size hyperparameter (|V|) remains outside the scope of CL drop, requiring separate trial-and-error search. The paper does not investigate multilingual or noisy speech settings. The optimal frame duration finding (22ms being the sweet spot) is specific to this codec architecture and training data and may not generalise to other codec families." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2505.02625" published_date: "2025-05-05" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: autoregressive_streaming_tts_decoders_in_modular_spoken_conversational role: supports claim: "Autoregressive streaming TTS decoders in modular spoken conversational agents produce higher-quality speech and smaller S2T-to-S2S accuracy gaps than non-autoregressive streaming decoders, at a modest latency cost." source: "§5.1, Table 1" evidence: "Larger R:W write chunks improve UTMOS (speech quality) at the cost of latency, while smaller values reduce latency at a modest quality cost (Table 4)." confidence: high relevance: high - claim_id: modular_spoken_conversational_agents_trained_on_tens_of role: supports claim: "Modular spoken conversational agents trained on tens of thousands of hours of synthesized speech-to-speech dialogue can match the performance of native SpeechLMs trained on millions of hours of unsupervised speech data." source: "§5.1, Table 1" evidence: "The 200K multi-turn dialogue samples are synthesized from Alpaca/UltraChat text data using CosyVoice 2 for responses and FishSpeech-1.5 for diverse instruction voices." confidence: high relevance: medium - claim_id: jointly_conditioning_a_tts_language_model_on_llm role: supports claim: "Jointly conditioning a TTS language model on LLM hidden states and text token embeddings via a learned gate fusion improves both semantic consistency (WER) and instruction-following quality over hidden-state-only conditioning." source: "§5.2, Table 2" evidence: "Rather than conditioning M_TTS on LLM hidden states alone, the model adaptively blends two representations per output position: a projected hidden state from the LLM (carrying contextual information) and a text token embedding (ensuring alignment with the textual response)." confidence: high relevance: high - claim_id: tts_language_model_pretraining_on_text_speech_pairs role: supports claim: "TTS language model pretraining on text-speech pairs is a critical prerequisite for stable convergence in modular SpeechLMs; initializing from a language model alone is insufficient." source: "§5.2, Table 3" evidence: "LLaMA-Omni 2 is a modular SpeechLM consisting of three main components: a speech encoder (Whisper-large-v3), an LLM backbone (one of the Qwen2.5-0.5B/1.5B/3B/7B/14B-Instruct variants), and a streaming speech decoder composed of a TTS language model and a causal flow matching vocoder." confidence: high relevance: high - claim_id: in_streaming_speech_generation_the_write_chunk_size role: supports claim: "In streaming speech generation, the write chunk size (W) primarily determines speech naturalness while the read chunk size (R) primarily determines text-speech alignment, with latency jointly determined by both." source: "§5.2, Table 4" evidence: "Streaming is controlled by a Read-Write (R:W) strategy: after every R LLM text tokens are generated, W speech tokens are produced by M_TTS and synthesized as a waveform chunk." confidence: high relevance: high limitations: - "All evaluations are conducted in English only. The model is trained on a single fixed output voice, and no multilingual or voice-diversity experiments are reported. Generalization to other languages or to emotionally expressive speech is untested." - "The model cannot modulate speech style (emotion, speaking rate, dialect) in response to paralinguistic cues in the input, because training data contains only conventional speech-to-speech dialogue. The authors note this as planned future work. The benchmarks used (SpokenQA accuracy, ChatGPT score) are narrow and do not cover naturalness in unconstrained conversational settings or robustness to noisy or accented input. Comparisons to Minmo (a concurrent work using 1.4M hours) are mentioned in related work but not included in the main experimental table, leaving the data efficiency claim partially unverified against the most directly comparable system." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2505.07916" published_date: "2025-05-12" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, VAE] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: jointly_training_a_speaker_encoder_with_an_autoregressive role: supports claim: "Jointly training a speaker encoder with an autoregressive TTS model yields better intelligibility and competitive speaker similarity compared to using a fixed speaker verification embedding." source: "§3.6, Table 4" evidence: "Ablation on speaker conditioning (§3.6, Table 4) confirms the learnable speaker encoder outperforms both fixed speaker embedding (pre-trained speaker verification model) and prompt-only conditioning in WER-SIM balance." confidence: high relevance: high - claim_id: zero_shot_voice_cloning_conditioned_on_untranscribed_reference role: complicates claim: "Zero-shot voice cloning conditioned on untranscribed reference audio produces lower word error rates than one-shot conditioning with a paired text-audio exemplar, at the cost of slightly reduced speaker similarity." source: "§3.2, Table 1; §3.5, Table 3" evidence: "The system supports two voice cloning paradigms: zero-shot (conditioning only on the speaker encoder's output from an untranscribed reference) and one-shot (additionally providing a paired text-audio exemplar as an in-context prompt)." confidence: high relevance: medium - claim_id: augmenting_a_vae_with_a_normalizing_flow_on role: supports claim: "Augmenting a VAE with a normalizing flow on the latent space (Flow-VAE) improves both waveform reconstruction quality and downstream TTS speaker similarity over a standard VAE with the same architecture." source: "§3.7, Tables 5–6" evidence: "The Flow-VAE extends a standard VAE by inserting a normalizing flow between the encoder's posterior distribution and the standard normal prior, enabling a more expressive latent space." confidence: high relevance: medium - claim_id: speaker_encoder_representations_trained_without_text_dependencies_support role: supports claim: "Speaker encoder representations trained without text dependencies support cross-lingual synthesis with high intelligibility, outperforming prompt-based systems that require transcribed reference audio in cross-lingual scenarios." source: "§3.5, Table 3" evidence: "Its core innovation is the jointly-trained speaker encoder, which takes a variable-length reference audio segment (without transcription) and compresses it to a fixed-size conditional vector." confidence: high relevance: medium - claim_id: disentangled_speaker_embeddings_from_a_task_specific_encoder role: supports claim: "Disentangled speaker embeddings from a task-specific encoder enable parameter-efficient per-speaker adaptation by fine-tuning only the speaker embedding vector, preserving base model generalization." source: "§4.3" evidence: "Three downstream extensions are documented: LoRA-based emotion control (separate LoRA modules per emotion category loaded at inference), text-to-voice generation (a compact model predicts speaker embeddings from natural language descriptions and discrete attribute tags), and professional voice cloning (PVC), which fine-tunes only the speaker embedding vector for a target speaker while keeping the AR Transformer frozen." confidence: high relevance: medium limitations: - "The training dataset is proprietary and the model size is not reported, which makes independent replication impossible. All results are from internal or public leaderboard evaluations only; no code release is indicated." - "The leaderboard evaluation (Artificial Analysis Arena) uses preference judgements that may be influenced by sample selection; the Arena snapshot is from a single date (May 12, 2025) and rankings can shift as more models are added. The claim of \"top position\" is time-bound." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2505.13000" published_date: "2025-05-19" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: directly_encoding_ssl_features_into_the_first_rvq role: supports claim: "Directly encoding SSL features into the first RVQ layer preserves significantly more semantic content than distilling SSL representations into codec tokens, particularly for tonal languages where pitch information is phonemically critical." source: "§4.2, Table 2" evidence: "DualCodec addresses the three-way tension through a dual-stream architecture that directly assigns SSL features to the first RVQ layer rather than distilling into it." confidence: high relevance: high - claim_id: neural_audio_codecs_operating_at_lower_frame_rates role: supports claim: "Neural audio codecs operating at lower frame rates with more quantization layers achieve superior audio quality at equivalent bitrates compared to higher frame rate codecs with fewer quantization layers." source: "§4.3, Table 3" evidence: "Dual encoding adds perceptual quality gains over plain DAC at the same frame rate (MUSHRA 79.5 vs 68.8 at 25Hz)." confidence: high relevance: high - claim_id: semantic_enhancement_of_the_first_rvq_layer_improves role: supports claim: "Semantic enhancement of the first RVQ layer improves downstream TTS speaker similarity as well as intelligibility, because higher semantic fidelity in RVQ-1 enables the waveform stream's quantizers to focus on acoustic detail rather than recovering content information." source: "§4.4, Table 4" evidence: "This residual formulation concentrates acoustic detail (timbre, noise, fine structure) in the waveform stream's quantizers, while the SSL stream is responsible solely for content and prosody." confidence: high relevance: medium - claim_id: the_quality_gap_between_distillation_based_and_direct role: complicates claim: "The quality gap between distillation-based and direct-encoding semantic codecs is substantially larger in Mandarin than in English, revealing a systematic limitation of distillation approaches for tonal languages." source: "§4.2, Table 2" evidence: "The same setup with distillation (C2) gives 17.8/14.4%, confirming that direct encoding from SSL features substantially outperforms distillation." confidence: high relevance: high limitations: - "The 12.5Hz DualCodec-based TTS systems consistently underperform their 25Hz counterparts on both WER and speaker similarity (Table 4, Table 6), indicating that the quality upper bound of the 12.5Hz variant is not yet competitive with the best open-source systems at 50Hz despite the frame rate reduction improving inference speed." - "The paper evaluates TTS only on Seed-TTS-Eval; no subjective TTS listening tests are reported, so the MUSHRA gains in codec reconstruction may not fully translate to perceived TTS naturalness. Speaker similarity scores with DualCodec-VALLE remain below those of MaskGCT baselines that use separate semantic and acoustic tokenizers, suggesting the unified approach has not yet matched the best-performing two-stage pipeline design. The DualCodec encoder is substantially heavier than baselines (628M vs 38M for Mimi) due to the frozen w2v-BERT-2.0 model, increasing training-time compute, though the decoder remains lightweight for inference." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2505.17589" published_date: "2025-05-23" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: differentiable_reward_optimization_over_discrete_speech_tokens_rather role: supports claim: "Differentiable reward optimization over discrete speech tokens, rather than synthesized audio, enables scalable post-training for LLM-based TTS without the computational overhead of full audio generation." source: "§2.2" evidence: "The post-training method, Differentiable Reward Optimization (DiffRO), bypasses full audio synthesis by operating directly on discrete speech tokens." confidence: high relevance: high - claim_id: supervised_multi_task_training_of_a_speech_tokenizer role: supports claim: "Supervised multi-task training of a speech tokenizer on tasks including emotion recognition, language identification, and speaker analysis captures richer paralinguistic content than ASR-only objectives, with measurable downstream gains in content consistency and speaker similarity." source: "§2.1, §5.3" evidence: "A Finite Scalar Quantization (FSQ) module is inserted into MinMo's voice encoder, and the entire tokenizer is then fine-tuned via supervised multi-task learning on 530K hours covering ASR, language identification (LID), speech emotion recognition (SER), audio event detection (AED), and speaker analysis." confidence: high relevance: high - claim_id: current_zero_shot_tts_systems_infer_emotional_tone role: complicates claim: "Current zero-shot TTS systems infer emotional tone primarily from text semantics; expression of text-unrelated emotions from a reference prompt remains a significant unsolved challenge." source: "§5.2.3, Table 9" evidence: "Emotion cloning results are more mixed: text-related emotion accuracy is strong (0.84–0.98 for happy/angry/sad with DiffRO-EMO), but text-unrelated emotion accuracy drops substantially for sad and angry categories, revealing that current models infer emotional tone primarily from text semantics rather than from the reference prompt." confidence: high relevance: medium - claim_id: scaling_training_data_and_model_size_in_two role: supports claim: "Scaling training data and model size in two-stage codec TTS produces consistent gains up to a point of diminishing returns, with data scaling showing stronger marginal returns than model size scaling beyond 0.5B parameters on current benchmarks." source: "§4.2, §5.1" evidence: "The only model that exceeds CosyVoice 3's speaker similarity in this evaluation is Seed-TTS, which the authors attribute to larger speaker diversity in Seed-TTS pretraining data." confidence: high relevance: high - claim_id: existing_tts_benchmarks_based_on_studio_quality_audiobook role: supports claim: "Existing TTS benchmarks based on studio-quality audiobook data underestimate the difficulty of in-the-wild voice cloning, where noisy reference audio and domain diversity expose robustness gaps not visible in standard evaluations." source: "§4.4" evidence: "The only model that exceeds CosyVoice 3's speaker similarity in this evaluation is Seed-TTS, which the authors attribute to larger speaker diversity in Seed-TTS pretraining data." confidence: high relevance: medium limitations: - "Speaker similarity lags Seed-TTS across all SEED-TTS-Eval conditions, and the authors attribute this to data volume differences rather than architectural factors. Since pretraining data is not released, the degree to which CosyVoice 3's improvements are reproducible without Alibaba's internal 1M-hour corpus is unclear." - "The DiffRO reward mechanism exhibits a trade-off between intelligibility and speaker similarity: RL post-training slightly reduces speaker similarity while improving WER. Multi-reward DiffRO with emotion targets further degrades pronunciation, suggesting that reward balancing in multi-objective post-training for TTS is an open engineering problem. Timbre control via textual instructions is absent and noted as future work. Singing voice generation remains unsatisfactory due to insufficient training data coverage in the tokenizer and LM stages. Performance on Korean and Japanese lags other languages due to data volume limitations and language-specific encoding challenges (kana conversion, multiple pronunciations)." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2506.10274" published_date: "2025-06-12" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec, TTS, evaluation] architecture: [] relevance: medium evidence_role: - infrastructure current_role: influential method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: no_discrete_audio_tokenizer_consistently_outperforms_others_across role: supports claim: "No discrete audio tokenizer consistently outperforms others across reconstruction, downstream discriminative tasks, and acoustic language modeling; the optimal tokenizer is task- and domain-dependent." source: "§3.4, Figure 4" evidence: "Downstream performance uses the DASB benchmark with frozen tokenizer representations and lightweight task-specific heads across 12 tasks (ASR, speaker ID/verification, emotion recognition, keyword spotting, intent classification, speech enhancement, speech separation, music genre classification, music source separation, sound event classification, audio source separation)." confidence: high relevance: high - claim_id: increasing_the_number_of_codebooks_improves_signal_reconstruction role: supports claim: "Increasing the number of codebooks improves signal reconstruction quality but often degrades downstream task performance by adding redundancy that burdens representation-level learning." source: "§3.2, §3.2 \"Impact of Codebook Size\"" evidence: "Increasing codebook count (bitrate) often hurts downstream performance by adding redundancy." confidence: high relevance: low - claim_id: semantic_distillation_aligning_early_rvq_layers_with_ssl role: supports claim: "Semantic distillation (aligning early RVQ layers with SSL features) improves phonetic content preservation in acoustic tokenizers but may reduce cross-domain generalization when the distillation source is speech-specific." source: "§4.2 \"Distillation Effect\", Table 16" evidence: "Acoustic tokenizers trained on multi-domain data underperform speech-specific or semantic tokenizers for TTS." confidence: high relevance: high - claim_id: domain_alignment_between_tokenizer_training_data_and_evaluation role: supports claim: "Domain alignment between tokenizer training data and evaluation domain is the dominant factor in discrete audio codec performance, outweighing quantization method or bitrate choices." source: "§4.2 \"Data Domains\"" evidence: "The ablation study (Section 4) uses ESPnet-Codec with a controlled DAC backbone to isolate the effects of quantization method (RVQ, SVQ, FSQ), training domain (speech, audio, music, multi-domain), sampling rate (16 kHz vs. 44.1 kHz), and semantic distillation from SSL features." confidence: high relevance: high - claim_id: continuous_speech_representations_e_g_wavlm_large_consistently role: supports claim: "Continuous speech representations (e.g., WavLM-large) consistently outperform all discrete tokenizers on discriminative tasks, with the gap widening in low-resource conditions." source: "§3.2, Table 7" evidence: "Continuous WavLM-large representations remain the ceiling, outperforming all discrete tokenizers across most tasks, with the gap widening in low-resource settings." confidence: high relevance: high limitations: - "The TTS and audio LM evaluations are conducted with constrained training budgets (VALL-E on LibriTTS only; 300M audio LM at half the original training compute), which limits the practical transferability of findings on TTS tokenizer ranking to large-scale production settings." - "The acoustic LM evaluation uses a single architecture (Qwen-2.5 based, 357M) and a single dataset (LibriHeavy), so findings on which tokenizers support better SLMs may not generalise to other LM architectures or training scales. The ablation study (Section 4) is restricted to a DAC-backbone framework; FSQ and SVQ conclusions may not transfer to other encoder-decoder designs. Evaluation metrics for audio generation (FAD, KLD, CLAP) conflate vocoder quality with language model quality, making it difficult to attribute performance differences to the tokenizer's representational properties versus its decoder quality. The paper does not evaluate streaming tokenizers under actual latency constraints, limiting guidance for real-time deployment. Trustworthiness considerations (voice deepfakes, bias) are raised as open concerns but not empirically evaluated." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2507.02380" published_date: "2025-07-03" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: minor method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: routing_llm_hidden_states_into_the_tts_module role: supports claim: "Routing LLM hidden states into the TTS module's embedding space can enable voice cloning in an end-to-end spoken chatbot without a dedicated speaker encoder." source: "§1.2" evidence: "JoyTTS projects Qwen-7B hidden states via an MLP into the CosyVoice2-based LLM-TTS embedding, achieving SS of 0.73 on seed-tts-zh using prompt audio as the only speaker reference." confidence: high relevance: medium - claim_id: jointly_training_an_llm_chat_module_with_a role: complicates claim: "Jointly training an LLM-Chat module with a TTS module in a chatbot pipeline can degrade intelligibility relative to running the TTS component standalone, even when speaker similarity improves." source: "§4, Table 1" evidence: "JoyTTS achieves WER of 5.09 compared to 1.45 for standalone CosyVoice2 on seed-tts-zh, despite closing the speaker similarity gap (JoyTTS SS 0.73 vs. CosyVoice2 SS 0.748)." confidence: high relevance: medium - claim_id: end_to_end_spoken_chatbot_systems_pairing_a role: supports claim: "End-to-end spoken chatbot systems pairing a 7B-parameter LLM with an autoregressive TTS module can achieve sub-2-second response latency on a single consumer GPU without specialised inference optimisations." source: "§4" evidence: "JoyTTS reports 1.8-second end-to-end latency on a single NVIDIA 4090D with no engineering optimisation applied." confidence: high relevance: high limitations: - "The evaluation is limited to a single Chinese benchmark (seed-tts-zh), leaving performance on English, multilingual, or spontaneous conversational speech uncharacterised. The WER gap between JoyTTS (5.09) and standalone CosyVoice2 (1.45) is large and unexplained; the paper does not ablate whether the regression originates from the joint training procedure, the conversational data distribution, or the MLP projection coupling. No subjective listening tests are reported, making it impossible to assess naturalness or perceived quality beyond intelligibility and speaker similarity metrics. Emotion control, identified as a target for future work, is not implemented in the current system." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.01348" published_date: "2025-07-08" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [VC, TTS] architecture: [autoregressive-LM, VAE, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [masked_parallel_codec_alternatives] claims: - claim_id: ctc_regularization_applied_before_vector_quantization_improves_the role: supports claim: "CTC regularization applied before vector quantization improves the temporal locality and temporal robustness of discrete speech content tokens." source: "§5.2, Table 2" evidence: "SpeechCodeVAE achieves 59% higher De-duplication Efficiency and approximately 9 times better Speed Robustness than CosyVoice-50Hz; ablation without CTC loss collapses Speed Robustness from 0.219 to 0.009, identifying CTC as the critical factor." confidence: high relevance: high - claim_id: multi_task_learning_with_tts_as_an_auxiliary role: supports claim: "Multi-task learning with TTS as an auxiliary objective compensates for data scarcity in foreign accent conversion, improving both convergence and output quality." source: "§3, §5.1, Table 1" evidence: "Joint FAC+TTS training on 370 hours of TTS data alongside 9.3 hours of FAC data yields 25% accentedness reduction and WER improvement from 14.4% to 9.1% relative to a standalone FAC baseline." confidence: high relevance: medium - claim_id: bert_style_masked_token_restoration_can_correct_stochastic role: supports claim: "BERT-style masked token restoration can correct stochastic local substitution errors introduced by autoregressive speech token decoding, improving acoustic continuity." source: "§5.3, Table 4" evidence: "Removing SpeechRestorer decreases TTS CMOS from 3.850 to 3.629, with the paper attributing the gain to error-correction of spurious token substitutions that cause acoustic discontinuities." confidence: high relevance: high - claim_id: token_level_post_processing_for_llm_speech_generation role: complicates claim: "Token-level post-processing for LLM speech generation can correct local substitution errors but fails to address higher-level failure modes such as word skipping and repetition." source: "§6" evidence: "The paper explicitly states that SpeechRestorer cannot fix skipped words or repetitions, as these require sequence-level rather than token-level correction." confidence: high relevance: high - claim_id: training_data_scale_rather_than_architectural_design_is role: refines claim: "Training data scale, rather than architectural design, is the primary driver of quality gaps between LLM-based TTS systems at different performance levels." source: "§5.3, Table 4" evidence: "SpeechAccentLLM trails NaturalSpeech2 in TTS naturalness (CMOS 3.850 vs. 3.944) despite comparable architecture; the gap is attributed to NS2 training on approximately two orders of magnitude more data." confidence: high relevance: medium limitations: - "The FAC evaluation is restricted to four L1 backgrounds from L2-ARCTIC and uses a single TTS model (LJSpeech-trained VITS) to generate native accent counterparts. Generalisation to other accents, speaking styles, or higher-quality native reference speech is untested." - "Prosody modelling is acknowledged as incomplete; the Variance Adapter models f0 only and does not capture rhythm, duration patterns, or prosodic phrasing beyond pitch. Timbre reconstruction quality is bounded by the frozen ECAPA-TDNN speaker encoder, which was not trained for the L2/accented-speech domain. SpeechRestorer cannot resolve sequence-level decoding failures (word skipping, repetition), leaving a category of LLM-generated errors unaddressed. The paper does not report total parameter counts for any module, which complicates direct comparison with other systems." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2506.23325" published_date: "2025-07-09" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN, hybrid] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: minimizing_shared_parameters_between_semantic_and_acoustic_encoding role: supports claim: "Minimizing shared parameters between semantic and acoustic encoding pathways in a speech codec reduces task conflict and enables simultaneous strong text alignment and high acoustic fidelity at low bitrates." source: "§3.1, §4.4, Table 4" evidence: "XY-Tokenizer's dual-tower architecture (parameters shared only in the RVQ module) achieves WER 0.13 and SPK-SIM 0.83 at 1 kbps; a matched single-channel model sharing encoder and RVQ across both tasks achieves SPK-SIM 0.77 with the same WER, confirming that parameter sharing degrades reconstruction." confidence: high relevance: high - claim_id: llm_based_asr_supervision_during_codec_pre_training role: supports claim: "LLM-based ASR supervision during codec pre-training provides stronger text alignment than SSL representation distillation for low-bitrate codecs." source: "§4.3, Table 3" evidence: "At approximately 1 kbps, XY-Tokenizer (LLM-based ASR, WER 0.13) substantially outperforms SpeechTokenizer variants (distillation-based, WER 0.18-0.34) and Mimi-8 (distillation-based, WER 0.28) on the ASR probing task, while maintaining comparable or better reconstruction quality than those baselines." confidence: high relevance: high - claim_id: representation_distillation_from_ssl_models_for_semantic_alignment role: complicates claim: "Representation distillation from SSL models for semantic alignment introduces reconstruction-quality penalties that worsen with distillation strength at low bitrates." source: "§4.3, Table 3" evidence: "SpeechTokenizer-x3 (5x stronger distillation than the official setting) achieves WER 0.18 but SPK-SIM only 0.48 at 1.5 kbps, compared to SpeechTokenizer-x1 (WER 0.34, SPK-SIM 0.65), demonstrating that stronger distillation systematically degrades acoustic fidelity." confidence: high relevance: low - claim_id: allowing_the_llm_decoder_to_train_freely_during role: complicates claim: "Allowing the LLM decoder to train freely during multi-task codec pre-training improves autoregressive text generation but degrades the encoder's transferable text-alignment representations." source: "§4.4, Table 5" evidence: "A trainable-LLM variant achieves lower LLM decoder WER (0.03 vs. 0.06 at 200K steps) but higher ASR probing WER (0.18 vs. 0.13), with the probing WER worsening progressively over 800K training steps, suggesting the encoder's text alignment capacity migrates into the flexible decoder." confidence: high relevance: high - claim_id: whisper_s_supervised_asr_pre_training_enables_better role: supports claim: "Whisper's supervised ASR pre-training enables better paralinguistic information preservation in codec encoder architectures than self-supervised alternatives." source: "§2, Table 1" evidence: "Preliminary auto-encoder experiments with frozen encoders show Whisper achieves SPK-SIM 0.68, STOI 0.88, and PESQ-NB 2.03 on LibriSpeech test-clean, compared to WavLM (SPK-SIM 0.53) and HuBERT (SPK-SIM 0.42), motivating Whisper as the codec encoder initialization." confidence: high relevance: high limitations: - "Evaluation uses only objective metrics (WER, SPK-SIM, STOI, PESQ). No subjective listening tests or MOS scores are reported, making it impossible to assess perceptual audio quality relative to competing systems." - "Achieving further reductions in bitrate below 1 kbps without performance degradation remains an open challenge noted by the authors. The scaling behavior of the two-stage training approach with respect to parameter count and dataset size is not characterized, leaving open questions about how to optimize training efficiency for larger variants. The LLM-based ASR decoder (Qwen2.5-0.5B) adds substantial parameter overhead during pre-training that is absent during inference, but the computational cost and training complexity of this stage relative to distillation-based approaches is not discussed." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2507.09070" published_date: "2025-07-11" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [VC] architecture: [autoregressive-LM, flow-matching] relevance: low evidence_role: - historical_context current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: audio_codec_and_self_supervised_speech_representations_inherently role: supports claim: "Audio codec and self-supervised speech representations inherently encode speaker identity, making timbre leakage a structural challenge in codec-based voice conversion." source: "§4.1, Table 1" evidence: "Speaker classification accuracy on LibriHeavy using EnCodec token IDs reaches 96.7%, HuBERT layer 9 discrete tokens 71.7%, and the authors' BEST-RQ tokenizer 82.05%, all substantially above chance." confidence: high relevance: high - claim_id: aligning_audio_encoder_outputs_to_speaker_independent_text role: supports claim: "Aligning audio encoder outputs to speaker-independent text embeddings via monotonic alignment search produces representations with negligible residual speaker identity." source: "§4.1, §5, Table 1" evidence: "The SemAlign-trained semantic encoder Q_ϕ achieves 2.84% speaker classification accuracy on LibriHeavy, versus 82.05% for the same encoder's raw tokenizer, confirming that the alignment objective removes speaker information that CTC loss alone cannot." confidence: high relevance: low - claim_id: zero_shot_voice_conversion_without_explicit_speaker_verification role: supports claim: "Zero-shot voice conversion without explicit speaker verification embeddings can achieve higher speaker similarity than systems that rely on them." source: "§4.2, Tables 2-3" evidence: "SemAlignVC achieves the highest SMOS (3.29), WavLM speaker similarity (0.95), ECAPA-TDNN (0.82), and Resemblyzer (0.89) among KNNVC, HierSpeech++, and UniAudio on VCTK and LibriHeavy evaluations, despite using only a reference mel spectrogram excerpt rather than a speaker embedding." confidence: high relevance: low - claim_id: aggressive_timbre_disentanglement_via_text_embedding_alignment_introduces role: complicates claim: "Aggressive timbre disentanglement via text-embedding alignment introduces intelligibility trade-offs when BERT-derived representations replace phoneme-based encodings." source: "§5, Table 3" evidence: "SemAlignVC achieves 12.31% WER on LibriHeavy, compared to HierSpeech++'s 8.24%, with the authors attributing the gap to occasional word substitutions caused by the semantic ambiguity of BERT token representations during generation." confidence: high relevance: low limitations: - "The audio tokenizer is trained on a proprietary internal dataset and is described as interchangeable, but its interaction with SemAlign has not been tested with public codecs. Results may not transfer directly to systems using EnCodec, SpeechTokenizer, or other publicly available tokenizers." - "The evaluation is English-only and uses VCTK and LibriHeavy, which are relatively clean audiobook and read-speech corpora. Performance on spontaneous speech, accented speech, or noisy conditions is not assessed. The baseline set is modest (three systems), and no ablation isolates the contribution of the flow matching acoustic model relative to SemAlign itself. The WER gap relative to HierSpeech++ remains unexplained beyond the synonym-substitution hypothesis, which is not directly tested." caveats: - "The connection to autoregressive codec TTS is contextual rather than a direct system contribution." - id: "2507.12197" published_date: "2025-07-16" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, singing] architecture: [autoregressive-LM, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [fully_autoregressive_multicodebook] claims: - claim_id: scaling_the_number_of_rvq_codebooks_in_a role: supports claim: "Scaling the number of RVQ codebooks in a discrete speech codec reduces information loss and improves reconstruction quality for expressive and challenging vocal content." source: "§3.2, Table 5" evidence: "QDAC reconstruction improves monotonically from 1 to 16 codebooks across PESQ, STOI, and SI-SDR; 16-codebook QDAC at 50Hz achieves PESQ 3.83 versus PESQ 2.98 for 8 codebooks, with similar gains on Mel distance and speaker similarity." confidence: high relevance: high - claim_id: higher_multi_codebook_reconstruction_fidelity_in_a_codec role: complicates claim: "Higher multi-codebook reconstruction fidelity in a codec does not necessarily translate into better speaker identity preservation in downstream zero-shot TTS." source: "§3.2, Table 6" evidence: "QTTS achieves Spk Sim 0.75 on SeedTTS-Easy compared to 0.82 and 0.81 for single-codebook CosyVoice v1 and v2, despite QDAC's superior reconstruction metrics in Table 5." confidence: high relevance: high - claim_id: semantic_disentanglement_in_audio_codecs_can_be_achieved role: supports claim: "Semantic disentanglement in audio codecs can be achieved by backpropagating ASR loss exclusively through the first RVQ codebook, enforcing content isolation without relying on general-purpose self-supervised representations." source: "§2.1.2, Table 5" evidence: "QDAC trains an AR-ASR module conditioned only on first-codebook tokens; the resulting WER at reconstruction (6.42 for 8cb/25Hz) is close to ground truth (6.01), indicating the first codebook encodes phoneme-level content while residual codebooks capture acoustic detail." confidence: high relevance: high - claim_id: multi_codebook_autoregressive_tts_admits_a_principled_speed role: refines claim: "Multi-codebook autoregressive TTS admits a principled speed-quality trade-off by choosing between strict hierarchical inter-codebook conditioning and a delayed multi-head parallel prediction scheme." source: "§2.2, §2.3, Tables 1, 3, 4" evidence: "Hierarchy Parallel (200M, dual-AR) and Multihead Delay (120M, parallel with fixed delay) achieve comparable TTFT at 512 tokens (26ms vs 24ms) but differ substantially in decode throughput; the Multihead variant reaches over 196K codebook tokens/s at short output lengths versus 105K for Hierarchy." confidence: high relevance: high - claim_id: mos_evaluations_in_zero_shot_tts_can_yield role: complicates claim: "MOS evaluations in zero-shot TTS can yield above-reference scores for synthesised speech, undermining direct absolute comparisons across studies." source: "§3.2, Table 6" evidence: "Ground truth speech achieves MOS 2.7 while QTTS, CosyVoice, and CosyVoice2 all score between 3.01 and 3.03 on the same test set, producing a ranking inconsistent with naturalness expectations." confidence: high relevance: medium limitations: - "Training data is not disclosed, the PGC-hard benchmark is proprietary, and no code or demo is available, making results difficult to reproduce or build upon." - "The evaluation compares only against the CosyVoice v1/v2 family, leaving open how QTTS performs relative to flow-matching systems, other multi-codebook approaches, or stronger autoregressive baselines. Speaker similarity is lower than both single-codebook baselines (0.75 vs 0.81-0.82), suggesting the multi-codebook generation pipeline needs further work to fully leverage the improved codec for speaker transfer. The paper positions singing and music synthesis as motivating use cases but does not evaluate on these tasks. Only the 8-codebook QTTS model is evaluated for TTS synthesis, leaving open whether 16-codebook generation would further improve or introduce new training challenges." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.16632" published_date: "2025-07-22" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, preference_optimized_autoregression] claims: - claim_id: interleaving_discrete_audio_tokens_with_text_tokens_in role: supports claim: "Interleaving discrete audio tokens with text tokens in a shared language model vocabulary enables end-to-end spoken dialogue systems that respond coherently to paralinguistic cues without requiring separate speech synthesis pipelines." source: "§3.1, §4.6" evidence: "Step-Audio 2 is a multi-modal LLM that processes raw audio as input and generates interleaved sequences of discrete text and audio tokens." confidence: high relevance: high - claim_id: reinforcement_learning_applied_to_audio_language_models_can role: supports claim: "Reinforcement learning applied to audio language models can improve reasoning efficiency in complex acoustic scenarios while preserving generation quality." source: "§3.4" evidence: "Supervised fine-tuning follows on 4B tokens, succeeded by multi-stage reinforcement learning using proximal policy optimization (PPO) and group relative policy optimization (GRPO) to improve reasoning efficiency and audio perceptual ability." confidence: high relevance: high - claim_id: retrieval_augmented_generation_and_external_tool_calling_can role: supports claim: "Retrieval-augmented generation and external tool calling can substantially reduce hallucination and expand capability (timbre switching, web-grounded responses) in large audio language models without architectural redesign." source: "§3.1, §4.5" evidence: "A key capability addition is tool calling, including a novel audio search tool that maintains a library of hundreds of thousands of speech samples with descriptions." confidence: high relevance: high - claim_id: comprehensive_multi_task_pre_training_across_asr_tts role: supports claim: "Comprehensive multi-task pre-training across ASR, TTS, translation, and conversation significantly improves spoken dialogue performance in low-resource languages and accented speech." source: "§3.2, §4.1" evidence: "Training proceeds in four stages: adaptor alignment on 100B ASR tokens with frozen encoder and LLM; vocabulary expansion and joint training on 256B mixed tokens; main pre-training on 800B tokens across ASR, TTS, translation, and speech-to-speech conversation data; and a cooldown phase on 200B high-quality tokens." confidence: high relevance: medium - claim_id: existing_audio_language_model_benchmarks_fail_to_capture role: complicates claim: "Existing audio language model benchmarks fail to capture fine-grained paralinguistic comprehension and voice-triggered tool invocation, leaving important capability dimensions systematically unmeasured." source: "§4.2, §4.5" evidence: "On the authors' StepEval-Audio-Paralinguistic benchmark evaluating 11 paralinguistic dimensions, Step-Audio 2 achieves 83.09% average accuracy compared to 49.64% for Kimi-Audio and 44.18% for Qwen-Omni (Table 2)." confidence: high relevance: high limitations: - "Two of the main evaluation benchmarks (StepEval-Audio-Paralinguistic, StepEval-Audio-Toolcall) are introduced by the authors themselves and have not been validated independently. Results on these benchmarks may overstate absolute capability levels even if relative comparisons are informative." - "Model size is not reported, which makes parameter-count comparisons with other open-source systems (Kimi-Audio, Qwen2.5-Omni) difficult to interpret fairly. The open-source mini variant uses Qwen2.5-7B as its backbone, but the full model's parameter count remains undisclosed." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.21138" published_date: "2025-07-22" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: rl_alignment_with_composite_perceptual_rewards_improves_multiple role: supports claim: "RL alignment with composite perceptual rewards improves multiple speech quality dimensions simultaneously over SFT-only baselines in autoregressive codec TTS." source: "§3.5, Table 8" evidence: "GRPO with combined WER + speaker similarity + DNSMOS rewards achieves gains in all three metrics over the SFT baseline, while single-reward models improve only their target dimension; TTS-1 WER drops from 7.9% (SFT) to 6.3% (RL-aligned)." confidence: high relevance: high - claim_id: audio_pre_training_on_large_scale_raw_speech role: supports claim: "Audio pre-training on large-scale raw speech substantially improves subsequent SFT quality in LLM-based TTS beyond what instruction-tuned LLM initialisation provides." source: "§3.4, Figure 5" evidence: "Initialising SFT from an audio pre-trained LLaMA-3.2-1B checkpoint yields lower SFT loss, approximately 15% lower WER, and approximately 3% higher speaker similarity than initialising from the base LLaMA-3.2-1B-Instruct checkpoint." confidence: high relevance: medium - claim_id: scaling_speechlm_parameter_count_in_autoregressive_codec_tts role: supports claim: "Scaling SpeechLM parameter count in autoregressive codec TTS consistently improves intelligibility and speaker fidelity across languages." source: "§4, Figure 8, Table 8" evidence: "TTS-1-Max (8.8B) achieves 5.1% overall WER and higher SIM across all 11 evaluated languages compared to TTS-1 (1.6B) at 6.3% WER, with performance correlating with pre-training loss differences." confidence: high relevance: high - claim_id: style_conditioning_via_discrete_text_tags_conflicts_with role: complicates claim: "Style conditioning via discrete text tags conflicts with speaker identity preservation in single-codebook codec TTS architectures." source: "§3.6" evidence: "Direct prepending of style markup tags during SFT produced no effect on synthesized speech because the single-codebook design entangles acoustic and semantic information; successful style control required constructing paired neutral/stylized utterances from the same speaker and applying LoRA fine-tuning." confidence: high relevance: high - claim_id: streaming_audio_delivery_in_autoregressive_tts_introduces_audible role: complicates claim: "Streaming audio delivery in autoregressive TTS introduces audible artifacts and volume inconsistencies at segment boundaries that require specific engineering mitigations independent of the generative model's quality." source: "§5.1" evidence: "Without concatenation restricted to non-voicing regions and context-aware decoder decoding with extended audio prompt context, segment boundaries introduce clicks and volume drops; these are engineering-layer problems independent of SpeechLM quality." confidence: high relevance: high limitations: - "Model weights are not publicly released, making independent benchmarking and replication impossible. All evaluations use proprietary or internal test sets; the internal TTS arena covers only English and uses approximately 20 annotators with a modest vote count per pair." - "The training data is drawn from a proprietary mixture of public and licensed sources whose exact composition is not disclosed, limiting reproducibility. The evaluation framework does not include standard public TTS benchmarks (e.g., LibriTTS or VCTK test sets), making direct numerical comparison with published systems that do report on these benchmarks difficult." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.18897" published_date: "2025-07-25" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: single_quantizer_neural_codecs_can_match_multi_quantizer role: supports claim: "Single-quantizer neural codecs can match multi-quantizer systems in perceived speech quality at far lower bitrates when combined with stabilized VQ spaces and asymmetric decoder architectures." source: "§4.3, Table 1" evidence: "HH-Codec achieves UTMOS 3.61 on LibriTTS test-clean at 0.3 kbps with a single quantizer, competitive with DAC's 4-quantizer configuration at 4 kbps (UTMOS 3.41) and markedly above SpeechTokenizer's single-quantizer variant at 0.75 kbps (UTMOS 1.26)." confidence: high relevance: high - claim_id: multi_layer_vq_training_with_single_layer_inference role: supports claim: "Multi-layer VQ training with single-layer inference, where auxiliary quantizer layers act as regularizers, substantially improves codebook utilization in high-compression single-quantizer settings." source: "§4.4, Tables 2-3" evidence: "SLM-VQ achieves 98% codebook utilization at 8192 entries versus 56% for Classic VQ and 92% for single-layer SLM-VQ, while improving UTMOS from 2.76 (Classic VQ) to 3.07 on LibriTTS test-other." confidence: high relevance: low - claim_id: dual_domain_supervision_combining_intermediate_mel_spectrogram_and role: supports claim: "Dual-domain supervision combining intermediate mel-spectrogram and final audio reconstruction objectives is critical for stable high-compression neural codec training." source: "§4.4, Table 2" evidence: "Reducing to single audio-domain supervision drops UTMOS from 3.07 to 1.85 and SPK-SIM from 0.64 to 0.33 on LibriTTS test-other, the largest degradation across all ablation variants." confidence: high relevance: high - claim_id: standard_adversarial_codec_training_recipes_break_down_at role: complicates claim: "Standard adversarial codec training recipes break down at extreme compression ratios, requiring architectural and procedural modifications to avoid collapse." source: "§1" evidence: "Below 0.3 kbps with existing methods, the paper documents adversarial training collapse, a 63% UTMOS drop below 30 tokens/s, 43% codebook utilization at 8192 entries, and minimal benefit from expanding training data, all addressed in HH-Codec through SLM-VQ and progressive training." confidence: high relevance: high limitations: - "HuBERT-based semantic distillation is trained on English, so multilingual performance of SLM-VQ is unknown. The downstream spoken language modeling experiment measures only training loss reduction rather than end-to-end TTS quality, leaving it unclear how the 24-token-per-second compression affects downstream synthesis intelligibility and naturalness. Training data conditions differ across compared baselines (WavTokenizer and DAC use larger or different datasets), which limits direct attribution of gains to architecture versus data. The ablation study is conducted on a subset of training data only (LibriTTS train-100/360, not the full training set including Emilia), which may underestimate some component contributions." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2025.acl-long.1043" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS] architecture: [flow-matching, hybrid] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: influential method_family: [semantic_acoustic_token_cascade] claims: - claim_id: replacing_gaussian_noise_with_a_learned_prior_as role: supports claim: "Replacing Gaussian noise with a learned prior as the starting point for flow matching reduces the number of required inference steps to one without needing a separate distillation stage." source: "§3.1, §3.3" evidence: "Vector Field Estimator (OT-CFM): Rather than regressing vectors from Gaussian noise x_0 to data x_1, the estimator regresses from the learned prior x_pr to x_1." confidence: high relevance: low - claim_id: flow_matching_tts_systems_trained_on_traditional_ot role: complicates claim: "Flow-matching TTS systems trained on traditional OT-CFM are data-hungry and fail to generalise when retrained on limited data, while neural codec-based systems remain effective with as few as 500 hours." source: "§4.2, Table 1" evidence: "OZSpeech consists of three components: FACodec (frozen): NaturalSpeech 3's factorized neural codec decomposes speech into 6 discrete token sequences encoding prosody (1 quantizer), content (2 quantizers), and acoustic details (3 quantizers), plus a separate timbre embedding." confidence: high relevance: high - claim_id: zero_shot_tts_systems_trained_exclusively_on_clean role: supports claim: "Zero-shot TTS systems trained exclusively on clean prompts degrade substantially in intelligibility when given noisy reference audio, with autoregressive codec models being especially vulnerable." source: "§4.4, Table 4" evidence: "Under noisy prompts (Table 4), all baselines degrade dramatically." confidence: high relevance: high - claim_id: factorised_codec_representations_that_balance_acoustic_and_semantic role: supports claim: "Factorised codec representations that balance acoustic and semantic attributes trade perceptual naturalness (UTMOS) for intelligibility (WER) relative to codecs that prioritise acoustic fidelity." source: "§4.2" evidence: "UTMOS is lower than F5-TTS and VALL-E (3.15 vs. 3.7+) due to codec trade-offs (FACodec balances acoustic and semantic representations)." confidence: high relevance: high - claim_id: fine_tuning_a_zero_shot_tts_model_on role: supports claim: "Fine-tuning a zero-shot TTS model on noise-augmented prompts preserves intelligibility and substantially recovers acoustic quality metrics under low-SNR conditions." source: "§4.4, Table 4" evidence: "Fine-tuning with noisy prompts further improves acoustic quality metrics." confidence: high relevance: low limitations: - "- UTMOS and speaker similarity lag behind F5-TTS and VoiceCraft; the prior-based approach trades acoustic naturalness for intelligibility and speed. - Duration predictor rounding errors (integer quantization of phoneme durations) introduce temporal domain artifacts. - FACodec dependency means reproduction requires NaturalSpeech 3's codec infrastructure. - Noise robustness fine-tuning improves non-WER metrics but was validated only on the QUT-NOISE database. - Future work: multilingual extension, adaptive noise filtering integration." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2025.acl-long.1498" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, codec] architecture: [autoregressive-LM, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: context_dependent_encoding_in_rvq_based_neural_audio role: supports claim: "Context-dependent encoding in RVQ-based neural audio codecs causes perceptually equivalent audio segments to produce divergent discrete token sequences, increasing prediction uncertainty in downstream codec language models." source: "§2, §2.3" evidence: "The paper first defines a consistency accuracy metric: given a full audio clip and a random slice extracted from it, consistency accuracy measures what fraction of the slice's RVQ token positions are identical between encoding the slice in isolation and encoding the same frames within the full-length clip." confidence: high relevance: high - claim_id: codec_token_consistency_and_downstream_autoregressive_tts_quality role: supports claim: "Codec token consistency and downstream autoregressive TTS quality are monotonically correlated: improvements in consistency accuracy reliably reduce word error rate and increase speaker similarity." source: "§5.2, Figure 4" evidence: "The reported correlation between consistency accuracy and WER (Figure 4) shows a clear monotonic relationship — improving consistency reliably reduces downstream WER." confidence: high relevance: high - claim_id: auxiliary_consistency_losses_applied_during_codec_training_can role: supports claim: "Auxiliary consistency losses applied during codec training can substantially increase token consistency with negligible impact on reconstruction quality." source: "§5.1, Table 1" evidence: "To mitigate DRI without shrinking the encoder's receptive field (which would harm reconstruction quality), the authors add two auxiliary MSE losses during codec training: Slice-consistency: A random 20% segment is carved out of each training audio clip and encoded in isolation to obtain Z_slice." confidence: high relevance: high - claim_id: consistency_constraint_methods_applied_to_codec_training_generalize role: supports claim: "Consistency constraint methods applied to codec training generalize across neural codec architectures and autoregressive LM backbones, as demonstrated by cross-system experiments on both EnCodec-VALL-E and FunCodec-UniAudio pipelines." source: "§5.2, Table 3" evidence: "Cross-codec generalizability (Table 3): Applying consistency constraint to EnCodec (as input to VALL-E) reduces WER from 9.02% to 5.13% at 960 h." confidence: high relevance: high - claim_id: in_rvq_codecs_deeper_codebook_layers_suffer_disproportionately role: supports claim: "In RVQ codecs, deeper codebook layers suffer disproportionately from context-induced inconsistency, because they encode fine-grained acoustic detail that is more sensitive to contextual perturbation than the semantic information stored in shallow layers." source: "§2.3, Appendix A.3, Table 6" evidence: "The analysis shows that: (1) all existing codecs suffer from low consistency — FunCodec 8 kbps has only 3.58% consistency across all 16 codebook layers; (2) consistency deglines substantially with deeper codebook layers in every codec examined, because shallow layers encode semantic (context-independent) information while deeper layers encode fragile fine-grained acoustic detail; and (3) the large receptive field of the RVQ encoder (up to 2718 samples at layer 18, per Table 5 in the paper) is the structural cause." confidence: high relevance: high limitations: - "The DRI fix is applied at codec training time; it does not address the underlying structural cause (large receptive fields in the convolutional encoder). A follow-up could explore whether directly reducing receptive field size with compensating distillation achieves similar or better consistency." - "The method is validated on English speech (LibriTTS and MLS, which is multilingual but dominated by English). Whether DRI and its mitigation generalize to tonal languages (where fine-grained acoustic context is phonemically contrastive) is unstudied." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.acl-long.65" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS] architecture: [autoregressive-LM, VAE, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hierarchical_ar_nar_codec_tts, hybrid_ar_token_continuous_renderer, masked_parallel_codec_alternatives] claims: - claim_id: discrete_codec_representations_introduce_a_quantifiable_fidelity_loss role: supports claim: "Discrete codec representations introduce a quantifiable fidelity loss relative to continuous mel-spectrogram representations even at high codebook counts, measurable in both WER and speaker similarity." source: "§5, Table 1" evidence: "Objective (Table 1, LibriSpeech test-clean, continuation task): - MELLE WER_C/WER_H: 1.47/1.98, SIM: 0.508 - VALL-E 2 WER_C/WER_H: 1.6/2.32, SIM: 0.504 - VALL-E WER_H: 3.8, SIM: 0.508 - Ground truth WER_H: 2.15, SIM: 0.668 (mel resynthesis: WER_H 2.24, SIM 0.617; EnCodec 8CB: WER_H 2.33, SIM 0.593) — confirming codec fidelity loss hypothesis MELLE achieves 47.9% relative WER reduction vs. VALL-E and 8.1% vs. VALL-E 2 on continuation." confidence: high relevance: high - claim_id: continuous_valued_autoregressive_speech_synthesis_can_achieve_robustness role: supports claim: "Continuous-valued autoregressive speech synthesis can achieve robustness and naturalness on par with codec-based two-stage systems when paired with appropriate regularization objectives." source: "§5.1, §5.2, Table 1, Table 3" evidence: "This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling." confidence: high relevance: high - claim_id: a_variational_latent_sampling_module_applied_to_continuous role: supports claim: "A variational latent sampling module applied to continuous spectrogram prediction provides diversity and robustness benefits analogous to top-p sampling for discrete tokens, without the instability caused by the high similarity of consecutive acoustic codes." source: "§3.2.2, §5.3, Table 4" evidence: "This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling." confidence: high relevance: high - claim_id: bypassing_the_non_autoregressive_second_stage_in_codec role: supports claim: "Bypassing the non-autoregressive second stage in codec language model pipelines reduces inference time while maintaining competitive output quality." source: "§5.4, Table 5" evidence: "Subjective (Table 3, 40 samples cross-sentence): - MELLE MOS: 4.20 ± 0.20 vs. GT 4.29 ± 0.16 (CMOS -0.032, p > 0.1 → not significantly different) - MELLE SMOS: 4.40 ± 0.22 vs. GT 3.94 ± 0.25 — higher than ground truth, suggesting the model's speaker reproduction is more consistent than inter-utterance variation in the reference - VALL-E 2 MOS: 4.08, CMOS: -0.085 Efficiency (Table 5): - MELLE: 5.49s for 10s speech vs. VALL-E 2: 7.32s (no NAR second pass) - MELLE-R2: 2.76s, MELLE-R4: 1.40s — 5x speedup at r=4 while still competitive Ablation (Table 4): Removing both LSM and SFL degrades cross-sentence WER_C from 1.47 to 23.21 — catastrophic failure." confidence: high relevance: high - claim_id: prediction_quality_in_continuous_valued_autoregressive_tts_degrades role: complicates claim: "Prediction quality in continuous-valued autoregressive TTS degrades gracefully with reduction factor increases, enabling a controllable quality-efficiency trade-off unavailable in discrete-token systems." source: "§5.1, Table 1, Table 2" evidence: "This module replaces top-p sampling for codec codes: it provides continuous diversity without the instability of discrete token sampling." confidence: high relevance: high limitations: - "- English-only evaluation; multilingual extension not attempted. - Vocoder quality bottleneck: uses open-source HiFi-GAN trained on 585h LibriTTS; Voicebox's proprietary vocoder trained on 60Kh provides higher quality ceiling. - Mel-spectrogram as the only continuous representation explored; VAE latent states suggested as future work. - SMOS exceeding ground truth may partly reflect the test setup's limitation (inter-speaker/inter-session variation in the reference set rather than genuine quality superiority). - No streaming or low-latency inference analysis." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.acl-long.654" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, codec] architecture: [GAN, VAE] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: codec_token_distributions_in_the_first_rvq_channel role: complicates claim: "Codec token distributions in the first RVQ channel are a meaningful bottleneck for autoregressive generation from text, independent of reconstruction quality." source: "§1, §3.3" evidence: "The net effect is that no single early channel dominates information content, making the first-channel token distribution lower-entropy and therefore more tractable for a text-conditioned autoregressive model to predict." confidence: high relevance: high - claim_id: redistributing_information_load_uniformly_across_the_first_few role: supports claim: "Redistributing information load uniformly across the first few RVQ codebook channels via parallel masked quantization consistently improves speaker similarity in downstream autoregressive TTS." source: "§4.4, Table 3" evidence: "In the first Nq=3 layers the quantizers operate in parallel: each receives only 1/Nq of the latent frame (the rest is masked), so each of the first three codebooks carries an equal, reduced share of the total information." confidence: high relevance: high - claim_id: a_fourier_based_decoder_with_a_self_attention role: supports claim: "A Fourier-based decoder with a self-attention module achieves better codec reconstruction quality than a transposed-convolution upsampler, without length extrapolation issues." source: "§3.2, Appendix G, Table 9" evidence: "The decoder replaces EnCodec's transposed-convolution upsampler with a Vocos-style Fourier-based decoder." confidence: high relevance: high - claim_id: codec_reconstruction_quality_does_not_scale_substantially_with role: supports claim: "Codec reconstruction quality does not scale substantially with training data volume beyond a few hundred hours, while domain generalization does benefit from larger and more diverse datasets." source: "Appendix A, Table 5" evidence: "Codec reconstruction (LibriTTS Test-Clean, Table 1): - Language-Codec with 4 codebooks (3.0 kbps): UTMOS 3.79, PESQ 3.27, STOI 0.949 — outperforms all 4-codebook baselines and several 8-codebook baselines (eg., Encodec-8 PESQ 2.72, STOI 0.939). - Language-Codec with 8 codebooks (6.0 kbps): UTMOS 4.04, PESQ 3.88, STOI 0.972 — best across all models on all metrics. - Results hold under noisy conditions (LibriTTS Test-Other, Table 6) and out-of-domain (LJSpeech, Table 7)." confidence: high relevance: high - claim_id: the_choice_of_underlying_codec_has_a_larger role: supports claim: "The choice of underlying codec has a larger impact on zero-shot TTS speaker similarity than on intelligibility, with codec swaps producing 10–15% SPK-SIM gains while WER differences remain within noise." source: "§4.3, Table 2" evidence: "Downstream zero-shot TTS (LibriSpeech Test-Clean, Table 2): - Replacing EnCodec with Language-Codec in VALL-E raises SPK-SIM from 0.612 to 0.700 (+14%) with negligible WER change. - MobileSpeech with Language-Codec: SPK-SIM 0.771, WER 2.9%, MOS-Q 4.20 vs. EnCodec baseline MOS-Q 3.91." confidence: high relevance: high limitations: - "- Language-Codec is trained and evaluated exclusively on speech; audio, music, and environmental sound domains are explicitly left as future work. The codec's suitability for general audio language models is therefore unvalidated. - MCRVQ prediction accuracy drops when more than 4 codebook channels are used in downstream models, suggesting the information-redistribution benefit weakens at higher bitrates. The mechanism for this degradation is not fully explained. - The paper evaluates downstream quality only with VALL-E and MobileSpeech; it is unclear whether the SPK-SIM gains extend to flow-matching or diffusion-based TTS backends. - No demo page is linked in the paper, making it difficult to subjectively verify the quality claims beyond the crowd-sourced MOS. - The internal 20,000-hour Chinese dataset is not publicly available, limiting full reproducibility of the 50k-hour training run." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2025.acl-long.682" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, SCA, evaluation] architecture: [autoregressive-LM, GAN, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, full_duplex_multistream_generation, preference_optimized_autoregression] claims: - claim_id: gan_based_vocoders_occupy_the_dominant_position_in role: supports claim: "GAN-based vocoders occupy the dominant position in production speech generation systems because their computational efficiency advantage over autoregressive and diffusion alternatives is orders of magnitude, at acceptable perceptual quality cost." source: "§3.3.1, §F.3, Table 8" evidence: "GAN-based vocoders (HiFi-GAN, BigVGAN) dominate in practice due to their computational efficiency and real-time factor orders of magnitude better than autoregressive or diffusion alternatives." confidence: high relevance: high - claim_id: continued_pretraining_from_a_text_language_model_checkpoint role: supports claim: "Continued pretraining from a text language model checkpoint improves SpeechLM convergence and downstream task performance compared to random initialization." source: "§4.2.1" evidence: "The synthesised empirical picture includes: HuBERT outperforms CPC and wav2vec 2.0 as a semantic tokenizer (GSLM); continued pretraining from TextLMs improves convergence and task performance over cold initialization (TWIST); interleaving text and speech tokens during pretraining improves cross-modal performance (SPIRIT-LM); larger TextLM checkpoints and more data continue to improve SpeechLMs (AudioPaLM, GLM-4-Voice with 13M hours of synthetic data)." confidence: high relevance: high - claim_id: semantic_tokenizers_and_acoustic_tokenizers_represent_complementary_capability role: supports claim: "Semantic tokenizers and acoustic tokenizers represent complementary capability profiles — strong semantic content fidelity versus strong acoustic reconstruction fidelity — and no single tokenizer type dominates both dimensions." source: "§3.1, §F.2, Table 6" evidence: "The survey distinguishes three objective families: semantic understanding tokenizers (HuBERT, WavLM, w2v-BERT) produce tokens that align well with text but sacrifice acoustic detail; acoustic generation tokenizers (neural codecs: SoundStream, EnCodec, DAC) preserve high-fidelity audio characteristics but often produce semantically inconsistent outputs; mixed-objective tokenizers (SpeechTokenizer, Mimi) distill semantic representations into the first quantizer layer of a codec architecture to capture both." confidence: high relevance: high - claim_id: post_alignment_of_speechlms_via_preference_optimization_addresses role: supports claim: "Post-alignment of SpeechLMs via preference optimization addresses qualitatively different failure modes (semantic inconsistency, token distribution mismatch) than post-alignment of text LLMs." source: "§4.2.3" evidence: "Training is covered across three stages: large-scale pretraining (with or without TextLM initialization), instruction tuning (cross-modal fine-tuning, chain-of-modality tuning), and post-alignment (DPO for semantic consistency in AlignSLM; token distribution alignment in SpeechAlign)." confidence: high relevance: high - claim_id: full_duplex_spoken_interaction_simultaneous_bidirectional_speech_with role: complicates claim: "Full-duplex spoken interaction — simultaneous bidirectional speech with interruption support — requires joint modeling of both speaker streams and remains an open research challenge." source: "§4.3" evidence: "Full-duplex systems that support simultaneous bidirectional communication — including user interruption and simultaneous response — are surveyed as an emerging frontier, with dGSLM, Parrot, Moshi, and LSLM as representative systems." confidence: high relevance: medium limitations: - "Coverage necessarily lags the field: systems published after mid-2024 receive limited treatment, and the overall corpus skews heavily toward English and Mandarin. The safety section identifies toxicity and speaker privacy risks but does not analyze them quantitatively. End-to-end training that backpropagates gradients from vocoder output to tokenizer input is flagged as a potentially high-value research direction but remains unexplored. The question of whether incorporating text modality fundamentally benefits or constrains speech intelligence is left open." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.acl-long.912" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [SCA, TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: autoregressive_streaming_speech_decoders_in_modular_spoken_conversational role: supports claim: "Autoregressive streaming speech decoders in modular spoken conversational agents substantially improve naturalness over non-autoregressive alternatives with comparable latency." source: "§5.1, Table 1" evidence: "Speech naturalness (UTMOS): LLaMA-Omni2 models score 4.19–4.22 across sizes, substantially above GLM-4-Voice (3.48) and LLaMA-Omni (3.67)." confidence: high relevance: high - claim_id: a_gate_fusion_mechanism_that_adaptively_blends_llm role: supports claim: "A gate fusion mechanism that adaptively blends LLM hidden states with text token embeddings as input to the TTS language model improves both instruction following quality and text-speech consistency over simpler additive fusion." source: "§5.2, Table 2" evidence: "M_TTS receives a gate-fused combination of LLM hidden states and text token embeddings: a learned sigmoid gate adaptively blends the two representations element-wise, allowing the model to exploit both broad contextual information from hidden states and precise textual content from embeddings." confidence: high relevance: high - claim_id: streaming_tts_pretraining_on_speech_dialogue_data_is role: complicates claim: "Streaming TTS pretraining on speech dialogue data is critical for quality: initializing from a text-only pretrained model degrades performance substantially, and training from scratch fails to converge." source: "§5.2, Table 3" evidence: "Ablation studies (Tables 2–5) confirm the importance of the gate fusion module, streaming TTS pretraining over offline or scratch initialization, and that multi-turn data consistently improves over single-turn data of the same size." confidence: high relevance: medium - claim_id: multi_turn_dialogue_training_data_consistently_outperforms_single role: supports claim: "Multi-turn dialogue training data consistently outperforms single-turn data of the same total size for modular speech language models across spoken QA and instruction-following benchmarks." source: "§5.3, Table 5" evidence: "Ablation studies (Tables 2–5) confirm the importance of the gate fusion module, streaming TTS pretraining over offline or scratch initialization, and that multi-turn data consistently improves over single-turn data of the same size." confidence: high relevance: high - claim_id: the_s2t_to_s2s_accuracy_gap_in_spoken role: supports claim: "The S2T-to-S2S accuracy gap in spoken question answering is a meaningful indicator of speech generation quality, and autoregressive TTS decoders reduce this gap relative to non-autoregressive alternatives." source: "§5.1, Table 1" evidence: "Spoken QA accuracy (S2S setting): LLaMA-Omni2-7B achieves 60.7% on Llama Questions and 31.3% on Web Questions, versus GLM-4-Voice's 50.7%/15.9% and LLaMA-Omni's 49.0%/23.7%." confidence: high relevance: high limitations: - "The model generates speech in a single fixed output style; it cannot modulate emotion, speaking rate, or dialect in response to the content or paralinguistic cues of the input speech. All evaluation is in English. The output voice is fixed during training (a single uniform voice for all responses), which limits expressiveness and speaker diversity. The system is inherently a response-after-input architecture and does not support full-duplex conversation. Latency, while adequate for real-time interaction, leaves room for further reduction via engineering optimization. Whether the gate fusion approach generalizes to multilingual settings or to more expressive speech styles is not explored." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.acl-long.937" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [codec] architecture: [VAE] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: domain_adaptive_partitioning_within_a_shared_codebook_resolves role: supports claim: "Domain-adaptive partitioning within a shared codebook resolves the performance degradation that plagues unified single-codebook codecs on mixed multi-domain audio." source: "§3.2, §5.3, Tables 2, 4, 6" evidence: "Removing the partitioned codebook raises Mel Distance from 0.344 to 0.487 on speech and from 0.396 to 0.506 on music in ablation; the full UniCodec outperforms WavTokenizer (unified) on all three domains in both objective and MUSHRA evaluation." confidence: high relevance: high - claim_id: self_supervised_mask_prediction_objectives_integrated_directly_into role: supports claim: "Self-supervised mask prediction objectives integrated directly into codec training improve semantic representation without requiring external pretrained SSL encoders." source: "§3.4, §5.2, Table 5" evidence: "The semantic training stage improves RAVDESS classification accuracy from 36.81 to 40.28 and Audio-MNIST from 69.84 to 70.94 compared to the codec without this stage; no auxiliary SSL model is used." confidence: high relevance: high - claim_id: joint_training_of_reconstruction_and_self_supervised_semantic role: complicates claim: "Joint training of reconstruction and self-supervised semantic objectives from scratch degrades single-codebook codec performance; sequential staging is required." source: "§3.4" evidence: "Preliminary experiments show that training reconstruction, mask prediction, and contrastive loss simultaneously is infeasible; the two-stage approach (acoustic training first, then semantic stage) is necessary for stable convergence." confidence: high relevance: high - claim_id: large_scale_diverse_audio_training_data_introduces_noise role: complicates claim: "Large-scale diverse audio training data introduces noise that degrades codec reconstruction quality, requiring targeted fine-tuning on curated high-quality subsets." source: "§4, §5.3, Appendix C, Table 6" evidence: "Training on 80K hours of mixed data without the fine-tuning stage raises Mel Distance by 0.103 on speech (0.448 vs 0.344) relative to the fine-tuned model; high-quality fine-tuning on LibriTTS clean, VCTK, and LJSpeech recovers most of the degradation." confidence: high relevance: high - claim_id: single_codebook_codecs_at_75_tokens_per_second role: supports claim: "Single-codebook codecs at 75 tokens per second can achieve acoustic reconstruction quality competitive with multi-layer RVQ codecs operating at 600 tokens per second when combined with domain-adaptive design." source: "§5.1, Table 3" evidence: "UniCodec PESQ 3.03 and STOI 0.949 on LibriTTS test-clean exceeds Encodec (2.72, 0.939) and SpeechTokenizer (2.61, 0.917) at 600 TPS, with eight times fewer tokens per second." confidence: high relevance: high limitations: - "Training is sensitive to noisy or low-quality input: large-scale noisy data alone degrades reconstruction quality, and the fine-tuning stage is essential. This dependency on curated data limits scalability." - "The single-codebook codec struggles to balance both acoustic reconstruction fidelity and semantic density across diverse domains simultaneously, as these objectives conflict. Streaming performance degrades relative to non-streaming inference. The paper does not demonstrate integration with an actual audio language model, leaving downstream generation quality unvalidated." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2025.acl-short.81" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, evaluation] architecture: [autoregressive-LM, transformer-enc-dec] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: long_form_training_audio_10_20_seconds_per role: supports claim: "Long-form training audio (10–20 seconds per segment) with explicit speaker identities improves zero-shot TTS quality for low-resource tonal languages compared to training on short-segment corpora." source: "§3.1, §4" evidence: "Three zero-shot TTS models are adapted to Vietnamese by fine-tuning on the 1,494-hour augmented set (940h long-form + 554h short-clip augmentation): - VALL-E: An autoregressive codec language model operating on EnCodec discrete tokens at the phoneme level." confidence: high relevance: medium - claim_id: multilingual_voice_cloning_models_such_as_xtts_v2 role: supports claim: "Multilingual voice-cloning models such as XTTS-v2 exhibit architectural failure modes on short input sequences that are not corrected by data augmentation with short clips." source: "§4" evidence: "Manual inspection reveals that XTTS-v2-based models generate rambling or redundant speech on short inputs, which the authors attribute to an architectural issue in XTTS-v2 rather than a data problem." confidence: high relevance: medium - claim_id: autoregressive_codec_language_models_vall_e_voicecraft_generalize role: supports claim: "Autoregressive codec language models (VALL-E, VoiceCraft) generalize better than Tortoise-based models to short-sentence inputs in low-resource language fine-tuning." source: "§4, Table 2" evidence: "Three zero-shot TTS models are adapted to Vietnamese by fine-tuning on the 1,494-hour augmented set (940h long-form + 554h short-clip augmentation): - VALL-E: An autoregressive codec language model operating on EnCodec discrete tokens at the phoneme level." confidence: high relevance: high - claim_id: a_dataset_curation_pipeline_based_on_dual_asr role: supports claim: "A dataset curation pipeline based on dual-ASR agreement filtering produces higher-quality transcriptions for audiobook audio than single-model transcription alone, enabling more reliable TTS training." source: "§2.1" evidence: "The authors construct PhoAudiobook via an automated pipeline applied to ~23K hours of Vietnamese audiobook audio scraped from sachnoiviet.net." confidence: high relevance: medium limitations: - "The paper does not evaluate code-switching scenarios (mixed Vietnamese-English input), which is relevant in practice. The dataset is audiobook domain only, so speaking style coverage is narrower than general-purpose datasets. All models are fine-tuned rather than trained from scratch, which means performance is bounded by the pre-trained model's multilingual capacity. The architecture issue observed with XTTS-v2 on short sentences is identified but not resolved. The dataset is released for non-commercial use only, which limits industrial adoption. It is also unclear how the system handles tonal phonology beyond phonemizer outputs, and no ablation on augmented data proportion is presented." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.findings-acl.101" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, SCA] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: decomposing_conversational_speech_synthesis_into_sequential_emotion_understandin role: supports claim: "Decomposing conversational speech synthesis into sequential emotion-understanding and speech-generation stages improves expressiveness over direct speech token prediction." source: "§6.5, Table 2" evidence: "Chain-Talker (DMOS-E 4.239) outperforms GPT-Talker's direct token prediction (3.913) and GPT-Talker_c, which adds emotion understanding without fully chaining it through caption prediction (4.102); the ablation w/o captions (4.084) confirms the caption conditioning is the key factor." confidence: high relevance: high - claim_id: natural_language_emotion_captions_are_more_effective_conditioning role: supports claim: "Natural language emotion captions are more effective conditioning signals for empathetic speech synthesis than discrete emotion category labels." source: "§6.4, §6.5, Tables 1 and 2" evidence: "Chain-Talker using empathetic captions (DMOS-E 4.239) outperforms Chain-Talker_e using emotion labels (4.127) and Chain-Talker_s using style labels (4.015); DMOS-C scores also show captions exceed ground-truth label quality (4.462 vs 4.327)." confidence: high relevance: medium - claim_id: supervised_asr_derived_semantic_speech_tokens_provide_more role: supports claim: "Supervised ASR-derived semantic speech tokens provide more interpretable and expressive conversational speech generation than unsupervised tokens that mix semantic and acoustic content." source: "§2.3, §6.5, Table 2" evidence: "The paper argues that HuBERT tokens in GPT-Talker contain entangled acoustic information that limits emotional comprehension; Chain-Talker's ASR-VQ tokens improve DDTW from 44.625 to 38.784 and ACC_m from 0.562 to 0.612." confidence: high relevance: high - claim_id: llm_driven_automatic_speech_emotion_captioning_with_multi role: supports claim: "LLM-driven automatic speech emotion captioning with multi-level attribute extraction produces higher-quality annotations than single-modality or keyword-based description approaches." source: "§6.4, Table 1" evidence: "CSS-EmCap (DMOS-C 4.462, SIM_G 0.694, DIS-2 0.296) outperforms Qwen2-Audio (4.212, 0.534, 0.174) and SECap (4.268, 0.617, 0.186); ablations confirm that removing either sentence-level style factors or dialogue-level emotion significantly degrades both quality and diversity." confidence: high relevance: medium - claim_id: autoregressive_chain_modeling_for_empathetic_conversational_speech_synthesis role: complicates claim: "Autoregressive chain modeling for empathetic conversational speech synthesis introduces latency that does not yet support real-time interaction requirements." source: "§Limitations" evidence: "Chain-Talker generates average empathetic responses of 2.5 seconds duration on an RTX 4080 with 32 GB VRAM; authors describe this as a gap relative to real-time dialogue and identify streaming inference as a necessary future direction." confidence: high relevance: high limitations: - "Chain-Talker is fine-tuned on only 384 hours of dialogue data drawn from daily conversational domains and predominantly young adult speakers. The authors explicitly note it may not capture the speaking styles of children or the elderly, and domain generalisation to non-conversational or non-English settings is untested." - "Inference latency is a practical bottleneck: the three-stage autoregressive pipeline produces responses averaging 2.5 seconds, which is insufficient for low-latency real-time dialogue. Streaming inference is identified as future work but not yet implemented." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.findings-acl.1051" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [unified_speech_text_language_model] claims: - claim_id: decoupling_speech_synthesis_from_llm_text_generation_via role: supports claim: "Decoupling speech synthesis from LLM text generation via a lightweight autoregressive module can preserve the base LLM's reasoning quality while achieving competitive streaming latency." source: "§6.4, Table 1" evidence: "LLMVoX is a plug-and-play autoregressive TTS frontend that converts streaming LLM text output into speech tokens in real time, decoupled from the LLM." confidence: high relevance: high - claim_id: end_to_end_speech_enabled_llms_that_fine role: supports claim: "End-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen." source: "§6.4, Table 1" evidence: "The paper reports evidence that end-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen." confidence: high relevance: medium - claim_id: a_single_layer_rvq_codec_is_sufficient_for role: supports claim: "A single-layer RVQ codec is sufficient for low-latency autoregressive TTS generation when paired with a compact decoder transformer, avoiding the complexity of multi-codebook prediction." source: "§3.1" evidence: "WavTokenizer (single-layer RVQ, 4096-entry codebook) encodes speech at 40–75 tokens/second at 24 kHz." confidence: high relevance: high - claim_id: streaming_tts_quality_improves_with_larger_decode_chunk role: supports claim: "Streaming TTS quality improves with larger decode chunk sizes, with WER and UTMOS gains achievable without substantially increasing end-to-end latency." source: "§6.4, Figure 6" evidence: "On the joint General+Knowledge QA streaming benchmark (Table 1), Whisper+LLaMA3.1-8B+LLMVoX achieves: - GPT-4o General QA score: 6.14 (highest among streaming models; Freeze-Omni: 3.48; Moshi: 2.71) - GPT-4o Knowledge QA score: 7.62 (highest; vs. 4.98 for Freeze-Omni) - WER: 3.70% (lowest; Freeze-Omni: 14.05%; LLaMA-Omni: 9.18%; CosyVoice2: 17.2%) - UTMOS: 4.05 (competitive; Freeze-Omni: 4.38; MiniCPM-o 2.6: 3.87) - End-to-end latency: 475ms (vs. Whisper+XTTS: 4200ms; LLaMA-Omni: 220ms) For Arabic (Table 3), LLMVoX streaming CER of 8.2% competes with non-streaming XTTS (CER 1.7%) while operating ~10× faster." confidence: high relevance: medium - claim_id: language_adaptation_of_a_codec_based_tts_module role: supports claim: "Language adaptation of a codec-based TTS module can be achieved by replacing training data alone, without architectural changes or explicit G2P conversion for the new language." source: "§6.5, Table 3" evidence: "No architectural changes needed for new languages — only data." confidence: high relevance: high limitations: - "LLMVoX is single-speaker — no voice cloning or speaker reference support. The Arabic model was trained on XTTS-synthesized data, so XTTS acts as an upper bound (CER 1.7% vs. LLMVoX 8.2%). The streaming pipeline does not yet extend to the ASR front-end. Latency with 70B LLMs exceeds 1.9s, making real-time use marginal. The quality improvement from larger chunk sizes (UTMOS 3.75→4.41) suggests that the 475ms latency figure is somewhat optimistic for maximum quality operation." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.findings-ijcnlp.49" published_date: "2025-07-27" entry_date: "2026-07-25" year: 2025 venue: "ACL" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: a_full_duplex_speech_language_model_can_perform role: supports claim: "A full-duplex speech language model can perform structured dialogue state tracking when DST is executed entirely through the model's text stream using explicit delimiter tokens." source: "§3, §5.3, Table 1" evidence: "J-Moshi-ext fine-tuned on synthesized JMultiWOZ data achieves JGA 65.69 and Slot F1 97.78 on the JMultiWOZ test set, with both metrics improving monotonically as training data volume increases from 25% to 100%." confidence: high relevance: high - claim_id: extending_full_duplex_speech_models_to_task_oriented role: complicates claim: "Extending full-duplex speech models to task-oriented dialogue does not close the gap with text-based systems on response generation quality." source: "§5.3, Table 1" evidence: "The proposed method reaches BLEU 0.129 and BERTScore 0.689, against text-based upper bounds of BLEU 0.364 and BERTScore 0.830 (T5-large), with the gap attributed to insufficient training data and to the difficulty of aligning text and audio modalities during response generation." confidence: high relevance: medium - claim_id: tts_data_quality_limitations_compound_into_downstream_training role: complicates claim: "TTS data quality limitations compound into downstream training noise when synthesized speech is used to construct full-duplex dialogue training corpora." source: "§5.3" evidence: "The multilingual OuteTTS model used for Japanese speech synthesis generated speech with reduced clarity, causing recognition errors during time-annotated tokenization that hindered the model's learning of coherent text-audio token sequences for response generation." confidence: high relevance: medium - claim_id: dst_timing_estimation_is_feasible_within_a_full role: supports claim: "DST timing estimation is feasible within a full-duplex spoken dialogue model but introduces substantial prediction error tied to the model's speech perception accuracy." source: "§5.1, §5.3" evidence: "DST initiation timing (defined as the time of `` token generation) achieved a MAE of 4.9 seconds, with discrepancies attributed to the difficulty of accurately predicting user utterance boundaries from the model's audio token representations alone." confidence: high relevance: medium limitations: - "All experiments are conducted in Japanese using J-Moshi, which was trained on less Japanese data than the original English Moshi. The authors explicitly note that performance on English equivalents (MultiWOZ, SpokenWOZ) could be substantially higher, and cross-lingual generalisability is untested." - "Evaluation is fully automatic (JGA, Slot F1, BLEU, BERTScore); no subjective listening tests or user studies are reported, leaving response naturalness, latency under real deployment conditions, and the effect of DST pauses on conversational flow unassessed. The experiment covers only the domains included in JMultiWOZ (travel planning), and extension to diverse or open-domain settings is planned but not yet evaluated. Future work includes experiments with the original Moshi on SpokenWOZ and DSTC11 datasets and application to additional dialogue domains." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2025.conll-1.9" published_date: "2025-07-31" entry_date: "2026-07-25" year: 2025 venue: "CoNLL" task: [TTS, evaluation] architecture: [] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: naturalness_ratings_do_not_fully_reflect_whether_tts role: complicates claim: "Naturalness ratings do not fully reflect whether TTS prosody is linguistically appropriate." source: "§5.3, Figure 3" evidence: "Syntactic Sensitivity Score precision and recall diverge from the MOS ranking across Tacotron2, SpeechT5, and Parler-TTS; specifically, recall does not follow MOS order, revealing that missed boundary errors have less impact on perceived naturalness than spurious boundary errors — a dimension MOS does not capture independently." confidence: high relevance: low - claim_id: punctuation_marks_are_the_dominant_surface_cue_for role: supports claim: "Punctuation marks are the dominant surface cue for intonational boundary placement in TTS systems, overriding syntactic and semantic information in ambiguous contexts." source: "§4.4, §5.3" evidence: "All three systems place boundaries reliably only when commas are present; without comma cues in garden path and attachment ambiguity sentences, systems default to the statistically most frequent pattern regardless of the intended structure, and semantic bias in prepositional phrase attachment has no measurable effect on boundary placement." confidence: high relevance: low - claim_id: transformer_based_tts_architectures_encode_partial_clause_boundary role: supports claim: "Transformer-based TTS architectures encode partial clause boundary information for prosody that LSTM-based systems do not." source: "§6.1, Figure 4" evidence: "LASSO regression selects `is_clause_boundary` as a significant predictor of pause duration for SpeechT5 and Parler-TTS but not for Tacotron2, which relies more on token position and length features." confidence: high relevance: low - claim_id: increasing_tts_training_exposure_to_syntactically_diverse_speech role: refines claim: "Increasing TTS training exposure to syntactically diverse speech can improve prosodic alignment with syntactic structure for structural cues, but not for semantic disambiguation cues." source: "§7.3, Figure 5" evidence: "Fine-tuning Parler-TTS on Jenny corpus prepositional phrase data produces more differentiated pause patterns correlated with syntactic function of ambiguous words; fine-tuning on synthetic semantic-bias data increases pause duration generally but does not teach the model to distinguish high from low attachment based on semantic context." confidence: high relevance: low limitations: - "The study measures intonational boundaries through duration proxies only (pause insertion and pre-boundary lengthening), omitting pitch contour and intensity changes that also contribute to boundary perception. The ToBI framework identifies multiple boundary strength levels that this methodology cannot distinguish." - "Evaluation is limited to English and to three specific TTS systems trained on audiobook speech, which under-represents prosodically disambiguating speech patterns relative to conversational speech." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2025.ccl-1.80" published_date: "2025-08-01" entry_date: "2026-07-25" year: 2025 venue: "workshop" task: [TTS] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: projecting_phoneme_representations_from_two_languages_into_a role: supports claim: "Projecting phoneme representations from two languages into a shared latent space reduces cross-lingual phoneme confusion and improves naturalness in code-switched speech synthesis." source: "§3.2, Table 2" evidence: "Cross-lingual phoneme latent space." confidence: high relevance: medium - claim_id: per_token_language_id_conditioning_helps_a_multilingual role: supports claim: "Per-token language ID conditioning helps a multilingual codec LM distinguish phonetic characteristics across languages in code-switched synthesis, though its impact is smaller than that of shared phoneme representations." source: "§3.3, Table 4" evidence: "The resulting latent representations H for Lao and English phonemes are fused into a shared space and concatenated with the phoneme sequence fed to the codec LM." confidence: high relevance: high - claim_id: codec_language_model_architectures_outperform_vae_based_seq2seq role: supports claim: "Codec language model architectures outperform VAE-based seq2seq systems for code-switched TTS when trained on monolingual data only." source: "§4.3.2, Table 3" evidence: "VITS baselines (Table 3) score substantially lower (CMOS –0.42 to –1.50), confirming that codec LM-based approaches outperform seq2seq VAE architectures for code-switching." confidence: high relevance: high - claim_id: code_switched_tts_systems_can_be_trained_effectively role: supports claim: "Code-switched TTS systems can be trained effectively from monolingual corpora alone, without requiring real bilingual or code-switched training audio." source: "§3.1, §4.1" evidence: "No real code-switched audio is used — the model learns cross-lingual synthesis entirely from monolingual sources." confidence: high relevance: medium limitations: - "- Proprietary Lao dataset is not publicly available, limiting reproducibility. - Evaluation uses a small number of listeners (10 Lao, 10 English), raising statistical concerns. - RMSE is used as the sole objective metric; no WER, CER, or SPK-SIM reported. - The approach does not generalize beyond Lao-English without new proprietary data per language. - No streaming or real-time inference analysis. - How the model handles intra-word code-switching (vs. inter-sentence) is not evaluated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2025.icnlsp-1.34" published_date: "2025-08-01" entry_date: "2026-07-25" year: 2025 venue: "workshop" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: preference_alignment_via_dpo_can_match_the_perceptual role: supports claim: "Preference alignment via DPO can match the perceptual quality of supervised fine-tuning on human-labeled TTS data when preference pairs are constructed automatically from model-generated samples." source: "§3.5, Table 1, Table 2, Table 5" evidence: "On VCTK, DPO fine-tuning with LLM-augmented texts (V4) achieves WER 0.013, SS 0.482, and UTMOS 4.108 from the base xTTSv2 checkpoint, outperforming classical fine-tuning on human-labeled data (V1: WER 0.041, SS 0.500, UTMOS 3.685) on intelligibility and naturalness. CMOS evaluation confirms no statistically significant perceptual difference (all CM p > 0.30)." confidence: high relevance: medium - claim_id: the_initialisation_point_before_dpo_fine_tuning_critically role: complicates claim: "The initialisation point before DPO fine-tuning critically determines outcome quality, with checkpoint diversity being a more important factor than label quality of the preceding supervised stage." source: "§3.6, Table 1, Table 2" evidence: "DPO applied from the ASR-fine-tuned checkpoint (L5/V5) consistently underperforms DPO from the base model (L3/V3), with V5 reaching WER 0.273 versus V3's WER 0.014. The explanation offered is that standard fine-tuning narrows generation diversity, leaving insufficient variation for DPO preference signal." confidence: high relevance: medium - claim_id: augmenting_unlabeled_tts_training_data_with_llm_generated role: supports claim: "Augmenting unlabeled TTS training data with LLM-generated texts improves synthesis naturalness when the dataset's own texts are structurally mismatched with natural language." source: "§3.5, Table 1, Table 2" evidence: "On VCTK, which contains newspaper sentences well-aligned with LLM output, adding 15,000 LLM-generated texts (V4 vs. V3) raises UTMOS from 4.009 to 4.108. On LJSpeech, which contains audiobook fragments (partial sentences), the improvement is smaller and WER increases, attributed to domain mismatch." confidence: high relevance: medium - claim_id: speaker_similarity_is_the_most_difficult_quality_dimension role: complicates claim: "Speaker similarity is the most difficult quality dimension to preserve when fine-tuning TTS on ASR-transcribed or automatically constructed data, even when naturalness and intelligibility are maintained." source: "§3.5, §3.7, Table 4, Table 5" evidence: "Human-labeled fine-tuning (L1/V1) achieves the highest SS scores across both datasets (SS 0.481 on LJSpeech, 0.500 on VCTK). DPO from the base model recovers intelligibility and naturalness but trails on SS. Statistical significance testing shows SS degradation is significant for checkpoint-initialised DPO (Exp 6 vs. Exp 1/2, p < 0.001), while naturalness differences are not significant." confidence: high relevance: medium limitations: - "All experiments use xTTSv2 as the sole base model; it is unknown whether the same training recipe and checkpoint-initialisation finding generalise to other LM-based TTS architectures. Evaluation covers English only despite xTTSv2 being multilingual." - "The comparison between fine-tuning from scratch on ASR-labeled versus human-labeled data is not performed; the human-labeled baseline starts from the same pre-trained xTTSv2 checkpoint, not a randomly initialised model, which limits conclusions about the true cost of label-free training. CMOS subgroup sample sizes (as few as 5 per accent in some comparisons) are too small for reliable accent-level conclusions. The bias implications of using ASR and LLM components for underrepresented speaker groups are acknowledged but not empirically quantified." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.22746" published_date: "2025-08-01" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [hybrid] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: composing_autoregressive_generation_across_fixed_length_token_blocks role: supports claim: "Composing autoregressive generation across fixed-length token blocks with parallel flow-matching denoising within each block can simultaneously provide KV-cache efficiency and bidirectional contextual refinement." source: "§3.1, Table 3" evidence: "Within each chunk, flow-matching denoising runs in parallel with bidirectional self-attention, exploiting full chunk context to refine all 25 tokens simultaneously." confidence: high relevance: high - claim_id: neural_codecs_using_finite_scalar_quantisation_can_preserve role: supports claim: "Neural codecs using finite scalar quantisation can preserve speaker similarity and intelligibility at frame rates (12.5 Hz) where STFT-based vocoders suffer significant quality degradation." source: "§4.3.3, Table 4" evidence: "This demonstrates that FSQ-based discrete tokens can match or exceed STFT-based reconstruction at rates where the stationarity assumption underlying STFT begins to break down." confidence: high relevance: high - claim_id: continuous_denoising_models_can_implicitly_classify_discrete_token role: supports claim: "Continuous denoising models can implicitly classify discrete token targets through appropriate embedding design, without requiring a separate discrete language model head." source: "§3.1" evidence: "This unification of discrete and continuous feature modeling avoids the instability of low-frame-rate mel spectrograms and simplifies downstream training without requiring a separate discrete language model head." confidence: high relevance: high - claim_id: reducing_the_token_frame_rate_is_a_more role: supports claim: "Reducing the token frame rate is a more tractable path to low-latency hybrid AR-diffusion TTS than increasing diffusion step efficiency alone, given the quadratic scaling of self-attention with sequence length." source: "§3.1, §4.3.2" evidence: "For diversity, Dragon-FM variants achieve Fréchet Audio Distance (FAD) scores of 1.9–2.6 across chunk size and flow-matching step configurations (Table 2), with shorter chunks (1s) and more FM steps generally yielding lower FAD." confidence: high relevance: high limitations: - "No MOS or SMOS listening test results are reported. All quality comparisons use SPK-SIM, WER, and FAD on an internal podcast dataset. The absence of subjective evaluation and fair comparison against published baselines (VALL-E 2, E2 TTS, NaturalSpeech 3) on a public benchmark makes it impossible to independently verify naturalness claims." - "The model is trained and evaluated on English podcast data only. Generalisation to other languages, controlled studio-quality TTS, and expressive speech domains is untested. The podcast use-case naturally emphasises diversity and disfluency tolerance over precise prosody control, so the evaluation protocol may not transfer to production TTS settings." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2504.10352" published_date: "2025-08-05" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, alignment_constrained_autoregression] claims: - claim_id: pseudo_autoregressive_generation_which_commits_spans_left_to role: supports claim: "Pseudo-autoregressive generation, which commits spans left-to-right within a bidirectional masked transformer, achieves constant inference steps regardless of target speech duration while maintaining temporal coherence." source: "§3, §5.4" evidence: "This creates span-level causal ordering (earlier spans committed before later ones) with O(1) inference steps regardless of target duration, since the number of steps equals 1/r' (a constant ratio of the target length)." confidence: high relevance: high - claim_id: a_model_trained_on_580_hours_of_english role: supports claim: "A model trained on 580 hours of English speech can match or exceed the intelligibility of NAR flow-matching systems trained on 100,000+ hours when temporal ordering is explicitly enforced during generation." source: "§5.3, Table 1" evidence: "PALLE two-stage achieves WER-H 2.83 and SIM-o 0.716 on the cross-sentence task of LibriSpeech test-clean — the best WER among all systems evaluated, outperforming F5-TTS (WER-H 2.77, SIM-o 0.705), E2-TTS (WER-H 2.92, SIM-o 0.756), and MaskGCT (WER-H 4.22, SIM-o 0.756), all of which are trained on 100k+ hours of data." confidence: high relevance: medium - claim_id: confidence_guided_iterative_nar_refinement_of_an_initial role: supports claim: "Confidence-guided iterative NAR refinement of an initial PAR generation substantially reduces word error rate with only a small number of additional inference steps." source: "§5.5, Figure 4" evidence: "This creates span-level causal ordering (earlier spans committed before later ones) with O(1) inference steps regardless of target duration, since the number of steps equals 1/r' (a constant ratio of the target length)." confidence: high relevance: medium - claim_id: temporally_unordered_nar_generation_produces_higher_alignment_errors role: supports claim: "Temporally unordered NAR generation produces higher alignment errors than span-level causal generation across both continuation and cross-sentence evaluation tasks." source: "§5.4, Table 3" evidence: "This creates span-level causal ordering (earlier spans committed before later ones) with O(1) inference steps regardless of target duration, since the number of steps equals 1/r' (a constant ratio of the target length)." confidence: high relevance: medium - claim_id: separate_model_capacity_for_each_generation_stage_is role: supports claim: "Separate model capacity for each generation stage is necessary; unifying PAR and NAR refinement into a single multitask model degrades cross-sentence intelligibility by approximately 20%." source: "§5.5" evidence: "PALLE two-stage achieves WER-H 2.83 and SIM-o 0.716 on the cross-sentence task of LibriSpeech test-clean — the best WER among all systems evaluated, outperforming F5-TTS (WER-H 2.77, SIM-o 0.705), E2-TTS (WER-H 2.92, SIM-o 0.756), and MaskGCT (WER-H 4.22, SIM-o 0.756), all of which are trained on 100k+ hours of data." confidence: high relevance: medium limitations: - "PALLE is evaluated only on English (LibriTTS). The 100-step inference (with 7 refinement steps) may still be too slow for the most latency-sensitive streaming applications despite the 10x speedup. Duration estimation for the cross-sentence task relies on a simple linear heuristic; errors in duration estimation lead to modest quality degradation (WER-H 2.83 vs. 2.62 with GT duration). The shared architecture between stage one and stage two (joint multitask fine-tuning) causes stage two loss to degrade stage one performance, suggesting that separate model capacity is required." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.14049" published_date: "2025-08-05" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: large_multilingual_tts_systems_built_on_semantic_token role: supports claim: "Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models." source: "§2.1, §5.1" evidence: "For semantic representations, the system uses a pretrained XLS-R wav2vec2.0 model (1B parameters, trained on 436K hours across 128 languages) with k-means clustering to produce a 10,000-token discrete phoneme dictionary at approximately 25 tokens per second." confidence: high relevance: high - claim_id: decoupling_the_text_to_semantic_and_semantic_to role: supports claim: "Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining." source: "§2, §4.1" evidence: "MahaTTS-v2 follows a two-stage pipeline: M1 converts text to semantic tokens, and M2 converts semantic tokens to an audio waveform — the two stages are trained independently with no joint optimisation required." confidence: high relevance: medium - claim_id: flow_matching_is_a_viable_replacement_for_diffusion role: supports claim: "Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity." source: "§2.3, §5.1" evidence: "M2 is a 300M parameter conditional flow matching model (Matcha-TTS-inspired) that maps semantic tokens to mel spectrograms." confidence: high relevance: medium - claim_id: intelligibility_in_low_resource_languages_with_limited_training role: complicates claim: "Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system." source: "§5.1, Table 2" evidence: "Performance is uneven: results on Assamese (42%) and Odia (73%) are substantially weaker than the leading baselines, suggesting that low-resource languages with limited training hours remain difficult even within this framework." confidence: high relevance: medium limitations: - "The evaluation relies exclusively on WER over 10 sentences per language with no MOS, SMOS, or naturalness scores. This makes it impossible to assess audio quality, expressiveness, or speaker similarity relative to baselines — core dimensions for a TTS system." - "English dominates the training set at 58%, which may explain strong English results but raises questions about whether true cross-lingual transfer or data dominance is responsible. The system lacks prosody and pace control conditioning in M1, which the authors flag as future work. Zero-shot speaker fidelity for M2 is acknowledged as limited compared to infilling-based approaches like Seamless. Fine-tuning introduces hallucination that requires careful intervention (freezing classification heads only), suggesting the LM component is sensitive to distribution shift. Languages with fewer training hours (Assamese: 48h, Dogri: 8h, Rajasthani: 20h) show substantially weaker results." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.04141" published_date: "2025-08-06" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: generating_semantic_and_acoustic_tokens_simultaneously_in_a role: supports claim: "Generating semantic and acoustic tokens simultaneously in a single autoregressive forward pass, rather than cascading semantic prediction before acoustic prediction, reduces word error rate and improves naturalness in zero-shot TTS." source: "§V.A, Table I" evidence: "Crucially, both token types are generated simultaneously in a single forward pass, maintaining temporal alignment while preserving content independence." confidence: high relevance: high - claim_id: combining_specialist_ssl_models_for_distinct_speech_attributes role: supports claim: "Combining specialist SSL models for distinct speech attributes (semantic content, acoustic texture, speaker identity) as frozen feature extractors enables more effective token-level disentanglement than using a single encoder for all attributes." source: "§III.A, Tables III–IV" evidence: "A custom tokenizer combines three frozen SSL models: Wav2Vec 2.0 (semantic features), BEATs (acoustic features), and CAM++ / Campplus (speaker features)." confidence: high relevance: high - claim_id: a_hybrid_ar_nar_design_that_enforces_independence role: supports claim: "A hybrid AR+NAR design that enforces independence at the coarse token level and interdependence at the fine-grained level distributes modeling complexity more effectively than an all-AR or all-NAR approach." source: "§V.B, Tables III–IV" evidence: "Crucially, both token types are generated simultaneously in a single forward pass, maintaining temporal alignment while preserving content independence." confidence: high relevance: high - claim_id: parallel_semantic_acoustic_modeling_improves_naturalness_and_intelligibility role: supports claim: "Parallel semantic-acoustic modeling improves naturalness and intelligibility without fully closing the speaker similarity gap relative to systems with dedicated speaker embedding refinement." source: "§V.A, Table I" evidence: "On speaker similarity (SMOS), Parallel GPT (4.08/3.92) is closely competitive with MaskGCT (4.05/3.91) but trails CosyVoice (4.23/4.11) — a gap the authors attribute to CosyVoice's specialized speaker embedding refinement." confidence: high relevance: medium limitations: - "Speaker similarity lags behind CosyVoice (SMOS gap ~0.15–0.2 on English), suggesting the parallel architecture does not yet fully leverage speaker conditioning. UTMOS scores, while competitive, do not reach ground-truth levels. Model size is not reported, making compute comparisons difficult. The internal Chinese dataset and preprocessing pipeline (Emilia + NCSSD) are not publicly released, limiting reproducibility on that front. Extending the framework to prosody control, emotion conditioning, or cross-lingual voice conversion is not explored. The subjective decoupling evaluation (Section V.C) relies on 90% evaluator agreement rather than a standardized metric, leaving quantitative disentanglement assessment as an open question." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.04585" published_date: "2025-08-06" entry_date: "2026-07-25" year: 2025 venue: "ACM MM" task: [TTS, SCA] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: matching_the_token_rates_of_speech_and_facial role: supports claim: "Matching the token rates of speech and facial landmark codecs enables frame-level synchronisation between synthesised speech and talking-face animations without post-hoc alignment." source: "§4.1.3, §4.2" evidence: "LmkCodec operates at 25 Hz with a codebook of size 1000, matching the speech tokenizer's rate and enabling frame-level alignment." confidence: high relevance: high - claim_id: llm_based_joint_prediction_of_interleaved_speech_and role: supports claim: "LLM-based joint prediction of interleaved speech and visual tokens in dialogue context outperforms cascaded speech-then-video generation on both emotional accuracy and lip synchronisation." source: "§6.2, §6.3, Table 2, Table 3" evidence: "Built on Qwen2.5-0.5B, EVSLM takes the serialized multimodal context and sequentially predicts: (1) the target emotion token, (2) interleaved facial expression and speech tokens via a bimodal hard alignment strategy." confidence: high relevance: high - claim_id: including_visual_dialogue_history_talking_face_animations_of role: supports claim: "Including visual dialogue history (talking-face animations of prior turns) improves emotion accuracy and speaker consistency in conversational speech synthesis relative to audio-text-only context." source: "§6.4, Table 4" evidence: "UniTalker frames the problem as Conversational Speech-Visual Synthesis (CSVS): given the multimodal history of a dialogue (text, speech, talking-face animations, speaker identity, and emotion labels for each prior turn) and the current turn's speaker and text, predict the target utterance's emotion, speech, and talking-face animations jointly." confidence: high relevance: medium - claim_id: low_rate_discrete_tokenisation_of_facial_landmarks_1 role: supports claim: "Low-rate discrete tokenisation of facial landmarks (1 token per frame) is more effective for LLM contextual modelling than higher-rate representations, even at a cost in geometric reconstruction fidelity." source: "§6.1, §6.2, Table 1, Table 2" evidence: "The LmkCodec's FSQ design with 1 token per frame also proves better than the 8-token-per-frame GRVQ variant, demonstrating that shorter sequences aid LLM contextual modeling even at some reconstruction cost." confidence: high relevance: high - claim_id: emotion_guided_conditioning_of_the_speech_renderer_including role: supports claim: "Emotion-guided conditioning of the speech renderer, including predicted emotion labels as explicit conditioning, improves measured emotional expressiveness over systems that rely on implicit contextual inference alone." source: "§6.4, Table 4" evidence: "In speech emotion expressiveness (MOS_SE), the margin is +0.111 over Empatheia." confidence: high relevance: medium limitations: - "The Talking-face Animations Renderer (EchoMimic) is a pre-trained third-party module that receives no additional fine-tuning in this pipeline. Its outputs are constrained by the quality ceiling and biases of the EchoMimic base model, limiting the paper's ability to attribute animation quality gains to UniTalker specifically vs. the renderer." - "Rendering latency is notable: speech generation takes approximately 2 seconds and animation rendering takes approximately 5 seconds per 25 frames on an RTX 4080 with 32 GB RAM, making the system unsuitable for real-time interaction in its current form. The paper acknowledges this and lists streaming optimisation as future work." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2507.20091" published_date: "2025-08-07" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [unified_speech_text_language_model] claims: - claim_id: codec_token_speech_lms_trained_on_tens_of role: supports claim: "Codec-token speech LMs trained on tens of thousands of hours of data do not develop prosodic understanding or generation as emergent capabilities through pre-training alone." source: "§1, §4.2–4.4" evidence: "A sentence-level [Global] token capturing aggregate prosodic extremity is optionally prepended to control expressive range during generation; the authors find it beneficial for generation tasks and detrimental for understanding tasks." confidence: high relevance: high - claim_id: replacing_codec_tokens_with_explicit_interpretable_word_level role: supports claim: "Replacing codec tokens with explicit, interpretable word-level prosody annotations enables a pre-trained text LLM to acquire prosody processing capabilities — including contrastive focus, emotion recognition, and style transfer — without task-specific fine-tuning." source: "§3.1, §4.2–4.4" evidence: "The decoder adapts StyleTTS2's duration predictor and prosody predictor — replacing their dependence on StyleTTS2's diffusion-based style embedding with conditioning on the word-level values generated by the LLM — while leaving all other StyleTTS2 modules frozen at their pre-trained weights." confidence: high relevance: high - claim_id: disentangling_prosodic_and_linguistic_content_in_the_token role: supports claim: "Disentangling prosodic and linguistic content in the token sequence trades voice quality fidelity for prosodic expressiveness and controllability." source: "§3.1, §5" evidence: "A sentence-level [Global] token capturing aggregate prosodic extremity is optionally prepended to control expressive range during generation; the authors find it beneficial for generation tasks and detrimental for understanding tasks." confidence: high relevance: high - claim_id: prosody_to_content_dependencies_detecting_emphasis_or_emotion role: supports claim: "Prosody-to-content dependencies (detecting emphasis or emotion from speech) are more difficult for codec-token LMs than content-to-prosody generation." source: "§4.3, Table 4" evidence: "On prosody understanding tasks, ProsodyLM produces a statistically significant 6.6% log-probability increase for emphasised words (emphasis detection on EmphAssess), while both codec-token baselines show near-zero signal." confidence: high relevance: high - claim_id: word_level_prosody_tokens_impose_only_marginal_degradation role: supports claim: "Word-level prosody tokens impose only marginal degradation on text language modelling capability when interleaved with transcription tokens in a joint sequence." source: "§4.5, Table 9" evidence: "Text perplexity on held-out audiobooks increases by only 2 points relative to a text-only fine-tuned Llama (11.70 vs. 13.87), indicating that prosody tokens add minimal interference to language modelling." confidence: high relevance: high limitations: - "ProsodyLM is trained and evaluated exclusively on audiobooks (Librilight). Audiobook prosody is stylised and read-speech in character; generalisation to spontaneous conversation, emotional dialogue, or cross-domain settings is untested and likely requires substantial data re-collection or domain adaptation." - "The decoder is frozen at StyleTTS2 pre-trained weights, which constrains output quality to that system's capability ceiling and limits the range of expressible voice characteristics. The five-dimensional prosody token cannot encode voice quality changes (breathiness, creakiness, vocal effort), which are important for fine-grained expressiveness." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.06262" published_date: "2025-08-08" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: token_verification_is_necessary_for_multi_token_prediction role: supports claim: "Token verification is necessary for multi-token prediction to be effective in autoregressive TTS: without it, WER increases from 3.07% to 14.37% and speaker similarity drops from 0.570 to 0.463." source: "§V, Table III" evidence: "Multi-Token Prediction (MTP) Modules." confidence: high relevance: high - claim_id: plug_and_play_mtp_modules_trained_on_a role: supports claim: "Plug-and-play MTP modules trained on a modest dataset can accelerate a frozen autoregressive TTS backbone by up to 1.48x without sacrificing generation quality on standard benchmarks." source: "§IV.B, Table I" evidence: "The backbone is never updated, ensuring no degradation of the original generation quality." confidence: high relevance: high - claim_id: under_quality_maximizing_inference_settings_mtp_with_verification role: supports claim: "Under quality-maximizing inference settings, MTP with verification can improve intelligibility beyond the backbone baseline, likely due to extended look-ahead context from the cascaded hidden states." source: "§IV.A, Table I" evidence: "At topk=100 (quality-maximizing setting), Llasa+ achieves WER 2.499% and SIM 0.575 on Seed-TTS-eval-en — both better than the Llasa baseline (WER 3.22, SIM 0.572) — with 1.42x speedup." confidence: high relevance: medium - claim_id: converting_a_non_causal_codec_decoder_to_a role: supports claim: "Converting a non-causal codec decoder to a causal streaming architecture via lightweight fine-tuning preserves approximately 95% of reconstruction quality, making streaming reconstruction viable without full retraining." source: "§IV.C, Table II" evidence: "The original XCodec2 transformer decoder is converted to a causal architecture to support streaming waveform reconstruction." confidence: high relevance: high - claim_id: attention_based_mtp_modules_substantially_outperform_mlp_based role: supports claim: "Attention-based MTP modules substantially outperform MLP-based equivalents of similar parameter count in both intelligibility and speaker similarity for TTS acceleration." source: "§IV.B, Table I" evidence: "Ablation studies confirm that: (1) attention-based MTP layers substantially outperform MLP replacements (WER 2.745 vs. 4.375 for MTP1 at topk=500); (2) the DeepSeek-V3-style input (feeding ground-truth tokens to MTP) underperforms the proposed hidden-state-only input due to train/inference mismatch; (3) a third MTP module provides less than 10% additional speedup at disproportionate cost." confidence: high relevance: medium limitations: - "Evaluation is English-only (LibriTTS training, Seed-TTS-eval-en test), and speaker generalization to out-of-distribution languages or accents is untested. The verification overhead (one additional LM forward pass for verification at each step) partially offsets the MTP speedup, especially at strict topk values. The 1.48x figure assumes topk=500, which allows some quality degradation; the fully lossless speedup (topk=100) is closer to 1.42x. Scaling MTP to larger models (Llasa-3B, Llasa-8B) is not investigated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.08715" published_date: "2025-08-12" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: llm_based_tts_architectures_when_fine_tuned_with role: supports claim: "LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS." source: "§4.2" evidence: "The architecture inherits CosyVoice's three-component pipeline: a transformer-based autoregressive LM that generates speech tokens conditioned on text and language identifiers (zh, ma, ta for Mandarin, Malay, Tamil), a pretrained flow-matching model that maps speech tokens to continuous acoustic features, and a pretrained HiFi-GAN vocoder for waveform synthesis." confidence: high relevance: high - claim_id: child_specific_vocal_characteristics_can_be_transferred_to role: supports claim: "Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model." source: "§2.2, §3.2" evidence: "Child-friendliness is introduced through speaker conditioning: x-vector embeddings extracted from children's speech in each of the three languages are incorporated as speaker representations, steering the model toward age-specific vocal characteristics including higher pitch and distinctive prosodic patterns." confidence: high relevance: medium - claim_id: foundation_tts_models_originally_trained_on_high_resource role: supports claim: "Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility." source: "§3.1, §4.3, Table 1" evidence: "MultiGen fine-tunes CosyVoice-300M, an existing multilingual zero-shot TTS model, for five epochs on domain-specific data covering three target languages." confidence: high relevance: medium - claim_id: intelligibility_gains_from_modern_tts_architectures_in_low role: supports claim: "Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited." source: "§4.3, Table 1" evidence: "The paper reports evidence that intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited." confidence: high relevance: medium limitations: - "Evaluation uses only 30 speech samples per language rated by 10 listeners each. The small scale means reported MOS differences, while large, should be treated cautiously — and no significance testing distinguishes genuine quality gaps from listener variability. All data is proprietary and not released, preventing independent replication." - "The Singaporean-accented Mandarin training set contains only 1,400 utterances from a single child speaker, raising questions about voice diversity and generalisation to other child speakers. The paper does not evaluate speaker similarity to the target child voices, making it difficult to assess how faithfully the age-appropriate characteristics are reproduced. Malay and Tamil models are not evaluated against CosyBase, so it is unclear whether fine-tuning adds value beyond what a capable pretrained model already provides for those languages. All evaluation is limited to Singapore's specific linguistic and cultural context; the approach's generalisability to other low-resource child-speech settings is undemonstrated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.08961" published_date: "2025-08-12" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, SCA, VC] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [unified_speech_text_language_model] claims: - claim_id: separating_the_token_used_for_llm_input_from role: supports claim: "Separating the token used for LLM input from the token used for generation output can resolve the information-level conflict that makes joint optimisation of understanding and generation tasks difficult in a shared-token speech LLM." source: "§DualSpeechLM, §Results and Analyses" evidence: "DualSpeechLM addresses the shared-token bottleneck by explicitly decoupling the input and output token spaces within a single end-to-end framework." confidence: high relevance: high - claim_id: training_a_speech_tokenizer_directly_against_a_text role: supports claim: "Training a speech tokenizer directly against a text LLM's next-token prediction objective is more effective at reducing the speech-text modality gap than optimising against ASR or SSL reconstruction losses alone." source: "§USTokenizer, Table 1" evidence: "What distinguishes this tokenizer is an understanding-driven loss: the quantized vectors are projected into a frozen LLaMA-3.2-1B's embedding space, and the tokenizer is trained to minimise next-token prediction loss on understanding tasks (ASR, emotion recognition, speech question answering)." confidence: high relevance: high - claim_id: understanding_task_supervision_produces_representations_that_transfer_to role: supports claim: "Understanding-task supervision produces representations that transfer to generation quality improvements, but the converse — generation-task supervision improving understanding — is weaker and less consistent." source: "§Ablation Study, §H. Discussion" evidence: "A semantic supervision loss trains the LLM to also predict USToken sequences for generation tasks, enforcing consistent semantic representations across pathways." confidence: high relevance: medium - claim_id: stochastic_conditioning_during_training_exposing_a_generation_module role: supports claim: "Stochastic conditioning during training (exposing a generation module to varied subsets of its conditioning signals) improves robustness to imperfect upstream predictions at inference time." source: "§Ablation Study, Table 6" evidence: "A Chain-of-Condition (CoC) strategy regularises AcousticGPT training by stochastically conditioning the module on one of three sources — prompt hidden states, predicted USTokens, or their concatenation — sampled with equal probability." confidence: high relevance: medium limitations: - "The entire evaluation is conducted at 4.5K hours of training data with parameter-efficient LoRA fine-tuning. The claim that USTokens reduce data requirements is plausible but untested at the scale (70K–570K hours) where competing systems are evaluated. Whether the dual-token architecture and the understanding-driven tokeniser remain advantageous at scale is an open question." - "No code or demo is linked in the paper, limiting reproducibility. The 4.5K-hour training regime excludes noisy, in-the-wild, and multilingual data, so generalisation to these conditions is untested despite the paper's stated future direction of expanding to multilingual and cross-domain data. The USTokenizer's understanding-driven loss requires a frozen LLM during tokeniser training, adding a significant computational overhead at the tokenisation stage (288% memory increase) even if this overhead disappears at DualSpeechLM inference. The model size of the full system (Phi-3.5-3B + AcousticGPT) is not explicitly stated in aggregate, and the AcousticGPT's token generation speed relative to real-time is not reported." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2504.12867" published_date: "2025-08-13" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: fine_grained_natural_language_emotion_descriptions_provide_richer role: complicates claim: "Fine-grained natural language emotion descriptions provide richer control over expressive speech synthesis than coarse categorical labels, at the cost of requiring emotion-specific training data." source: "§2.1, §6.1" evidence: "Training is two-stage: a standard TTS pretraining phase on large synthetic neutral speech (VoiceAssistant, 3,234h English; Belle, 6,418h Chinese), followed by emotional fine-tuning on EmoVoice-DB, LAION Got Talent, and in-house Chinese data." confidence: high relevance: medium - claim_id: parallel_phoneme_token_prediction_as_a_secondary_output role: supports claim: "Parallel phoneme token prediction as a secondary output head reduces intelligibility errors in LLM-based TTS, particularly on challenging inputs such as rare words and tongue twisters." source: "§6.2.1, §6.2.2, Table 5, Table 6" evidence: "EmoVoice-PP (Phoneme Boost variant) adds a parallel prediction head that simultaneously outputs phoneme tokens (~11Hz) alongside audio tokens (~17Hz)." confidence: high relevance: high - claim_id: llm_pretraining_initialisation_meaningfully_benefits_emotion_controllable_tts role: supports claim: "LLM pretraining initialisation meaningfully benefits emotion-controllable TTS: models without it show substantially higher word error rates and weaker emotion transfer." source: "§6.2.4, Table 8" evidence: "On the Chinese Secap test set, EmoVoice-PP achieves the best WER (7.6), emotion similarity (0.7939), and recall (0.434) among evaluated models, outperforming CosyVoice2 and GPT-series models, which struggle with non-English prosody and timbre." confidence: high relevance: medium - claim_id: automatic_emotion_similarity_metrics_e_g_emotion2vec_cosine role: supports claim: "Automatic emotion similarity metrics (e.g. emotion2vec cosine similarity) correlate reasonably at the system level but poorly at the utterance level with human perceptual judgments, limiting their utility for fine-grained model comparison." source: "§7, Table 10" evidence: "On the English EmoVoice-DB test set (objective), EmoVoice(1.5B) achieves the highest emotion similarity (0.9118) and recall rate (0.424) among open systems, with WER of 2.62 and UTMOS of 4.35." confidence: high relevance: medium - claim_id: multimodal_llms_are_not_yet_reliable_judges_of role: supports claim: "Multimodal LLMs are not yet reliable judges of emotional speech quality, exhibiting both low correlation with human ratings and inter-run instability." source: "§7, Table 10" evidence: "The ablation on LLM initialization shows it is essential: removing pretrained Qwen2.5 weights raises WER from 2.73 to 6.16 for the base model, confirming that LLM language understanding transfers meaningfully to emotional TTS quality." confidence: high relevance: medium limitations: - "The English model is trained and evaluated entirely on synthetic data generated by GPT-4o-audio. Both EmoVoice-DB (training) and the test set are GPT-4o-audio outputs, creating circularity: the model learns to mimic GPT-4o-audio's synthesis style rather than natural human emotional speech. Generalisation to real human emotional recordings or to out-of-distribution TTS systems is undemonstrated." - "The emotion recall evaluation omits three of seven emotion categories (disgusted, fearful, surprised) due to low recognition accuracy from emotion2vec, which limits the scope of the emotional expressiveness claims." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.09767" published_date: "2025-08-13" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: parameter_efficient_fine_tuning_with_lora_can_restore role: supports claim: "Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance." source: "§3.2, §5.1" evidence: "Because the LoRA module is language-specific and optional, inference on other languages simply omits it, leaving cross-lingual performance unaffected in principle." confidence: high relevance: medium - claim_id: bpe_based_multilingual_tts_models_systematically_underperform_on role: supports claim: "BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive." source: "§1, §5.2" evidence: "On the accent stress-test (Test Set 2), the baseline CosyVoice 2 achieves accent correctness of only 0.498 — near chance — even when kana transcription is supplied to resolve kanji ambiguity." confidence: high relevance: medium - claim_id: phoneme_tag_injection_via_special_tokens_can_provide role: supports claim: "Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans." source: "§5.2" evidence: "A follow-up probe using 50 katakana-only sentences confirms that the phoneme-tag mechanism does not bleed pronunciation control into spans outside the tags: accent correctness is 0.829 for UtterTune vs. 0.796 for the baseline, with no significant difference." confidence: high relevance: high - claim_id: speaker_similarity_in_zero_shot_tts_is_robust role: supports claim: "Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data." source: "§5.1, Table 1" evidence: "Only the embeddings of the two new tokens and the LoRA matrices are updated during fine-tuning; the original embedding matrix remains frozen." confidence: high relevance: medium limitations: - "Evaluation uses two custom test sets authored or curated by the single researcher-author, with no publicly available benchmark comparison. The listening test crowd cohort (16 workers for MOS, 15 for accent correctness) is small for a task with high inter-rater variance. Results should be treated as promising but not yet independently validated." - "The approach requires users to supply phonemic transcription and accent markup, which demands linguistic expertise most end users lack. The paper targets only standard Tokyo Japanese; dialectal variation and code-switching remain untested, as the author acknowledges. The LoRA module is trained on a single female-heavy corpus (JSUT) and 100-speaker JVS, which may limit generalisation to speaker types not well represented in that data. Applicability to other prosodically complex languages — Mandarin tones, Scandinavian pitch accent — is left for future work." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.11326" published_date: "2025-08-15" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, diffusion] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: freezing_the_backbone_llm_and_routing_modality_specific role: supports claim: "Freezing the backbone LLM and routing modality-specific tokens to separate expert sets can preserve pre-trained text understanding capabilities during speech generation fine-tuning." source: "§3.2" evidence: "A deterministic modality-based routing strategy assigns text tokens exclusively to the original (frozen) text experts and speech tokens exclusively to the newly trained speech experts." confidence: high relevance: high - claim_id: instruction_conditioned_tts_systems_trained_on_tag_derived role: supports claim: "Instruction-conditioned TTS systems trained on tag-derived description datasets exhibit significant performance degradation when faced with figurative or metaphorical natural language at inference time." source: "§1, §4.1" evidence: "On the alignment dimensions — which directly measure description-following — MoE-TTS outperforms both commercial systems (OA: 3.61 vs 3.26 ElevenLabs, 3.46 MiniMax on in-domain)." confidence: high relevance: medium - claim_id: commercial_speech_synthesis_products_are_not_immune_to role: supports claim: "Commercial speech synthesis products are not immune to the out-of-domain description challenge, suggesting that this generalisation gap is not solved by scale alone." source: "§4.2, Table 2" evidence: "On in-domain descriptions, MoE-TTS is competitive with the commercial systems on basic speech quality dimensions but does not lead; MiniMax achieves the highest OS (3.83 vs 3.82 for MoE-TTS)." confidence: high relevance: medium - claim_id: modality_separation_techniques_from_multimodal_vision_language_research role: supports claim: "Modality separation techniques from multimodal vision-language research transfer meaningfully to the speech domain, reducing catastrophic forgetting without requiring multi-modal data mixing during pre-training." source: "§2.2, §3.2" evidence: "During training, only the speech-modality expert weights are updated; the LLM backbone remains entirely frozen throughout." confidence: high relevance: medium limitations: - "The evaluation is based on 20 in-domain and 40 out-of-domain test samples, annotated by 21 evaluators. These are very small test sets for drawing strong comparative conclusions. Both the in-domain and out-of-domain test sets were constructed by the MoE-TTS authors, introducing potential design bias toward cases where the proposed approach excels." - "The system currently supports only English text descriptions, due to the limited scope of available open-source description-based TTS datasets. The LLM architecture sensitivity is unexplored — all experiments use Qwen3-4B, and the impact of model scale (smaller or larger LLM backbones) on the MoE approach is left for future work. The diffusion and VAEGAN components are adapted from Stable Audio without fine-tuning on description-based data, and their contribution to description alignment is not ablated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-0253" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: maintaining_dynamically_updated_compressed_memory_across_sentences_improves role: supports claim: "Maintaining dynamically updated compressed memory across sentences improves naturalness and coherence in paragraph-level TTS compared to methods that use fixed-window preceding sentences." source: "§3.4.1, Table 1" evidence: "On the internal Chinese Mandarin audiobook test set (Table 1): | System | MOS | CoMOS | SpeechBERT | CER | SPK-SIM (mean, var) | |---|---|---|---|---|---| | Proposed | 3.796 | 3.992 | 80.448 | 4.140% | 85.685 (0.019) | | MMCE-Qformer | 3.557 | 3.885 | 79.031 | 5.075% | 85.110 (0.021) | | CLAP-RAG | 3.489 | 3.717 | 78.892 | 6.234% | 84.920 (0.037) | | Baseline | 3.468 | 3.460 | 77.776 | 5.850% | 85.051 (0.035) | The proposed method achieves the best naturalness (MOS), coherence (CoMOS), content accuracy (CER), and speaker consistency (lowest SPK-SIM variance) while using only fixed 64-token context — the same cost as MMCE-Qformer but without requiring 5 full preceding sentences." confidence: high relevance: medium - claim_id: speech_context_representations_are_more_effective_than_text role: supports claim: "Speech context representations are more effective than text context representations for guiding prosodic coherence in autoregressive LM-based TTS, due to the one-to-many relationship between text and speech." source: "§3.4.2, Table 1" evidence: "Ablation confirms that both speech memory (Mem-S) and text memory (Mem-T) contribute, with speech memory having the larger effect (likely because speech captures richer prosodic information than text)." confidence: high relevance: high - claim_id: applying_bidirectional_attention_to_prefix_tokens_via_a role: supports claim: "Applying bidirectional attention to prefix tokens via a prefix mask enhances in-context learning in decoder-only TTS LMs without compromising autoregressive generation consistency." source: "§2.3, §3.4.2, Table 1" evidence: "Prefix mask replaces the standard causal mask for the memory/text prefix tokens, applying bidirectional attention to them while maintaining unidirectional attention over generated speech tokens." confidence: high relevance: high - claim_id: excessively_long_variable_length_inference_prompts_increase_hallucination role: supports claim: "Excessively long variable-length inference prompts increase hallucination and content errors in autoregressive TTS, while fixed-length context representations mitigate this instability." source: "§3.4.1, Table 1" evidence: "Inference requires only one preceding context plus fixed-length 64-token memory, compared to 5 full preceding sentences (MMCE-Qformer) or variable-length prompts up to ~900 tokens (CLAP-RAG)." confidence: high relevance: high limitations: - "- Evaluation is monolingual (Chinese Mandarin only); generalizability to other languages is unconfirmed. - The gap to Ground Truth (MOS 4.406 vs 3.796) remains substantial; paragraph-level naturalness is still an open problem. - No code release; replication requires re-implementing CAM on a CosyVoice backbone. - The model is evaluated only on single-speaker audiobooks; how it handles multi-speaker paragraphs is unknown. - The fixed 32-embedding Perceiver Resampler bottleneck may lose fine-grained phonetic detail." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-0319" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: llm_based_zero_shot_tts_systems_are_more role: supports claim: "LLM-based zero-shot TTS systems are more sensitive to noise in audio prompts than speaker-embedding-based approaches, because their in-context learning mechanism preserves the acoustic environment of the prompt." source: "§1" evidence: "Noise-robust zero-shot TTS task (subjective + objective): | System | Prompt | MOS | SMOS | SECS | WER | +FLOPs | |--------|--------|-----|------|------|-----|--------| | LauraTTS | Clean | 4.03 | 3.99 | 0.827 | 2.33% | — | | LauraTTS | Noisy | 3.46 | 2.80 | 0.642 | 21.37% | — | | LauraTTS + MP-SENet | Noisy | 4.01 | 3.85 | 0.811 | 2.54% | 38.93G | | NR-LauraTTS | Noisy | 4.02 | 3.98 | 0.827 | 2.44% | 1.10G | NR-LauraTTS achieves SECS of 0.827 — matching LauraTTS with clean prompts exactly — while LauraTTS+MP-SENet reaches only 0.811 due to SE artifacts corrupting speaker identity." confidence: high relevance: medium - claim_id: performing_speech_enhancement_in_the_discrete_acoustic_token role: supports claim: "Performing speech enhancement in the discrete acoustic token domain outperforms waveform-domain SE methods in both speech quality and computational efficiency, achieving higher DNSMOS scores at roughly one-third the FLOPs." source: "§4.1, Table 1" evidence: "Speech enhancement task (DNSMOS P.835, 1s signals): | Method | SIG | BAK | OVRL | FLOPs | |--------|-----|-----|------|-------| | CMGAN | 3.54 | 4.05 | 3.26 | 31.68G | | MP-SENet | 3.56 | 4.09 | 3.30 | 38.93G | | Codec Denoiser | 3.62 | 4.11 | 3.36 | 9.96G | The codec denoiser outperforms both waveform-domain baselines in all DNSMOS dimensions at 3x lower compute." confidence: high relevance: high - claim_id: waveform_domain_speech_enhancement_introduces_artifacts_that_degrade role: supports claim: "Waveform-domain speech enhancement introduces artifacts that degrade speaker identity in the enhanced prompt, resulting in lower speaker similarity in downstream zero-shot TTS compared to codec-domain denoising." source: "§4.2, Table 3" evidence: "Speech enhancement task (DNSMOS P.835, 1s signals): | Method | SIG | BAK | OVRL | FLOPs | |--------|-----|-----|------|-------| | CMGAN | 3.54 | 4.05 | 3.26 | 31.68G | | MP-SENet | 3.56 | 4.09 | 3.30 | 38.93G | | Codec Denoiser | 3.62 | 4.11 | 3.36 | 9.96G | The codec denoiser outperforms both waveform-domain baselines in all DNSMOS dimensions at 3x lower compute." confidence: high relevance: high - claim_id: predicting_only_the_first_two_rvq_groups_of role: supports claim: "Predicting only the first two RVQ groups of clean acoustic tokens is sufficient for effective token-domain speech enhancement; predicting more groups increases complexity without improving quality." source: "§4.1, Table 2" evidence: "Prediction of only the first two groups is motivated by LauraTTS's architecture (first two groups encode essential speaker properties) and by the empirical finding that predicting more groups increases complexity without improving SE metrics. 2." confidence: high relevance: high - claim_id: the_vq_bottleneck_of_neural_codecs_acts_as role: complicates claim: "The VQ bottleneck of neural codecs acts as an implicit noise filter during quantization, providing a structural advantage for denoising in the token domain relative to signal-domain methods." source: "§4.1" evidence: "Codec Denoiser (the novel component): A two-stage model operating entirely in the discrete acoustic token domain: 1." confidence: high relevance: high limitations: - "- Specific to LauraTTS + Encodec (FunCodeec); adaptation to other codecs (DAC, SoundStream, EnCodec original) requires retraining. - Training uses synthetic noise (DNS Challenge 2022 + WHAM!); real-world noise types (music, babble, room impulse responses) may not be fully covered. - No ablation on LauraTTS model size or the effect of codec VQ count K on denoiser difficulty. - Intelligibility (WER 2.44%) is slightly worse than clean-prompt LauraTTS (2.33%), though close. - No evaluation of the codec denoiser on other TTS or SE tasks beyond the paired experiment." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-0355" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [codec, evaluation] architecture: [] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: codec_noise_robustness_does_not_reliably_predict_across role: supports claim: "Codec noise robustness does not reliably predict across bitrate regimes, and clean-speech quality metrics are a poor proxy for noisy-condition performance." source: "§4, Figure 1 in paper" evidence: "At the highest bitrate, DAC outperforms EnCodec in noise robustness; at 3 kbps, EnCodec surpasses DAC on most downstream metrics under noise. HiFi-Codec matches EnCodec in clean-speech perceptual quality but degrades far more under noise." confidence: high relevance: high - claim_id: neural_speech_codecs_exhibit_non_linear_input_output role: supports claim: "Neural speech codecs exhibit non-linear input-output behavior (violations of additivity and homogeneity) that partially explains their robustness failures under noise and signal overlap." source: "§5, Figure 2 in paper" evidence: "Additivity improves with bitrate across all evaluated codecs, and codecs with better additivity (DAC) show stronger noise robustness. HiFi-Codec and SpeechTokenizer exhibit uneven homogeneity at extreme gains, aligning with their higher degradation in speaker and emotion metrics." confidence: high relevance: high - claim_id: codec_evaluation_using_only_clean_speech_reconstruction_metrics role: complicates claim: "Codec evaluation using only clean-speech reconstruction metrics provides an incomplete characterization of codec behavior for real-world speech processing pipelines." source: "§4" evidence: "Perceptual quality (PESQ) converges to similar values across codecs under severe noise, masking large differences in phonetic content (WER), speaker identity (EER), and emotional fidelity (SER-ACC)." confidence: high relevance: high - claim_id: low_frequency_spectral_emphasis_introduced_by_time_domain role: supports claim: "Low-frequency spectral emphasis introduced by time-domain training losses is a consistent artifact across neural speech codecs and can cause unintended spectral coloration." source: "§6, Figure 3 in paper" evidence: "All five evaluated codecs exhibit low-frequency boosting below 100 Hz under sine sweep analysis, most pronounced in EnCodec; the effect is attributed to waveform-level L1/L2 losses used in training." confidence: high relevance: high limitations: - "The evaluation is limited to five codecs, all of which use residual vector quantization. More recent codecs with different quantization strategies (e.g., fully transformer-based or finite scalar quantization approaches) are not included. The noise conditions, while diverse, use fixed SNR ranges and do not cover all real-world degradation types (e.g., codec artifacts, packet loss, or non-stationary noise sources). All test data are English; whether these robustness rankings generalize across languages and speaking styles is unexamined. The emotion preservation evaluation relies on a single emotion recognizer (emotion2vec) trained on limited data, and speaker identity assessment uses a single ECAPA-TDNN model, which may introduce evaluator-specific biases. No human perceptual evaluation is included." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "interspeech-2025-0464" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [VC, codec] architecture: [autoregressive-LM] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: explicit_mutual_information_minimisation_at_the_codec_embedding role: supports claim: "Explicit mutual information minimisation at the codec-embedding level provides effective prosody-timbre disentanglement for voice conversion." source: "§3.5, Table 3" evidence: "Removing the MI loss (L_MI) from the full system leads to a notably higher normalised F0 distance in the prosody-from-source scenario (3.28 vs. 2.82), while quality and timbre metrics change only modestly, isolating prosody control as the primary benefit of the MI objective." confidence: high relevance: high - claim_id: in_context_learning_codec_lms_can_serve_as role: supports claim: "In-context learning codec LMs can serve as controllable VC backbones when augmented with prosody-disentangling encoder modules." source: "§3.4, §3.5, Table 2, Table 3" evidence: "The proposed system builds on VALL-E X's ICL mechanism and outperforms VALL-E X in speaker similarity (ASV 0.91 vs. 0.84), intelligibility (WER 0.101 vs. 0.115), naturalness (MOS 4.36 vs. 4.19), and prosody alignment (F0 distance 2.70 vs. 3.10) in the prompt-based scenario." confidence: high relevance: high - claim_id: prosody_disentanglement_at_the_codec_level_introduces_a role: complicates claim: "Prosody disentanglement at the codec level introduces a small trade-off in absolute codec reconstruction fidelity compared to the unmodified encoder." source: "§3.3, Table 1" evidence: "PACE's ASV score (0.662) and NISQA score (3.98) are lower than the baseline EnCodec encoder (0.681, 4.17), though the gap does not substantially affect system-level VC performance." confidence: high relevance: high - claim_id: prosody_from_source_and_prosody_from_prompt_are role: refines claim: "Prosody-from-source and prosody-from-prompt are distinct capability axes in voice conversion; systems strong at one do not automatically handle the other." source: "§3.5, Table 3" evidence: "VALL-E X supports only prosody-from-prompt and is excluded from the source-prosody evaluation; TriAAN-VC and ProsoVC support only source-prosody and are excluded from the prompt-prosody evaluation. Only the proposed system is evaluated in both modes." confidence: high relevance: low limitations: - "All evaluation is conducted on LibriTTS-clean-100 and test-clean, a relatively clean single-domain corpus with 247 speakers. Generalisation to noisy environments, expressive or emotional speech, or cross-lingual settings is not tested." - "The 54-hour training dataset is modest for a codec language model approach; it is unclear whether the disentanglement quality degrades with longer or more expressive source utterances. No code or demo is reported, limiting reproducibility. The paper does not ablate the number of RVQ codebooks or the sensitivity of the MI-minimisation trade-off weight (lambda_MI), leaving the robustness of the disentanglement objective undercharacterised. Prosody is operationalised solely through f0 and UV binary flags; richer prosodic dimensions such as energy, speaking rate, and phrase-level structure are not captured." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "interspeech-2025-0468" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [codec] architecture: [VAE, GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: directly_encoding_ssl_features_as_a_first_class role: supports claim: "Directly encoding SSL features as a first-class codec stream produces stronger semantic preservation in RVQ-1 tokens than distillation from an SSL model, particularly for tonal languages where pitch fidelity is critical." source: "§4.2, Table 2" evidence: "The SSL encoding stream takes 16th-layer features from a frozen w2v-BERT-2.0 model (600M parameters), downsamples them via 1D average pooling to the target frame rate (factor 2 for 25 Hz, factor 4 for 12.5 Hz), and passes them through a ResNet encoder (ConvNeXt blocks, 13M parameters) before VQ quantisation." confidence: high relevance: high - claim_id: operating_a_neural_codec_at_lower_frame_rates role: supports claim: "Operating a neural codec at lower frame rates with more RVQ layers at fixed token rate improves audio quality over higher-frame-rate codecs with fewer layers at the same bitrate." source: "§4.3, Table 3" evidence: "The waveform encoding stream follows the DAC architecture — a CNN encoder with strided convolutions reaches the same frame rate, then N-1 RVQ layers quantise the residual between the waveform feature and the RVQ-1 feature; the decoder sums both streams back before upsampling to audio." confidence: high relevance: high - claim_id: semantic_quality_of_rvq_1_tokens_is_a role: supports claim: "Semantic quality of RVQ-1 tokens is a primary determinant of downstream TTS intelligibility in autoregressive codec-based systems, independent of codec audio reconstruction quality." source: "§4.4, Table 4" evidence: "On codec audio reconstruction quality (LibriSpeech-test-clean, normalised to 75 tokens/s, ~0.75 kbps), DualCodec 25Hz with 16384 codebook (G2) achieves MUSHRA 86.2 and UTMOS 4.08, exceeding WavTokenizer-large (MUSHRA 81.0, UTMOS 3.87) and Mimi (MUSHRA 72.8)." confidence: high relevance: high - claim_id: an_ssl_based_semantic_stream_in_a_codec role: supports claim: "An SSL-based semantic stream in a codec encoder can improve perceptual audio quality beyond what waveform-only codecs achieve, even when using the same decoder architecture." source: "§4.3, Table 3" evidence: "The waveform encoding stream follows the DAC architecture — a CNN encoder with strided convolutions reaches the same frame rate, then N-1 RVQ layers quantise the residual between the waveform feature and the RVQ-1 feature; the decoder sums both streams back before upsampling to audio." confidence: high relevance: high limitations: - "The 12.5 Hz DualCodec-based TTS lags behind the 25 Hz variant in both WER and speaker similarity, indicating that the more aggressive downsampling introduces a ceiling on semantic accuracy that affects TTS quality. The paper acknowledges this gap as the primary remaining challenge." - "The SSL model (w2v-BERT-2.0, 600M parameters, frozen) is required at TTS training time but not inference. This makes the training pipeline heavier than pure waveform codec approaches. It is also unclear whether the approach generalises to SSL models other than w2v-BERT-2.0, or whether the chosen 16th layer feature is optimal across languages beyond English and Mandarin." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "interspeech-2025-0551" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [alignment_constrained_autoregression] claims: - claim_id: selectively_applying_monotonic_attention_constraints_to_automatically_identified role: supports claim: "Selectively applying monotonic attention constraints to automatically identified alignment heads in decoder-only LLMs reduces word repetition and omission errors without degrading performance on general test sets." source: "§3.1, §3.2, §5.1, Table 1" evidence: "A recursive formula aggregates these probabilities into monotonic attention scores — analogous to monotonic multihead attention in encoder-decoder models (Raffel et al. 2017, Ma et al. 2019) but adapted for decoder-only self-attention." confidence: high relevance: medium - claim_id: forced_aligner_based_monotonic_alignment_methods_for_llm role: supports claim: "Forced-aligner-based monotonic alignment methods for LLM TTS suffer from out-of-domain generalization failure when training data is diverse, while alignment-free approaches are more robust." source: "§5.1, Table 1" evidence: "VALL-E R (via Phn-Cosy-S-Ali) significantly raises general test WER (Seed-ZH CER: 4.03%→5.08%, Seed-EN WER: 5.32%→6.40%) indicating poor out-of-domain generalization, while SMA avoids this degradation." confidence: high relevance: medium - claim_id: in_decoder_only_llms_used_for_tts_alignment role: supports claim: "In decoder-only LLMs used for TTS, alignment characteristics are concentrated in a small subset of attention heads (typically 2–3 out of the full multi-head set), and applying monotonic constraints only to those heads is critical for preserving generation quality." source: "§3.2, §5.2, Figure 2" evidence: "A recursive formula aggregates these probabilities into monotonic attention scores — analogous to monotonic multihead attention in encoder-decoder models (Raffel et al. 2017, Ma et al. 2019) but adapted for decoder-only self-attention." confidence: high relevance: medium - claim_id: monotonic_attention_fine_tuning_for_llm_based_tts role: supports claim: "Monotonic attention fine-tuning for LLM-based TTS scales to large model sizes and large-scale multilingual training corpora without degrading speaker similarity." source: "§5.1, Table 1" evidence: "Stepwise Monotonic Attention (SMA): At each acoustic token generation step i, the model computes selection probabilities over phoneme positions using sigmoid-activated energies from the self-attention mechanism." confidence: high relevance: medium limitations: - "The method fine-tunes a pre-trained model; it remains unclear whether SMA should be applied from scratch or only as a fine-tuning step. Hard-case CER/WER remains substantially above general test performance even after SMA (e.g., Seed-ZH-Hard CER 8.53% vs. Seed-ZH CER 2.33%), meaning severe repetition/omission cases are not fully resolved. The selection of alignment heads via the diagonal ratio heuristic is validated empirically but not theoretically. Extension to streaming/online inference (where the monotonic state must be maintained across steps) is not discussed. No subjective (MOS/SMOS) evaluation is reported, making naturalness trade-offs unclear." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-0669" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [codec, TTS] architecture: [hybrid] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: supervised_phonetic_data_ctc_and_phoneme_classification_can role: supports claim: "Supervised phonetic data (CTC and phoneme classification) can replace SSL pseudo-label distillation as the phonetic supervision signal in hybrid speech tokenizers, achieving superior phonetic representation without requiring a pretrained SSL model." source: "§3.3, §5.1, Table 1" evidence: "CTC is shown to be the dominant phonetic supervision signal in ablations. - Phoneme classification: a linear projection applies cross-entropy against TIMIT phoneme labels (10% of training batch)." confidence: high relevance: high - claim_id: ctc_character_match_loss_is_the_dominant_driver role: supports claim: "CTC character-match loss is the dominant driver of phonetic encoding quality in RVQ-based tokenizers, contributing more than phoneme classification alone." source: "§5.3, Table 4" evidence: "Ablation over components: CTC loss is the single most important addition (ABX-within drops from 21.2 to 4.42 with CTC added alone); phoneme classification adds further modest gains; the transformer encoder is necessary to avoid reconstruction degradation when CTC is added." confidence: high relevance: high - claim_id: a_transformer_encoder_inserted_before_the_rvq_quantizer role: supports claim: "A transformer encoder inserted before the RVQ quantizer improves phonetic representation, but requires stochastic skip-connection dropout during training to prevent the network from bypassing it." source: "§3.2, §5.3, Table 5" evidence: "To stabilize training and prevent the transformer from being bypassed, a stochastic skip-connection is used: during training the input to the RVQ is sampled from (a) transformer output only (p=0.3), (b) skip (encoder output, p=0.1), or (c) their average (p=0.6)." confidence: high relevance: low - claim_id: hybrid_tokenizers_that_optimize_phonetic_encoding_via_direct role: supports claim: "Hybrid tokenizers that optimize phonetic encoding via direct supervision can approach pure acoustic codecs in reconstruction quality while substantially surpassing SSL-distilled baselines." source: "§5.1, Table 2" evidence: "Only EnCodec (the pure acoustic topline) exceeds PAST on reconstruction (SISNR=7.49, PESQ=3.88)." confidence: high relevance: high - claim_id: speech_tokenizers_that_better_encode_phonetic_structure_yield role: supports claim: "Speech tokenizers that better encode phonetic structure yield stronger downstream speech language model performance on lexical discrimination benchmarks." source: "§5.2, Table 3" evidence: "Speech language modeling (sWUGGY, identical 300M LM trained per tokenizer): PAST achieves 71.8/57.5 (Inter/OOV) vs. SpeechTokenizer 63.7/55.6, X-Codec 55.1/52.9, and even k-means HuBERT 67.9/55.4." confidence: high relevance: high limitations: - "PAST requires labeled phoneme/character data, which constrains multilingual scalability — the paper explicitly acknowledges this and targets it as future work. PAST's reconstruction quality (SISNR=4.84) falls below pure EnCodec (SISNR=7.49), reflecting the phonetic-acoustic trade-off. The model is 185M parameters, larger than some baseline tokenizers, partly due to the transformer encoder. Evaluation is English-only on clean speech; robustness to noise, accents, and spontaneous speech is not assessed." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "interspeech-2025-0704" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: computing_rewards_directly_on_codec_tokens_rather_than role: supports claim: "Computing rewards directly on codec tokens rather than synthesized audio reduces the computational cost of RLHF for codec LM-based TTS systems without sacrificing reward signal quality." source: "§3.1" evidence: "Instead of decoding codec tokens to audio before computing rewards, DiffRO trains a reward model that operates directly on predicted codec token sequences." confidence: high relevance: high - claim_id: differentiable_reward_optimization_via_gumbel_softmax_can_replace role: supports claim: "Differentiable reward optimization via Gumbel-Softmax can replace PPO-style policy gradient training for codec LM fine-tuning, enabling direct backpropagation through the reward model." source: "§3.1, §2.2" evidence: "To make this differentiable for end-to-end optimization, Gumbel-Softmax replaces the argmax token sampling operation, allowing gradients to flow from the reward signal back through the LM parameters without PPO's policy gradient loop." confidence: high relevance: high - claim_id: a_multi_task_reward_model_trained_on_codec role: supports claim: "A multi-task reward model trained on codec tokens can simultaneously improve pronunciation accuracy, emotion expressiveness, and audio quality in a single RL training phase." source: "§3.2, §4.3" evidence: "A codec-based speech understanding model is trained jointly on four downstream tasks: ASR (WER), speech emotion recognition (SER, cross-entropy), speech quality assessment (SQA, MOS prediction via 5-class CE), and audio event detection." confidence: high relevance: high - claim_id: rlhf_fine_tuning_on_mandarin_and_english_data role: supports claim: "RLHF fine-tuning on Mandarin and English data generalizes to unseen languages (Japanese, Korean) when the reward signal targets phonetic accuracy rather than language-specific patterns." source: "§4.2, Table 2" evidence: "Multilingual generalization without explicit RL data: Japanese WER improves from 9.13% to 6.36%, Korean from 7.43% to 5.41%, because the ASR reward teaches the LM to produce phonetically accurate codec tokens in the codec's implicit multilingual space." confidence: high relevance: medium - claim_id: quality_attributes_governed_primarily_by_the_flow_matching role: supports claim: "Quality attributes governed primarily by the flow-matching and vocoder stages (MOS, speaker characteristics) are largely resistant to optimization by LM-level RLHF alone." source: "§4.3.2, Table 4" evidence: "However, the FM and vocoder largely denoise the output, so audio MOS change is modest (3.77–3.86), highlighting that quality is mostly determined by post-LM stages." confidence: high relevance: medium limitations: - "DiffRO only optimizes the LM component; the FM and vocoder are frozen, which limits the impact of rewards that primarily affect post-LM processing (e.g., speaker age/gender, which are partly determined by the FM). The MTR reward model requires a large in-house multi-task labeled corpus (13,000+ hours) that is not publicly available, limiting reproducibility. The approach is validated only on Mandarin-focused data; generalization to languages with distinct phonological properties (tonal, agglutinative) needs further study. Constructing diverse RL training data remains nontrivial." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-0874" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [SCA] architecture: [autoregressive-LM, hybrid] relevance: high evidence_role: - architecture_variant current_role: influential method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: separating_user_and_agent_stream_representations_using_a role: supports claim: "Separating user and agent stream representations — using a pretrained speech encoder for input and a neural codec for generation — allows full-duplex S2S models to bypass LLM speech pretraining without sacrificing conversation quality." source: "§3, §6.1, §6.2" evidence: "Because the user stream uses a pretrained encoder rather than codec tokens, no speech-text pretraining of the LLM backbone is required — a sharp contrast with prior duplex systems such as Moshi." confidence: high relevance: high - claim_id: codec_personalisation_through_fine_tuning_on_target_speaker role: supports claim: "Codec personalisation through fine-tuning on target-speaker data can recover audio quality at half the bitrate of an untuned codec, as measured by MOS, CER, and speaker similarity." source: "§6.3, Table 4" evidence: "Codec personalisation for the agent voice is enabled by fine-tuning NanoCodec on 21k utterances from the target speaker; this is practical precisely because agent and user modalities are handled by separate components." confidence: high relevance: high - claim_id: turn_level_alignment_between_text_and_speech_tokens role: supports claim: "Turn-level alignment between text and speech tokens in duplex training is sufficient to learn barge-in behaviour; word-level alignment provides no measurable improvement." source: "§3.1" evidence: "The model is trained with multi-channel next-token prediction: text and speech targets are maintained in separate channels, aligned at the turn level (not word level), with text weighted 3× relative to speech in the loss." confidence: high relevance: high - claim_id: full_duplex_end_to_end_models_remain_at role: supports claim: "Full-duplex end-to-end models remain at a reasoning disadvantage compared to cascaded oracle systems, though the gap narrows as backbone LLM quality increases." source: "§6.2, Table 3" evidence: "On reasoning quality (Table 3, GPT-4o-mini scores), the proposed model outperforms Moshi across all five evaluation sets despite using a far smaller backbone; it is competitive with an oracle cascaded system on two of five sets." confidence: high relevance: medium - claim_id: open_source_availability_of_training_code_and_model role: complicates claim: "Open-source availability of training code and model weights is a critical bottleneck for research progress in full-duplex spoken dialogue, given the historical concentration of such systems in closed industrial labs." source: "§1" evidence: "Because the user stream uses a pretrained encoder rather than codec tokens, no speech-text pretraining of the LLM backbone is required — a sharp contrast with prior duplex systems such as Moshi." confidence: high relevance: medium limitations: - "The backbone is TinyLlama-1.1B, a relatively small LLM. The reasoning gap between the end-to-end model and the GT+LLM cascaded oracle is real and acknowledged, particularly on QA tasks. Scaling to a larger backbone remains untested and its interaction with the duplex architecture is an open question." - "Training data is entirely synthetic (TTS-generated user and agent speech) except for the ASR-QA portion, which introduces a domain mismatch with natural conversation. The fixed 0.64-second silence inserted before agent turns is a hard-coded heuristic that will affect latency in practice and may not generalise to more varied conversational pacing. The evaluation does not include a listening test (MOS via human raters) for duplex conversation quality — UTMOS and GPT score are proxies. The first-response latency metric is not comparable to Moshi because Moshi's proactive interruption behaviour makes the metric inapplicable, which limits direct system comparison." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "interspeech-2025-0989" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS, evaluation] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: speaker_diversity_in_training_data_is_a_stronger role: complicates claim: "Speaker diversity in training data is a stronger driver of zero-shot TTS generalization than audio quality or dataset size alone, as a 10-speaker high-quality dataset fails catastrophically on unseen speakers despite controlled recording conditions." source: "§4.3, Table 3" evidence: "The model trained solely on HiFiTTS (10 speakers) fails badly on unseen speakers (WER 10.38%, SSIM 0.059), confirming that speaker diversity rather than audio quality alone drives zero-shot performance." confidence: high relevance: medium - claim_id: mixed_bandwidth_audio_in_large_scale_speech_corpora role: supports claim: "Mixed-bandwidth audio in large-scale speech corpora degrades codec and vocoder training, making bandwidth estimation and filtering an essential step in high-bandwidth TTS data preparation." source: "§1, §2.3" evidence: "Bandwidth Estimation: Following HiFiTTS methodology, the maximum frequency with ≥-50 dB power relative to peak is estimated from the first 30 seconds of each audiobook." confidence: high relevance: high - claim_id: restoring_punctuation_and_capitalization_to_asr_derived_transcripts role: supports claim: "Restoring punctuation and capitalization to ASR-derived transcripts is feasible at scale via text matching (87% coverage) with neural prediction for remaining cases, and meaningfully improves transcript quality for TTS prosody modeling." source: "§2.1" evidence: "Text Preprocessing: Punctuation and capitalization (PC) are restored either by matching MLS transcripts to original audiobook text (successful for 87%) or by predicting PC using NeMo DistilBERT (remaining 13%)." confidence: high relevance: medium - claim_id: providing_per_utterance_quality_metadata_wer_cer_bandwidth role: supports claim: "Providing per-utterance quality metadata (WER, CER, bandwidth, speaker count) rather than applying fixed thresholds increases dataset utility by allowing downstream researchers to select quality-volume trade-offs appropriate to their application." source: "§2.5, §2.6" evidence: "The model trained solely on HiFiTTS (10 speakers) fails badly on unseen speakers (WER 10.38%, SSIM 0.059), confirming that speaker diversity rather than audio quality alone drives zero-shot performance." confidence: high relevance: medium limitations: - "The dataset is English-only. Audio quality is not filtered by SNR, accepting noise present in LibriVox recordings; this is intentional (modern TTS can handle noise) but may affect some applications. The 44.1 kHz subset's bandwidth ranges from 13–22 kHz (mixed-bandwidth within the subset), which may complicate training for systems requiring uniform bandwidth. No listening-test based MOS evaluation is reported; evaluation relies on automatic metrics (SQUIM-MOS, SSIM). The Koel-TTS codec details are not specified in this paper. Future work on non-English and non-audiobook sources is mentioned but not pursued." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-1084" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: mamba_based_sequence_models_can_match_or_exceed role: supports claim: "Mamba-based sequence models can match or exceed the quality of larger Transformer-based TTS systems while enabling real-time streaming inference on CPU hardware." source: "§4.5, §4.6, Table 1" evidence: "SMAM+MLM (26M params) achieves MOS 4.02 and CER 2.73%, matching Lee et al. (2024) at 263M params (MOS 4.00, CER 4.01%) while reducing first-token latency from 26.5s to 0.065s on a single-threaded CPU." confidence: high relevance: medium - claim_id: iterative_depthwise_refinement_of_rvq_tokens_substantially_improves role: supports claim: "Iterative depthwise refinement of RVQ tokens substantially improves codec TTS quality over single-pass parallel depth prediction." source: "§4.7, Table 1" evidence: "Replacing MLM depthwise decoding with a single-pass no-masking baseline (SMAM+noMLM) causes a significant drop in all quality metrics (MOS from 4.02 to 3.89, CER from 2.73% to 4.12%, UTMOS from 4.13 to 3.83) with negligible change in RTF and latency." confidence: high relevance: high - claim_id: objective_speaker_similarity_metrics_based_on_embedding_cosine role: complicates claim: "Objective speaker similarity metrics based on embedding cosine distance do not reliably predict subjective speaker similarity as judged by human listeners." source: "§4.6, Table 1" evidence: "SMAM+MLM scores SECS 0.816 (below Lee et al.'s 0.863) but achieves higher SMOS of 3.36 vs. 3.27, indicating a divergence between embedding-space distance and perceptual similarity that has practical implications for zero-shot TTS evaluation." confidence: high relevance: medium - claim_id: depthwise_decoding_strategies_for_rvq_present_an_explicit role: supports claim: "Depthwise decoding strategies for RVQ present an explicit quality-speed trade-off that system designers can exploit based on deployment constraints." source: "§3.3, §4.5, §4.6, Table 1" evidence: "SMAM+MLM (iterative, 3 passes) achieves MOS 4.02 and RTF 0.701, while SMAM+INR (single forward pass) achieves MOS 3.97 and RTF 0.568, demonstrating a consistent quality-speed trade-off across both objective and subjective evaluations." confidence: high relevance: medium limitations: - "Evaluation is limited to LibriTTS test-clean (English, read speech), leaving performance on spontaneous speech, noisy environments, and non-English languages uncharacterized. The SECS speaker similarity scores for the proposed models fall below the strongest baseline (Lee et al. 2024), indicating room for improvement in speaker faithfulness despite strong subjective SMOS scores. The paper does not release code, limiting reproducibility and adoption. RTF comparisons are not fully apples-to-apples since baselines generate complete utterances in batch mode while SMAM operates incrementally. Future directions mentioned include a fully streaming pipeline covering codec processing and applying depthwise decoding strategies to decoder-only speech language models." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-1289" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [codec] architecture: [hybrid] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: constant_frame_rate_coding_introduces_temporal_redundancy_in role: supports claim: "Constant-frame-rate coding introduces temporal redundancy in neural speech codecs by allocating equal resolution to silence and phonetically dense regions alike." source: "§1, §2.1" evidence: "Frame rate allocation is determined by non-parametric temporal entropy computed over each resolution's receptive field window." confidence: high relevance: high - claim_id: dynamically_allocating_coarser_temporal_frames_to_low_entropy role: supports claim: "Dynamically allocating coarser temporal frames to low-entropy speech regions reduces token sequence length without proportional degradation in reconstruction quality." source: "§4.2, Table 1" evidence: "Notably, certain mixed-granularity configurations at lower average frame rates (below 75Hz) achieve higher UTMOS scores than the 75Hz DAC+TFC configuration, indicating that forced fine-grained coding of low-entropy regions carries an unnecessary reconstruction cost." confidence: high relevance: high - claim_id: reducing_the_number_of_encoded_frames_at_equivalent role: supports claim: "Reducing the number of encoded frames at equivalent bitrate can improve intelligibility, suggesting that sequence compactness benefits autoregressive downstream models independently of bitrate." source: "§4.2, Table 1" evidence: "At 6kbps, the WER advantage is more pronounced (4.13 → 2.85), suggesting that sequence-length reduction — rather than just bitrate — also benefits intelligibility." confidence: high relevance: high - claim_id: variable_frame_rate_allocation_and_variable_bitrate_control role: supports claim: "Variable frame rate allocation and variable bitrate control are orthogonal axes in neural codec design and can be combined additively." source: "§3, §5" evidence: "TFC is a plug-and-play module built on top of the DAC codec backbone (24kHz, 75Hz frame rate)." confidence: high relevance: high limitations: - "Evaluation is restricted to a single codec backbone (DAC) and a single dataset (LibriTTS). No human listening study is reported; all quality judgements rest on UTMOS, STOI, WER, and spectral distances. Generalisability to other architectures or acoustic conditions is untested." - "The paper does not report downstream TTS or speech LM experiments, so the claimed benefit of shorter token sequences for generation quality and latency is prospective rather than demonstrated. The entropy-based frame allocation heuristic uses fixed hyperparameters (bin count N, smoothing σ) without ablation; sensitivity to these choices is unknown. Training with mixed granularity ratios uses a fixed heuristic allocation that may not be optimal. The granularity ratios at inference must be set by the user; no automatic target-rate optimisation is described." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "interspeech-2025-1538" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [VC] architecture: [autoregressive-LM] relevance: low evidence_role: - historical_context current_role: minor method_family: [unified_speech_text_language_model] claims: - claim_id: conditioning_acoustic_generation_on_explicitly_predicted_text_tokens role: supports claim: "Conditioning acoustic generation on explicitly predicted text tokens reduces intelligibility errors in autoregressive voice conversion relative to purely acoustic-domain approaches." source: "§3.3.3, Table 1" evidence: "Removing text token generation from StarVC raises WER from 6.27% to 7.30% and SECS-WavLM drops from 0.472 to 0.382; StarVC achieves the lowest WER and CER among all compared systems including diffusion-based CosyVoice (8.24%/4.27%)." confidence: high relevance: high - claim_id: multi_stage_training_that_initializes_voice_conversion_with role: supports claim: "Multi-stage training that initializes voice conversion with ASR pretraining improves both content preservation and speaker similarity relative to single-stage training." source: "§3.3.3, Table 1" evidence: "Removing multi-stage training degrades SECS-Res from 0.835 to 0.812 and raises WER from 6.27% to 7.24%; multi-stage training is the single largest contributor in the ablation study." confidence: high relevance: low - claim_id: objective_speaker_embedding_metrics_and_perceptual_speaker_similarity role: complicates claim: "Objective speaker embedding metrics and perceptual speaker similarity ratings can diverge for codec-based voice conversion systems trained with strong linguistic objectives." source: "§3.3.1, §3.3.2, Tables 1-2" evidence: "StarVC scores marginally below CosyVoice on SECS-Res (0.835 vs. 0.839) and SECS-WavLM (0.472 vs. 0.478), yet exceeds CosyVoice on subjective SMOS (3.98 vs. 3.94), suggesting embedding-based metrics underestimate perceived similarity for this system class." confidence: high relevance: high - claim_id: autoregressive_voice_conversion_systems_can_produce_explicit_transcription role: refines claim: "Autoregressive voice conversion systems can produce explicit transcription output alongside converted audio at negligible additional cost, enabling inline content verification without separate ASR inference." source: "§3.3.1, Table 1" evidence: "StarVC generates text tokens with WER-Text of 4.95% and CER-Text of 1.51% as a byproduct of the VC decoding process, providing word-level content verification as part of the conversion pipeline." confidence: high relevance: high limitations: - "Subjective MOS evaluation involves only 20 listeners and 20 source-target pairs, making the reported SMOS and NMOS advantages over CosyVoice and OpenVoice V2 (all within overlapping confidence intervals) difficult to interpret as significant." - "The evaluation covers English only on a single clean corpus (LibriTTS test-clean). Generalization to cross-lingual conversion, noisy conditions, or longer conversational utterances is untested. The three-stage training pipeline requires 180 GPU-hours on 8 H100s, representing a substantial compute cost that may limit practical adoption. Data augmentation relies on OpenVoice V2-synthesized speech, which could propagate artifacts from that system into StarVC's training distribution. Whether the text-before-speech decoding constraint generalizes to expressive or emotional speech conversion, where prosody is not captured by a pure transcription, remains an open question." caveats: - "The connection to autoregressive codec TTS is contextual rather than a direct system contribution." - id: "interspeech-2025-1641" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [alignment_constrained_autoregression] claims: - claim_id: explicit_phoneme_position_supervision_during_autoregressive_codec_training role: supports claim: "Explicit phoneme position supervision during autoregressive codec training eliminates alignment errors more effectively than phoneme identity prediction or monotonic decoding constraints." source: "§4.2.1, Table 2; §4.3, Table 4" evidence: "VALL-E R, the closest prior baseline, reduces deletion errors well (8.37% hard CER) but its greedy decoding variant degrades severely (19.33%), showing that phoneme identity prediction alone does not solve the fundamental alignment ambiguity in repeated-character contexts." confidence: high relevance: high - claim_id: alignment_failures_in_codec_language_model_tts_including role: supports claim: "Alignment failures in codec language model TTS — including phoneme skipping, repetition, and one-to-many correspondence — are fundamentally a training-objective problem rather than an inference-time problem." source: "§4.2.1, §4.3, Table 4" evidence: "An alignment analysis on the hard set confirms that VALL-E makes 1,656 phoneme-skipping errors and 813 repetition errors; the proposed method eliminates all three error categories entirely (zero occurrences of skipping, repetition, and one-to-many alignment errors in Table 4)." confidence: high relevance: high - claim_id: jointly_predicting_phoneme_identity_and_position_introduces_conflicting role: supports claim: "Jointly predicting phoneme identity and position introduces conflicting signals that degrade pronunciation accuracy compared to position-only prediction." source: "§4.2.1, Table 2" evidence: "The ablation VALL-E BOTH (joint identity + position prediction) achieves 6.07% hard CER but introduces more substitution errors than position-only prediction, suggesting that conflicting phoneme identity and position signals interfere with pronunciation accuracy." confidence: high relevance: medium - claim_id: robustness_improvements_in_autoregressive_codec_tts_can_be role: supports claim: "Robustness improvements in autoregressive codec TTS can be achieved without changes to inference-time decoding strategy or additional duration prediction stages." source: "§3.3, §4.2.1, Table 1" evidence: "VALL-E R, the closest prior baseline, reduces deletion errors well (8.37% hard CER) but its greedy decoding variant degrades severely (19.33%), showing that phoneme identity prediction alone does not solve the fundamental alignment ambiguity in repeated-character contexts." confidence: high relevance: high limitations: - "The model is trained and evaluated exclusively on Mandarin using a proprietary G2P toolkit and character-level duration annotations from the WenetSpeech4TTS dataset. Generalisation to languages without character-aligned duration labels, or to datasets where forced-alignment quality is lower, is untested." - "The approach requires phoneme duration annotations at training time to construct the position sequence, which constrains its applicability to datasets with reliable forced alignments. At inference, the enrollment speech prompt must include a duration estimate — the paper does not discuss what happens when this estimate is imprecise." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-1776" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [hybrid] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, semantic_acoustic_token_cascade, masked_parallel_codec_alternatives] claims: - claim_id: multi_task_joint_training_on_synthesis_editing_and role: supports claim: "Multi-task joint training on synthesis, editing, and continuation tasks improves speech synthesis quality over single-task training in non-autoregressive codec models." source: "§3.2, §3.3, Tables 1, 3" evidence: "In all three tokenizer configurations (ST, STDAC, HuDAC), multi-task SpeechSEC consistently outperforms the corresponding single-task baseline on MOS, voice preservation, WER, and CER, with gains confirmed by ablation." confidence: high relevance: high - claim_id: in_multi_task_speech_generation_training_editing_tasks role: refines claim: "In multi-task speech generation training, editing tasks primarily contribute intelligibility improvements while continuation tasks primarily contribute acoustic quality and voice preservation." source: "§3.3, Table 3" evidence: "Ablation removing the editing task increases WER by up to 4.4 points with minimal audio quality change; removing continuation degrades MOS by up to 0.18 and voice preservation by up to 0.06 with smaller intelligibility effects." confidence: high relevance: high - claim_id: non_autoregressive_masked_token_prediction_frameworks_can_unify role: supports claim: "Non-autoregressive masked token prediction frameworks can unify speech synthesis, editing, and continuation tasks through task-specific input conditioning within a single model." source: "§2, §3.2, Table 2" evidence: "SpeechSEC handles all three tasks with a shared Conformer backbone, differentiating tasks via a Task Register embedding and per-task masking strategies, achieving competitive quality on editing (MOS 3.93) and continuation (MOS 3.63) alongside synthesis." confidence: high relevance: high - claim_id: the_choice_of_semantic_and_acoustic_token_extractor role: complicates claim: "The choice of semantic and acoustic token extractor significantly affects absolute synthesis quality in codec-based TTS, even when model architecture and training are held constant." source: "§3.2, Table 1" evidence: "With the same SpeechSEC architecture and training scheme, MOS ranges from 3.65 (STDAC) to 4.20 (ST) across the three tokenizer configurations, and voice preservation from 0.61 to 0.72, indicating that tokenizer quality is a dominant factor." confidence: high relevance: high limitations: - "The evaluation is restricted to LibriTTS-R, a clean studio-quality English corpus, leaving generalization to noisy, spontaneous, or multilingual speech untested. Cross-system comparisons with SoundStorm use independently reported numbers from separate evaluations, weakening the claim of surpassing prior state of the art. Model parameter count is not reported, preventing meaningful comparisons of capacity-normalised performance. Speech continuation lacks intelligibility metrics (WER, CER) by design, limiting interpretability of those results. The paper does not evaluate the editing task on real-world editing scenarios beyond random masking." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "interspeech-2025-1993" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS, VC] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: embedding_watermark_detection_directly_into_codec_encoder_training role: supports claim: "Embedding watermark detection directly into codec encoder training is a viable alternative to post-hoc or hard-coded watermark gates for protecting open-source zero-shot TTS models." source: "§2.2, §3.3.1" evidence: "By modifying the codec encoder's training objective, the paper embeds watermark detection implicitly: when the encoder encounters watermarked audio, it is trained to reconstruct a silent (all-zero) waveform rather than the original signal; for clean audio, reconstruction proceeds normally." confidence: high relevance: high - claim_id: neural_codec_architectures_are_a_natural_intervention_point role: supports claim: "Neural codec architectures are a natural intervention point for access-control in speaker-conditioned TTS because they mediate all speaker information transfer from prompt to synthesis." source: "§1, §2.3" evidence: "The key observation is that virtually all state-of-the-art zero-shot TTS systems route the speaker prompt through a pre-trained codec encoder before passing speaker information to the generative model." confidence: high relevance: high - claim_id: training_time_augmentation_with_common_audio_distortions_substantially role: supports claim: "Training-time augmentation with common audio distortions substantially improves a codec's robustness to watermark removal attacks without degrading reconstruction quality on clean audio." source: "§2.2, Table 1, Table 2" evidence: "To make the codec robust against watermark removal attacks, 13 types of audio distortions (speed adjustment, resampling, echo, noise, low/high/band-pass filters, smoothing, dynamic range changes, and MP3/AAC/EnCodec compression) are applied to training utterances as augmentation." confidence: high relevance: high - claim_id: codec_level_defenses_for_voice_cloning_create_a role: supports claim: "Codec-level defenses for voice cloning create a structural barrier to adaptation attacks because TTS models trained on modified codec distributions cannot be trivially swapped to unprotected codecs without retraining." source: "§2.3, §3.3.2" evidence: "The TTS model is retrained on clean data using the modified codec encoder, meaning that adapting it back to a standard codec would require substantial retraining effort — forming the primary barrier against adaptation attacks." confidence: high relevance: high limitations: - "The defense is effective only against speech watermarked with the specific watermarking model (AudioSeal) used during codec training. A copyrighted voice that is unwatermarked — or that is protected with a different, unseen watermarking system — receives no protection. The attacker simply needs to avoid using an AudioSeal-watermarked prompt." - "The evaluation is conducted only on the VALL-E architecture and EnCodec backbone; whether the approach generalizes to other zero-shot TTS architectures (e.g., flow-matching or diffusion-based systems) is not tested. All data is clean studio speech (LibriSpeech/LibriTTS-R); robustness in noisy or spontaneous speech conditions is unknown. The speed adjustment and low-pass filter attacks are reported as failures for the codec detector, though the authors argue these attacks also degrade clean prompts, partially neutralising them as practical bypass routes. No listening test compares watermark-rejected output against human expectations of what a \"protection failure\" looks and sounds like." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-2328" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS, evaluation] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: statistical_watermarking_techniques_developed_for_text_language_models role: supports claim: "Statistical watermarking techniques developed for text language models can be applied to autoregressive speech generation models by treating speech tokens as text tokens and applying token-distribution reweighting during generation." source: "§4, Table 1" evidence: "KGW and DiPmark are adapted to SpiritLM's HuBERT-based speech token vocabulary with no architectural modification to either system, achieving TPR@1%FPR up to 98.9% on C4." confidence: high relevance: high - claim_id: statistical_watermark_detection_in_autoregressive_speech_systems_is role: complicates claim: "Statistical watermark detection in autoregressive speech systems is degraded by the lossy round-trip of audio decode followed by re-encode, which disrupts the token-level statistical signal." source: "§3.2, §5.2, Table 6" evidence: "On Dolly-CW, KGW (δ=1.0) drops from 99% to 17% TPR@1%FPR after re-encoding; DiP-reweight (α=0.4) drops from 99% to 53%, demonstrating that the mismatch is the dominant failure mode regardless of watermark method." confidence: high relevance: high - claim_id: distortion_free_watermarking_methods_preserve_generated_audio_quality role: supports claim: "Distortion-free watermarking methods preserve generated audio quality at the cost of reduced detection power compared to logit-bias methods." source: "§5.1, Tables 1–4" evidence: "DiP-reweight maintains FAD scores near the no-watermark baseline (0.0067 vs. 0.0051 on C4 at α=0.4) while KGW (δ=2.0) raises FAD to 0.0494; correspondingly, DiP-reweight peaks at 47.5% vs. 98.9% TPR@1%FPR." confidence: high relevance: medium - claim_id: the_strength_of_a_statistical_watermark_signal_before role: complicates claim: "The strength of a statistical watermark signal before audio rendering is not a reliable predictor of detection accuracy in deployed conditions, because re-encoding introduces a new and substantial source of signal loss." source: "§5.2, Table 5" evidence: "Before re-encoding, KGW (δ=1.0) achieves 97.1% TPR@1%FPR on C4; after re-encoding this falls to 54.3%, illustrating that near-perfect pre-decoding detection does not guarantee practical detectability." confidence: high relevance: medium limitations: - "The framework is evaluated only on SpiritLM, which uses HuBERT-based discrete speech units. Applicability to neural codec-based AR speech models (e.g., those using EnCodec or Mimi) is untested, and re-encoded mismatch characteristics may differ substantially across codec types." - "No mitigation for the re-encoded mismatch is proposed; the paper frames this as an open problem without a solution pathway. The evaluation uses Frechet Audio Distance to assess distortion but does not include subjective listening tests, leaving the perceptual impact of the KGW-reweight quality degradation unverified. Robustness against adversarial attacks on the watermark (e.g., re-synthesis, noise injection) is not evaluated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-2447" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: speculative_decoding_adapted_for_speech_can_reduce_autoregressive role: supports claim: "Speculative decoding adapted for speech can reduce autoregressive inference latency without measurable degradation in subjective naturalness or speaker similarity." source: "§4.1, §4.2" evidence: "WER rises from 3.67% to 5.70%, which the authors attribute to draft model training on data roughly 1/300th the size of CosyVoice 2's proprietary training corpus rather than to the speculative decoding mechanism itself — the draft model in isolation reaches 16.13% WER, whereas SSD's verification keeps it at 5.70%." confidence: high relevance: high - claim_id: speech_token_sequences_exhibit_many_to_one_mappings role: supports claim: "Speech token sequences exhibit many-to-one mappings to perceived quality, enabling relaxed acceptance criteria that improve decoding throughput over strict token-distribution matching." source: "§2.2, §4.3" evidence: "The tolerance factor ablation (Table 3) shows that varying β from 0 to 0.4 consistently improves LM-RTF while quality metrics (NMOS, SS) remain stable, suggesting that speech token identity is weakly coupled to perceptual quality — a key property that makes the relaxed acceptance criterion viable." confidence: high relevance: high - claim_id: initialising_a_lightweight_draft_model_from_the_upper role: supports claim: "Initialising a lightweight draft model from the upper layers of the target model provides immediate vocabulary alignment and reduces the data requirements for draft model training." source: "§2.3" evidence: "Initialisation uses the pre-trained upper layers (layers 19–24) of the target, giving immediate vocabulary alignment." confidence: high relevance: medium - claim_id: inference_stage_acceleration_of_autoregressive_tts_is_achievable role: supports claim: "Inference-stage acceleration of autoregressive TTS is achievable without fine-tuning the target model, preserving deployment flexibility for frozen production systems." source: "§2, §4.1" evidence: "This selective fine-tuning is roughly analogous to adapter tuning: the draft model learns speech-specific generation patterns in its trainable lower layers while the frozen upper layers preserve the semantic representations of the full target model." confidence: high relevance: high limitations: - "The WER increase (3.67% → 5.70%) is unexplained beyond a data-scale hypothesis. The draft model's limited training data (LibriTTS, ~580h vs. CosyVoice 2's proprietary corpus) is identified as the likely cause, but this is not verified experimentally — e.g., by scaling draft training data or by ablating with matched data." - "Evaluation is restricted to a single target model (CosyVoice 2) on a single English benchmark. Generalisation of SSD to multilingual systems, streaming inference contexts, or multi-codebook AR models (e.g., VALL-E-style RVQ decoding) is not explored. The tolerance factor β is treated as a fixed hyperparameter tuned on objective metrics; its interaction with speaker diversity and domain shift is unexamined. The reported 1.4× speedup measures LM-RTF only and does not account for the draft model's own compute overhead in the total pipeline time." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "interspeech-2025-2536" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [TTS, evaluation] architecture: [] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: speech_enhancement_and_automated_quality_filtering_are_necessary role: supports claim: "Speech enhancement and automated quality filtering are necessary preprocessing steps for training current-generation TTS systems on in-the-wild audio." source: "§3.3, §5.3, Table 4" evidence: "All four TTS systems trained on TITW-Easy (enhanced, DNSMOS-filtered) converge and produce intelligible speech, while TransformerTTS and MQTTS fail to converge on TITW-Hard (unenhanced); GradTTS-DiffWave UTMOS falls from 2.18 to 1.29 and DNSMOS from 2.39 to 1.47 when switching from Easy to Hard training data." confidence: high relevance: low - claim_id: training_tts_models_directly_on_raw_social_media role: supports claim: "Training TTS models directly on raw social media audio without enhancement remains too unstable for most current model architectures." source: "§3.2, §5.3, Table 4" evidence: "TITW-Hard data (DNSMOS 2.38) caused divergence in two of four tested systems despite heuristic data selection; only VITS and GradTTS-DiffWave converged, and both showed degraded quality relative to their TITW-Easy counterparts." confidence: high relevance: low - claim_id: automated_perceptual_quality_metrics_such_as_utmos_and role: complicates claim: "Automated perceptual quality metrics such as UTMOS and DNSMOS do not consistently predict intelligibility performance at inference time." source: "§5.2, §5.3, Table 2, Table 4" evidence: "TITW-Easy and TITW-Hard training sets differ modestly in WER (9.1% vs. 9.3%), but GradTTS-DiffWave trained on Hard exhibits WER of 26.2% vs. 11.9% for Easy; the DNSMOS gap (2.38 vs. 2.78) understates the intelligibility collapse in the generated speech." confidence: high relevance: low - claim_id: automatic_speaker_recognition_datasets_derived_from_in_the role: supports claim: "Automatic speaker recognition datasets derived from in-the-wild audio provide a scalable and ethically beneficial source for TTS corpus construction." source: "§3, Table 1" evidence: "VoxCeleb1 (1,251 speakers, YouTube-sourced) yields 173–189 hours of TTS-trainable speech through a fully automated pipeline, and its single-speaker-per-segment guarantee enables direct pairing of synthetic and genuine speech for deepfake detection benchmarks." confidence: high relevance: low limitations: - "Table 1 and the paper's introduction text report contradictory sample counts: the introduction states TITW-Hard contains 189 hours and TITW-Easy 173 hours, while Table 1 shows the opposite (Easy 189h, Hard 173h). The Key Results and Claims sections above use Table 1 values, which are consistent with Table 2 quality scores (Easy > Hard). Readers consulting the paper should verify against the released dataset directly." - "The dataset is currently English-only; the authors leave multilingual extension to future work, which limits utility for underrepresented language TTS. Evaluation uses only automated metrics (UTMOS, DNSMOS, MCD, WER) without human listening tests, so the perceptual quality of models trained on TITW relative to studio-trained counterparts is not directly measured. The DNSMOS threshold of 3.0 for TITW-Easy was set empirically and may not generalise to other downstream models or tasks." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "interspeech-2025-2564" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [unified_speech_text_language_model, full_duplex_multistream_generation] claims: - claim_id: large_scale_monolingual_pre_training_followed_by_stereo role: supports claim: "Large-scale monolingual pre-training followed by stereo dialogue fine-tuning enables spoken dialogue models to acquire language-specific conversational behaviors." source: "§5, Table 3" evidence: "J-Moshi, trained on J-CHAT and stereo Japanese dialogue, exhibits more speech overlaps (5.0s/min) and more IPUs (53.2/min) than English Moshi (1.2s overlap, 35.1 IPUs), consistent with Japanese conversational norms." confidence: high relevance: medium - claim_id: synthetic_spoken_dialogue_generated_by_multi_stream_tts role: supports claim: "Synthetic spoken dialogue generated by multi-stream TTS improves language capability in full-duplex dialogue models when added to fine-tuning data." source: "§4.3, Table 2" evidence: "J-Moshi-ext (trained with 602 hours of TTS-synthesized dialogue added) achieves meaningfulness 2.30 versus J-Moshi's 2.19, a statistically distinguishable improvement, with no degradation in naturalness." confidence: high relevance: medium - claim_id: neural_audio_codecs_pre_trained_on_one_language role: complicates claim: "Neural audio codecs pre-trained on one language can transfer to another with minimal acoustic degradation, but the autoregressive language model component requires substantial retraining to achieve acceptable dialogue quality." source: "§4.3, Table 2" evidence: "Mimi re-synthesis of Japanese speech degrades by approximately 0.5 MOS from ground truth, while J-Moshi (with RQ-Transformer adapted) degrades by more than 1 MOS, identifying the language model as the primary quality bottleneck." confidence: high relevance: high - claim_id: morphological_density_differences_across_languages_affect_full_duplex role: complicates claim: "Morphological density differences across languages affect full-duplex dialogue model training dynamics: languages with higher phoneme-to-token ratios produce sparser text-to-audio token alignments that may require adjusted training objectives." source: "§5" evidence: "Japanese data preprocessing results in 88% PAD tokens in text sequences versus 65% for English in Moshi, reflecting that kanji characters encode more phonemes per token, and the authors flag this as a design consideration for future Japanese-specific training objectives." confidence: high relevance: high limitations: - "Both J-Moshi and J-Moshi-ext score above 1 MOS below the Mimi re-synthesis ceiling, indicating the autoregressive RQ-Transformer component is a major quality bottleneck in Japanese. Mimi itself degrades by approximately 0.5 MOS from ground truth when applied to Japanese without adaptation, suggesting codec fine-tuning for Japanese will be necessary for production-quality systems." - "The comparison with English Moshi in Table 3 is not conducted under identical experimental conditions (different test sets, possibly different prompt lengths), so turn-taking statistics should be interpreted as indicative rather than rigorously controlled. The 24.6% overall WER of the TTS-synthesized augmentation data introduces noise, and the effect of this noise on specific error categories is not analyzed. The paper does not evaluate spoken dialogue content quality beyond naturalness and meaningfulness, leaving turn-taking appropriateness and response coherence unmeasured." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "interspeech-2025-2726" published_date: "2025-08-17" entry_date: "2026-07-25" year: 2025 venue: "Interspeech" task: [codec] architecture: [GAN, hybrid] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: staged_codec_training_that_separates_quantizer_optimization_from role: supports claim: "Staged codec training that separates quantizer optimization from decoder optimization can improve single-codebook reconstruction quality beyond joint training." source: "§3.6, Table 3" evidence: "DS-Codec's two-stage framework (mirror Stage 1 to train the quantizer, non-mirror Stage 2 to specialize the decoder) outperforms APCodec+'s single-stage joint training at both stages: Stage 1 UTMOS 4.123 vs. 4.113, PESQ 2.768 vs. 2.632; final model UTMOS 4.214 vs. 4.186 on LibriSpeech." confidence: high relevance: high - claim_id: a_mirrored_encoder_decoder_constraint_during_quantizer_training role: supports claim: "A mirrored encoder-decoder constraint during quantizer training reduces the input-output MSE of the quantization module, producing more robust codebooks." source: "§3.5, Figure 2" evidence: "VQ loss curves during Stage 1 show the mirrored structure achieves lower quantization MSE than the non-mirrored structure across training epochs, even though the non-mirrored structure has lower VQ loss; the paper interprets smaller MSE as higher codebook fidelity." confidence: high relevance: low - claim_id: product_quantization_over_multiple_small_sub_codebooks_enables role: supports claim: "Product quantization over multiple small sub-codebooks enables large effective codebook sizes while preserving the single-token-per-frame interface required by LLM-based TTS." source: "§2.2.2, Table 1" evidence: "DS-Codec-PQ combines four 16-code VQ modules to produce a 65,536-code effective codebook, indexed as a single integer, achieving UTMOS 4.214 and PESQ 2.882 at 1.28kbps with 80 tokens/second." confidence: high relevance: high - claim_id: a_stronger_decoder_does_not_straightforwardly_compensate_for role: complicates claim: "A stronger decoder does not straightforwardly compensate for a weaker quantizer when both are trained jointly." source: "§3.6, Table 3" evidence: "APCodec+'s joint training with a non-mirrored (stronger) decoder in Stage 1 yields lower Stage 1 PESQ (2.632) than DS-Codec's mirror Stage 1 with a weaker mirrored decoder (PESQ 2.768), suggesting quantizer quality dominates reconstruction fidelity early in training." confidence: high relevance: low limitations: - "Evaluation is limited to English read speech (LibriSpeech) and one supplementary in-domain set (LJSpeech). Performance on noisy, spontaneous, or multilingual speech is untested. Model size is not reported, making it impossible to assess parameter efficiency relative to BigCodec (159M) or DAC (74M). The UTMOS and PESQ metrics used are objective proxies for perceptual quality; no formal subjective listening study is reported." - "The Stage 2 improvement from retaining decoder weights vs. reinitializing them is stated but not ablated directly — the comparison to APCodec+ involves multiple differences (stage design, weight retention, architecture), so the isolated effect of weight retention is unclear." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2508.15827" published_date: "2025-08-18" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [unified_speech_text_language_model] claims: - claim_id: in_speech_models_reasoning_depth_and_response_latency role: supports claim: "In speech models, reasoning depth and response latency are not fundamentally in conflict when the model's token generation rate substantially exceeds the real-time audio playback rate." source: "§1, §3.1" evidence: "On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model's 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale." confidence: high relevance: high - claim_id: interleaving_silent_reasoning_tokens_with_spoken_response_tokens role: supports claim: "Interleaving silent reasoning tokens with spoken response tokens at a fixed ratio can improve accuracy on structured reasoning tasks while reducing audible output length." source: "§2.3, §4.4, Table 2" evidence: "On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model's 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale." confidence: high relevance: high - claim_id: the_thinking_before_speaking_paradigm_when_applied_directly role: supports claim: "The \"thinking-before-speaking\" paradigm, when applied directly to speech, produces user-facing latency or verbosity that impairs conversational quality independently of reasoning correctness." source: "§1, §2.2" evidence: "On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves: - Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%). - Contextual Reasoning: 68.1% average, up from the base model's 64.0% (+4.1%), competitive with the 7B model (75.1%). - Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output. - First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale." confidence: high relevance: medium - claim_id: multi_stage_training_separating_modality_alignment_reasoning_transfer role: supports claim: "Multi-stage training — separating modality alignment, reasoning transfer, and acoustic synthesis — is an effective strategy for progressively adapting an existing speech LLM to a new generation paradigm." source: "§3.3" evidence: "Training proceeds in five stages: (1) audio adapter alignment with the language backbone; (2) mixed mathematical pretraining in thinking-before-speaking format; (3) textual thinking-in-speaking training (language model parameters only); (4) acoustic transfer — audio encoder fine-tuning with the frozen LLM; (5) Talker training with the entire Thinker frozen." confidence: high relevance: medium - claim_id: synthetic_speech_based_mathematical_reasoning_datasets_constructed_from role: supports claim: "Synthetic speech-based mathematical reasoning datasets constructed from text corpora via TTS can provide sufficient training signal for spoken reasoning capabilities." source: "§3.2, §4.4" evidence: "To create training data for interleaved reasoning, the authors introduce SPOKEN-MATH-PROBLEMS-3M, a 3M-sample dataset constructed from public text-based maths QA corpora (GSM8K, Orca-Math, MetaMath, SimpleOP)." confidence: high relevance: medium limitations: - "The entire evaluation uses a single benchmark (Spoken-MQA) focused on mathematics. There is no assessment of speech naturalness, intelligibility, or reasoning accuracy on open-domain conversational tasks. Reported latency claims refer to the absence of a pre-speech reasoning phase rather than to measured real-time performance metrics." - "The fixed 2:8 interleaving ratio is derived from a throughput estimate for a specific GPU configuration; it is not adaptive and may be suboptimal for different deployment environments or model sizes. The training data is entirely synthetic — both the audio (produced by CosyVoice2-0.5B) and the reasoning traces (constructed algorithmically from text datasets). Whether the model generalises to naturalistic spoken queries beyond maths problems is untested. The GPT-based verification stage for dataset quality introduces a dependency on a proprietary closed model that is not reproducible." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2508.15442" published_date: "2025-08-21" entry_date: "2026-07-25" year: 2025 venue: "EMNLP" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [preference_optimized_autoregression] claims: - claim_id: token_level_entropy_is_a_statistically_significant_predictor role: supports claim: "Token-level entropy is a statistically significant predictor of hallucination in autoregressive LM-based TTS, enabling unsupervised detection of likely error regions before synthesis is complete." source: "§2.1, §2.2" evidence: "Uncertainty is measured via token-level entropy aggregated at character and utterance levels, adapting uncertainty-based hallucination detection from NLP to the speech modality." confidence: high relevance: high - claim_id: reformulating_autoregressive_speech_token_generation_as_trajectory_flow role: supports claim: "Reformulating autoregressive speech token generation as trajectory flow optimization via GFlowNets enables reward-proportional sampling that reduces hallucination without modifying the inference procedure." source: "§3.1, §3.2" evidence: "GOAT treats the autoregressive generation of speech tokens as a trajectory flow optimization problem." confidence: high relevance: high - claim_id: an_intrinsic_reward_derived_from_the_backbone_lm role: supports claim: "An intrinsic reward derived from the backbone LM's own probability distribution is sufficient to drive distribution alignment in post-training, without requiring human annotations or an external reward model." source: "§3.2.2" evidence: "Rather than relying on an external reward model or labeled data, GOAT uses an intrinsic reward: the cumulative log-probability of the generated token sequence under the backbone LM, sharpened by an inverse temperature T (0 < T < 1)." confidence: high relevance: medium - claim_id: fine_grained_subtrajectory_optimization_is_essential_for_hallucination role: supports claim: "Fine-grained subtrajectory optimization is essential for hallucination suppression in long and complex utterances; degrading to trajectory-level balance significantly reduces the error reduction." source: "§4.4.5, Table 2" evidence: "Ablation (Table 2): Enhanced SubTB reduces test-hard CER from 13.72% to 6.61%; degrading to plain Trajectory Balance yields 11.72–11.84%, demonstrating that fine-grained subtrajectory optimization is essential." confidence: high relevance: medium - claim_id: post_training_alignment_for_hallucination_suppression_in_one role: supports claim: "Post-training alignment for hallucination suppression in one language generalizes to unseen languages, with cross-lingual models retaining meaningful gains over the untuned baseline." source: "§4.4.4, Table 1" evidence: "Cross-lingual generalization holds: models trained on one language and evaluated on the other still outperform the baseline." confidence: high relevance: medium limitations: - "- Evaluated on a single backbone (CosyVoice 2); generalization to other LM-based TTS architectures (VALL-E, BASE TTS, LLaSA) is asserted but not demonstrated. - The uncertainty-hallucination correlation explains only part of hallucination causes (correlation 0.636–0.649); prosody errors show no consistent uncertainty pattern and are not effectively addressed. - The internal reward assumes the backbone LM assigns higher probability to higher-quality sequences — an assumption the authors acknowledge is a hypothesis. - Training still requires H100 GPU access (~70 GB VRAM per GPU), which is non-trivial despite the small data requirement. - Reward temperature is a sensitive hyperparameter; the sweet spot (0.825) was found empirically and may differ for other backbones or domains. - No subjective listening evaluation (MOS/MUSHRA) is reported; quality improvements are measured entirely through ASR-based error rates and the automatic UTMOS predictor." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.16332" published_date: "2025-08-22" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, VC, singing] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: joint_pre_training_on_speech_and_singing_voice role: supports claim: "Joint pre-training on speech and singing voice data improves generation quality in both domains, with singing data contributing to expressive speech prosody richness and speech data compensating for the limited size of singing corpora." source: "§V-A, Table I" evidence: "On singing voice, Vevo2 substantially outperforms zero-shot TTS baselines in N-CMOS (gap > 1.5) and SS-CMOS (gap > 0.9), validating the benefit of joint training." confidence: high relevance: medium - claim_id: chromagram_based_prosody_tokenization_provides_a_notation_free role: supports claim: "Chromagram-based prosody tokenization provides a notation-free, octave-invariant representation that bridges the F0 distribution gap between speech and singing, enabling unified prosody control without expert MIDI annotations." source: "§III-A" evidence: "Two custom VQ-VAE tokenizers are introduced: - Prosody tokenizer (6.25 Hz, 56.25 bps, codebook 512): encodes chromagram features, which are octave-free (bridging the F0 distribution gap between speech and singing) and notation-free (extractable without MIDI annotation)." confidence: high relevance: high - claim_id: autoregressive_models_initialized_from_llm_checkpoints_can_achieve role: supports claim: "Autoregressive models initialized from LLM checkpoints can achieve competitive zero-shot TTS quality at low codec frame rates (12.5 Hz) when paired with strong post-training, partially compensating for the sequence granularity disadvantage relative to higher frame-rate systems." source: "§V-A, Table VII" evidence: "On the SeedTTS benchmark (regular speech), post-trained Vevo2 achieves WER 3.64 (en) / 2.94 (zh) and SIM-o 0.693 / 0.754, competitive with CosyVoice 2 and MaskGCT despite using a lower frame-rate (12.5 Hz vs 25–50 Hz) content-style tokenizer." confidence: high relevance: high - claim_id: single_objective_preference_alignment_in_multi_capability_speech role: supports claim: "Single-objective preference alignment in multi-capability speech models degrades complementary objectives; joint multi-objective optimization is necessary to avoid quality regressions on non-targeted capabilities." source: "§V-F, Figure 5" evidence: "Single-objective optimization degrades the complementary objective (intelligibility-only training drops melody accuracy from 65% to 50%); joint optimization achieves both gains." confidence: high relevance: medium - claim_id: inference_time_pitch_shift_applied_to_prosody_tokens role: complicates claim: "Inference-time pitch shift applied to prosody tokens improves speaker similarity in voice and singing conversion tasks, at the cost of a minor intelligibility degradation due to train-inference distribution mismatch." source: "§V-E, Table VI" evidence: "Inference-time controllability: The system flexibly recombines text, prosody source, style reference, and timbre reference to perform TTS, VC, SVS, SVC, speech/singing editing, duration control (97%+ accuracy via chromagram length scaling), and pitch region control (via F0 shift before prosody token extraction)." confidence: high relevance: high limitations: - "- Training-inference mismatch when applying pitch shift: content-style tokens from pitch-shifted audio are out-of-distribution, causing slight WER degradation. The paper flags future work to include pitch shift augmentation during training. - Singing voice data used (7K hours source-separated from in-the-wild songs) may have variable quality. - The 12.5 Hz frame rate of the content-style tokenizer slightly limits VC intelligibility compared to Vevo-FM (50 Hz), as shown in Table VIII. - Melody-MOS for Vevo2 in SVC is slightly lower than the FM-only Vevo2-FM, suggesting a quality-style tradeoff when adding text input. - Generalization to tonal languages in singing is not specifically evaluated." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.16790" published_date: "2025-08-22" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec, TTS] architecture: [diffusion, transformer-enc-dec, autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [codec_tokenizer_infrastructure, hybrid_ar_token_continuous_renderer] claims: - claim_id: text_conditioning_in_the_codec_decoder_rather_than role: supports claim: "Text conditioning in the codec decoder, rather than in the language model alone, is a viable lever for achieving extreme compression rates in speech tokenization without adversarial training." source: "§3.1, Table 4" evidence: "In speech LM contexts — TTS, spoken dialogue — the target text is always available, so conditioning on it effectively offloads content information from the compressed token to the conditioning signal, enabling much more aggressive compression." confidence: high relevance: high - claim_id: a_single_end_to_end_training_objective_flow role: supports claim: "A single end-to-end training objective (flow-matching loss) is sufficient to jointly optimise quantization and reconstruction in a speech codec, eliminating the need for multi-stage pipelines." source: "§3.1, §4.2.2" evidence: "TaDiCodec reframes the codec problem as an end-to-end diffusion autoencoder trained with a single flow-matching loss." confidence: high relevance: high - claim_id: the_reconstruction_generation_gap_the_degradation_in_intelligibility role: supports claim: "The reconstruction-generation gap — the degradation in intelligibility when tokens are predicted by a language model rather than encoding reference speech — varies substantially across tokenizer architectures and is not captured by reconstruction metrics alone." source: "§4.3, Figure 3" evidence: "The low token rate (6.25 Hz vs. 25–50 Hz for typical AR TTS) means each second of speech is predicted as only 6–7 tokens, drastically shortening the generation sequences." confidence: high relevance: high - claim_id: lower_token_rates_in_speech_tokenizers_can_improve role: supports claim: "Lower token rates in speech tokenizers can improve autoregressive TTS intelligibility by shortening prediction sequences and reducing error accumulation, particularly on linguistically challenging inputs." source: "§4.3, Table 5" evidence: "The low token rate (6.25 Hz vs. 25–50 Hz for typical AR TTS) means each second of speech is predicted as only 6–7 tokens, drastically shortening the generation sequences." confidence: high relevance: high - claim_id: binary_spherical_quantization_without_a_commitment_loss_achieves role: supports claim: "Binary Spherical Quantization without a commitment loss achieves stable end-to-end training of a speech codec and produces superior representations to standard VQ under equal codebook sizes." source: "§4.2.2, Table 4" evidence: "The encoder is an 8-layer Llama-style bidirectional Transformer operating on mel-spectrogram frames; its output is downsampled and projected to a 14-dimensional latent, then quantized using Binary Spherical Quantization (BSQ) — an implicit-codebook scheme that projects encoder embeddings onto a unit hypersphere and binarises each dimension, yielding a codebook of size 2¹⁴ = 16,384 tokens without a learnable codebook or commitment loss." confidence: high relevance: high limitations: - "TaDiCodec's text-aware decoder is not a general-purpose audio codec: it requires a transcript at both training and inference time. The strong reconstruction and TTS results are conditional on text availability; performance at 6.25 Hz without text conditioning is not competitive (WER exceeds 10% at 12.5 Hz without text, per Table 4). This limits applicability to codec-transmission, speech enhancement, or any scenario where transcriptions are unavailable." - "The diffusion decoder introduces multi-step inference latency. At 32 steps, decoding speed is acceptable for generation but higher than GAN vocoders; reducing to 5 steps degrades quality noticeably. The authors propose distillation as future work but have not yet demonstrated single-step performance." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.19098" published_date: "2025-08-26" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, VAE] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: replacing_discrete_neural_codec_tokens_with_continuous_vae role: supports claim: "Replacing discrete neural codec tokens with continuous VAE latents in an autoregressive TTS framework reduces the number of AR decoding steps and lowers inference RTF without sacrificing speech quality." source: "§5.1, Table 2" evidence: "On LibriSpeech-PC test-clean Subset-B (1,127 samples, F5-TTS evaluation protocol): - CLEAR-Large: WER 1.88%, SPK-SIM 0.59, UTMOS 4.22 - CLEAR-Base: WER 2.21%, SPK-SIM 0.59, UTMOS 4.22 - Compared to DiTAR (600M, Emilia-100k): CLEAR-Large achieves 0.51% absolute WER reduction (21.3% relative) at lower parameter count and with far fewer AR decoding steps On LibriSpeech test-clean Subset-A (40 samples, NaturalSpeech3 protocol): - CLEAR-Large: WER 1.74%, SPK-SIM 0.56, UTMOS 4.26 - CLEAR-Base: WER 1.83%, SPK-SIM 0.55, UTMOS 4.21 - Outperforms VALL-E (WER 6.11%), MegaTTS (WER 2.32%), NaturalSpeech2 (WER 1.94%) Inference efficiency (Table 2, 10-second generation on single GPU): - CLEAR-Base: RTF 0.18, avg. 78 AR decoding steps (vs. VALL-E 750 steps / RTF 1.03; MELLE 620 steps / RTF 0.55; F5-TTS RTF 0.31) - CLEAR-Large: RTF 0.29 Subjective evaluation on Subset-B vs. F5-TTS and CosyVoice-2: - CLEAR-Base: N-MOS 4.09, Q-MOS 4.14, S-MOS 4.02, CMOS +0.04 (outperforms ground truth on N-MOS and Q-MOS) Streaming synthesis (Table 4, chunk size 4): first-frame latency 96ms, WER 2.34%, UTMOS 4.27 — largely matching non-streaming performance." confidence: high relevance: high - claim_id: a_per_token_mlp_rectified_flow_head_that role: supports claim: "A per-token MLP rectified flow head that operates independently on each AR hidden state enables streaming speech synthesis with latency competitive with non-streaming systems." source: "§5.3, Table 4" evidence: "Importantly, the MLP operates independently on each conditioning vector, so denoising can start as soon as the first AR hidden state is available — enabling streaming without the full-sequence wait imposed by DiT-based heads." confidence: high relevance: high - claim_id: high_compression_ratio_vae_latents_2048x_downsampling_can role: supports claim: "High-compression-ratio VAE latents (2048x downsampling) can achieve reconstruction quality comparable to ground truth on standard benchmarks while producing sequences short enough to benefit AR sequence modeling." source: "§3.4, Appendix C.1" evidence: "To maintain reconstruction quality under this aggressive compression, parameter-free shortcut connections perform space-to-channel reshaping with channel averaging in each downsampling block and the reverse in each upsampling block." confidence: high relevance: medium - claim_id: continuous_ar_tts_systems_trained_with_a_joint role: supports claim: "Continuous AR TTS systems trained with a joint language model and flow-matching head in a single stage can match or exceed the WER of two-stage cascaded discrete AR systems trained on comparable data." source: "§5.1, Table 1" evidence: "The system has three components trained end-to-end in a single stage: Wav-VAE." confidence: high relevance: high - claim_id: objective_speaker_similarity_metrics_underestimate_the_perceptual_voice role: supports claim: "Objective speaker similarity metrics underestimate the perceptual voice fidelity of AR systems that rely entirely on in-context learning from an audio prompt, as shown by divergence between WavLM-TDNN scores and subjective S-MOS ratings." source: "§5.2, Table 3, Appendix D.2" evidence: "For zero-shot synthesis, a short audio prompt and its transcription are prepended as context: `[S, x_prompt, x_target, T, y_prompt]`; the model then autoregressively completes y_target." confidence: high relevance: medium limitations: - "The paper explicitly acknowledges: (1) objective speaker similarity (WavLM-TDNN) lags behind models that inject speaker embeddings directly into the generator (e.g., CosyVoice, NaturalSpeech3 with FACodec); CLEAR relies entirely on in-context learning from the audio prompt with no explicit speaker conditioning, which is a design choice that limits speaker fidelity under high compression. (2) Evaluation is English-only; multilingual generalization is untested. (3) Sequential single-token AR generation; multi-token prediction could improve throughput further. Additionally, training data for VAE (LibriTTS, 585h) is much smaller than TTS training data (LibriHeavy, 50kh), and the interaction between VAE compression ratio and downstream TTS quality could be investigated more rigorously in full-scale settings." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2508.19205" published_date: "2025-08-26" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [hybrid] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: influential method_family: [semantic_acoustic_token_cascade] claims: - claim_id: extreme_acoustic_codec_compression_single_codebook_vae_at role: supports claim: "Extreme acoustic codec compression (single-codebook VAE at 7.5 Hz) can achieve superior perceptual quality over multi-codebook discrete codecs operating at much higher frame rates." source: "§3.3, Table 3" evidence: "The acoustic tokenizer comparison (Table 3) shows that at 7.5 Hz — 40× fewer tokens than WavTokenizer at 40 Hz — the proposed tokenizer achieves PESQ 3.068 and UTMOS 4.181 on LibriTTS test-clean, beating all discrete codec baselines including DAC (UTMOS 3.433) and WavTokenizer 75 Hz (UTMOS 4.049)." confidence: high relevance: high - claim_id: long_form_multi_speaker_tts_benefits_from_separate role: supports claim: "Long-form multi-speaker TTS benefits from separate acoustic and semantic tokenizers trained with task-specific objectives rather than a single unified codec." source: "§2.1" evidence: "A separate semantic tokenizer mirrors the encoder architecture but is trained via an ASR proxy task, producing content-aligned representations; this component is discarded after pre-training and used only to provide hybrid acoustic-semantic context tokens during generation." confidence: high relevance: high - claim_id: scaling_the_llm_backbone_in_a_next_token role: supports claim: "Scaling the LLM backbone in a next-token diffusion speech system yields consistent gains in perceptual quality, speaker similarity, and expressiveness." source: "§3.1, Table 1" evidence: "VibeVoice (paper name: VIBEVOICE) resolves the context-length bottleneck through two innovations that work in tandem: an ultra-compressed acoustic tokenizer and a next-token diffusion generation framework." confidence: high relevance: high - claim_id: token_level_diffusion_conditioned_on_llm_hidden_states role: supports claim: "Token-level diffusion conditioned on LLM hidden states enables streaming speech generation without the codebook constraints of discrete autoregressive systems." source: "§2.2" evidence: "The LLM processes this hybrid context and produces hidden states that condition a lightweight token-level diffusion head (4 transformer layers) adapted from the MAR image generation framework." confidence: high relevance: high - claim_id: tts_systems_optimised_for_long_form_conversational_content role: supports claim: "TTS systems optimised for long-form conversational content retain competitive performance on short-utterance benchmarks without dedicated fine-tuning." source: "§3.2, Table 2" evidence: "On the SEED short-utterance benchmark (CommonVoice English/Chinese), VibeVoice-1.5B achieves CER of 1.16% and SIM of 0.744 for Chinese, and WER of 3.04% and SIM of 0.689 for English, despite being primarily trained for long-form content." confidence: high relevance: low limitations: - "Training data is not disclosed. The paper is from Microsoft Research but does not specify the data composition, size, or any cleaning procedures, making it impossible to assess whether the reported gains are attributable to architecture or data advantage." - "The model is limited to English and Chinese; other languages produce unpredictable outputs. The system does not model overlapping speech — a significant gap for realistic conversational audio. The subjective evaluation used only 8 long-form test conversations, which is a narrow sample; standard benchmark evaluations (SEED) are short-utterance only and do not capture the long-form quality the paper targets. Speaker similarity at 7.5 Hz remains below the best short-utterance systems (e.g., Seed-TTS at 0.762 SIM for English), suggesting the compressed representation sacrifices some speaker identity fidelity. The maximum of 4 speakers is a hard constraint imposed by the context design." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2509.00503" published_date: "2025-08-30" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [VC, codec] architecture: [autoregressive-LM, transformer-enc-dec] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: adaptive_entropy_based_segmentation_of_discrete_speech_tokens role: supports claim: "Adaptive entropy-based segmentation of discrete speech tokens preserves more task-relevant linguistic information than fixed-length downsampling at equivalent compression ratios." source: "§6.1, Table 3; §6.2, Table 4" evidence: "The 7 Hz high-compression setting still outperforms fixed-length pooling at 12.5 Hz on ASR (WER 10.5% vs 6.8%), demonstrating that adaptive boundaries preserve more linguistic structure than uniform windows at the same compression level." confidence: high relevance: high - claim_id: optimal_token_granularity_differs_systematically_between_understanding_and role: supports claim: "Optimal token granularity differs systematically between understanding and generation tasks: recognition-oriented tasks (ASR, ST) benefit from moderate compression near phoneme rate, while voice conversion requires finer token density to maintain acoustic fidelity." source: "§5.1, Table 1; §6.3, Table 5" evidence: "This confirms that generation tasks are less tolerant of compression than understanding tasks." confidence: high relevance: high - claim_id: ssl_derived_semantic_tokens_at_standard_rates_50 role: supports claim: "SSL-derived semantic tokens at standard rates (50 Hz) contain substantial redundancy that can be removed without degrading — and occasionally improving — downstream task performance." source: "§6.1, Table 3" evidence: "The paper reports evidence that sSL-derived semantic tokens at standard rates (50 Hz) contain substantial redundancy that can be removed without degrading — and occasionally improving — downstream task performance." confidence: high relevance: high - claim_id: entropy_boundaries_in_compressed_token_sequences_align_with role: supports claim: "Entropy boundaries in compressed token sequences align with linguistically meaningful units: 15 Hz compression achieves 83.2% phoneme boundary alignment, while 7 Hz aligns primarily with word boundaries (89.7%)." source: "Appendix C.3, Table 11" evidence: "At 15 Hz (moderate compression), the entropy-guided method achieves WER 5.6% and CER 2.9% on LibriSpeech test-clean — outperforming both HuBERT original (50 Hz, WER 6.2%) and HuBERT deduplicated (26 Hz, WER 5.9%), while reducing sequence length by 70%." confidence: high relevance: high limitations: - "Voice conversion quality degrades noticeably with compression: entropy-guided 15 Hz already falls below HuBERT deduplicated (Q-MOS 3.85 vs 4.12), and the gap widens at higher compression. For generation tasks, this framework does not improve on simply using deduplicated tokens — only for understanding tasks does compression help." - "The framework is evaluated exclusively on HuBERT-derived tokens; whether the entropy-based approach transfers to other SSL features (WavLM, w2v-BERT) or supervised tokenizers (S3, FACodec) is untested. The entropy LLM requires pre-training on 20k hours of speech, adding a pipeline step beyond vanilla k-means clustering. The method is tested on English only, and its behaviour on morphologically complex or tonal languages — where token redundancy patterns may differ — remains unknown. All evaluations use the English MLS training corpus, so generalisation across domains and recording conditions is unexplored." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.00685" published_date: "2025-08-31" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: optimizing_lm_based_tts_with_preference_data_across role: supports claim: "Optimizing LM-based TTS with preference data across multiple quality dimensions simultaneously is more effective than per-metric or naively combined ranking approaches." source: "§4.5, Table 2" evidence: "Objective metrics (Table 2): - MPO achieves CER 3.9, SPK-SIM 0.577, prosody (log F0 RMSE) 0.279 vs. SFT baseline CER 4.72, SPK-SIM 0.548, prosody 0.337. - A model trained on combined rankings (naive merge of three per-metric ranks) reaches CER 4.3, SPK-SIM 0.564, prosody 0.218—better on prosody but worse on intelligibility and speaker similarity than MPO. - Single-metric DPO variants confirm that optimizing for one dimension improves that metric but degrades others." confidence: high relevance: medium - claim_id: vanilla_dpo_applied_to_lm_based_tts_without role: supports claim: "Vanilla DPO applied to LM-based TTS without regularization causes progressive degradation of generation capability, with CER rising sharply after 10k training steps." source: "§4.4, Table 1" evidence: "Regularization ablation (Table 1): Without the CE constraint, CER worsens from 4.72 (SFT) to 6.41 (10k steps) and 14.52 (15k steps)." confidence: high relevance: medium - claim_id: adding_a_cross_entropy_loss_on_preferred_sequences role: supports claim: "Adding a cross-entropy loss on preferred sequences as a regularizer during DPO training prevents model collapse without sacrificing alignment gains." source: "§3.2, §4.4" evidence: "The combined training loss is L = λ·L_DPO + L_CE, where L_CE is the standard cross-entropy objective on the preferred sequences." confidence: high relevance: medium - claim_id: multi_dimensional_preference_alignment_in_lm_based_tts role: supports claim: "Multi-dimensional preference alignment in LM-based TTS produces output quality comparable to ground truth in subjective pairwise preference tests." source: "§4.5, Figure 3" evidence: "Subjective ABX test (Figure 3): MPO is preferred over the SFT baseline in 52.3% of comparisons (22.4% ties); MPO beats combined-ranking in 40.2% vs. 30.7% (29.1% ties); MPO is competitive with ground truth in pairwise preference." confidence: high relevance: medium limitations: - "- Evaluation is entirely on an internal Mandarin TTS corpus; there is no open benchmark comparison, making it difficult to assess absolute quality relative to published systems. - The model size is not disclosed and the base LM architecture details are partially withheld (internal codec), limiting reproducibility. - The 100-hour preference optimization corpus is Mandarin only; whether the approach generalizes to multilingual or cross-lingual zero-shot scenarios is untested. - The preference set construction requires running three separate quality scorers (ASR, speaker verification, F0) over 10 generated candidates per input—a non-trivial computational overhead not quantified in the paper. - The optimal value of λ (set to 10) and the preference margin thresholds (0.1 for SPK-SIM and prosody) are not ablated; sensitivity to these hyperparameters is unknown. - Comparison against SpeechAlign, UNO, and RIO is discussed only qualitatively; a controlled quantitative comparison against these direct baselines would strengthen the claims." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.02020" published_date: "2025-09-02" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: influential method_family: [unified_speech_text_language_model] claims: - claim_id: reducing_speech_tokenizer_frame_rate_to_12_5hz role: supports claim: "Reducing speech tokenizer frame rate to 12.5Hz with explicit semantic supervision produces tokens that enable more stable text-to-token modelling over long dialogue sequences than higher-rate tokenizers without semantic injection." source: "§2.1, §4.1, Table 1" evidence: "Crucially, the quantised features are passed through a semantic decoder that predicts the original Whisper features (explicit semantic supervision), stabilising text-to-token modelling over long sequences." confidence: high relevance: high - claim_id: a_dual_transformer_architecture_for_multi_layer_rvq role: supports claim: "A dual-transformer architecture for multi-layer RVQ prediction achieves lower first-packet latency than the delay-pattern while providing stronger contextual conditioning from prior turns." source: "§2.2" evidence: "This dual-transformer avoids the delay-pattern's drawbacks (weak contextual conditioning, high first-packet latency) and achieves under 100ms first-packet latency." confidence: high relevance: medium - claim_id: autoregressive_tts_systems_trained_on_multi_speaker_dialogue role: supports claim: "Autoregressive TTS systems trained on multi-speaker dialogue data with text-speech interleaved formatting can infer and adjust prosody and emotion from implicit conversational context without explicit emotion labels." source: "§3.2, §4.3, Table 3" evidence: "Emotion inference (interactive chat): 76.7–93.3% accuracy across 6 emotions (surprise, sadness, happiness, concern, apology, anger) inferred from implicit contextual cues without explicit emotion labels." confidence: high relevance: high - claim_id: sentence_by_sentence_multi_speaker_dialogue_tts_systems role: supports claim: "Sentence-by-sentence multi-speaker dialogue TTS systems produce more coherent prosody across turns than approaches that concatenate monologue TTS outputs or model a mixed audio track." source: "§4.4, Table 4" evidence: "Three-stage curriculum training: pretraining on 1.1M hours monologue, post-training on 300k hours 2–5 speaker dialogue data, then task-specific SFT (eg. 15h emotion corpus for chat, 50h podcast hosts for podcast fine-tuning)." confidence: high relevance: medium - claim_id: fine_tuning_a_post_trained_dialogue_tts_model role: supports claim: "Fine-tuning a post-trained dialogue TTS model on as little as 50 hours of speaker-specific data is sufficient to produce synthesis that is perceptually indistinguishable from human recordings in a majority of trials." source: "§4.4, Figure 4" evidence: "Three-stage curriculum training: pretraining on 1.1M hours monologue, post-training on 300k hours 2–5 speaker dialogue data, then task-specific SFT (eg. 15h emotion corpus for chat, 50h podcast hosts for podcast fine-tuning)." confidence: high relevance: medium limitations: - "- Currently limited to 3-minute dialogues with up to 4 speakers; scaling requires extending training corpus. - English speaker similarity (SIM 0.665) lags Mandarin (0.736), attributed to limited English voice diversity in training data — a data rather than architectural limitation. - Trails Mimi on PESQ metrics, likely because Mimi was trained on a massive English-only corpus closely matching LibriSpeech. - Emotion fine-tuning is demonstrated for a single distinctive female voice; generalisation to arbitrary voices and more nuanced emotional transitions is not evaluated. - No ablation of the semantic supervision contribution vs. the lower frame rate independently." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2502.11128" published_date: "2025-09-03" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: autoregressive_context_from_the_preceding_mel_spectrogram_frame role: supports claim: "Autoregressive context from the preceding mel-spectrogram frame provides a more informative prior for per-token flow matching, enabling high-quality synthesis with fewer function evaluations than a standard Gaussian prior." source: "§4.2, §6.2, Table 3" evidence: "Rather than using a standard Gaussian prior, FELLE initialises the flow-matching ODE for frame i from the previous predicted frame x_{i-1} via a Gaussian centred at that frame (variance sigma^2 = 0.1)." confidence: high relevance: high - claim_id: hierarchical_coarse_to_fine_spectral_decomposition_within_each role: supports claim: "Hierarchical coarse-to-fine spectral decomposition within each flow-matching step improves speaker similarity over holistic or fully decoupled generation approaches in continuous-token autoregressive TTS." source: "§4.2, §6.2, Table 3" evidence: "Token-wise coarse-to-fine flow matching (C2F-FM)." confidence: high relevance: high - claim_id: continuous_mel_spectrogram_autoregressive_models_can_match_discrete role: supports claim: "Continuous mel-spectrogram autoregressive models can match discrete-token systems on intelligibility while substantially surpassing them on speaker similarity in zero-shot TTS." source: "§6.1, Table 2" evidence: "Evaluations on LibriSpeech test-clean use two zero-shot schemes: continuation (same utterance, first 3 s as prompt) and cross-sentence (different utterance from the same speaker as prompt)." confidence: high relevance: high - claim_id: the_number_of_flow_matching_function_evaluations_introduces role: complicates claim: "The number of flow-matching function evaluations introduces a trade-off in autoregressive TTS: moderate NFE improves intelligibility but excessive NFE degrades both intelligibility and speaker similarity." source: "§6.3, Figure 3" evidence: "Ablations confirm all three novel components contribute: removing the previous-frame prior degrades both WER and SIM; replacing C2F-FM with holistic (HFM) or decoupled (DFM) flow matching degrades one or both metrics." confidence: high relevance: high limitations: - "- The model operates at 16 kHz with 80-band mel features; extension to 24 kHz or higher fidelity representations is not addressed. - Inference is significantly slower than non-autoregressive systems due to the frame-by-frame autoregressive loop compounded with 3 flow-matching NFEs per frame. Streaming or parallel decoding is not explored. - Training and evaluation are English-only (LibriSpeech); multilingual zero-shot capability is not tested. - The LM parameter count is not stated, making it difficult to assess compute efficiency relative to MELLE or VALL-E. - The MOS evaluation uses automatic prediction (RAMP+) rather than human listeners; discrepancies with human MOS are possible, especially when comparing systems at similar quality levels. - Coarse-to-fine decomposition uses a fixed downsampling strategy (even-indexed frames); more principled spectral decompositions (e.g., sub-band filterbanks) are unexplored." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2506.21619" published_date: "2025-09-03" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, GAN, hybrid] relevance: high evidence_role: - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: tying_the_duration_target_embedding_to_the_semantic role: supports claim: "Tying the duration-target embedding to the semantic positional embedding table enables near-perfect token-count precision in autoregressive TTS without sequence-level supervision." source: "§ Proposed Method — Duration Control, Table 4" evidence: "For duration control (Table 4), token number error rate is below 0.02% at 1× scaling and below 0.07% even at 0.75× compression on SeedTTS test-zh — near-perfect precision for an autoregressive model." confidence: high relevance: high - claim_id: gradient_reversal_layer_training_effectively_disentangles_emotional_prosody role: supports claim: "Gradient Reversal Layer training effectively disentangles emotional prosody from speaker timbre in zero-shot TTS, enabling independent control of expression and identity from separate reference signals." source: "§ Proposed Method — Emotional Control, Table 2" evidence: "A Gradient Reversal Layer (GRL) during training forces `e` to be invariant to speaker timbre, while a frozen speaker perceiver conditioner extracts `c` (timbre)." confidence: high relevance: medium - claim_id: gpt_latent_fusion_between_the_autoregressive_semantic_module role: supports claim: "GPT-latent fusion between the autoregressive semantic module and the flow-matching acoustic module improves phonetic clarity under high emotional expressiveness, as measured by WER on the emotional test set." source: "§ Semantic-to-Mel Module, Table 2" evidence: "IndexTTS2 is a three-module cascade: a Text-to-Semantic (T2S) autoregressive transformer, a Semantic-to-Mel (S2M) flow-matching model, and a BigVGANv2 vocoder." confidence: high relevance: high - claim_id: a_three_stage_curriculum_that_separates_base_capability role: supports claim: "A three-stage curriculum that separates base capability training from emotion-conditioner fine-tuning is essential for emotional expressiveness; removing it causes EMOS to collapse from 4.22 to 2.82." source: "§ Experiment Results — Emotional Performance, Table 2" evidence: "Training uses a three-stage curriculum: (1) full dataset without emotion conditioning to establish base capability; (2) 135-hour emotional subset with emotion conditioner trainable and GRL active; (3) full dataset fine-tune with all conditioners frozen for robustness." confidence: high relevance: medium - claim_id: knowledge_distillation_from_a_large_reasoning_llm_into role: supports claim: "Knowledge distillation from a large reasoning LLM into a small fine-tuned model provides a usable soft emotion routing interface for TTS that outperforms instruction-tuned baselines on naturalness and prosody." source: "§ Text-to-Emotion, Table 3/5" evidence: "Natural language emotion control (T2E, Table 3/5) outperforms CosyVoice2 on all four MOS dimensions (SMOS 3.875 vs. 2.973)." confidence: high relevance: medium limitations: - "The emotional training data (135 hours across 361 speakers) is modest and partly commercial, raising questions about reproducibility and demographic coverage. The GRL-based disentanglement works for the 7 basic emotions defined, but the emotion space is discrete and limited — continuous or compositional emotion representations are not addressed. The paper notes WER degrades slightly on AISHELL-1 relative to IndexTTS, suggesting the emotional training may slightly hurt clean-speech intelligibility. Duration control is evaluated only at up to 1.25× scaling; extreme compression or expansion is untested. The code release is stated as planned but not confirmed at submission." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.04685" published_date: "2025-09-04" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: content_adaptive_token_allocation_in_acoustic_tokenisers_achieves role: supports claim: "Content-adaptive token allocation in acoustic tokenisers achieves better reconstruction quality than fixed-rate designs at equal or lower token budgets." source: "§4.2, Table 1" evidence: "VARSTok at 30.95 Hz achieves UTMOS 3.8949 on LibriTTS test-clean, surpassing the 40 Hz WavTokenizer (3.6107) while using 23% fewer tokens; the 36.81 Hz configuration (UTMOS 4.000) nearly matches the 75 Hz WavTokenizer (4.025) with fewer than half the tokens." confidence: high relevance: high - claim_id: dynamically_segmented_speech_tokens_carry_more_semantically_discriminative role: supports claim: "Dynamically segmented speech tokens carry more semantically discriminative information than uniformly sampled tokens at the same average rate." source: "§4.3, Table 2" evidence: "All VARSTok configurations outperform the 40 Hz WavTokenizer on all four ARCH benchmark classification tasks (emotion, digit recognition, intent), despite operating at lower average frame rates." confidence: high relevance: high - claim_id: encoding_token_duration_implicitly_in_the_vq_codebook role: supports claim: "Encoding token duration implicitly in the VQ codebook index eliminates the need for auxiliary duration predictors and preserves compatibility with autoregressive speech language models." source: "§3.4, §4.4, Table 3" evidence: "The implicit duration coding scheme maps each cluster's content index k and duration d to a single token ID D = (d-1)*K + k, enabling a standard cross-entropy autoregressive model to generate variable-rate token sequences without modification; MOS and WER improve over the fixed-rate baseline in zero-shot TTS." confidence: high relevance: high - claim_id: more_aggressive_temporal_compression_in_variable_rate_tokenisers role: complicates claim: "More aggressive temporal compression in variable-rate tokenisers trades reconstruction quality for token efficiency beyond a practical compression threshold." source: "§4.2, Table 1" evidence: "Increasing S_max from 2 to 8 reduces the average frame rate from 46.5 Hz to 22.38 Hz but degrades UTMOS from 4.038 to 3.647 and PESQ from 2.069 to 1.453 on LibriTTS test-clean; the optimal configuration (tau=0.7, S_max=4 at 30.95 Hz) sits at the knee of this trade-off curve." confidence: high relevance: high - claim_id: inference_speed_in_autoregressive_speech_lms_depends_primarily role: refines claim: "Inference speed in autoregressive speech LMs depends primarily on sequence length rather than vocabulary size, so variable-rate tokenisers with expanded vocabularies still accelerate decoding." source: "§J, Table 5" evidence: "VARSTok (tau=0.6) achieves RTF 0.487 versus 0.766 for the 40 Hz WavTokenizer baseline (36% speedup) despite expanding the token vocabulary from K to K*S_max = 16,384 entries, because shorter sequences reduce the dominant cost of attention computation over more function evaluations." confidence: high relevance: high limitations: - "Evaluation is restricted to English (LibriTTS). The clustering algorithm relies on cosine similarity in a WavTokenizer embedding space trained on English read speech; whether the density-peak boundaries remain meaningful for other languages, spontaneous speech, or emotionally expressive styles is untested." - "Speaker similarity under more aggressive compression (tau=0.6, 26.29 Hz) does show a statistically modest decline in objective SIM (0.880 vs 0.918 for the baseline), and while subjective SMOS remains comparable, the long-tail impact on voices far from the training distribution is unknown. Codebook collapse becomes severe for K above 4096 in the expanded index space, suggesting that very large vocabulary configurations require dedicated regularisation strategies not addressed here. The clustering algorithm is not differentiable, so joint end-to-end training with a downstream TTS model is not straightforward." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.05863" published_date: "2025-09-06" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: preference_based_alignment_dpo_improves_intelligibility_in_multilingual role: supports claim: "Preference-based alignment (DPO) improves intelligibility in multilingual autoregressive TTS systems beyond what supervised fine-tuning achieves." source: "§4.3, §5.2, Table 2, Table 3" evidence: "LatinX (DPO) reduces WER across nearly all 30 cross-lingual language pairs compared to the supervised fine-tuned baseline, and outperforms XTTSv2 in most pairs; Romanian-source conditions show particularly large gains (e.g., ro-to-es at 0.45% WER)." confidence: high relevance: high - claim_id: automated_speaker_similarity_metrics_based_on_speaker_encoder role: complicates claim: "Automated speaker similarity metrics based on speaker encoder embeddings do not reliably reflect human perceptual judgments of voice identity in zero-shot TTS." source: "§5.2, §6, Table 4, Table 5" evidence: "XTTSv2 achieves higher Sim-O scores than both LatinX models, yet human evaluators strongly prefer LatinX speaker similarity (SMOS 3.63/3.54 vs. 3.24); the paper explicitly flags this as a divergence between objective and subjective evaluation." confidence: high relevance: medium - claim_id: dpo_alignment_in_tts_involves_a_trade_off role: complicates claim: "DPO alignment in TTS involves a trade-off: optimizing for intelligibility and objective similarity can reduce naturalness MOS and, in some language conditions, perceptual similarity relative to the fine-tuned baseline." source: "§5.2, §6, Table 5, Table 6" evidence: "LatinX (DPO) improves WER and Sim-E over the fine-tuned model but achieves lower average MOS (3.35 vs. 3.41) and lower SMOS in several cross-lingual conditions; the paper attributes this partly to the codec introducing artifacts that cap perceptual quality." confidence: high relevance: medium - claim_id: lossy_neural_audio_codecs_set_a_perceptual_quality role: complicates claim: "Lossy neural audio codecs set a perceptual quality ceiling in codec-based TTS that preference alignment cannot overcome, because the model learns to replicate codec artifacts introduced during reference encoding." source: "§6" evidence: "The paper notes that the VQ-VAE codec is lossy and the model learns to reproduce its artifacts, limiting the maximum perceptual quality achievable regardless of post-training alignment method." confidence: high relevance: high limitations: - "The evaluation uses an internal test set of unseen speakers with no publicly named benchmark, and the human rating pool is predominantly English and Portuguese native speakers. Conclusions about multilingual naturalness and similarity, especially for French, Italian, and Romanian, should be treated with caution." - "The DPO preference signal is constructed solely from WER and speaker similarity; no prosody, naturalness, or rhythm metric is incorporated, which likely explains the MOS regression relative to the fine-tuned baseline. The preference labeling is fully automated with no human verification of winner/loser assignments. The real-time factor of 4.85 makes the system unsuitable for real-time applications, and the authors note that non-autoregressive architectures are a necessary direction. The Romanian evaluation suffers from very small rater counts and predominantly non-native listeners, undermining the interpretation of the unusually high SMOS scores that exceed real audio." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.09174" published_date: "2025-09-11" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: decoupling_semantic_training_objectives_from_acoustic_token_prediction role: supports claim: "Decoupling semantic training objectives from acoustic token prediction substantially reduces knowledge degradation in speech-to-speech LLMs." source: "§5.1, Table 4" evidence: "EchoX's Echo training, which generates speech targets from the model's own semantic hidden states, raises average QA accuracy from 24.3 (T2C without Echo) to 37.1 (EchoX-3B) on the same data, compared to 12.8 for direct interleaved training." confidence: high relevance: high - claim_id: unit_based_speech_token_compression_via_language_model role: supports claim: "Unit-based speech token compression via language-model segmentation improves downstream accuracy and reduces sequence length without sacrificing audio quality." source: "§5.3, Table 5, Figure 7" evidence: "Unit language achieves 4.57 length ratio vs. 9.31 for raw units while improving accuracy on all three QA benchmarks and maintaining comparable audio quality in spectral comparison." confidence: high relevance: high - claim_id: streaming_inference_in_speech_llms_can_be_achieved role: supports claim: "Streaming inference in speech LLMs can be achieved with minimal accuracy degradation when the segmentation boundary is determined by semantic similarity rather than fixed length." source: "§5.4, Table 6" evidence: "EchoX's cosine-similarity trigger reduces first-token latency from 138 to 27 tokens at 3B scale with less than 1.5 percentage points of accuracy drop on any benchmark." confidence: high relevance: medium - claim_id: training_data_efficiency_in_speech_llms_may_depend role: complicates claim: "Training data efficiency in speech LLMs may depend more on the training paradigm than on data volume." source: "§4.2, Table 2" evidence: "EchoX achieves competitive performance on spoken QA against models trained on millions of hours using only approximately 6,200 hours, but this result holds specifically for factual QA and has not been tested on broader spoken dialogue tasks." confidence: high relevance: medium - claim_id: speech_naturalness_and_response_helpfulness_are_not_jointly role: complicates claim: "Speech naturalness and response helpfulness are not jointly optimised by the same training signal in speech-to-speech LLMs." source: "§Appendix C, Figure 8" evidence: "Human evaluation shows EchoX wins clearly on helpfulness but performs only competitively on naturalness, reflecting a training objective focused on semantic correctness rather than prosodic quality." confidence: high relevance: medium limitations: - "Speech quality is assessed only via brief spectral comparison (Figure 7) and a 5-rater human study. No perceptual quality metric (MOS, DNSMOS) or automatic speech recognition accuracy on generated audio is reported as a primary evaluation result, making it difficult to characterise the system's output quality independently of QA accuracy." - "The evaluation benchmarks are limited to factual knowledge QA (Llama Questions, Web Questions, TriviaQA). It is unclear whether Echo training retains its advantage on open-ended dialogue, instruction following, or longer-form conversational tasks. The human evaluation was conducted with only five raters on one dataset (AlpacaEval), which limits statistical confidence in the naturalness comparison. The model trains on synthesised assistant audio from GPT-SoVITS, which may introduce a fixed timbre bias and limit voice diversity. The streaming threshold and window size are fixed hyperparameters with no ablation reported on their sensitivity." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2509.09550" published_date: "2025-09-11" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [hybrid, GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: fsq_based_neural_audio_codecs_develop_inherent_redundancy role: supports claim: "FSQ-based neural audio codecs develop inherent redundancy in their code representations, enabling multiple independently trained encoders to produce radically different code sequences that nonetheless decode to perceptually equivalent audio." source: "§4.2, Table 2" evidence: "Encoder distillation experiment on NeuCodec showing only 2% element-wise code match between original (635M params) and distilled (42M params) encoders, while cosine similarity between pre-quantization projections is 0.73 and reconstruction metrics (WER, CER, STOI, PESQ) remain within 0.5% WER of each other." confidence: high relevance: high - claim_id: fsq_codecs_are_substantially_more_robust_to_bit role: supports claim: "FSQ codecs are substantially more robust to bit-level transmission errors than RVQ codecs of comparable codebook size, maintaining intelligibility at bit-flip rates an order of magnitude higher than the RVQ failure threshold." source: "§5, Figure 3" evidence: "Binary symmetric channel simulation on LibriSpeech test-clean showing FSQ codecs (NeuCodec, Distill-NeuCodec, StableCodec) maintain stable STOI and PESQ up to 10% bit-flip probability, whereas RVQ codecs (EnCodec, DAC) experience sharp quality collapse above 1% bit-flip rate." confidence: high relevance: high - claim_id: the_perturbation_robustness_advantage_of_fsq_over_rvq role: refines claim: "The perturbation robustness advantage of FSQ over RVQ is structural: FSQ's fixed-grid quantization maps bit-flip perturbations to bounded, predictable steps in the embedding space, whereas RVQ code perturbations produce arbitrarily large embedding-space displacements." source: "§4.2, §6" evidence: "Implicit codebook confusion matrices show that 93% of level predictions between original and distilled encoders are either correct or off by exactly one neighbouring level, confirming local structure in the FSQ embedding space." confidence: high relevance: low - claim_id: fsq_based_codec_architectures_that_achieve_competitive_intelligibility role: complicates claim: "FSQ-based codec architectures that achieve competitive intelligibility metrics often rely on very large pretrained semantic encoders, concentrating most of the parameter budget in a component that contributes indirectly to the quantization benefit." source: "§3, §4.1, Table 2" evidence: "Wav2Vec2-BERT-large accounts for 600M of NeuCodec's 635M total parameters; replacing it with DistilHuBERT reduces the model to 42M parameters with only a 0.5% WER increase, suggesting the semantic representation is partially substitutable without sacrificing the FSQ robustness property." confidence: high relevance: high limitations: - "Evaluation uses WER, CER, STOI, and PESQ but no perceptual naturalness metric (MOS or MUSHRA), making it impossible to assess how NeuCodec compares to state-of-the-art codecs on speech generation quality. The perturbation experiment uses a binary symmetric channel that does not reflect real-world transmission protocols (e.g., structured burst errors or packet loss), so robustness claims cannot be extrapolated directly to deployment scenarios." - "Training includes 1,000 hours of proprietary data, partially limiting reproducibility. The model is evaluated only at 16kHz and 24kHz; the 24kHz upsampling decoder is trained on only 2,600 hours of data, and high-fidelity (48kHz) audio is not addressed. The paper does not evaluate whether FSQ robustness properties are preserved when the codec is used as a tokenizer for downstream TTS or language model tasks, which is the stated application motivation." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.11425" published_date: "2025-09-14" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec, TTS] architecture: [GAN, autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: injecting_multimodal_guidance_directly_into_the_encoder_latent role: supports claim: "Injecting multimodal guidance directly into the encoder latent space of a neural codec improves reconstruction quality beyond similarity-based supervision of the quantized layer." source: "§2.3.1, §3.2.1, Table 2" evidence: "FuseCodec-Fusion, which fuses global semantic and contextual vectors into Z via additive fusion, achieves WER 3.99, ViSQOL 3.47, and PESQ 3.13 on LibriSpeech test-clean, outperforming FuseCodec-Distill (ViSQOL 3.43, PESQ 3.06) and FuseCodec-ContextAlign, both of which only supervise Q(1) without modifying the latent." confidence: high relevance: high - claim_id: neural_codecs_trained_with_joint_semantic_and_contextual role: supports claim: "Neural codecs trained with joint semantic and contextual supervision generalize to unseen languages without multilingual training data." source: "§3.3, Table 5" evidence: "FuseCodec, trained exclusively on English LibriSpeech train-clean-100, achieves the best WER and ViSQOL in the majority of 7 tested languages from Multilingual LibriSpeech and outperforms all baselines on PESQ by at least 0.3 in most languages." confidence: high relevance: high - claim_id: codec_representations_enriched_with_semantic_and_contextual_signals role: supports claim: "Codec representations enriched with semantic and contextual signals support stronger downstream task generalization (emotion recognition, audio event classification) than acoustic-only codecs." source: "§3.2.2, Table 3" evidence: "On CodecSUPERB at 4 kbps, FuseCodec-Fusion achieves emotion recognition accuracy of 73.96% and audio signal quality 0.785, versus 66.18% and 0.697 for EnCodec at 6 kbps. All FuseCodec variants exceed SpeechTokenizer, EnCodec, and DAC on audio signal quality." confidence: high relevance: high - claim_id: fine_grained_temporal_alignment_between_text_tokens_and role: complicates claim: "Fine-grained temporal alignment between text tokens and acoustic frames improves local interpretability but is constrained relative to global supervision strategies." source: "§2.3.3, §3.2.1, Table 2" evidence: "FuseCodec-ContextAlign, which aligns contextual embeddings to RVQ tokens via a windowed similarity matching algorithm, achieves WER 4.15 and ViSQOL 3.18, lagging FuseCodec-Fusion (WER 3.99, ViSQOL 3.47) and FuseCodec-Distill (ViSQOL 3.43). The paper attributes this to constrained local alignment limiting global contextual guidance." confidence: high relevance: high - claim_id: distilling_both_semantic_self_supervised_speech_model_and role: supports claim: "Distilling both semantic (self-supervised speech model) and contextual (language model) signals into codec token supervision outperforms semantic-only distillation for perceptual naturalness." source: "§3.2.1, Table 2" evidence: "FuseCodec-Distill achieves UTMOS 3.65 and Similarity 0.996 on LibriSpeech test-clean, while codecs using only semantic distillation (SpeechTokenizer, Mimi, X-Codec2) score below 3.55 UTMOS and fail to consistently match speaker similarity." confidence: high relevance: high limitations: - "The codec is trained on LibriSpeech train-clean-100 (100 hours, English read speech), which limits conclusions about robustness to spontaneous speech, diverse accents, or noisy conditions. Multilingual generalization results are promising but the training data and BERT model are English-only, leaving the mechanism behind cross-lingual transfer unclear. The ASR transcription step (wav2vec 2.0) introduces an error-prone intermediate representation during training; the effect of ASR errors on contextual embedding quality is not quantified. Model size is not reported, making it difficult to assess computational cost relative to baselines. The TTS evaluation compares only against other codec-based systems trained on LibriTTS and does not benchmark against the strongest current TTS models." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.13068" published_date: "2025-09-16" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, codec, VC] architecture: [autoregressive-LM, VAE] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: cascaded_residual_codec_architectures_can_enforce_attribute_disentanglement role: supports claim: "Cascaded residual codec architectures can enforce attribute disentanglement through structure rather than through adversarial training objectives." source: "§2.1, §3.3.3, Table 3" evidence: "MSR-Codec achieves clean separation of timbre, prosody, and semantic content by having each stream operate on residuals from the previous stage, without adversarial disentanglement loss; VC experiments confirm independent manipulation of each attribute." confidence: high relevance: high - claim_id: explicit_prosodic_supervision_in_a_dedicated_codec_stream role: supports claim: "Explicit prosodic supervision in a dedicated codec stream promotes measurable disentanglement of pitch from speaker identity." source: "§2.1.2, §3.3.3, Table 3" evidence: "VQ1 (prosody stream) is trained with MSE loss against ground-truth F0 and spectral energy; prosody-only VC achieves low ΔF0,tar (12.3-14.2 Hz) while maintaining high SIM-src (0.59-0.64), confirming that prosody and timbre are independently manipulable." confidence: high relevance: high - claim_id: disentangled_codec_designs_can_achieve_competitive_speaker_similarity role: supports claim: "Disentangled codec designs can achieve competitive speaker similarity at lower bitrates than undifferentiated RVQ codecs." source: "§3.3.1, Table 1" evidence: "MSR-Codec-424 achieves SPK-SIM 0.80 at 424 bps, higher than WavTokenizer (0.67 at 900 bps) and X-Codec (0.72 at 1000 bps), attributed to the time-invariant timbre stream which preserves speaker identity without scaling with utterance length." confidence: high relevance: high - claim_id: data_efficient_tts_systems_built_on_factorized_codec role: supports claim: "Data-efficient TTS systems built on factorized codec representations can achieve competitive intelligibility relative to larger models trained on more data." source: "§3.3.2, Table 2" evidence: "The 0.2B MSR-Codec-524 TTS model trained on 45k hours achieves WER 3.07% on Seed-TTS-eval English, outperforming Llasa-1B trained on 250k hours (WER 3.22%) and FireRedTTS-0.4B trained on 150k hours (WER 3.82%)." confidence: high relevance: high - claim_id: signal_fidelity_codec_metrics_stoi_pesq_and_speaker role: complicates claim: "Signal-fidelity codec metrics (STOI, PESQ) and speaker similarity diverge at low bitrates, making holistic quality assessment difficult." source: "§3.3.1, Table 1" evidence: "MSR-Codec-424 achieves the highest SPK-SIM (0.80) among codecs at comparable bitrates but lower STOI (0.84) and PESQ-WB (1.82) than some baselines, indicating that speaker preservation and signal-level fidelity are optimized differently by the multi-stream design." confidence: high relevance: high limitations: - "No subjective listening test (MOS/MUSHRA) is reported for any condition; all quality comparisons rely on automatic metrics (UTMOS, STOI, PESQ, WER, SPK-SIM). Conclusions about perceived naturalness cannot be confirmed from the available data." - "The VC evaluation protocol is small in scope: 8 target speakers from VCTK and 100 source utterances from LibriTTS. Generalisation to more diverse speakers, accents, or noisy conditions is not assessed. The FreGAN vocoder operates at 16 kHz, and the Mel-spectrogram-based pipeline may impose a quality ceiling relative to waveform-domain codecs. The TTS model is evaluated only on English; the codec was trained on Mandarin and English data but multilingual TTS capability is not demonstrated. Model size figures for the codec itself are not reported; only the TTS model size (0.2B) is provided." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2412.16846" published_date: "2025-09-17" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, VAE] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: distributional_training_objectives_for_continuous_ar_speech_modeling role: supports claim: "Distributional training objectives for continuous AR speech modeling achieve higher intelligibility than regression-based alternatives." source: "§TTS Evaluation, Table 2; §Ablation Study, Table 5" evidence: "KALL-E with KL divergence loss achieves WER 1.94 / CER 0.96 on Seed-TTS test sets, below all discrete-token and regression-based baselines; ablation replacing Flow-VAE with Stable Audio VAE (near-zero KL weight, approaching a plain autoencoder) collapses CER from 2.79 to 40.09 at the same latent dimension." confidence: high relevance: medium - claim_id: low_frame_rate_continuous_representations_reduce_autoregressive_tts role: supports claim: "Low frame-rate continuous representations reduce autoregressive TTS inference compute by over an order of magnitude without sacrificing synthesis quality." source: "§TTS Evaluation, Table 3, Table 4" evidence: "KALL-E at 12.5 Hz requires 7,947 GFLOPs to synthesize 10 seconds vs. 122,170 for Llasa-1B at 50 Hz, while achieving higher MOS (4.17 vs. 3.92) and lower WER (1.94 vs. 3.6) on the same test set." confidence: high relevance: high - claim_id: objective_speaker_similarity_metrics_for_zero_shot_tts role: complicates claim: "Objective speaker similarity metrics for zero-shot TTS are unreliable for cross-system comparisons when decoder architectures differ in their use of reference audio." source: "§TTS Evaluation, Table 2, Table 3" evidence: "Discrete-token systems (Seed-TTS SIM 0.796, FireRedTTS SIM 0.635) score differently on objective SPK-SIM than KALL-E (SIM 0.646/0.568), but KALL-E receives higher listener naturalness ratings; the authors attribute the gap to those systems conditioning the waveform decoder on the reference utterance at decode time, which inflates the metric independent of perceived speaker fidelity." confidence: high relevance: medium - claim_id: test_time_adaptation_from_a_single_reference_utterance role: supports claim: "Test-time adaptation from a single reference utterance improves speaker similarity in continuous-representation AR TTS without requiring full model retraining." source: "§Test Time Training; §TTS Evaluation, Table 2" evidence: "KALL-E (TTT) improves SPK-SIM from 0.568 to 0.611 on test-en using N=200 latent sequences sampled from the reference utterance's Flow-VAE distribution, with WER remaining stable at 1.90." confidence: high relevance: medium - claim_id: increasing_vae_kl_regularization_weight_trades_reconstruction_fidelity role: complicates claim: "Increasing VAE KL regularization weight trades reconstruction fidelity for a latent space structure that is more suitable for downstream generative modeling." source: "§VAE Evaluation, Table 1; §Ablation Study, Table 5" evidence: "Flow-VAE uses KL weight 32 and scores PESQ-WB 3.26 at 512-dim/12.5 Hz, below Stable Audio VAE (3.11) at the same frame rate with near-zero KL weight; however, Stable Audio VAE's latent space causes CER to collapse when used as the AR LM encoder, demonstrating that reconstruction quality and generation compatibility impose conflicting constraints on VAE training." confidence: high relevance: medium limitations: - "Objective speaker similarity remains below discrete-token systems that condition their decoders on the reference audio, suggesting the Flow-VAE's information bottleneck trades some speaker detail for a more LM-friendly latent space. The TTT procedure assumes the transcript of the reference utterance is available, which may not hold in all deployment settings. Overfitting risk in TTT is real: CER rises after N=200 in ablation, limiting the effective adaptation set size. Evaluations are conducted solely on the Seed-TTS test sets; generalization to other benchmarks, out-of-distribution speakers, or noisy acoustic conditions is not assessed. Training data composition differs from the most directly comparable system (Llasa-1B), making it difficult to fully isolate architecture from data quality as the source of WER gains." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.13670" published_date: "2025-09-17" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: knowledge_distillation_from_a_non_causal_high_complexity role: supports claim: "Knowledge distillation from a non-causal, high-complexity teacher can effectively recover reconstruction quality degraded by model causalization and channel pruning in low-latency streamable neural codecs." source: "§IV.A, Table I" evidence: "StreamCodec2 (NH→CL direct KD) achieves PESQ 2.744 and ViSQOL 4.313 vs. 2.650 and 4.290 for the undistilled student at identical 910 MFLOPs and 5.4 M parameters, with all gains p < 0.01." confidence: high relevance: high - claim_id: multi_stage_distillation_pipelines_do_not_provide_additive role: complicates claim: "Multi-stage distillation pipelines do not provide additive quality gains over direct teacher-to-student distillation in neural codec compression." source: "§IV.A, Table I" evidence: "Both indirect distillation schemes (NH→CH→CL and NH→NL→CL) underperform direct distillation across all objective metrics, with ViSQOL 4.294 and 4.305 vs. 4.313 for direct, suggesting that intermediate steps dilute rather than refine the knowledge transferred." confidence: high relevance: high - claim_id: fully_causal_neural_codec_architectures_incur_a_meaningful role: complicates claim: "Fully causal neural codec architectures incur a meaningful quality penalty relative to non-causal counterparts at the same bitrate, which knowledge distillation only partially closes." source: "§IV.A, Table I" evidence: "Even with the best distillation strategy, StreamCodec2 (NH→CL) scores PESQ 2.744 vs. the teacher's 3.132 and ViSQOL 4.313 vs. 4.463, leaving a gap that reflects the fundamental constraint of causal-only processing." confidence: high relevance: high - claim_id: distillation_loss_weighting_in_codec_training_requires_careful role: refines claim: "Distillation loss weighting in codec training requires careful calibration; excessively large weights degrade reconstruction quality by shifting the learning objective toward teacher imitation." source: "§IV.B, Figure 3" evidence: "ViSQOL for StreamCodec2 (NH→CL) peaks at lambda_KD = 0.01 and declines for weights above this value (tested at 0.002, 0.005, 0.01, 0.02, 0.05), consistent with the reconstruction objective being displaced by over-fitting the teacher's intermediate representations." confidence: high relevance: high limitations: - "The evaluation uses only objective metrics (LSD, STOI, PESQ, ViSQOL); it is unclear whether the statistically significant gains over the undistilled student are perceptually meaningful. Comparisons are limited to the authors' own student and teacher variants with no benchmarking against published competing streamable codecs (SoundStream, EnCodec, or others) under matched latency and bitrate conditions, making it difficult to assess where StreamCodec2 stands in the broader landscape. Experiments use a single dataset (LibriTTS at 16 kHz), leaving generalisation to other languages, domains, or sampling rates unconfirmed. Future directions noted by the authors include improving reconstruction quality further and evaluating on additional audio datasets." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.14784" published_date: "2025-09-18" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [hybrid] relevance: medium evidence_role: - negative_evidence - architecture_variant current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: aligning_autoregressive_decoder_representations_with_pre_trained_asr role: supports claim: "Aligning autoregressive decoder representations with pre-trained ASR encoder semantics improves content consistency in continuous-feature TTS." source: "§2.3, §3.2, Table 1, Table 2" evidence: "Representation alignment reduces WER 6.3 to 5.3 on LibriTTS ablation and produces 25%, 40%, and 30% relative CER/WER reductions on seed-tts-eval test-zh, test-en, and test-hard at 170K-hour scale. Training convergence accelerates by over 3.3x." confidence: high relevance: high - claim_id: using_the_target_acoustic_domain_mel_spectrogram_as role: contradicts claim: "Using the target acoustic domain (mel-spectrogram) as an intermediate alignment target helps continuous AR TTS systems." source: "§3.2, Table 1" evidence: "Aligning transformer decoder outputs to mel-spectrograms (Exp 2, Table 1) raises WER from 6.3 to 6.7 and degrades SS1/SS2, while alignment to SenseVoice-Large ASR representations (Exp 1) reduces WER to 5.3." confidence: high relevance: low - claim_id: continuous_feature_ar_tts_achieves_competitive_content_accuracy role: complicates claim: "Continuous-feature AR TTS achieves competitive content accuracy but lags behind discrete-token systems on speaker similarity in zero-shot voice cloning." source: "§3.3, Table 2" evidence: "MELA-TTS achieves best CER on seed-tts-eval test-zh but lower SS1 (0.59 vs. F5-TTS 0.65) and SS2 (0.68 vs. F5-TTS 0.74) on test-en; attributed to the diffusion module's local context constraint versus discrete-token systems that condition on all prior tokens and the full prompt." confidence: high relevance: high - claim_id: unified_interleaved_training_on_streaming_and_offline_sequences role: supports claim: "Unified interleaved training on streaming and offline sequences enables continuous AR TTS to operate in both modes at comparable quality without separate model variants." source: "§2.4, §3.2, §3.3, Table 1, Table 2" evidence: "Streaming mode achieves WER 5.0 versus offline 5.2, and SS1/SS2 parity, on LibriTTS. At 170K-hour scale, streaming WER is 2.5 versus offline 2.4 on test-en." confidence: high relevance: low - claim_id: continuous_feature_tts_without_speech_tokenization_scales_effectively role: supports claim: "Continuous-feature TTS without speech tokenization scales effectively with training data, reducing content error rates and approaching discrete-token system performance." source: "§3.3, Table 2" evidence: "Scaling from LibriTTS (585h) to 170K-hour in-house data reduces English WER from 5.2 to 2.4 and improves SS1 from 0.48 to 0.59, bringing MELA-TTS close to CosyVoice 2.0 (2.6 WER) and CosyVoice 3.0 (2.5 WER)." confidence: high relevance: high limitations: - "The diffusion module operates on local context only (the immediately adjacent latent vector and the previous mel chunk), preventing it from attending to the full prompt speech. This is identified as the likely structural cause of the consistent speaker similarity gap versus discrete-token systems across test-en and test-hard. The limitation is left as future work with no proposed fix." - "Main comparison results rely on a proprietary 170K-hour multilingual dataset that is not publicly available, limiting reproducibility of the top-line numbers. Ablations are conducted on LibriTTS only (585 hours), which may not reflect behavior at full scale. No comparison to DiTAR on test-hard is available, making it unclear whether MELA-TTS's robustness advantage on challenging inputs holds relative to the closest continuous AR baseline. Code and demo are not released with the preprint." caveats: - "This non-autoregressive or hybrid TTS system provides direct comparative evidence for the concept." - id: "2509.14882" published_date: "2025-09-18" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA] architecture: [autoregressive-LM] relevance: high evidence_role: - architecture_variant current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, semantic_acoustic_token_cascade, unified_speech_text_language_model] claims: - claim_id: a_single_transformer_flattened_architecture_can_match_or role: supports claim: "A single-Transformer flattened architecture can match or surpass hierarchical speech LM designs on acoustic consistency tasks when both use identical data and comparable parameter budgets." source: "§4.3, Table 1" evidence: "Llama-Mimi-1.3B outperforms CSM-1.3B on all SALMon acoustic consistency dimensions and speaker similarity (0.346 vs. 0.320) under controlled training conditions." confidence: high relevance: high - claim_id: flattening_rvq_tokens_into_a_single_autoregressive_sequence role: complicates claim: "Flattening RVQ tokens into a single autoregressive sequence creates an inherent acoustic-linguistic trade-off: strong acoustic performance comes at the cost of weaker linguistic benchmarks relative to SSL-based phonetic-token approaches." source: "§4.3, Table 1" evidence: "Llama-Mimi-1.3B achieves best acoustic consistency but underperforms TWIST-1.3B on sWUGGY (68.7 vs. 71.7) and T-Story Cloze (64.0 vs. 69.9), attributed to the Q-fold sequence length increase from RVQ token flattening." confidence: high relevance: high - claim_id: increasing_the_number_of_rvq_quantizers_in_a role: complicates claim: "Increasing the number of RVQ quantizers in a flattened speech LM improves audio quality but degrades spoken content coherence, because longer token sequences shift modeling capacity toward acoustic reconstruction." source: "§4.4, Table 5" evidence: "Ablation with Q∈{2,4,8} shows Q=8 achieves best Audiobox-Aesthetics scores and speaker similarity (0.474) but worst content quality (2.54), while Q=2 yields content quality (3.53) comparable to TWIST-1.3B." confidence: high relevance: high - claim_id: applying_a_higher_loss_weight_to_semantic_tokens role: refines claim: "Applying a higher loss weight to semantic tokens in speech LM training shifts the acoustic-linguistic balance toward linguistic accuracy, but causes measurable degradation in acoustic consistency and speaker similarity." source: "§4.4, Table 3" evidence: "Llama-Mimi-1.3B with semantic weight λ=100 gains on sBLIMP (55.4 vs. 54.3) and T-Story Cloze (68.4 vs. 64.0), but loses substantially on room consistency (74 vs. 92) and speaker similarity (0.196 vs. 0.346)." confidence: high relevance: high - claim_id: scaling_model_size_in_flattened_speech_lms_consistently role: supports claim: "Scaling model size in flattened speech LMs consistently improves performance across both acoustic and linguistic tasks, with the largest gains in spoken content quality." source: "§4.4, Table 4" evidence: "Llama-Mimi-8B improves content quality (4.03 vs. 3.01) and T-Story Cloze (67.6 vs. 64.0) over the 1.3B model, with qualitative analysis showing more semantically coherent long-form continuations." confidence: high relevance: high limitations: - "Evaluations are conducted exclusively on English speech using LibriSpeech prompts; generalisation to other languages, speakers, or acoustic conditions is untested. No human listening tests are reported; all acoustic and linguistic evaluations rely on automated metrics." - "The paper evaluates only speech continuation, not text-conditioned generation or dialogue. Whether the acoustic-linguistic trade-off in flattened designs persists when text tokens are added to the sequence (as in Moshi's inner monologue approach) is left as an open question. The 8B model is evaluated only on a subset of tasks, and the cost of the Q-fold sequence length increase at large scale is not fully characterised. Training Mimi weights end-to-end jointly with the LM is not explored; the frozen codec is an architectural constraint, not a systematic choice." caveats: - "This paper applies autoregressive speech-token generation in a spoken-agent architecture." - id: "2509.14946" published_date: "2025-09-18" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: automated_mixing_of_isolated_paralinguistic_audio_events_with role: supports claim: "Automated mixing of isolated paralinguistic audio events with clean conversational speech, combined with voice conversion for timbre alignment, can produce effective fine-tuning data for paralinguistic TTS." source: "§4.2.1, Table 3" evidence: "CosyVoice2 SFT on SynParaSpeech achieves PMOS 3.31 versus 2.35 for NVS SFT and 1.88 baseline; F5-TTS SFT on SynParaSpeech similarly achieves PMOS 3.10 versus 1.49 for NVS SFT." confidence: high relevance: medium - claim_id: dpo_with_naturally_paired_utterances_with_vs_without role: supports claim: "DPO with naturally paired utterances (with vs. without paralinguistic events) can improve paralinguistic speech quality beyond SFT alone." source: "§4.2.1, Table 3" evidence: "CosyVoice2 DPO-Joint achieves PMOS 3.46 and QMOS 4.12 versus SFT-only PMOS 3.31 and QMOS 4.01; joint SFT+DPO outperforms staged DPO across all subjective dimensions." confidence: high relevance: medium - claim_id: standard_automatic_speech_quality_metrics_are_unreliable_for role: complicates claim: "Standard automatic speech quality metrics are unreliable for evaluating paralinguistic TTS systems." source: "§4.2.1, Table 3" evidence: "All models fine-tuned on paralinguistic data show degraded CER, SECS, and UTMOSv2 relative to baselines, despite achieving higher subjective NMOS and QMOS; laughter vocalizations raise CER because ASR transcribes them as \"ha ha\"." confidence: high relevance: medium - claim_id: few_shot_in_context_prompting_with_paralinguistic_examples role: supports claim: "Few-shot in-context prompting with paralinguistic examples improves paralinguistic event detection in multimodal LLMs." source: "§4.2.2, Table 4" evidence: "SynParaSpeech prompt tuning raises accuracy and macro F1 for both Kimi Audio and Qwen 2.5 Omni over no-context baselines, with optimal performance at 5-shot context for both models." confidence: high relevance: medium - claim_id: increasing_prompt_context_length_for_in_context_learning role: complicates claim: "Increasing prompt context length for in-context learning in paralinguistic event detection yields diminishing returns and eventually degrades performance." source: "§4.2.2, Table 4" evidence: "Kimi Audio accuracy peaks at 5-shot (0.382) and declines at 7-shot (0.371 for Kimi Audio); Qwen 2.5 Omni also peaks at 5-shot (0.473) and declines at 7-shot (0.423), reflecting input overload beyond an optimal context window." confidence: high relevance: medium limitations: - "The corpus is Chinese-only and covers only 6 paralinguistic categories, limiting direct applicability to other languages or richer paralinguistic inventories. The paralinguistic events are inserted synthetically rather than captured in natural speech, which may introduce distribution mismatch relative to genuinely spontaneous vocalizations." - "Standard objective metrics (CER, UTMOSv2, SECS) degrade after paralinguistic fine-tuning and do not reflect the true quality improvement observed in subjective evaluation. This leaves the field without reliable automatic metrics for paralinguistic speech, requiring human listener studies for valid comparisons. The paper does not evaluate zero-shot generalization to unseen paralinguistic categories or cross-lingual transfer." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.15845" published_date: "2025-09-19" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [flow-matching, autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: conditioning_a_flow_matching_model_to_generate_speaker role: supports claim: "Conditioning a flow matching model to generate speaker embeddings from natural language descriptions enables automatic voice assignment for characters without requiring reference audio." source: "§2.2, §3.3, Table 2" evidence: "The TTT model, trained with OT-CFM to map text descriptions (age, gender, personality, identity) to speaker embeddings, achieves CMS of 2.866 and sex accuracy above 96% for adult demographics; Qwen3-Embedding-0.6B outperforms T5-Large and RoBERTa-Large on all character attribute metrics." confidence: high relevance: medium - claim_id: providing_llm_derived_emotion_scene_instructions_as_an role: supports claim: "Providing LLM-derived emotion-scene instructions as an additional conditioning signal to TTS improves emotional expressiveness without substantially degrading intelligibility." source: "§3.3, Table 3" evidence: "CA-Instruct-TTS achieves MOS-E of 4.15 vs. 3.67 and MOS-N of 3.33 vs. 3.10 compared to the instruction-free CA-TTS baseline, with WER increasing only from 2.39% to 2.54%." confidence: high relevance: medium - claim_id: embedding_based_speaker_generation_from_text_degrades_for role: complicates claim: "Embedding-based speaker generation from text degrades for child voices due to acoustic similarity and training data artifacts." source: "§3.3, Figure 2" evidence: "The TTT model shows notably lower age accuracy for the child category (74.38% with TTT-Qwen3-0.6B) compared to adult categories (90%+). The paper attributes this to pre-pubertal voice similarity and to adult speakers imitating child voices in available training data." confidence: high relevance: medium - claim_id: larger_semantically_richer_text_encoders_improve_text_conditioned role: supports claim: "Larger, semantically richer text encoders improve text-conditioned speaker embedding generation across demographic categories." source: "§3.3, Table 2" evidence: "TTT-Qwen3-0.6B achieves substantially higher CMS (2.866) and age accuracy across most categories compared to TTT-T5-Large (CMS 2.375) and TTT-RoBERTa-Large (CMS 2.359), suggesting that language understanding capacity from pre-trained LLMs transfers to voice attribute generation." confidence: high relevance: medium - claim_id: the_flow_matching_framework_extends_naturally_from_speech role: refines claim: "The flow matching framework extends naturally from speech waveform generation to speaker embedding synthesis." source: "§2.2" evidence: "The OT-CFM framework is applied directly to the generation of continuous speaker embedding vectors from noise, using the same DiT backbone and conditioning architecture as the speech-side flow matching decoder. The speaker embedding space is treated as the target data distribution." confidence: high relevance: medium limitations: - "All evaluations use proprietary internal data. No comparisons to published expressive audiobook TTS systems are included, limiting the ability to independently verify the reported gains in emotional expressiveness." - "The TTT model struggles with child voice generation, an acknowledged limitation tied to data scarcity and label noise in child speech corpora. Future work is noted toward collecting authentic child speech." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.15969" published_date: "2025-09-19" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hierarchical_ar_nar_codec_tts, fully_autoregressive_multicodebook] claims: - claim_id: fully_autoregressive_streaming_tts_can_achieve_first_packet role: supports claim: "Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation." source: "§4, Table 1, Table 3" evidence: "On LibriSpeech long (Table 2, full-stream evaluation against CosyVoice2 under matched conditions), VoXtream achieves substantially lower WER (3.24% vs. 6.11%) and wins the naturalness preference study (57% vs. 31%, p < 5e-10), though CosyVoice2 retains higher SPK-SIM (0.685 vs. 0.564) due to its non-autoregressive flow-matching decoder." confidence: high relevance: high - claim_id: training_data_scale_is_the_primary_driver_of role: supports claim: "Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive." source: "§4, Table 1, Table 2" evidence: "Among mid-scale systems (comparable training data), VoXtream-NS (non-streaming variant) achieves the best SPK-SIM (0.537 on SEED) and UTMOS (3.89 on SEED) and attains the second-best WER on LibriSpeech test-clean (2.99%) across all systems evaluated." confidence: high relevance: medium - claim_id: borrowing_frozen_depth_transformer_weights_from_a_large role: supports claim: "Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training." source: "§3, Table 4" evidence: "Critically, the DT weights are borrowed frozen from the CSM model trained on a large-scale dataset — this cross-model distillation substantially improves output quality without requiring large-scale training." confidence: high relevance: medium - claim_id: full_stream_input_processing_introduces_only_marginal_quality role: supports claim: "Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS." source: "§4, Table 1" evidence: "The streaming variant VoXtream (full-stream) introduces only minor degradations: WER rises from 2.99% to 3.15% on LibriSpeech test-clean; SPK-SIM and UTMOS are essentially unchanged." confidence: high relevance: high - claim_id: non_autoregressive_flow_matching_decoders_used_in_chunk role: supports claim: "Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents." source: "§4, Table 3" evidence: "With `torch.compile`, the full system achieves a first-packet latency of 102 ms and an RTF of 0.17, meaning it runs more than 5 times faster than real time." confidence: high relevance: high limitations: - "Speaker similarity remains lower than large-scale systems trained on hundreds of thousands of hours, particularly in full-stream mode (SPK-SIM 0.458 on LibriSpeech test-clean vs. 0.587 for CosyVoice2 trained on 167k hours). The system is English-only; multilingual extension is not addressed. Prosody and speaking rate are not explicitly controllable at inference. The use of a frozen CSM depth transformer introduces a dependency on an external large-scale model. Long-form streaming beyond 10-15 second utterances is identified as future work. Performance in adverse or spontaneous speech conditions (outside the training domain) is untested." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.20378" published_date: "2025-09-20" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [alignment_constrained_autoregression] claims: - claim_id: sub_sentence_emotion_conditioning_produces_more_accurate_emotional role: supports claim: "Sub-sentence emotion conditioning produces more accurate emotional dynamics in TTS than sentence-level conditioning." source: "§4.1, Table 1" evidence: "Emo-FiLM outperforms CosyVoice2 by 9.1% on DTW and 12.7% on ESD DTW, with higher EMOS on both test sets, when comparing word-level emotion labels against global prompt conditioning." confidence: high relevance: medium - claim_id: feature_wise_linear_modulation_is_an_effective_mechanism role: supports claim: "Feature-wise Linear Modulation is an effective mechanism for injecting word-level conditioning signals into the text representations of autoregressive LLM-TTS systems." source: "§4.2, Table 2" evidence: "Ablation replacing the FiLM layer with simple addition increases FEDD DTW from 49.62 to 70.5, demonstrating that the non-linear affine modulation is essential beyond mere feature fusion." confidence: high relevance: high - claim_id: self_supervised_speech_emotion_models_provide_useful_word role: supports claim: "Self-supervised speech emotion models provide useful word-aligned supervision for fine-grained emotional TTS when combined with forced alignment." source: "§2.1, §4.2, Table 2" evidence: "emotion2vec frame-level features aligned via MFA and pooled to word boundaries form the annotation backbone; removing word-level data tuning causes the most severe degradation (FEDD DTW 49.62 to 133.97)." confidence: high relevance: medium - claim_id: frame_averaged_emotion_similarity_metrics_are_insufficient_for role: complicates claim: "Frame-averaged emotion similarity metrics are insufficient for evaluating intra-utterance emotional dynamics in TTS output." source: "§3.4" evidence: "The authors note that Emo SIM averages frame-level emotion vectors, potentially obscuring dynamic information, and introduce DTW as a complementary metric that more sensitively distinguishes the models' ability to track emotional transitions." confidence: high relevance: medium - claim_id: explicit_emotion_classification_as_an_auxiliary_training_objective role: supports claim: "Explicit emotion classification as an auxiliary training objective improves fine-grained emotion control in multi-task TTS training." source: "§4.2, Table 2" evidence: "Removing the emotion classification loss increases FEDD DTW from 49.62 to 73.96, a degradation comparable to removing the FiLM layer entirely." confidence: high relevance: medium limitations: - "The FEDD dataset is small (1,000 utterances, 5 speakers) and constructed partly via concatenation of emotionally distinct segments, which may not capture naturally occurring emotional transitions. The method is not evaluated for spontaneous or conversational speech, where emotion boundaries are less well-defined. The backbone (CosyVoice2) is frozen, so the approach inherits any limitations of that system in speaker diversity or prosodic range. Model size and speaker generalization are not reported." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.17021" published_date: "2025-09-21" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: exposure_bias_produces_measurable_quality_degradation_in_autoregressive role: supports claim: "Exposure bias produces measurable quality degradation in autoregressive codec TTS, with the effect scaling with output sequence length." source: "§3.4, Figure 3; §3.2, Table 1" evidence: "Token prediction accuracy under free-running inference is consistently lower than under teacher forcing, with the gap widening over sequence position; WER improvement from hybrid training is larger on long-form LibriSpeech (6.23→4.21%) than on short Seed-TTS utterances (4.83→4.64%)." confidence: high relevance: high - claim_id: hybrid_training_that_mixes_teacher_forced_and_self role: supports claim: "Hybrid training that mixes teacher-forced and self-generated tokens reduces WER and improves speaker similarity in autoregressive LM-based TTS relative to standard teacher-forcing fine-tuning." source: "§3.2, Table 1; §3.3, Figure 2" evidence: "Prompt-guided hybrid fine-tuning of CosyVoice2 on LibriSpeech reduced WER from 6.23% to 4.21% and raised speaker similarity from 0.74 to 0.78 on LibriSpeech test-clean (Table 1); human MOS scores approach ground-truth quality on 30-sample evaluation." confidence: high relevance: high - claim_id: the_benefits_of_exposure_bias_mitigation_in_autoregressive role: complicates claim: "The benefits of exposure-bias mitigation in autoregressive TTS diminish for short utterances where prediction errors have less opportunity to accumulate." source: "§3.2, Table 1" evidence: "On Seed-TTS utterances under 10 seconds, WER improvement is smaller (4.83→4.64) compared to LibriSpeech, where longer sequences amplify the compounding effect of distributional mismatch." confidence: high relevance: high - claim_id: eos_misprediction_rate_serves_as_a_reliable_proxy role: supports claim: "EOS misprediction rate serves as a reliable proxy for training-time exposure bias severity, enabling adaptive control of self-conditioning intensity." source: "§3.5; §3.6, Table 2" evidence: "EOS-guided adaptive scheduling tracks premature termination events across iterations; ablation shows removing this component raises WER from 4.21% to 4.98% and drops speaker similarity from 0.80 to 0.72 on LibriSpeech test-clean." confidence: high relevance: medium limitations: - "The evaluation is limited to LibriSpeech and Seed-TTS with a single speaker conditioning setup; generalisation to multilingual or highly expressive TTS domains is untested. The method is applied only as fine-tuning on top of existing CosyVoice models, leaving open whether hybrid training from scratch would yield similar or greater benefits. The MOS evaluation uses only 30 samples, which provides limited statistical power for assessing naturalness improvements. The 1.5x training overhead, while modest, is an additional cost relative to standard fine-tuning." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.17516" published_date: "2025-09-22" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, GAN] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: decoupling_the_training_pairing_between_the_speech_prompt role: supports claim: "Decoupling the training pairing between the speech prompt and the synthesis target (rather than using identical prompt/target audio) improves text-prosody alignment in style-controllable TTS, at the cost of an inflated speaker-similarity score under the coupled setup." source: "§3.3, Table 2" evidence: "Switching from non-decoupled to decoupled prompt/target training raises S-MOS from 3.45±0.09 to 3.82-3.93±0.06-0.07 while reducing speaker-similarity (SS) from 0.87 (interpreted as over-similar timbre/prosody) to 0.69-0.8." confidence: high relevance: medium - claim_id: providing_an_autoregressive_speech_language_model_with_explicit role: supports claim: "Providing an autoregressive speech language model with explicit surrounding-sentence text context improves narrative coherence in long-form speech generation beyond what sentence-isolated synthesis achieves." source: "§2.1, §3.3, Table 1" evidence: "Adding pre/post-context text sequences raises Chapter M-MOS from 3.88±0.07 (CosyVoice2 baseline, sentence-level synthesis) to 4.13±0.09 (Infer-ctx), and a qualitative example shows the model generating contextually appropriate laughter for a sentence with no explicit emotional markers when given the preceding sentence's context." confidence: high relevance: high - claim_id: the_strength_of_the_prompt_target_decoupling_threshold role: complicates claim: "The strength of the prompt/target decoupling threshold used during training trades off speaker-identity stability against naturalness, rather than improving both simultaneously." source: "§3.3, Table 2" evidence: "Sweeping the voiceprint-similarity clustering threshold shows a lower threshold reduces speaker-similarity (with occasional timbre discontinuities) but yields slightly higher S-MOS, while a higher threshold approaches the non-decoupled model's inflated similarity at the cost of lower S-MOS." confidence: high relevance: medium - claim_id: self_distillation_from_a_pretrained_emotional_tts_model role: supports claim: "Self-distillation from a pretrained emotional TTS model, filtered by objective quality metrics, can mitigate the scarcity of high-intensity emotional training data and improve controllable emotional expressiveness." source: "§2.2, §3.3, Table 3, Table 4" evidence: "On the CV3-Eval text-unrelated test set, the model trained with 500 hours of emotional data plus 5,300 hours of self-distilled augmentation achieves larger high- vs. low-intensity F1 discriminability than a CosyVoice2-instruct baseline (e.g. angry ΔF1 0.31 vs. 0.07) and higher mixed-emotion S-MOS (3.87±0.09 vs. 3.35±0.07)." confidence: high relevance: medium limitations: - "All training data (1M-hour audiobook corpus, 100K-hour context-aware corpus, 500-hour emotional corpus, 5K-hour augmentation set) and all three test sets (Test-NAR, Test-DIA, Test-CHAP) are proprietary and internally constructed; no code, model weights, or evaluation data are released, which limits independent verification of the reported gains." - "The evaluation is confined to Chinese-language audiobook content with a single subjective rater pool (50 native Chinese speakers); generalization to other languages or narration styles is untested. The decoupling-threshold ablation reports trends across tested values without identifying a clearly optimal setting, and the paper notes the risk of persona instability at high thresholds is hypothesized rather than directly measured across the full range. The authors flag future work on scaling chapter-level context data and applying reinforcement learning for further quality improvement." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.18531" published_date: "2025-09-23" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [preference_optimized_autoregression] claims: - claim_id: reinforcement_learning_with_automatically_verifiable_rewards_can_optimize role: complicates claim: "Reinforcement learning with automatically verifiable rewards can optimize the measured objective while degrading unmeasured perceptual qualities such as prosodic naturalness." source: "§4.3, Table 1, Figure 2" evidence: "GRPO trained on a harmonic-mean reward of CER and token likelihood reduced CER to the lowest value among all tested systems (2.20%) but produced the lowest human-preference ELO (753.7), with log-F0 pitch distributions showing collapse toward monotone speech relative to the baseline." confidence: high relevance: medium - claim_id: adding_an_auxiliary_reward_term_intended_to_fix role: complicates claim: "Adding an auxiliary reward term intended to fix a known RL side effect can destabilize training instead of correcting the targeted attribute." source: "§4.4" evidence: "Extending the GRPO reward with a speaker-similarity utility increased similarity scores but inflated CER to 42.63% and produced degenerate non-terminating outputs (missing end-of-sequence tokens), consistent with reward hacking rather than genuine prosody improvement." confidence: high relevance: medium - claim_id: small_scale_iterative_human_preference_optimization_can_recover role: supports claim: "Small-scale iterative human preference optimization can recover perceptual speech qualities lost to metric-driven RL training without requiring a large static preference dataset." source: "§4.5, Table 1" evidence: "Using approximately 200 human preference pairs per round with a moving DPO reference, three iterative rounds raised ELO from 753.7 (GRPO) to a peak of 1190.1 at round 2, exceeding three commercial TTS baselines in human preference, while CER recovered to 3.30-3.60% by rounds 2-3." confidence: high relevance: medium - claim_id: the_benefit_of_successive_rounds_of_iterative_preference role: refines claim: "The benefit of successive rounds of iterative preference optimization is not monotonic and can saturate or reverse as the policy converges toward its own reference." source: "§4.5" evidence: "ELO peaked at round 2 (1190.1) and declined at round 3 (1064.2); the authors attribute this to a narrowing policy-reference gap reducing the informativeness of newly collected preference pairs in later rounds." confidence: high relevance: medium limitations: - "Beyond the evaluation scope, the paper does not report an ablation over the DPO sharpness parameter (beta) or over preference-pair budget per round, so it is unclear how sensitive the round-2 peak is to these choices. The mechanism behind the round-3 ELO decline is offered as a hypothesis (narrowing policy-reference gap) rather than directly verified. The GRPO speaker-similarity failure is also reported as a single run rather than across multiple reward-weighting configurations, leaving open whether a different weighting could avoid the observed instability." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.19025" published_date: "2025-09-23" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: deterministic_nearest_neighbor_codeword_selection_in_residual_vector role: supports claim: "Deterministic nearest-neighbor codeword selection in residual vector quantization is fragile to small input perturbations, producing codeword reassignments that compound across RVQ stages into audible reconstruction artefacts." source: "§2.1, Figure 1" evidence: "Codeword-shift analysis on Encodec's first RVQ stage (24 kHz, 6 kbps) shows that although the noisy codeword usually matches the clean top-1 choice, a pronounced long tail of shifts (k>1) occurs when 15 dB SNR DEMAND noise is added to 120 clean VCTK utterances." confidence: high relevance: low - claim_id: neural_speech_codec_noise_robustness_can_be_improved role: supports claim: "Neural speech codec noise robustness can be improved without any paired noisy-clean training data by simulating perturbation-induced instability directly at the quantization step." source: "§3.4, Table 1" evidence: "Fine-tuning Encodec and WavTokenizer with distance-weighted probabilistic top-K sampling, trained exclusively on clean speech, improves SI-SDR, PESQ, STOI, and UTMOS at both 15 dB and 10 dB SNR with statistically significant paired t-tests (e.g., Encodec UTMOS 3.475 to 3.586 at 15 dB)." confidence: high relevance: high - claim_id: training_on_paired_noisy_clean_data_achieves_stronger role: complicates claim: "Training on paired noisy-clean data achieves stronger robustness than perturbation-only training under the specific noise conditions it was exposed to, but this advantage does not transfer to unseen noise types and comes at the cost of clean-speech quality." source: "§3.5.1, Tables 2-3" evidence: "A noise-exposed fine-tuning baseline (Closest*) outperforms the proposed clean-data-only method on some noisy-speech metrics (e.g., PESQ) under matched training/test noise, yet degrades clean-speech scores and is outperformed by the proposed method on all four metrics across three held-out noise types not present in either method's training." confidence: high relevance: low - claim_id: introducing_training_time_quantization_perturbations_in_a_curriculum role: refines claim: "Introducing training-time quantization perturbations in a curriculum-like schedule, from the finest residual quantizer stage toward the coarsest, stabilizes robustness training more effectively than perturbing all quantizer stages simultaneously." source: "§3.5.3, Figure 2" evidence: "PESQ and UTMOS improve steadily as probabilistic top-K sampling is progressively rolled out from the 6th to the 1st VQ of Encodec's RVQ, whereas applying the same sampling to all six VQs at once (\"Direct Top-K\") yields inferior results due to premature perturbation of core structural features." confidence: high relevance: low limitations: - "The evaluation is confined to a VCTK subset (a single-domain, studio-quality English read-speech corpus) and to relatively mild noise conditions (10-15 dB SNR from DEMAND). Whether the robustness gains extend to more severe noise, non-additive distortions (e.g., reverberation, codec cascading, packet loss), or multilingual/spontaneous speech is untested. The method is validated on only two codecs (Encodec, WavTokenizer); both use RVQ-style quantization, so its applicability to non-RVQ or single-codebook designs with different codebook geometries is unconfirmed. The paper itself notes extending the framework to more streamable architectures and integrating it with large speech-language models as future directions, but does not attempt either." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.19186" published_date: "2025-09-23" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure] claims: - claim_id: greedy_per_layer_code_selection_in_residual_vector role: supports claim: "Greedy per-layer code selection in residual vector quantization leaves quantization error on the table that can be recovered purely at test time, without retraining the codec." source: "§5.2, Table 3" evidence: "Beam-search encoding (B=16) lowers average L2 quantization error on LibriTTS from 5.096 to 4.625 for EnCodec and from 22.29 to 21.81 for HiFi-Codec relative to greedy (B=1) encoding of the same pre-trained checkpoints." confidence: high relevance: high - claim_id: reductions_in_rvq_quantization_error_obtained_by_a role: supports claim: "Reductions in RVQ quantization error obtained by a better search strategy translate into measurable gains on standard reconstruction-quality metrics, not just the raw error term." source: "§5.3, Table 1" evidence: "Increasing beam size from 1 to 16 improves PESQ, STOI, NISQA, and SI-SNR simultaneously for both EnCodec and HiFi-Codec on LibriTTS test-clean." confidence: high relevance: low - claim_id: adding_more_rvq_codebooks_higher_bit_rate_does role: refines claim: "Adding more RVQ codebooks (higher bit-rate) does not by itself eliminate the suboptimality of greedy encoding; the search-strategy gap persists even at high codebook counts." source: "§5.5, Table 4" evidence: "At 24 kbps with 32 codebooks, EnCodec's greedy encoding still trails beam-search encoding (PESQ 3.670 vs. 3.691, NISQA 3.992 vs. 4.001), a gap of comparable relative size to the one seen at 6 kbps with 8 codebooks." confidence: high relevance: low - claim_id: a_naive_implementation_of_wider_search_codec_encoding role: complicates claim: "A naive implementation of wider-search codec encoding introduces a latency cost that scales with search width, and only a parallel hardware-aware implementation avoids this trade-off." source: "§5.6, Table 5" evidence: "Sequential CPU beam-search encoding increases inference time by 285% from B=1 to B=16 (43.53 ms to 167.6 ms per 5-second clip), while a GPU-parallelized implementation of the same algorithm increases by only about 9% (6.780 ms to 7.366 ms) over the same range." confidence: high relevance: high limitations: - "The evaluation is confined to two pre-trained codec checkpoints (EnCodec, HiFi-Codec); the paper does not test the algorithm on more recent RVQ-GAN codec designs (e.g. DAC-style codecs) or on codecs with substantially larger codebook sizes, where the exponential O(S^L) exhaustive-search cost the method is designed to avoid is even more pronounced. Beam size B and the per-step candidate width k are always set equal in the experiments, so their individual contributions to the error reduction are not disentangled. All reported gains are measured on intrinsic reconstruction-quality metrics (PESQ, STOI, NISQA, SI-SNR, mel distance); the paper does not evaluate whether the improved codec reconstructions change downstream outcomes for systems that consume the codes, such as word error rate in a codec-token TTS or speech language model pipeline. The non-verbal vocalization test set is an internal, non-public dataset, limiting reproducibility of that portion of the evaluation." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.19592" published_date: "2025-09-23" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [fully_autoregressive_multicodebook, masked_parallel_codec_alternatives] claims: - claim_id: decoding_codebooks_of_a_multi_codebook_acoustic_frame role: supports claim: "Decoding codebooks of a multi-codebook acoustic frame with explicit intra-frame dependencies (iteratively) yields a generated token distribution closer to the ground truth than decoding all codebooks in parallel under an independence assumption." source: "§3.3.1, Fig. 2e" evidence: "Across all tested frame-stacking factors (1, 2, 4), every autoregressive- or MaskGIT-local-transformer configuration achieves lower Fréchet Distance than every parallel-sampled configuration, including the unstacked parallel baseline." confidence: high relevance: high - claim_id: offloading_intra_frame_codebook_decoding_to_a_small role: supports claim: "Offloading intra-frame codebook decoding to a small auxiliary transformer lets a primary acoustic decoder predict multiple codec frames per generation step, substantially increasing throughput without retraining the underlying codec at a lower frame rate." source: "§2.4, §3.3.2, Table 1, Fig. 2f" evidence: "At a frame-stacking factor of 2, the autoregressive local-transformer model reaches 2.1x throughput over the unstacked parallel baseline while improving Fréchet Distance and keeping WER, speaker similarity, and MOS within or better than baseline; the MaskGIT variant reaches 3.1x throughput at comparable quality." confidence: high relevance: high - claim_id: parallel_independent_codebook_prediction_degrades_disproportionately_not_just role: complicates claim: "Parallel independent codebook prediction degrades disproportionately, not just proportionally, as more codebook information is packed into a single decoding step." source: "§3.3.2" evidence: "Applying parallel sampling to a 2x frame-stacked model (instead of routing through the local transformer) increases unseen-speaker Fréchet Distance by 67% relative to the unstacked parallel baseline and lowers MOS." confidence: high relevance: medium - claim_id: the_throughput_gains_of_iterative_masked_prediction_decoding role: complicates claim: "The throughput gains of iterative masked-prediction decoding for acoustic codebooks come at a quality cost that grows sharply once the number of sampling steps is small relative to the number of tokens being resolved per step." source: "§3.3.2, Fig. 2a" evidence: "At a stacking factor of 4, the MaskGIT local transformer with 3 sampling steps decoding 32 tokens per step (8 codebooks × 4 stacked frames) shows a significant MOS drop relative to baseline, while the autoregressive local transformer at the same stacking factor does not exhibit this drop." confidence: high relevance: high limitations: - "Robustness to unseen speakers degrades substantially at higher frame-stacking factors: unseen-speaker speaker similarity falls from 0.765 at stacking factor 1 to 0.642 (AR LT) and 0.624 (MaskGIT LT) at stacking factor 4, which the authors themselves flag by recommending high stacking only \"when not needing zero-shot functionality.\"" - "All experiments build on a single base system (Koel-TTS) and a single codec (NanoCodec, FSQ-based, 8 codebooks at 21.5 fps); it is untested whether the same tradeoffs hold for RVQ-based codecs, different codebook counts, or other primary-decoder architectures. The MaskGIT local transformer's degradation at high stacking is attributed to using only 3 sampling steps, but the paper does not run the ablation that would confirm more steps recover quality, leaving the speed-quality Pareto frontier for MaskGIT only partially characterized. Training data is described only as \"the same 18k hours of data as in the Koel-TTS paper,\" with no further specification of language, speaker count, or domain in this paper itself." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2505.10599" published_date: "2025-09-25" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, GAN, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer] claims: - claim_id: combining_discrete_emotion_labels_and_continuous_dimensional_emotion role: supports claim: "Combining discrete emotion labels and continuous dimensional emotion representations within one conditioning scheme, rather than choosing one, expands the practically controllable range of emotional TTS beyond what either representation supports alone." source: "§4.4, §4.6, Figure 4" evidence: "Semi-supervised training across label-only, ADV-only, and jointly-annotated datasets raises ADV-space controllable coverage from 70% (training on jointly-annotated data alone) to 89.35% (full semi-supervised training), including synthesis of previously unseen ADV combinations." confidence: high relevance: medium - claim_id: the_reliability_of_continuous_emotion_control_depends_heavily role: complicates claim: "The reliability of continuous emotion control depends heavily on how the continuous space is quantized into controllable units, not just on the underlying representation." source: "§4.4, Table 2, §4.6" evidence: "Replacing the clustering-based nonlinear ADV quantizer with linear binning drops Spearman's Rank Correlation for perceived linear emotion control across all three ADV dimensions (e.g., valence SRC falls from 0.92 to 0.57), showing that imbalanced raw annotation distributions bias the model toward overrepresented regions of the space unless the quantizer compensates." confidence: high relevance: medium - claim_id: predicting_an_emotion_conditioning_signal_directly_from_input role: supports claim: "Predicting an emotion conditioning signal directly from input text allows emotional TTS to operate without an explicit emotion label or reference at inference time." source: "§4.5, §4.6, Table 3" evidence: "An ADV predictor infers pseudo-ADV tokens from text alone; removing it biases synthesis toward neutral emotion and drops pairwise preference against baselines from 67.33%/58.60% to 46.88%/28.50% in end-to-end preference tests." confidence: high relevance: medium - claim_id: fine_grained_control_over_continuous_emotion_representations_remains role: complicates claim: "Fine-grained control over continuous emotion representations remains bounded by the consistency of the human annotations used to train the control mechanism." source: "§Limitations" evidence: "The authors note that subjective inter-annotator variation in ADV labels degrades linear control accuracy, and that texts with ambiguous emotional attributes cause the ADV predictor to infer inappropriate values, since the same text can express different emotions in different contexts." confidence: high relevance: medium limitations: - "Training and evaluation are restricted to English-language datasets; the paper does not report whether the semi-supervised ADV/label fusion strategy transfers to multilingual emotional speech. The ADV predictor's text-only inference mode is explicitly noted to fail on texts with ambiguous emotional attributes, since a single sentence can carry different emotions depending on context that the model cannot observe. The framework also depends on the quality and consistency of ADV annotations, which the authors identify as inherently noisy due to subjective variation across human annotators; they suggest more data and more consistent annotation as the primary mitigation rather than a modeling fix. Speaker conditioning uses a fixed embedding averaged from a speaker's neutral-emotion utterances rather than zero-shot cloning from an arbitrary reference clip, which is a narrower speaker-control setting than several of the LLM-based baselines it is compared against." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.20802" published_date: "2025-09-25" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: many_transformer_layers_in_autoregressive_llm_tts_backbones role: supports claim: "Many transformer layers in autoregressive LLM-TTS backbones contribute little to synthesis quality and can be removed with minimal loss in naturalness and speaker similarity." source: "§4.1, Table 1" evidence: "Halving CosyVoice 2's depth to 12 layers (39.7% fewer parameters) increases Seed-TTS WER by only 0.68 and decreases NMOS by 0.13, while speaker similarity and UTMOS remain essentially unchanged." confidence: high relevance: high - claim_id: layer_importance_criteria_developed_for_pruning_text_only role: refines claim: "Layer-importance criteria developed for pruning text-only LLMs do not transfer directly to speech generation backbones; intelligibility-grounded criteria are needed to identify prunable layers correctly." source: "§2.1, §4.2, Table 2" evidence: "Cosine-based layer importance (input/output latent similarity) diverges from WER-based importance in TTS backbones, and substituting it for the proposed WER-based criterion increases WER from 1.59 to 1.74 and CER from 0.54 to 0.61 on LibriTTS test-clean." confidence: high relevance: high - claim_id: knowledge_distillation_can_recover_most_of_the_performance role: supports claim: "Knowledge distillation can recover most of the performance lost from aggressive layer pruning in speech-generation LLMs using only a small fraction of the original pretraining data." source: "§3, §4.1, Table 1b" evidence: "Fine-tuning pruned variants required under 5% of the original pretraining data (25% of LibriTTS for CosyVoice 2, an upper-bounded 12.5% of LibriHeavy for LLaSA) yet speaker similarity and UTMOS changed by at most 0.045 and 0.04 respectively relative to the uncompressed backbones." confidence: high relevance: medium - claim_id: robustness_to_layer_pruning_varies_substantially_across_llm role: complicates claim: "Robustness to layer pruning varies substantially across LLM-TTS backbones depending on how redundant their transformer layers are, so a single pruning ratio does not generalize uniformly." source: "§4.1" evidence: "LLaSA showed a larger relative quality drop after 50% layer pruning (speaker similarity −0.045, UTMOS −0.04) than CosyVoice 2 at the same pruning ratio, attributed to LLaSA's WLI values being more uniformly high across layers, indicating less exploitable redundancy." confidence: high relevance: medium limitations: - "The evaluation covers only two backbones (CosyVoice 2 and LLaSA-1B) and English-only test sets (LibriTTS test-clean, Seed-TTS test-en), so it is untested whether the WER-based pruning criterion and dynamic distillation scheme generalize to other LLM-TTS architectures, multilingual settings, or streaming inference. The more aggressive pruning configurations (9-layer CosyVoice 2) trade a larger, unquantified increase in WER for additional speed and memory gains, and the paper does not characterize where this trade-off becomes unacceptable for deployment. Computing WLI itself requires running WER evaluation over a data subset for each candidate layer removal, adding an upfront cost to the pruning procedure that is not reported in wall-clock terms." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.21968" published_date: "2025-09-26" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [codec] architecture: [GAN, VAE] relevance: medium evidence_role: - infrastructure current_role: active_evidence method_family: [codec_tokenizer_infrastructure, semantic_acoustic_token_cascade] claims: - claim_id: a_shared_single_codebook_can_be_structured_with role: supports claim: "A shared single codebook can be structured with overlapping, nested domain partitions rather than rigid disjoint splits, improving both reconstruction and downstream generation quality relative to a rigid-split design." source: "§4.4, Tables 3–4" evidence: "Ablating codebook design at fixed codebook size and data scale, the nested codebook reduces reconstruction WER from 4.21 (rigid-split) to 3.99 and downstream TTS generation WER from 6.26 to 4.99 on LibriSpeech-PC test-clean." confidence: high relevance: low - claim_id: distilling_frame_level_representations_from_multiple_domain_specific role: supports claim: "Distilling frame-level representations from multiple domain-specific self-supervised teacher models into one acoustic codec can improve reconstruction and generation quality across all the covered domains simultaneously, not just the domain of a single teacher." source: "§3.3, §4.4, Tables 3–4" evidence: "Adding multi-domain distillation (WavLM for speech, MuQ for vocal/music, BEATs for sound) on top of the nested codebook further raises the speech-partition code-usage ratio from 37.1% to 38.2% and improves downstream TTS generation WER from 4.99 to 4.51." confidence: high relevance: high - claim_id: unifying_multiple_audio_domains_into_a_single_shared role: complicates claim: "Unifying multiple audio domains into a single, shared quantization codebook does not close the gap with domain-specific single-layer codecs on every reconstruction metric, even when the unified model uses a larger codebook and lower token rate." source: "§4.2, Table 1" evidence: "On LibriSpeech test-clean, AUV's PESQ-WB (2.40) and SPK-SIM (0.81) trail dedicated speech codecs such as DAC (4.01, 0.95) and are roughly on par with, not clearly better than, BigCodec and UniCodec." confidence: high relevance: high - claim_id: larger_unified_codebooks_intended_to_accommodate_more_audio role: complicates claim: "Larger unified codebooks intended to accommodate more audio domains can hurt downstream autoregressive generation quality by increasing the modeling burden on the generative model consuming the tokens, even when reconstruction quality is unaffected." source: "§4.3, Table 4" evidence: "Scaling the codebook from 16,384 to 20,480 entries (C0 vs. C2) yields slightly worse downstream generation WER (4.51 vs. 4.89) despite improved reconstruction metrics; training an autoregressive model on codes from a still-larger 131,072-entry codebook (MagiCodec) reportedly failed outright." confidence: high relevance: high limitations: - "The paper reports no total parameter count for the AUV encoder-decoder, limiting direct efficiency comparison with baselines whose sizes are known. Downstream generative evaluation is restricted to a single autoregressive TTS backbone (EmoVoice) trained on a comparatively small 1K-hour subset of LibriSpeech, so it is unclear whether the reconstruction and generation gains hold at larger downstream training scales or with non-autoregressive generators. Speaker similarity in the downstream TTS setting remains low in absolute terms (SPK-SIM ≈ 0.43–0.44) across all codecs tested, including AUV, suggesting the codec-level improvements shown here do not yet translate into strong speaker fidelity for generated speech. The domain-label input used during training but withheld at inference creates a train/inference mismatch whose effect on codebook index selection is only indirectly probed via the reported index-distribution statistics, not directly ablated." caveats: - "This paper contributes codec infrastructure used by speech language models rather than a complete TTS system." - id: "2509.22062" published_date: "2025-09-26" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [fully_autoregressive_multicodebook, semantic_acoustic_token_cascade] claims: - claim_id: injecting_explicit_linguistic_structure_into_the_primary_codebook role: supports claim: "Injecting explicit linguistic structure into the primary codebook of a neural speech codec reduces the downstream language model's learning burden and improves synthesis intelligibility." source: "§4.3, Table 4" evidence: "Removing the semantic distillation loss during codec-conditioned TTS training raises WER from 3.31% to 3.97% on SeedTTS-test, 9.74% to 11.83% on PGC-Hard, and 16.57% to 18.34% on PGC-Poly, with SIM and UTMOS also degrading." confidence: high relevance: high - claim_id: an_automatic_speech_recognition_model_can_serve_as role: supports claim: "An automatic speech recognition model can serve as an effective semantic teacher for codec distillation, as an alternative to self-supervised speech representation models." source: "§E.2, Table 5" evidence: "S3Codec distills Whisper encoder embeddings (rather than HuBERT/SSL features) into the first RVQ level; a small model trained with S3Codec reaches 3.30% WER on SeedTTS-test vs. 4.21% for an otherwise identical model using undistilled DAC tokens." confidence: high relevance: high - claim_id: fully_autoregressive_tts_systems_that_omit_an_explicit role: complicates claim: "Fully autoregressive TTS systems that omit an explicit continuous acoustic-feature conditioning stage (e.g., mel-spectrogram or speaker-similarity-vector guidance) tend to underperform hybrid AR+NAR or flow-matching systems on speaker similarity even when intelligibility is competitive." source: "§4.2, Table 2-3" evidence: "CaT-TTS reports SIM of 0.668-0.678 across test sets versus 0.71-0.80 for Seed-TTS, CosyVoice 2/3, and F5-TTS, despite comparable or better WER among AR-only baselines." confidence: high relevance: high - claim_id: test_time_parallel_decoding_with_learned_input_dependent role: refines claim: "Test-time parallel decoding with learned, input-dependent aggregation weights can reduce autoregressive error accumulation at near-zero added latency, but the achievable robustness gain is bounded by how many parallel streams are used, trading GPU utilization against benefit." source: "§4.3, §3.3, Figure 3-4" evidence: "MAPI ablation across increasing parallel-stream counts shows WER improving and becoming more stable across 10 repeated inferences per sample, while the authors note GPU resource utilization rises correspondingly and stream count must be tuned per deployment scenario." confidence: high relevance: high limitations: - "Training relies on an unreleased proprietary corpus (~200k hours, ~85% Chinese / ~15% English), and neither code nor a demo is available, which limits independent verification of the reported results." - "The semantic-distillation ablation (removing the loss) and the MAPI ablation are both run on smaller sub-datasets and reduced-size \"CaT-TTS-small\" models rather than the full 0.4B system, so it is not established that the same magnitude of gains transfers to the full-scale model. Speaker similarity remains a clear weak point relative to hybrid and NAR baselines, which the authors attribute to the deliberate absence of continuous acoustic conditioning rather than treat as a target for improvement. The evaluation is also dominated by Chinese-language and Chinese-out-of-domain test sets (PGC-Hard, PGC-Poly, Seed-TTS test-zh/test-hard), with comparatively less English-language evidence." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.24570" published_date: "2025-09-29" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, VC, evaluation] architecture: [autoregressive-LM] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [scaled_general_ar_codec_tts] claims: - claim_id: automated_pipelines_combining_expressive_tts_voice_conversion_and role: supports claim: "Automated pipelines combining expressive TTS, voice conversion, and LLM-based instruction generation can produce large-scale paired speech style editing data without manual recording or annotation, provided a multi-criterion filtering step is applied." source: "§2.1, §2.2, Fig. 3" evidence: "The three-stage pipeline (EmoCapTTS synthesis + Chatterbox voice conversion + Qwen3-8B instruction generation) yields 382 hours and ~100,000 pairs from EARS and Expresso source material, filtered to WER < 10, style similarity > 0.5, and speaker similarity > 0.5 *(§2.1, §2.2, Fig. 3)*." confidence: medium relevance: medium - claim_id: fine_grained_diverse_natural_language_instructions_improve_both role: supports claim: "Fine-grained, diverse natural-language instructions improve both in-domain accuracy and cross-domain generalization of instruction-guided speech style editing models relative to coarse, templated instruction sets." source: "§4.2, Table 2" evidence: "LlasaEdit trained on ISSE outperforms the same architecture trained on ESD across WER, style similarity, speaker similarity, and UTMOS in-domain (8.06 vs. 10.07 WER; 0.68 vs. 0.64 style-sim), and the ISSE-trained model's cross-domain performance on ESD exceeds the ESD-trained model's in-domain performance on several metrics *(§4.2, Table 2)*." confidence: medium relevance: medium - claim_id: instruction_guided_style_editing_models_trained_on_narrow role: complicates claim: "Instruction-guided style editing models trained on narrow, templated-instruction datasets fail catastrophically when evaluated on more diverse, fine-grained instruction distributions." source: "§4.2, Table 2" evidence: "The ESD-trained LlasaEdit model, when evaluated on the ISSE test set, produces a WER of 68.17, compared to 10.07 on its own in-domain ESD test set, indicating the model does not generalize beyond the coarse single-attribute instructions it was trained on *(§4.2, Table 2)*." confidence: medium relevance: medium - claim_id: isolating_style_variation_from_speaker_identity_in_synthetically role: complicates claim: "Isolating style variation from speaker identity in synthetically generated paired training data requires an explicit voice-conversion correction step, since expressive TTS models conditioned on style descriptions alone conflate style and timbre changes." source: "§2.1" evidence: "EmoCapTTS-generated stylized speech differs from the anchor speech in timbre because the model lacks explicit speaker control; a separate voice conversion stage (Chatterbox) is needed to re-align target timbre to the anchor speaker before the pair can be used to define a style-only edit *(§2.1)*." confidence: medium relevance: medium limitations: - "The dataset and benchmark are limited to English, which the authors explicitly flag as constraining applicability to multilingual editing scenarios. The generated portion of ISSE (292 of 382 hours) is itself the product of a TTS+VC synthesis pipeline rather than real recordings, so any systematic biases or artifacts introduced by EmoCapTTS or Chatterbox could propagate into models trained on it; the quality-filtering thresholds (WER < 10, similarity > 0.5) are relatively loose and their effect on downstream editing fidelity is not separately ablated. The benchmark comparison is against a single alternative dataset (ESD) and a single model architecture (LlasaEdit); no comparison is made against other instruction-guided editing systems such as InstructSpeech, and no ablation isolates the individual contribution of instruction diversity versus raw data scale." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.24650" published_date: "2025-09-29" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS, VC] architecture: [autoregressive-LM, diffusion] relevance: high evidence_role: - core_evidence current_role: influential method_family: [semantic_acoustic_token_cascade, hybrid_ar_token_continuous_renderer] claims: - claim_id: a_differentiable_scalar_quantization_bottleneck_applied_to_hidden role: supports claim: "A differentiable scalar-quantization bottleneck applied to hidden states, rather than used as a discrete prediction target, can induce semantic/acoustic task separation inside a continuous autoregressive TTS model without an external speech tokenizer." source: "§4.3, Table 6" evidence: "Removing the FSQ bottleneck from an otherwise identical hierarchical architecture increases ZH-hard-case CER from 18.19% to 24.92%, while FSQ dimensionality shows a non-monotonic optimum around 128-256 dimensions rather than monotonic improvement with capacity." confidence: high relevance: high - claim_id: explicitly_separating_acoustic_detail_recovery_into_a_dedicated role: supports claim: "Explicitly separating acoustic detail recovery into a dedicated residual module improves robustness on complex inputs beyond what a single semantic language model plus a diffusion decoder achieves." source: "§4.4, Table 7" evidence: "Removing the RALM (TSLM output feeding the diffusion decoder directly, architecturally close to DiTAR) degrades EN-WER from 2.98% to 4.34% and ZH-hard-case CER from 18.19% to 25.0%; removing the historical acoustic embedding from the RALM input degrades results further." confidence: high relevance: high - claim_id: learning_rate_schedule_design_not_just_architecture_materially role: complicates claim: "Learning-rate schedule design, not just architecture, materially affects zero-shot speaker similarity in large-scale continuous TTS training." source: "§4.5, Table 8" evidence: "A two-phase Warmup-Stable-Decay schedule's decay phase alone improves ZH-hard-case CER from 13.22% to 8.87% and SIM by 4.4 points over the stable-phase-only checkpoint on an otherwise identical model." confidence: high relevance: medium - claim_id: classifier_free_guidance_strength_in_diffusion_based_tts role: complicates claim: "Classifier-free guidance strength in diffusion-based TTS decoders trades off intelligibility against speaker similarity non-monotonically, with both very low and very high guidance scales degrading both metrics simultaneously." source: "§4.6, Table 9" evidence: "CFG scale 1.0 (no guidance) yields EN-WER 16.32% and SIM 55.1%, while scale 5.0 yields EN-WER 12.78% and SIM 60.7%; the optimum at scale 2.0 achieves EN-WER 1.85% and SIM 72.9%, with degradation on both sides of the optimum." confidence: high relevance: medium - claim_id: removing_dependency_on_a_pre_trained_discrete_speech role: refines claim: "Removing dependency on a pre-trained discrete speech tokenizer does not require sacrificing zero-shot voice cloning quality relative to discrete-token-based open-source TTS systems." source: "§4.2, Table 3" evidence: "On SEED-TTS-EVAL, the fully continuous VoxCPM reports SIM of 72.9% (EN) and 77.2% (ZH), exceeding the discrete-token-based IndexTTS2 and CosyVoice2 baselines on the same benchmark." confidence: high relevance: high limitations: - "Multilingual capability is limited to Chinese and English by construction; the paper explicitly reports uncertain generalization to other languages, and prosody/emotion control lacks any intuitive or precise user-facing conditioning mechanism. The causal audio VAE operates at 16kHz, which the authors acknowledge falls short of the 24kHz or 44.1kHz sampling rates typically expected for high-fidelity applications. Baseline comparisons draw on official implementations or numbers reported in prior papers rather than a uniformly controlled re-evaluation, so cross-system rankings on tables that mix reproduced and self-reported numbers should be read cautiously. The training corpus (1.8M hours) is internal and not released, which limits independent reproduction of the full-scale result even though code and weights for the trained model are public." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.25131" published_date: "2025-09-29" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [SCA, TTS] architecture: [autoregressive-LM, flow-matching, hybrid] relevance: high evidence_role: - core_evidence current_role: active_evidence method_family: [hybrid_ar_token_continuous_renderer, unified_speech_text_language_model] claims: - claim_id: chunking_text_into_aligned_segments_with_a_short role: supports claim: "Chunking text into aligned segments with a short token-delay before speech decoding reduces error accumulation in long-form autoregressive speech generation." source: "§4.2, Table 6" evidence: "Removing chunk-based decoding raises Long-TTS-Eval error rates above those of concurrent long-form TTS baselines, and with it enabled MGM-Omni-TTS-2B achieves EN-hard WER 26.26 versus 42.48-98.61 for CosyVoice2, MOSS-TTSD-v0.5, and Higgs-Audio-v2." confidence: high relevance: high - claim_id: multi_token_parallel_decoding_is_not_restricted_to role: supports claim: "Multi-token parallel decoding is not restricted to RVQ speech tokenizers and can be applied effectively to finite scalar quantization (FSQ) tokenizers." source: "§4.2, Table 6" evidence: "Increasing parallel decoding size on the CosyVoice2 FSQ tokenizer maintains TTS quality on Seed-TTS-Eval while cutting inference RTF by roughly 3x at parallel size 4." confidence: high relevance: high - claim_id: increasing_the_parallel_decoding_size_trades_off_synthesis role: complicates claim: "Increasing the parallel decoding size trades off synthesis error rate against inference speed rather than improving both simultaneously." source: "§4.2" evidence: "Larger parallel sizes in the ablation slightly raise audio error rate even as they substantially accelerate inference, leading the authors to select a parallel size of 4 as a balance point." confidence: high relevance: medium - claim_id: separating_multimodal_reasoning_from_speech_synthesis_into_distinct role: refines claim: "Separating multimodal reasoning from speech synthesis into distinct model components can improve long-form audio understanding without sacrificing speech generation efficiency." source: "§4.1.1, §4.1.3, Figure 5, Table 5b" evidence: "The dual-track brain-mouth design lets the MLLM handle needle-in-the-haystack audio inputs up to 4,500 seconds while the SpeechLM independently achieves the lowest RTF among compared long-form TTS systems." confidence: high relevance: high limitations: - "The long-form evaluation itself is partly self-authored: Long-TTS-Eval is introduced by this paper, and while its construction and normalized-text scoring procedure are documented, results on it cannot yet be cross-checked against independent replications. The comparison in Table 5b is limited to three baseline systems, and the qualitative long-speech examples in the appendix (a classical Chinese poem and a code-switched English-Chinese poem) are illustrative rather than a systematic error analysis. The paper does not report results on emotion or prosody control, nor does it evaluate robustness to reference audio recorded in noisy or far-field conditions. The 32B MLLM variant's long-form and vision-speech results are mixed relative to the 7B variant (e.g., lower TextVQA-Speech and EN-hard performance context is not directly reported for 32B TTS), suggesting scaling benefits are not uniform across all sub-tasks." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." - id: "2509.26514" published_date: "2025-09-30" entry_date: "2026-07-25" year: 2025 venue: "arXiv" task: [TTS] architecture: [autoregressive-LM, flow-matching, GAN] relevance: high evidence_role: - core_evidence current_role: influential method_family: [hybrid_ar_token_continuous_renderer, preference_optimized_autoregression] claims: - claim_id: decoupling_instruction_interpretation_from_speech_rendering_via_an role: supports claim: "Decoupling instruction interpretation from speech rendering, via an explicit intermediate textual feature representation, can match or exceed instruction-annotated controllable TTS systems without requiring any manually labeled instruction-speech data." source: "§3.2, Table 1" evidence: "BatonVoice-1.7B reaches 57.6% emotion accuracy using 0 hours of manual instruction annotation, exceeding CosyVoice (43.8%, 556 hours) and CosyVoice2 (37.8%, 1,500 hours) on the same Seed-TTS-derived English emotion benchmark." confidence: high relevance: medium - claim_id: when_conditioning_signals_are_represented_as_explicit_text role: supports claim: "When conditioning signals are represented as explicit text rather than learned embeddings, a downstream generation model can gain synthesis quality from a stronger upstream language model without any retraining of the generation model itself." source: "§3.5, Figure 3b" evidence: "Holding BatonTTS fixed and only changing the external \"conductor\" LLM that produces the vocal plan, emotion accuracy rises monotonically from 29.8% (Qwen3-1.7B) through 39.8% (Qwen3-80B) and 47.8% (Qwen3-Max) to 57.6% (Gemini-2.5-Pro)." confidence: high relevance: high - claim_id: automatic_emotion_classification_accuracy_and_human_judged_naturalness role: complicates claim: "Automatic emotion-classification accuracy and human-judged naturalness can diverge, so strong performance on an LLM-judged style-control metric does not guarantee a favorable human preference outcome against commercial systems." source: "§3.3, Table 2" evidence: "On a free-form instruction-following test set judged by trained human annotators, BatonVoice wins only 56% of comparisons against CosyVoice and just 30% against Minimax-2.5-HD, with annotators specifically citing weaker fluency and naturalness." confidence: high relevance: medium - claim_id: numerical_structured_representations_of_prosodic_control_targets_transfer role: refines claim: "Numerical, structured representations of prosodic control targets transfer more precisely to a conditioned TTS decoder than free-text qualitative descriptions of the same target style." source: "§B.2, Table 5" evidence: "On a RAVDESS reconstruction task, the structured numerical vocal-plan format achieves 1.54 Mel-Cepstral Distortion versus 2.62 for an equivalent caption-based qualitative description, and ablating any single numerical feature (pitch, energy, or spectral centroid) increases MCD." confidence: high relevance: medium limitations: - "The paper's headline emotion-control results across both English and Chinese benchmarks rely entirely on an LLM (Gemini-2.5-Pro) as an automatic judge rather than human raters; the one benchmark that does use human evaluation shows BatonVoice losing to the top closed-source baseline on naturalness and fluency (30% win rate against Minimax-2.5-HD), a gap not visible in the automated emotion-accuracy metric." - "The vocal-plan representation is limited to pitch, energy, and spectral centroid; the authors note that finer-grained paralinguistic features such as emphatic stress and non-verbal vocalizations are not captured and are left to future work. Best reported results depend on using a large, capable, and likely costly external LLM (Gemini 2.5 Pro) as the conductor at inference time; using the in-house model of the same size as the orchestra as its own conductor (Qwen3-1.7B, 29.8% accuracy) performs far worse, so the practical quality of the system is bottlenecked by conductor access and cost, not by BatonTTS alone. Cross-lingual generalization is demonstrated on one unseen language (Chinese) with instructions and text machine-translated by the same LLM family used for judging, which leaves open how the approach performs on languages more typologically distant from English or evaluated independently of the conductor/judge model." caveats: - "This paper directly evaluates an autoregressive speech-token or codec-language-model TTS system." claim_clusters: - id: codec_lm_enables_zero_shot_in_context_voice_cloning claim: Conditional language modeling over discrete speech tokens enables zero-shot voice cloning through in-context acoustic prompting. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2306.12925' - '2308.16692' - '2402.08093' - '2403.16973' - '2406.02430' - '2406.04904' - '2409.00750' - '2409.03283' - '2412.15649' - '2502.05512' - '2502.07243' - '2503.01710' - '2503.14345' - '2507.02380' - 2025.acl-long.1043 - 2025.acl-long.654 - 2025.acl-short.81 - '2508.04141' - '2508.09767' - interspeech-2025-0319 - interspeech-2025-1993 - '2508.16332' - '2502.11128' - '2506.21619' - '2509.15969' contradicting_papers: [] refining_papers: - iclr-2025-cuFzE8Jlvb - 2025.findings-naacl.279 - '2505.07916' - '2505.17589' - '2507.12197' - interspeech-2025-0989 - '2509.05863' - '2412.16846' - '2509.14784' - '2509.24650' caveats: - Prompt length, prompt noise, and training-set speaker coverage remain major confounds in reported speaker-similarity gains. last_reviewed: '2026-07-25' - id: semantic_acoustic_hierarchy_balances_coherence_and_fidelity claim: Separating semantic and acoustic token prediction improves linguistic coherence while retaining high-fidelity waveform detail. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2301.11325' - '2305.09636' - '2308.16692' - '2410.00037' - '2410.03751' - '2411.01156' - '2502.11946' - '2502.17239' - '2505.13000' - '2506.10274' - '2506.23325' - 2025.acl-long.1043 - 2025.acl-long.1498 - 2025.acl-long.682 - 2025.findings-acl.101 - '2508.14049' - '2508.04141' - '2508.19205' - '2506.21619' - '2509.09174' - '2509.11425' - '2509.24650' contradicting_papers: [] refining_papers: - '2502.06490' - interspeech-2025-1776 - '2509.14882' caveats: - Semantic supervision can suppress prosody or speaker information when the separation objective is too strong. last_reviewed: '2026-07-25' - id: tokenizer_quality_is_downstream_generation_bottleneck claim: Tokenizer semantic alignment, bitrate, and reconstruction quality materially constrain downstream speech-language-model performance. status: strongly_supported confidence: high supporting_papers: - '2305.02765' - '2306.12925' - '2308.16692' - '2408.16532' - '2410.03751' - '2412.02612' - '2502.11946' - 2025.naacl-srw.6 - '2505.17589' - '2506.10274' - 2025.acl-long.1498 - interspeech-2025-0669 - interspeech-2025-1084 - '2509.11425' - '2509.22062' contradicting_papers: [] refining_papers: - '2507.12197' - '2509.24650' caveats: - Codec reconstruction rankings do not transfer consistently across TTS architectures, domains, or languages. last_reviewed: '2026-07-25' - id: rvq_structure_supports_staged_generation claim: Residual-vector-quantizer structure supports staged generation in which early codebooks carry linguistic or speaker information and later codebooks refine acoustic detail. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2308.16692' - '2406.07855' - '2502.17239' - iclr-2025-868masI331 - '2505.13000' - '2506.10274' - 2025.acl-long.1498 - interspeech-2025-0468 contradicting_papers: [] refining_papers: [] caveats: - The assumed semantic-to-acoustic ordering is tokenizer-dependent and is not cleanly present in every RVQ codec. last_reviewed: '2026-07-25' - id: autoregressive_decoding_creates_latency_cost claim: Causal token-by-token speech generation creates a deployment-relevant latency and throughput cost relative to parallel decoding. status: strongly_supported confidence: high supporting_papers: - '1609.03499' - '2305.09636' - '2406.07855' - '2408.02622' - '2505.02625' - '2507.02380' - 2025.acl-long.912 - 2025.findings-acl.1051 - interspeech-2025-2447 - '2509.15969' contradicting_papers: [] refining_papers: - 2025.findings-acl.101 - '2509.22062' - '2509.25131' caveats: - Streaming can reduce time to first audio without eliminating total sequential decoding cost. last_reviewed: '2026-07-25' - id: alignment_failures_remain_primary_ar_robustness_limit claim: Autoregressive codec TTS remains vulnerable to omissions, repetitions, insertions, and runaway decoding caused by text–speech alignment failures. status: strongly_supported confidence: high supporting_papers: - '2401.07333' - '2406.05370' - '2407.08551' - '2507.16632' - interspeech-2025-0253 - interspeech-2025-0551 - interspeech-2025-1641 - '2508.15442' contradicting_papers: [] refining_papers: - '2507.01348' caveats: - WER captures only part of this failure class and can miss perceptually severe prosodic or looping errors. last_reviewed: '2026-07-25' - id: explicit_alignment_constraints_improve_robustness claim: Explicit monotonic, duration, phoneme-interleaving, or sequence-reordering constraints improve autoregressive TTS robustness. status: strongly_supported confidence: high supporting_papers: - '2401.07333' - '2404.03204' - '2406.02430' - '2406.07855' - '2410.17799' - '2411.00774' - iclr-2025-cuFzE8Jlvb - '2507.21138' - interspeech-2025-0551 - interspeech-2025-0874 - interspeech-2025-2447 - '2509.20378' contradicting_papers: [] refining_papers: - '2509.05863' - '2509.11425' caveats: - Several methods depend on external aligners or duration labels that may fail on spontaneous or low-resource speech. last_reviewed: '2026-07-25' - id: scaling_data_and_models_improves_ar_codec_tts claim: Increasing speech data and model capacity generally improves intelligibility, speaker fidelity, and expressive generalization in autoregressive codec TTS. status: strongly_supported confidence: high supporting_papers: - '2301.02111' - '2303.03926' - '2305.07243' - '2305.11000' - '2306.12925' - '2402.01912' - '2402.05755' - '2402.08093' - '2403.16973' - '2406.02430' - '2406.18009' - '2409.03283' - '2409.05377' - '2410.17799' - '2411.00774' - '2411.19842' - '2502.04128' - '2502.05512' - '2502.11946' - iclr-2025-868masI331 - '2505.17589' - '2507.21138' - 2025.acl-long.654 - interspeech-2025-0551 - interspeech-2025-2564 - '2509.13068' - '2509.14784' - '2509.14882' - '2509.15969' - '2509.18531' - '2509.24570' contradicting_papers: [] refining_papers: - '2507.01348' - 2025.acl-long.937 - interspeech-2025-0989 caveats: - Most scaling comparisons jointly change data, model size, tokenizer, and training recipe, so causal attribution remains weak. last_reviewed: '2026-07-25' - id: multilingual_codec_lms_transfer_voice_across_languages claim: Multilingual codec language models can transfer speaker identity across languages from short acoustic prompts without paired bilingual recordings. status: strongly_supported confidence: high supporting_papers: - '2303.03926' - '2308.16692' - '2406.04904' - '2412.15649' - 2025.acl-short.81 - '2508.08715' - interspeech-2025-0551 contradicting_papers: [] refining_papers: - 2025.findings-naacl.279 caveats: - Accent naturalness and speaker similarity often trade off, and evidence is concentrated in a small set of language pairs. last_reviewed: '2026-07-25' - id: unified_speech_text_lms_enable_spoken_generation claim: A shared autoregressive vocabulary over speech and text enables unified understanding and spoken generation in one language-model backbone. status: strongly_supported confidence: high supporting_papers: - '2305.11000' - '2306.12925' - '2402.05755' - '2407.05407' - '2410.03751' - '2410.17799' - '2411.17607' - '2412.02612' - '2412.10117' - '2503.14345' - 2025.acl-long.937 - 2025.findings-ijcnlp.49 - '2508.04141' - '2508.04585' - '2508.08961' - interspeech-2025-2328 - '2509.02020' - '2509.11425' contradicting_papers: [] refining_papers: - '2509.21968' caveats: - Unified training can introduce task interference and still trails cascades on some factual and task-specific metrics. last_reviewed: '2026-07-25' - id: multistream_autoregression_enables_full_duplex_dialogue claim: Parallel or delayed autoregressive speech streams enable streaming and full-duplex dialogue without explicit turn segmentation. status: strongly_supported confidence: high supporting_papers: - '2408.02622' - '2410.00037' - '2410.11190' - '2410.17799' - '2411.00774' contradicting_papers: [] refining_papers: - 2025.acl-long.682 - 2025.findings-ijcnlp.49 - interspeech-2025-0874 - interspeech-2025-2564 caveats: - Many duplex results use synthetic conversations or offline simulations rather than live interactive evaluation. last_reviewed: '2026-07-25' - id: preference_optimization_improves_multiple_quality_dimensions claim: Preference and reinforcement-learning post-training can improve intelligibility, speaker similarity, and perceptual quality beyond supervised autoregressive TTS. status: strongly_supported confidence: high supporting_papers: - '2406.00654' - '2406.02430' - '2410.03751' - '2412.10117' - '2502.04128' - '2502.11946' - '2504.02407' - '2505.17589' - '2507.16632' - '2507.21138' - 2025.acl-long.682 - interspeech-2025-0704 - '2509.00685' - '2509.14946' contradicting_papers: [] refining_papers: - 2025.icnlsp-1.34 - '2509.05863' - '2509.18531' caveats: - Reward models can be gamed, and gains often depend on proprietary evaluators or self-generated preference data. last_reviewed: '2026-07-25' - id: parallel_and_masked_models_challenge_full_autoregression claim: Masked and parallel codec generators can match autoregressive quality while reducing decoding latency, challenging the need for fully causal generation. status: contested confidence: medium supporting_papers: - '2305.09636' - 2025.acl-long.65 - '2508.04141' - interspeech-2025-1776 contradicting_papers: - '2310.00704' - '2509.15969' - '2509.19592' refining_papers: - '2509.22062' caveats: - Comparisons are frequently unmatched in data, compute, tokenizer, or prompt conditions. last_reviewed: '2026-07-25' - id: continuous_representations_avoid_quantization_tradeoffs claim: Autoregressive generation over continuous speech representations can avoid quantization loss while preserving language-model-style sequence prediction. status: emerging confidence: medium supporting_papers: - 2025.findings-naacl.184 - '2412.16846' contradicting_papers: [] refining_papers: [] caveats: - Evidence remains limited, and continuous prediction may reintroduce expensive rendering or unstable regression objectives. last_reviewed: '2026-07-25' - id: speaker_and_style_control_conflict_with_token_disentanglement claim: Separating linguistic content, speaker identity, and style in codec tokens improves controllability but introduces fidelity and identity trade-offs. status: strongly_supported confidence: high supporting_papers: - '2209.03143' - '2305.09636' - '2308.16692' - '2402.08093' - '2406.02430' - '2406.04904' - '2406.05370' - '2412.15649' - '2502.06490' - '2503.01710' - '2505.07916' - '2505.13000' - '2507.09070' - '2507.12197' - '2507.22746' - '2508.04141' - '2507.20091' - interspeech-2025-0989 - '2506.21619' - '2509.13068' contradicting_papers: [] refining_papers: - '2502.07243' - iclr-2025-dGSOn7sdWg - '2507.21138' - interspeech-2025-0464 - '2508.16332' - '2509.14882' - '2509.15845' - '2509.24570' - '2509.24650' caveats: - Reported disentanglement is often measured by probing classifiers rather than intervention-based listening tests. last_reviewed: '2026-07-25' - id: automatic_metrics_do_not_fully_predict_perceived_quality claim: Automatic intelligibility and signal-quality metrics do not reliably predict perceived naturalness for autoregressive codec speech. status: strongly_supported confidence: high supporting_papers: - '2310.00704' - '2403.16973' - '2406.00654' - '2410.00037' - '2503.14345' - '2504.12867' - interspeech-2025-0355 - '2509.18531' contradicting_papers: [] refining_papers: - '2412.16846' - '2509.14946' caveats: - Metric disagreement varies by codec, dataset, listener protocol, and error type; no single replacement metric is established. last_reviewed: '2026-07-25' - id: sampling_controls_diversity_but_reduces_determinism claim: Stochastic token sampling provides prosodic diversity but increases instability and complicates deterministic evaluation of autoregressive TTS. status: strongly_supported confidence: medium supporting_papers: - '2301.02111' - '2406.05370' - '2407.08551' - 2025.acl-long.65 contradicting_papers: [] refining_papers: [] caveats: - Candidate reranking and rejection sampling improve robustness at additional inference cost. last_reviewed: '2026-07-25' method_families: - id: historical_autoregressive_precursors name: Historical autoregressive speech precursors summary: Sample-level waveform and attention-based acoustic models that established autoregressive speech generation before codec-token TTS. papers: - '1609.03499' - '1703.10135' - '2106.15561' open_questions: - Which architectural lessons from sample- and frame-level autoregression remain relevant once generation operates over compact learned tokens? - id: hierarchical_ar_nar_codec_tts name: Hierarchical AR+NAR codec TTS summary: A causal model predicts the first or semantic code stream, followed by non-autoregressive or staged prediction of residual acoustic codebooks. papers: - '2301.02111' - '2305.09636' - '2308.16692' - '2310.00704' - '2404.03204' - '2406.18009' - '2502.07243' - '2503.01710' - iclr-2025-868masI331 - '2505.02625' - 2025.acl-long.65 - 2025.acl-long.912 - interspeech-2025-1776 - '2509.14882' - '2509.15969' open_questions: - Can a single decoder retain the quality benefits of staged RVQ prediction without the deployment complexity of separate AR and NAR models? - id: fully_autoregressive_multicodebook name: Fully autoregressive multi-codebook modeling summary: Inter-frame and intra-frame token dependencies are both modeled causally, often with delay patterns, depth transformers, or multi-scale factorization. papers: - '2310.00704' - '2507.12197' - '2509.15969' - '2509.19592' - '2509.22062' open_questions: - When does strict causal dependency across codebooks justify its latency cost over delayed or parallel codebook prediction? - id: semantic_acoustic_token_cascade name: Semantic–acoustic token cascades summary: Separate semantic and acoustic token streams divide linguistic coherence from waveform fidelity, with a second stage rendering acoustic detail. papers: - '2209.03143' - '2301.11325' - '2305.09636' - '2306.12925' - '2308.16692' - '2402.08093' - '2402.13236' - '2407.05407' - '2409.00750' - '2410.00037' - '2410.03751' - '2411.01156' - '2502.06490' - '2502.07243' - '2502.11946' - '2502.17239' - '2503.14345' - '2504.10344' - 2025.naacl-demo.12 - '2505.13000' - '2506.10274' - '2506.23325' - 2025.acl-long.1043 - 2025.acl-long.1498 - 2025.acl-long.682 - 2025.acl-long.937 - 2025.findings-acl.101 - '2508.14049' - '2508.04141' - interspeech-2025-0468 - interspeech-2025-0669 - interspeech-2025-1776 - '2508.19205' - '2509.00503' - '2506.21619' - '2509.04685' - '2509.09174' - '2509.11425' - '2509.14882' - '2509.15845' - '2509.21968' - '2509.22062' - '2509.24650' open_questions: - How much explicit semantic supervision is needed before token disentanglement begins to sacrifice speaker and prosodic fidelity? - id: hybrid_ar_token_continuous_renderer name: Autoregressive tokens with continuous renderers summary: Autoregressive semantic or codec-token prediction is paired with diffusion, flow-matching, or continuous acoustic decoding for final waveform fidelity. papers: - '2106.15561' - '2305.07243' - '2310.00704' - '2406.02430' - '2407.05407' - '2409.03283' - '2410.00037' - '2411.00774' - '2411.17607' - '2412.02612' - '2412.10117' - '2412.15649' - '2502.07243' - '2502.11946' - '2502.17239' - '2503.14345' - 2025.findings-naacl.184 - 2025.findings-naacl.279 - 2025.naacl-srw.6 - '2505.02625' - '2505.07916' - '2505.17589' - '2507.02380' - '2507.09070' - '2507.16632' - 2025.acl-long.65 - 2025.acl-long.682 - 2025.acl-long.912 - 2025.findings-acl.101 - 2025.ccl-1.80 - '2504.10352' - '2508.14049' - '2508.04141' - '2508.04585' - '2508.08715' - '2504.12867' - '2508.11326' - interspeech-2025-0253 - interspeech-2025-0704 - interspeech-2025-0874 - '2508.16332' - '2508.16790' - '2508.19098' - '2502.11128' - '2506.21619' - '2509.14946' - '2509.15845' - '2509.17516' - '2505.10599' - '2509.24650' - '2509.25131' - '2509.26514' open_questions: - Do hybrid renderers provide consistent quality gains once latency, parameter count, and training data are matched? - id: unified_speech_text_language_model name: Unified speech–text language models summary: Speech and text tokens share a decoder-only language-model backbone for understanding, generation, translation, and conversational response. papers: - '2209.03143' - '2305.09636' - '2305.11000' - '2306.12925' - '2402.05755' - '2408.02622' - '2408.16725' - '2410.00037' - '2410.03751' - '2410.11190' - '2410.17799' - '2411.00774' - '2411.17607' - '2412.02612' - '2412.15649' - '2502.11946' - '2502.17239' - '2503.14345' - iclr-2025-dGSOn7sdWg - 2025.naacl-demo.12 - '2505.02625' - '2507.02380' - '2507.16632' - 2025.acl-long.682 - 2025.acl-long.912 - 2025.findings-acl.101 - 2025.findings-acl.1051 - 2025.findings-ijcnlp.49 - '2508.04585' - '2507.20091' - '2508.08961' - interspeech-2025-0874 - interspeech-2025-2564 - '2508.15827' - '2509.02020' - '2509.09174' - '2509.14882' - '2509.25131' - interspeech-2025-1538 open_questions: - How can unified models avoid speech-task interference and factual-knowledge degradation while preserving text reasoning capability? - id: full_duplex_multistream_generation name: Full-duplex and multi-stream generation summary: Parallel user and agent token streams, delay schedules, or interleaving schemes enable streaming dialogue without explicit turn boundaries. papers: - '2403.16973' - '2408.02622' - '2410.00037' - '2410.11190' - '2410.17799' - '2411.00774' - 2025.acl-long.682 - 2025.findings-ijcnlp.49 - interspeech-2025-0874 - interspeech-2025-2564 open_questions: - Which duplex evaluation protocols reliably separate semantic latency, acoustic latency, interruption handling, and conversational naturalness? - id: alignment_constrained_autoregression name: Alignment-constrained autoregression summary: Phoneme interleaving, monotonic constraints, duration guidance, chain-of-thought plans, and sequence reordering reduce omissions, repetitions, and runaway decoding. papers: - '2401.07333' - '2404.03204' - '2406.02430' - '2406.07855' - iclr-2025-cuFzE8Jlvb - '2504.10352' - interspeech-2025-0551 - interspeech-2025-1641 - '2509.20378' open_questions: - Can robust monotonic alignment be learned without forced aligners or auxiliary duration labels at large multilingual scale? - id: continuous_token_autoregression name: Continuous-token autoregression summary: Autoregressive models predict continuous speech representations or latent patches, avoiding vector-quantization distortion and codebook complexity. papers: - '2305.07243' - '2407.08551' - iclr-2025-cuFzE8Jlvb - 2025.findings-naacl.184 open_questions: - Can continuous autoregression retain stable sampling and efficient sequence lengths at production-scale speech diversity? - id: masked_parallel_codec_alternatives name: Masked and parallel codec alternatives summary: Masked prediction, parallel codebook decoding, and non-autoregressive generation challenge the necessity of fully causal codec-token synthesis. papers: - '2408.16725' - '2409.00750' - '2507.01348' - 2025.acl-long.65 - interspeech-2025-1776 - '2509.19592' open_questions: - Under matched data and compute, where is the crossover between autoregressive robustness and parallel decoding efficiency? - id: codec_tokenizer_infrastructure name: Codec and tokenizer infrastructure summary: Neural codecs and tokenizers optimize bitrate, semantic alignment, codebook utilization, and reconstruction for downstream language modeling. papers: - '2210.13438' - '2301.02111' - '2305.02765' - '2306.00814' - '2308.16692' - '2402.13236' - '2408.16532' - '2409.05377' - '2411.19842' - '2502.06490' - '2504.10344' - 2025.findings-naacl.184 - 2025.naacl-srw.6 - '2505.13000' - '2506.10274' - '2506.23325' - '2507.18897' - 2025.acl-long.1498 - 2025.acl-long.654 - 2025.acl-long.937 - interspeech-2025-0355 - interspeech-2025-0464 - interspeech-2025-0468 - interspeech-2025-0669 - interspeech-2025-1289 - interspeech-2025-2726 - '2508.16790' - '2509.00503' - '2509.04685' - '2509.09550' - '2509.11425' - '2509.13068' - '2509.13670' - '2509.19025' - '2509.19186' - '2509.21968' open_questions: - Which tokenizer properties predict downstream TTS quality better than reconstruction metrics alone? - id: preference_optimized_autoregression name: Preference- and reward-optimized autoregression summary: Human feedback, learned rewards, reinforcement learning, and differentiable token-level objectives post-train autoregressive speech generators. papers: - '2406.00654' - '2406.02430' - '2410.03751' - '2412.10117' - '2502.04128' - '2502.11946' - '2504.02407' - '2505.17589' - '2507.16632' - 2025.acl-long.682 - 2025.icnlsp-1.34 - interspeech-2025-0704 - '2508.15442' - '2508.16332' - '2509.00685' - '2509.05863' - '2509.14946' - '2509.18531' - '2509.26514' open_questions: - How can multi-objective reward optimization improve intelligibility, speaker fidelity, and naturalness without reward hacking or loss of diversity? - id: scaled_general_ar_codec_tts name: Scaled general-purpose autoregressive codec TTS summary: General-purpose decoder-only codec TTS systems rely primarily on data and model scale, prompting, and standard causal next-token prediction. papers: - '2303.03926' - '2402.01912' - '2406.04904' - '2406.05370' - '2502.05512' - 2025.naacl-long.591 - '2507.21138' - 2025.acl-short.81 - 2025.conll-1.9 - '2507.22746' - '2508.06262' - '2508.09767' - interspeech-2025-0319 - interspeech-2025-0989 - interspeech-2025-1084 - interspeech-2025-1993 - interspeech-2025-2328 - interspeech-2025-2447 - interspeech-2025-2536 - '2412.16846' - '2509.14784' - '2509.17021' - '2509.20802' - '2509.24570' open_questions: - What are the compute-optimal scaling relationships among model size, speech hours, tokenizer bitrate, and sequence length? reassessment_queue: - id: parallel_and_masked_models_challenge_full_autoregression type: claim_status reason: The central architecture comparison is confounded by unmatched tokenizers, data, and compute. trigger: At least two controlled studies compare causal and masked/parallel decoding with the same tokenizer, data, parameter count, and evaluation protocol. due: 2027-01 current_assessment: contested watch_for: - Matched AR versus NAR codec-decoder ablations. - Latency-quality Pareto comparisons including time to first audio and total generation time. - id: scaled_general_ar_codec_tts type: method_family reason: The general-purpose family may conceal distinct architectures as the corpus grows. trigger: Two or more coherent subfamilies with at least five papers each become identifiable from decoder or conditioning structure. due: 2027-01 current_assessment: active_evidence watch_for: - Repeated decoder structures currently grouped only by general causal next-token prediction. - id: continuous_representations_avoid_quantization_tradeoffs type: claim_status reason: Continuous-token autoregression has a small evidence base relative to discrete codec modeling. trigger: Three independent systems reproduce quality or efficiency advantages under matched codec-token baselines. due: 2027-01 current_assessment: emerging watch_for: - Independent continuous-token autoregressive TTS systems. - Matched quantization-fidelity and inference-cost ablations. - id: hybrid_ar_token_continuous_renderer type: method_family reason: The hybrid family spans several distinct flow, diffusion, and codec-decoder placements and is the largest family in the concept. trigger: At least two stable renderer-placement subfamilies each contain five or more papers with distinct latency-quality behavior. due: 2027-01 current_assessment: active_evidence watch_for: - Repeated distinctions between AR semantic planning plus flow rendering, diffusion rendering, and parallel codec decoding. - id: semantic_acoustic_token_cascade type: method_family reason: The semantic-acoustic family groups multiple supervision and factorization mechanisms that may warrant separate treatment. trigger: Distillation-based, supervised-ASR, and dual-tokenizer branches each reach five papers with internally consistent evidence. due: 2027-01 current_assessment: active_evidence watch_for: - Replicated differences between SSL distillation, ASR-supervised semantic tokens, and independently trained semantic/acoustic tokenizers. open_questions: - Can autoregressive codec TTS eliminate alignment failures without external forced alignment, duration labels, or expensive candidate reranking? - Which tokenizer properties best predict downstream naturalness, intelligibility, speaker fidelity, and generation efficiency across architectures? - Under matched data, compute, and tokenization, where does causal autoregression remain superior to masked or parallel generation? - How should streaming and full-duplex systems be evaluated so that semantic latency, acoustic latency, interruption handling, and naturalness are measured separately? - Can preference optimization improve multiple perceptual dimensions without collapsing prosodic diversity or exploiting imperfect reward models? trend_notes: - The field moves from sample- and frame-level autoregression toward compact semantic and codec tokens after 2022, with VALL-E-style hierarchical AR+NAR generation becoming a dominant template. - By 2024–2025, hybrid systems increasingly retain autoregressive semantic planning while delegating acoustic rendering to flow-matching, diffusion, or parallel codec decoders. - Alignment robustness, streaming latency, and tokenizer quality replace basic waveform fidelity as the main differentiators among mature codec-language-model TTS systems. - Unified speech-text and full-duplex language models broaden autoregressive codec generation from TTS into conversational agents, but controlled comparisons with cascades remain limited. - Reward-based post-training and learned preference optimization emerge in 2024–2025 as a distinct strategy for improving autoregressive speech quality without changing the base decoder architecture.