TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

arXiv · 2026 · Preprint

Trung Dang et al. (Hume AI, Dartmouth College) · → Paper · Demo: ✗ · Code: ✗

Introduces a text-synchronous acoustic tokenizer that maps speech features one-to-one onto text tokens, enabling a single autoregressive LLM stream (with a flow matching head) to jointly generate text and speech with near-zero content hallucination and substantially higher inference throughput.

Problem

LLM-based TTS systems typically discretize speech with fixed-frame-rate acoustic codecs (e.g. 75 Hz, 50 Hz, or 25 Hz residual codecs). Because speech carries far more acoustic information per second than text, this produces speech token sequences that are an order of magnitude longer than, and temporally asynchronous with, the corresponding text tokens. This mismatch inflates the context window needed for transformer attention (quadratic cost), slows both training and inference, and forces architectures to fall back on complex interleaving or hierarchical semantic-token schemes to keep text and speech loosely aligned. The authors argue this asynchrony is also a root cause of two failure modes: (1) content hallucinations in TTS (word skipping, insertion, unstable duration), because the model lacks an inductive monotonic bias tying generated audio to specific text units; and (2) a “modality gap” in spoken language modeling (SLM), where adding speech tokens to an LLM’s context degrades its text reasoning and commonsense performance relative to a text-only model. Prior unified speech-text models (Spirit-LM, LST, LLaMA-Omni) either interleave low-bit-rate discrete semantic units (e.g. HuBERT tokens) with text, which sacrifices acoustic fidelity, or condition a separately-trained speech decoder on LLM hidden states without feeding the generated audio back into the LLM’s own context, leaving the model unable to perceive or adapt to its own acoustic output.

Method

TADA (Text-Acoustic Dual-Alignment) addresses this by constructing a tokenization scheme that establishes a strict one-to-one correspondence between text tokens and variable-duration acoustic segments, then modeling the resulting synchronous stream autoregressively with a flow matching (diffusion) head.

Joint speech-text tokenizer. The tokenizer has three components. (1) An aligner: a CTC model built on a Wav2Vec2-large backbone (self-supervised speech representation), fine-tuned to predict frame-level positions for each LLM subword token via Viterbi forced alignment; an intermediate character-level CTC loss and a curriculum-based token-selection scheme stabilize training against the LLM’s large (128,256-token) vocabulary. (2) An encoder: a CNN feature extractor followed by a transformer that aggregates variable-length audio segments into one latent vector per text token, using a constrained local-attention mask so each token’s feature is derived only from its own aligned audio span; a VAE-style reparameterization (mean vector plus sampled variance, following VibeVoice’s approach) produces the final latent. (3) A decoder, architecturally symmetric to the encoder, that expands the sparse per-token latents back into a dense sequence and synthesizes the waveform via a transformer followed by a multi-layer CNN; a second, frozen-encoder streaming variant uses purely local attention (only the current and preceding attention block) to allow low-latency autoregressive decoding. The tokenizer (TADA-Codec) is trained with a composite objective: multi-scale mel-spectrogram loss, GAN generator/discriminator losses, a multi-scale feature-matching loss, a CTC-based semantic (grapheme prediction) loss, and a KL-divergence term against a standard normal prior.

TADA language model. The generative backbone continues training from Llama 3.2 (1B and 3B variants). Text and acoustic embeddings are additively fused at each step, with the acoustic stream shifted by K positions relative to text to allow lookahead. Rather than interleaving text and speech tokens as separate positions (as in Spirit-LM/LST), TADA fuses them into a single synchronous stream, which the authors show accommodates roughly a ten-fold increase in audio duration within a fixed context window (682s vs. 73s for an interleaved 3 Hz text / 25 Hz speech setup at a 2048-token budget, §4.1). The LLM’s final hidden state feeds both a standard text LM head and a flow matching head. The flow matching head is conditioned on this hidden state and jointly predicts (a) the continuous acoustic latent for the next text-aligned token and (b) the number of preceding/succeeding “blank” acoustic frames for that token, the latter encoded via Bit Diffusion with Gray coding to make discrete frame counts diffusable. At inference, an Euler ODE solver samples the flow matching head (typically 10 steps) with classifier-free guidance applied to the acoustic features only. A lightweight step-level rejection sampling mechanism (a 3-layer MLP speaker-embedding head) can reject generated segments whose speaker embedding drifts from the voice-cloning prompt.

To counter the modality gap, the paper introduces Speech Free Guidance (SFG) (also called text-only guidance in places): during inference, logits are computed as a blend λ_SFG · z(text-speech) + (1 − λ_SFG) · z(text-only), where the text-only branch is obtained by running the same model with speech context masked out (using an explicit acoustic-mask embedding and stochastic audio dropout during training). This lets the model trade off speech-context benefits against text-only linguistic accuracy at a chosen operating point, with negligible inference overhead when both branches are batched together.

Training is staged: an initial phase at context length 192 (~200k steps), followed by a context-extension phase to 1,024 (1B model) or 2,048 (3B model) tokens (~200k steps). Alongside the flow matching loss, the LLM is trained with a text cross-entropy loss and a KL-divergence knowledge-distillation loss against the frozen base Llama model to preserve prior text capability.

Key Results

Tokenizer reconstruction (Table 1). At its dynamic 2-3 fps rate, TADA-Codec reconstructs speech at CER 0.14 and speaker similarity (SIM) 83.6, on par with fixed-rate baselines operating at far higher frame rates (EnCodec 75 Hz, DAC-24kHz 75 Hz, Mimi 12.5 Hz, Higgs v2 25 Hz, VibeVoice 7.5 Hz), and achieves the best objective MOS (oMOS 3.34) among all tokenizers compared, including the ground-truth reference resynthesis pipeline reported.

Voice cloning (Table 2, SeedTTS-Eval / LibriTTSR-Eval). TADA-1B reaches CER 0.73 / 0.55 and SIM 77.9 / 80.2 on SeedTTS-Eval and LibriTTSR-Eval respectively, comparable in stability (CER) to specialized two-stage systems trained on smaller, cleaner data (XTTS-v2, IndexTTS2) and comparable in speaker similarity to top-tier cloning systems (Higgs Audio V2, VibeVoice 1.5B), while training on substantially less data (270K hours vs. up to 10M hours for Higgs Audio V2). Its objective MOS (2.79) trails all baselines, which the authors attribute to the decoder not yet being specialized for the LLM’s output distribution. Measuring content hallucinations as samples with CER > 0.15, TADA has zero hallucinated samples versus 41 (FireRedTTS-2), 24 (Higgs Audio V2), and 17 (VibeVoice 1.5B).

Inference efficiency (Table 2). TADA-1B achieves RTF 0.09, several times lower than baselines (XTTS-v2: 0.19, VibeVoice 1.5B: 0.51, IndexTTS2: 0.58, FireRedTTS-2: 0.76), attributable to its 2-3 fps generation rate, despite the added cost of prompt alignment and a flow matching sampling step at each LLM decode step (Table 5: 4-10 flow matching steps add ~50-75% per-token latency over a text-only LLM step, but overall RTF still comes out far lower because so many fewer steps are needed per second of audio).

Long-form expressive TTS (Table 3, EARS). Base TADA-3B shows speaker drift in long-form generation (SIM 67), which text-free guidance and online rejection sampling largely correct (SIM 74.7), landing TADA second only to IndexTTS on this benchmark and second in subjective naturalness (sMOS 3.78) and similarity (sSIM 4.18).

Spoken language modeling (Table 4). In text-only mode, TADA outperforms base Llama-1B/3B-Instruct on Seamless Interaction perplexity (fine-tuned on spoken-text distribution) but trails the base models on sSC/tSC accuracy, indicating some text-pretraining capacity is traded away to accommodate the speech modality. In text-speech mode, TADA-3B-ML (with SFG at λ=0.5) surpasses all other text-speech baselines including the much larger SpiritLM-7B and TWIST-7B/13B on tSC (94.7 vs. 82.9/74.1/76.4), and approaches or exceeds its own text-only accuracy on both sSC and tSC, at negligible additional inference cost (RTF overhead of 0.01 for batch-size-1 SFG).

Novelty Assessment

The core novelty is architectural: a text-synchronous, variable-duration tokenization scheme that removes the frame-rate/sequence-length mismatch between text and speech altogether, rather than mitigating it via interleaving, hierarchical semantic tokens, or a decoupled speech decoder. This is a genuinely different design point from prior unified speech-text models (Spirit-LM, LST) and prior codec-based TTS-LMs (VALL-E-style, CosyVoice, FireRedTTS-2), which all operate on fixed-frame-rate acoustic units. The flow matching head jointly predicting acoustic content and discretized frame duration (via Bit Diffusion) is also a specific new mechanism for this setting. Speech Free Guidance is an incremental but practically useful contribution: it is a straightforward classifier-free-guidance-style logit blend, but applied to the novel problem of trading off text-only vs. text-speech inference modes within a single synchronous-stream model. The training recipe itself (continued pretraining of Llama 3.2, standard adversarial/mel/feature-matching codec losses) is largely conventional; the contribution is concentrated in the tokenization/alignment design and its consequences for context efficiency and hallucination.

Field Significance

High significance

high — TADA demonstrates that removing the text-speech frame-rate mismatch via explicit synchronous alignment, rather than working around it, can simultaneously improve inference efficiency, virtually eliminate content hallucinations, and narrow (though not close) the SLM modality gap, all within a single unified autoregressive stream. This offers a concrete architectural alternative to the interleaving and hierarchical-semantic-token strategies that dominate current unified speech-text modeling, backed by direct, controlled comparisons against multiple strong contemporaneous baselines (VibeVoice, FireRedTTS-2, Higgs Audio V2, IndexTTS2, Spirit-LM).

Claims

• supports: Establishing an explicit one-to-one alignment between generated acoustic units and text tokens, rather than relying on global attention over asynchronous sequences, substantially reduces content hallucination in autoregressive TTS.

  Evidence: Under a CER > 0.15 hallucination threshold across combined SeedTTS-Eval and LibriTTSR-Eval samples, the synchronously-aligned model produced zero hallucinated samples versus 41 (FireRedTTS-2), 24 (Higgs Audio V2), and 17 (VibeVoice 1.5B). (§6.2, Table 2)

• supports: Synchronizing acoustic and text token rates removes the sequence-length bottleneck that otherwise forces a trade-off between context window size and audio duration, enabling much longer audio to fit within a fixed LLM context and improving inference throughput.

  Evidence: At a shared 2048-token budget, synchronous single-stream modeling accommodates 682s of audio versus 73s for an interleaved 3 Hz text / 25 Hz speech setup; TADA-1B’s measured RTF (0.09) is several times lower than baselines generating fixed-rate discrete tokens (0.19-0.76). (§4.1, Table 2)

• complicates: Blending text-only and text-speech inference logits can substantially narrow, but does not uniformly eliminate, the modality-gap degradation that speech conditioning introduces into an LLM’s text reasoning and commonsense performance, and the residual gap varies by benchmark.

  Evidence: With Speech Free Guidance (λ=0.5), text-speech mode surpasses all other text-speech baselines and slightly exceeds pure text-only accuracy on the topic-story-cloze benchmark, but text-speech mode without guidance still trails the base text-only model by a wide margin on story-cloze accuracy (66.5-66.8 vs. 73.8-79.4). (§6.3, Table 4)

• complicates: Compressing acoustic information into compact, per-text-token continuous latents can trade away perceptual audio quality relative to fixed higher-frame-rate codecs, even when content accuracy and speaker similarity remain competitive.

  Evidence: TADA-1B records the lowest objective MOS (2.79) among all compared voice-cloning systems in Table 2 despite matching or beating several baselines on CER and speaker similarity; the authors attribute this to the decoder not yet being specialized to the LLM’s output distribution. (§6.2)

Limitations and Open Questions

The paper reports several limitations directly. Long-form, expressive generation (EARS benchmark) exhibits speaker drift that requires two auxiliary mitigations, text-free guidance and online rejection sampling, to reach competitive speaker similarity; without them, base TADA-3B’s SIM score (67) trails all listed baselines. Objective perceptual quality (oMOS) consistently lags behind baseline systems across both the tokenizer-reconstruction and voice-cloning evaluations, which the authors attribute to an undertrained decoder rather than a fundamental limitation of the representation. In text-only mode, TADA underperforms the base Llama models on commonsense/story-cloze accuracy, indicating that adapting a text LLM to jointly model speech consumes some of its original text-pretraining capacity even before any speech is added to the context. The tokenizer pipeline depends on external ASR transcription (Parakeet-TDT / Whisper-V3) and a Wav2Vec2-based CTC aligner during training data preparation, adding pipeline complexity and a potential source of alignment noise on out-of-domain or low-resource languages not covered by the seven-language training mix. No code or public demo is provided, limiting independent verification of the reported results.

Wiki Connections

• Flow Matching — TADA’s generative head is a flow matching model jointly predicting continuous acoustic latents and discretized frame durations, conditioned on LLM hidden states rather than modeling discrete acoustic codes autoregressively.
• Neural Audio Codec — introduces TADA-Codec, a VAE-based tokenizer whose frame rate is dynamically tied to text tokens (2-3 fps) rather than fixed, and evaluates it directly against EnCodec, DAC, Mimi, and Higgs v2.
• Zero-Shot TTS — evaluates zero-shot voice cloning from short reference prompts on SeedTTS-Eval, LibriTTSR-Eval, and EARS, comparing against XTTS-v2, IndexTTS2, Higgs Audio V2, VibeVoice, and FireRedTTS-2.
• Spoken Language Model — evaluates the same unified model as a spoken language model on perplexity and story-cloze benchmarks, comparing against Spirit-LM and TWIST and introducing Speech Free Guidance to mitigate the modality gap.
• Self-Supervised Speech — the tokenizer’s forced-alignment aligner is built on a Wav2Vec2-large backbone, using self-supervised pretraining as the foundation for the CTC-based text-acoustic alignment.
• Speaker Adaptation — voice cloning relies on prompt-based speaker conditioning plus a dedicated online rejection-sampling mechanism that scores generated segments against a reference speaker embedding to correct speaker drift.
• VibeVoice — TADA adopts VibeVoice’s VAE reparameterization approach for its latent sampling and is directly compared as the closest continuous-token baseline throughout Tables 1-3.
• FireRedTTS-2 — used as a long-conversational-speech baseline in the voice-cloning and hallucination-rate comparisons, where it shows the highest hallucination count among compared systems.
• IndexTTS2 — used as a strong two-stage baseline for stability and long-form expressive generation, outperforming TADA on the EARS benchmark’s speaker similarity metric.
• Qwen3-TTS — cited as a recent low-frame-rate LLM-based TTS system in the related-work discussion of the shift toward reduced token rates.
• Seed-TTS — SeedTTS-Eval, one of TADA’s primary voice-cloning benchmarks, originates from this paper.
• CosyVoice — cited among LLM-based TTS systems using supervised semantic tokens as an intermediate representation, the design pattern TADA’s synchronous tokenization is positioned against.
• VALL-E — cited as the foundational neural codec language model establishing the discretize-then-autoregressively-model paradigm that TADA’s synchronous tokenization departs from.