ACL · 2026 · Conference

Jun-Hak Yun et al. (Korea University) · → Paper · → Preprint · Demo: ✓ · Code: ✓

Generates speech together with a text-described background scene in one flow-matching model, using a dual-stream multimodal diffusion transformer where transcript-aligned speech features and environment-prompt tokens interact through joint attention, and regularizes it by aligning intermediate features to a speech SSL teacher (WavLM) and an audio SSL teacher (ATST-Frame) at the same time.

Problem

Environment-aware TTS asks for intelligible speech embedded in a specified acoustic scene (for example “birds chirping”). Text-to-audio models produce plausible soundscapes but not precise linguistic content, while TTS models are trained on clean speech, and the two are usually built as separate pipelines. Earlier text-prompted environment-aware systems, VoiceLDM (U-Net latent diffusion with cross-attention to CLAP and SpeechT5 embeddings) and VoiceDiT (DiT with AdaLN environment conditioning), inject the environment description as a global condition, and the authors argue this leaves a speech-environment mismatch because speech and background never interact at the token level. Reference-audio approaches (UmbraTTS, DAIEN-TTS, IDEA-TTS) need a recording of the target environment, which the text-prompted setting avoids.

Method

ImmersiveTTS operates in the latent space of the frozen AudioLDM2 VAE: a 16 kHz waveform is converted to a 64-bin log-mel spectrogram, encoded into an 8-channel latent downsampled 4x in time and frequency, and decoded back to mel and then to waveform with a pretrained HiFi-GAN vocoder (§3.2, Appendix A). The generator is a velocity field trained with a rectified-flow (flow matching) objective, with timesteps drawn from a logit-normal distribution as in SD3, and sampled with an Euler ODE solver at 25 NFEs (§3.1, §3.5).

The backbone follows the Flux variant of the multimodal diffusion transformer (MM-DiT): 12 double-stream blocks followed by 18 single-stream blocks, 6 attention heads, hidden size 1024, about 450M trainable parameters (§4.1). Two text inputs drive it. The environment prompt is encoded twice: frozen Flan-T5-Large token embeddings form the environmental context stream, and a frozen CLAP sentence embedding, passed through an MLP and added to the timestep embedding, modulates AdaLN in every block (a “dual-granularity” condition). The content prompt (the transcript) goes through a Glow-TTS-style text encoder with monotonic alignment search and a duration predictor, giving a frame-level prior mel representation; a convolutional mapper (taken from VoiceDiT) projects it into the VAE latent space and it is concatenated channel-wise with the noisy latent to form the speech stream (§3.3). In the double-stream blocks the two streams keep separate weights but share joint attention, so speech tokens can attend to individual environment tokens; only the speech stream continues into the single-stream blocks. A speaker embedding from a pretrained WavLM-based speaker-verification model, computed on a speaker prompt clip, is projected into the text encoder (Appendix A).

Figure 1: Overview of ImmersiveTTS. A dual-stream MM-DiT backbone conditions the speech stream on content prompt-aligned linguistic features. At the same time, Flan-T5 token embeddings drive the environmental context stream, and CLAP embeddings modulate AdaLN for global conditioning. The model is trained with flow matching and domain-specific REPA objectives.

The second contribution is a domain-specific representation alignment (REPA) loss (§3.4). Hidden states from intermediate speech-stream layers are projected by small MLPs and aligned, via negative cosine similarity after temporal interpolation or pooling to a common length, with the final-layer features of two frozen self-supervised teachers: WavLM-Large, a speech SSL model fed the clean LibriTTS speech before mixing (targeting linguistic content), and ATST-Frame-Base, a general-audio SSL model fed the mixed speech-plus-environment audio (targeting the acoustic scene). Both teacher losses are weighted 1 and placed in the MM-DiT stage (6th or 10th of the 12 double-stream blocks, Appendix C). The total loss is the sum of the Glow-TTS prior and duration losses, the flow-matching loss and the REPA loss, all weighted 1 (§3.5). Dual classifier-free guidance masks content and environment prompts independently with probability 0.1 during training and uses separate scales at inference (ω_env = ω_cont = 3).

Training data are synthetic mixtures: LibriTTS train-clean-360 speech mixed with WavCaps clips filtered to 340k non-speech sounds at an SNR drawn uniformly from 2 to 10 dB, with 15% of examples left clean (§4.1). Training runs 400k steps on two RTX A6000 GPUs at batch size 8 per GPU.

Key Results

All three environment-aware systems were trained from scratch by the authors on the same corpus and step count, with per-baseline dual-CFG scales tuned on an evaluation set (§5.1). Subjective scores are 5-point MOS from 20 native English speakers on Amazon Mechanical Turk over 30 utterances per test set, with attention-check samples (Appendix B).

On the AudioCaps test set (real recordings containing speech and background), ImmersiveTTS (25 NFEs) scores SN-MOS 4.20, EC-MOS 3.48, ON-MOS 3.47, WER 8.06, FAD 5.80 and CLAP 0.308, against VoiceDiT (200 NFEs) at 3.47 / 3.44 / 2.63 / 11.68 / 9.07 / 0.263 and VoiceLDM (200 NFEs) at 3.41 / 3.33 / 2.55 / 16.45 / 8.75 / 0.229, so it is ahead of both baselines on all six metrics there (Table 1). The EC-MOS lead over VoiceDiT is small (3.48 vs 3.44, each ±0.07). The reconstructed upper bound scores 4.08 / 4.16 / 3.49; ImmersiveTTS’s SN-MOS is above it, while its EC-MOS is well below it. Ground-truth WER on this set is 22.29, because background audio masks the speech for the Whisper-Large-v3 ASR.

On the augmented set (Seed-TTS test-en speech mixed with AudioCaps non-speech clips), ImmersiveTTS has the highest SN-MOS (4.18 vs 3.45 VoiceDiT, 3.32 VoiceLDM) and ON-MOS (3.23 vs 3.12, 2.91), the lowest WER (4.48 vs 7.08, 11.20) and FAD (3.92 vs 5.37, 6.98) and the highest CLAP (0.207 vs 0.134, 0.118), but VoiceDiT has the highest EC-MOS (3.38 vs 3.32 for ImmersiveTTS and 3.24 for VoiceLDM) (Table 2). Speaker-similarity S-MOS on this set is 3.15 for ImmersiveTTS, equal to VoiceDiT (3.15) and above VoiceLDM (2.97), against 3.18 for reconstructed audio (Table 6).

In single-task evaluation (Table 3), TTS on LibriTTS test gives ImmersiveTTS the lowest WER of the three environment-aware models (9.89 vs 11.08 VoiceDiT, 14.01 VoiceLDM), but VoiceDiT is ahead on UTMOS (3.33 vs 3.23) and SECS (0.8942 vs 0.8859). For text-to-audio on AudioCaps test, ImmersiveTTS has the lowest FAD (7.81 vs 8.23, 8.71) and highest CLAP (0.323 vs 0.302, 0.288).

Dedicated single-task models remain far ahead on their own metrics (Table 7): CosyVoice 2 (WER 5.23, SECS 0.9227) and CosyVoice 3 (5.98, 0.9152) on TTS, and AudioLDM2-Audio (FAD 3.79, CLAP 0.407) and TangoFlux (FAD 2.96, CLAP 0.502) on TTA. A cascade that mixes CosyVoice 2 speech with TangoFlux background beats ImmersiveTTS on all three objective metrics on both test sets: WER 6.76 vs 8.06, FAD 4.01 vs 5.80 and CLAP 0.452 vs 0.308 on AudioCaps (Table 8); 2.28 vs 4.48, 3.11 vs 3.92 and 0.287 vs 0.207 on the augmented set (Table 9). An AudioLDM2-Speech plus AudioLDM2-Audio cascade has much worse WER (41.33 and 35.78), though its FAD (5.36 and 3.39) and CLAP (0.365 and 0.236) are better than ImmersiveTTS’s. These cascades were compared on objective metrics only.

The teacher ablation on AudioCaps (Table 4) starts from a no-REPA base at WER 11.21, FAD 9.64, CLAP 0.236. Single teachers trade domains: WavLM gives 10.97 / 8.02 / 0.231, ATST-Frame gives 13.77 / 8.78 / 0.271 (worse WER, better CLAP), and the unified speech-audio encoder USAD gives 9.04 / 7.93 / 0.239. Among dual teachers, WavLM + USAD gives 8.95 / 7.33 / 0.248, USAD + ATST 8.94 / 8.20 / 0.266, and WavLM + ATST (the final model) 8.06 / 5.80 / 0.308, the best value in every column of the table. Moving one of the two alignments into the single-stream stage weakens WavLM + ATST to 9.67 / 5.97 / 0.287 (Table 5). Across NFEs, ImmersiveTTS at 9 steps already shows lower WER and FAD and higher CLAP than both 200-NFE baselines (§5.4, Figure 2).

Novelty Assessment

The main new element is the domain-specific, dual-teacher REPA objective: REPA had already been used in speech generation with a single teacher or with CTC plus a speech SSL model (A-DMA, Vevo2), and here a speech SSL teacher on the clean source and an audio SSL teacher on the mixture supervise the same hidden states, with ablations that separate the two teachers’ effects and compare them against a single unified teacher (USAD). The MM-DiT backbone is the image-generation Flux design moved to audio, and the speech pathway reuses the Glow-TTS text encoder and duration model and VoiceDiT’s mel-to-latent mapper, so the architecture is largely an adaptation; its specific choice, letting speech tokens attend to token-level environment features rather than only a global CLAP vector, is not ablated on its own. The comparison is fair in that both baselines were retrained on identical data, but only two text-prompted environment-aware baselines are included and no reference-audio environment-aware system is compared.

Field Significance

Moderate — The paper provides a controlled comparison showing that, for joint speech-plus-scene generation, supervising one generator with two domain-specialized SSL teachers works better than either teacher alone or a unified teacher, and that a flow-matching MM-DiT reaches better objective and listener scores than retrained diffusion baselines at one eighth of the sampling steps. It also documents honestly that a cascade of strong single-task models mixed afterwards still wins on objective metrics, which frames unified environment-aware TTS as a coherence argument rather than a quality one so far.

Claims

  • supports: In joint generation of speech and background audio, aligning a generator’s hidden states simultaneously with a speech-specialized SSL teacher and an audio-specialized SSL teacher improves both intelligibility and scene fidelity more than a single teacher or a unified speech-audio teacher.

    Evidence: On AudioCaps, the no-alignment base scores WER 11.21, FAD 9.64, CLAP 0.236; WavLM + ATST-Frame alignment reaches 8.06 / 5.80 / 0.308, against 8.95 / 7.33 / 0.248 for WavLM + USAD and 9.04 / 7.93 / 0.239 for USAD alone. (§5.3, Table 4)

  • complicates: A single representation-alignment teacher from one audio domain shifts quality toward that domain at the cost of the other when speech and environmental sound are generated together.

    Evidence: The audio SSL teacher ATST-Frame alone raises CLAP from 0.236 to 0.271 but worsens WER from 11.21 to 13.77, while WavLM alone lowers WER to 10.97 with CLAP 0.231; the pattern holds when the alignment is applied in the single-stream blocks (ATST WER 14.57, CLAP 0.261). (§5.3, Table 4; §Appendix C, Table 5)

  • complicates: Unified environment-aware TTS models do not yet match cascades that mix the outputs of strong dedicated TTS and text-to-audio models on objective intelligibility, fidelity and prompt-alignment metrics.

    Evidence: CosyVoice 2 speech mixed with TangoFlux background scores WER 6.76, FAD 4.01, CLAP 0.452 on AudioCaps and 2.28 / 3.11 / 0.287 on the augmented set, against 8.06 / 5.80 / 0.308 and 4.48 / 3.92 / 0.207 for the unified model; no listening test compares the two. (§Appendix F.2, Tables 8 and 9)

  • refines: With separate classifier-free guidance scales for transcript and scene description, strong scene guidance mainly costs intelligibility while strong transcript guidance mainly costs acoustic realism and scene alignment.

    Evidence: With ω_cont fixed at 3, raising ω_env from 3 to 5 or more increases WER from 8.06 to over 10.92 with small FAD changes; with ω_env fixed at 3, raising ω_cont increases FAD from 4.62 to 10.27 and lowers CLAP from 0.325 to 0.232, with the lowest WER at ω_cont = 7. (§Appendix D, Figures 4 and 5)

  • supports: Flow-matching generators for environment-aware TTS can match or exceed diffusion-based counterparts with far fewer function evaluations.

    Evidence: At 9 NFEs the model already has lower WER and FAD and higher CLAP than VoiceLDM and VoiceDiT at 200 NFEs on AudioCaps, and the main results use 25 NFEs. (§5.4, Figure 2; §5.1, Table 1)

Limitations and Open Questions

The most consequential limitation is that all training data are synthetic mixtures of clean LibriTTS read speech with WavCaps clips at 2-10 dB SNR; the authors note that this may not capture acoustic interactions in real recordings and that robustness across SNR conditions and scene difficulty is unexplored (§Limitations). Related to this, the unified model is beaten on every objective metric by a simple CosyVoice 2 + TangoFlux mixing cascade (Tables 8 and 9), and the argument that unified modelling gives better speech-background coherence is not tested with a listening comparison against that cascade.

  • Only two environment-aware baselines are compared, both text-prompted and both retrained by the authors; reference-audio systems such as UmbraTTS and DAIEN-TTS are not evaluated.
  • On the augmented test set VoiceDiT has slightly higher environmental-consistency MOS (3.38 vs 3.32), and on single-task TTS VoiceDiT is ahead on UTMOS and SECS (Table 3), so the gains are concentrated in intelligibility, speech naturalness and overall blend rather than scene fidelity or speaker similarity.
  • Listening tests use 30 utterances per test set, and ground-truth audio was excluded from the MOS tests (only VAE-vocoder reconstructions serve as reference), so absolute scores are not anchored to real recordings.
  • WER on the real AudioCaps set is unreliable as an absolute measure: ground truth itself scores 22.29.
  • Output is limited to 16 kHz through a frozen AudioLDM2 VAE and HiFi-GAN, and the model has no explicit control over prosody, speaking style or emotion (§Limitations).
  • The joint-attention design is not ablated against a global-conditioning variant with the same REPA, so its individual contribution is not isolated.

Wiki Connections

  • Flow Matching — Trains the MM-DiT velocity field with a rectified-flow objective and logit-normal timestep sampling, and samples at 25 Euler steps against 200-step diffusion baselines.
  • Self-Supervised Speech — Uses frozen WavLM-Large features on clean speech as a representation-alignment target during training, paired with a general-audio SSL teacher (ATST-Frame) for the scene.
  • Zero-Shot TTS — Conditions on a speaker embedding extracted from a reference clip and reports speaker similarity on unseen LibriTTS test and Seed-TTS test-en speakers (SECS 0.8859, S-MOS 3.15), as a secondary capability.
  • Subjective Evaluation — Uses separate crowd-sourced MOS ratings for speech naturalness, environmental consistency and overall speech-background blending, plus speaker-similarity MOS.
  • UmbraTTS — Reference-audio environment-aware TTS on a flow-matching backbone, discussed as the alternative strategy to text-prompted environments but not compared.
  • DAIEN-TTS — Reference-audio environment-aware TTS using a speech-environment separation module, also discussed but not compared.
  • A-DMA — Earlier speech-side representation alignment (CTC plus a speech SSL model) for diffusion TTS training, which this paper extends to two domain-specific teachers.
  • Vevo2 — Prior use of REPA with w2v-BERT 2.0 features in a flow-matching speech and singing acoustic model, cited as a precedent.
  • CosyVoice 2 — Single-task TTS baseline and the speech half of the CosyVoice 2 + TangoFlux cascade that outperforms ImmersiveTTS on objective metrics.
  • CosyVoice 3 — Additional single-task TTS baseline with lower WER and higher speaker similarity than all environment-aware models.
  • HiFi-GAN — Pretrained vocoder used unchanged to convert decoded mel spectrograms to 16 kHz waveforms.
  • LibriTTS — Source of the clean speech (train-clean-360) mixed with WavCaps sounds for training and of the single-task TTS test set.