TED-TTS: Training-Free Intra-Utterance Emotion and Duration Control for Text-to-Speech Synthesis

arXiv · 2026 · Preprint

Qifan Liang et al. (National University of Singapore) · → Paper · Demo: ✓ · Code: ✓

Presents the first training-free framework for intra-utterance emotion and duration control in pretrained zero-shot TTS, restructuring how segment-level conditioning is accessed during autoregressive decoding rather than retraining the acoustic model.

Problem

Existing controllable TTS methods mostly apply a single emotion or prosodic condition uniformly across an entire utterance, which does not match how humans naturally shift emotional expression and pacing within a single utterance. The few prior methods that attempt intra-utterance control (e.g. WeSCon) rely on large-scale, non-public, time-aligned emotional speech datasets or multi-stage training pipelines, which limits their transferability across TTS backbones and their practical deployability. This raises the question the paper addresses directly: can stable, segment-level emotion transitions and duration control be achieved at inference time, without retraining the underlying TTS model at all?

Method

TED-TTS operates entirely at inference time on a frozen, pretrained zero-shot TTS backbone configured identically to IndexTTS2, focusing on its text-to-semantic (T2S) autoregressive module. Given user text decomposed into M segments, each assigned a condition embedding combining a shared speaker-identity embedding with segment-specific emotion conditioning, the method introduces two mechanisms.

For emotion control, a 2D causal attention mask disentangles condition visibility from semantic context: tokens belonging to segment m can attend only to that segment’s condition embedding (not other segments’), while standard causal attention across all text and semantic tokens is preserved for semantic coherence. Because autoregressive semantic-token generation has no explicit segment boundaries, the mask transitions are driven by an online Monotonic Stream Alignment (MSA) algorithm: a Bayesian-style filter that maintains a belief distribution over text positions, propagates it forward at each decoding step via a monotonicity-enforcing transition operator (Predict), selects the most reliable attention head by measuring agreement with the predicted alignment across all layers and heads (Select), and updates the belief by combining the monotonic prior with the selected head’s (Gaussian-smoothed) attention observation (Update). This produces a stable alignment trajectory used to trigger mask transitions between segments.

For duration control, each segment’s target duration is converted to a target semantic-token count via the codec’s token rate, and a duration embedding (indexed by cumulative target length, with its embedding table tied to the semantic positional embedding table) conditions each segment. Because actual generation speed can drift from the target during decoding, a local duration embedding steering mechanism uses MSA’s alignment estimate to compute the discrepancy between text progress and semantic progress within the active segment, then applies a proportional controller to adjust the effective target length and re-query the duration table only for the active segment. Separately, a global EOS steering mechanism suppresses the end-of-semantic token in non-final segments (preventing premature termination) and progressively biases the EOS logit in the final segment as the target token budget is approached, to ensure globally consistent sequence termination without needing an explicit endpoint predictor.

To eliminate manual segment-level prompt engineering, the authors construct MED-TTS, a 30,000-sample (14,965 Chinese / 15,092 English) text dataset with multi-segment emotion and duration annotations, generated via a GPT-4o content-generation step and a DeepSeek-Chat segmentation-and-annotation step (7 core emotions, smooth or abrupt emotional progressions), followed by automatic filtering and manual verification. Qwen3-8B is LoRA fine-tuned on MED-TTS to automatically convert raw input text into structured, segment-level emotion-duration prompts at inference time; this LLM fine-tune is the only trained component in the entire pipeline.

Key Results

Against MaskGCT, F5-TTS, Spark-TTS, CosyVoice2, and IndexTTS2 (all evaluated by synthesizing segments independently and concatenating, since they lack native intra-utterance control), TED-TTS achieves the best or near-best objective results on most metrics across English and Chinese, under both speech-emotion-prompt and text-emotion-prompt settings, with consistent gains on DNSMOS-Pro (transition smoothness) and speaker similarity. WER/CER and automatic emotion-recognition scores are not always best but remain comparable to the IndexTTS2 baseline, which the authors frame as expected given the method adds no new training signal. On duration-scaled synthesis (five scaling factors from 0.75x to 1.25x), TED-TTS attains the best DNSMOS-Pro, NISQA, and OVRL scores in both languages and the lowest average semantic-token-count error across all five factors, reducing error by 3.53 and 2.41 percentage points on average relative to ablated variants without local steering or global EOS control respectively, and by 5.07 points relative to the unmodified baseline. In a 15-listener subjective study across four MOS criteria (SMOS, NMOS, EMOS, SPMOS), TED-TTS is state-of-the-art or highly competitive despite performing one-shot multi-segment generation, versus baselines that synthesize each segment independently. An ablation on the alignment mechanism shows raw attention-based alignment is unstable; adding the monotonic-stream constraint reduces boundary mean absolute error to 0.216, and adding top-k (k=3) reliable-head selection reduces it further to 0.157. Efficiency analysis shows real-time factor increases of 22.7%/24.5% (English/Chinese) from adding emotion conditioning and a further 17.5%/17.7% from adding duration steering, remaining below 1.0 (real-time) throughout.

Novelty Assessment

The central claim, that intra-utterance multi-emotion and duration control can be achieved purely at inference time without any modification to the pretrained acoustic model’s parameters, is the paper’s most distinctive contribution, and the ablations (isolating full-text access, alignment tracking, local steering, and global EOS control) provide reasonably direct evidence that each proposed mechanism does what it claims rather than the overall gain being attributable to some confound. The Monotonic Stream Alignment algorithm is a specific, well-motivated solution to a real problem (raw attention maps are too noisy and non-monotonic to drive reliable segment-boundary transitions), evaluated with its own dedicated metric (alignment MAE) rather than only indirectly through downstream synthesis quality. The one caveat is that “training-free” applies only to the TTS acoustic model itself: automatic prompt construction still depends on a LoRA-fine-tuned Qwen3-8B model trained on the paper’s own MED-TTS dataset, so the full pipeline is not training-free end to end, only the speech-generation component is.

Field Significance

Tip

High, this is presented as the first training-free framework for intra-utterance emotion and duration control, directly addressing the practical deployability problem (large annotated datasets, multi-stage training, poor cross-model transferability) that limited prior intra-utterance control methods like WeSCon. Because the core mechanism operates purely at inference time on standard autoregressive T2S decoding, it is plausibly transferable to other autoregressive TTS backbones beyond IndexTTS2 without architectural modification, which is a meaningfully different point in the design space from training-based approaches to the same problem.

Claims

• supports: Segment-level conditioning can be applied within a single autoregressive TTS decoding pass, without any model retraining, by restricting attention between tokens and their corresponding segment condition embeddings while preserving standard causal attention over semantic content.

  Evidence: The 2D causal attention mask that isolates each segment’s condition embedding while preserving global causal semantic context achieves smoother transitions (higher DNSMOS-Pro and speaker similarity) than ablated variants restricting text access or removing alignment tracking, without any parameter updates to the underlying pretrained TTS model. (§3.2, §5.2, Table 3)

• supports: Tracking token-level text-to-semantic alignment online via a Bayesian-style monotonic belief-propagation filter over multi-head attention observations is more reliable for driving segment-boundary transitions in autoregressive TTS than using raw attention maps directly.

  Evidence: Adding a monotonic-stream constraint reduces segment-boundary mean absolute error from an unstable raw-attention baseline to 0.216, and further incorporating top-k (k=3) reliable-head selection as the attention observation reduces it to 0.157. (§5.2, Figure 5)

• supports: Local, per-step duration correction combined with a separate global end-of-sequence bias term achieves more accurate segment-level duration control in autoregressive TTS than either mechanism alone.

  Evidence: Ablating local duration embedding steering causes the largest duration-accuracy degradation across scaling factors (semantic token number error rising from 3.211% to 11.594% at 1.25x scaling), while ablating global EOS steering causes a smaller but consistent degradation, and the combined method achieves the lowest error at every tested scaling factor. (§5.2, Table 4)

• complicates: Training-free, inference-time control methods for autoregressive TTS incur measurable computational overhead that scales with the number of composed control mechanisms.

  Evidence: Adding segment-aware emotion conditioning increases real-time factor by 22.7%/24.5% (English/Chinese) over baseline, and further adding duration steering raises the total increase to 40.2%/42.2%, though real-time factor remains below 1.0 throughout. (§5.2, Figure 6)

Limitations and Open Questions

• The authors state the framework does not explicitly model gradual emotion transitions between adjacent segments; emotional variation is controlled segment-wise rather than as a continuous trajectory, which may limit representation of intermediate emotional states.
• Duration control precision is bounded by the duration representation already learned by the pretrained baseline TTS model, since no parameter updates are made; the authors note this may not support strictly linear or fine-grained timing control under highly expressive or out-of-domain conditions.
• The MED-TTS dataset used to fine-tune the prompt-construction LLM is itself synthetically generated by GPT-4o and DeepSeek-Chat, so any systematic biases or stylistic artifacts of those models could propagate into the segmentation and emotion-description supervision, despite the manual verification step.
• The method’s generality beyond IndexTTS2 is argued conceptually (restructuring conditioning during standard autoregressive decoding, no architectural modification needed) but not empirically demonstrated on a second autoregressive TTS backbone in this paper.

Wiki Connections

• Emotion Synthesis — introduces a training-free mechanism for smooth intra-utterance emotion transitions, in contrast to prior methods that apply a single emotion uniformly across an utterance or require multi-stage training for intra-utterance control.
• Instruction-Conditioned TTS — accepts natural language emotion descriptions per segment as conditioning, generated automatically by a fine-tuned LLM from raw input text.
• Prosody Control — introduces explicit local duration embedding steering and global EOS logit modulation to control segment-level speaking pace independently of content and speaker identity.
• Subjective Evaluation — conducts a 15-listener study across four MOS criteria (SMOS, NMOS, EMOS, SPMOS) covering speaker similarity, transition naturalness, emotion alignment, and speaking-rate accuracy.
• Zero-Shot TTS — operates entirely on top of a frozen, pretrained zero-shot TTS backbone (configured as IndexTTS2) and explicitly aims to preserve its zero-shot synthesis capability without additional training.
• IndexTTS2 — TED-TTS’s TTS architecture and duration-embedding design directly follow this paper’s configuration; TED-TTS adds training-free segment-level control on top of it without modification.
• MaskGCT — a non-autoregressive baseline compared against in both emotion- and duration-control evaluations.
• F5-TTS — a non-autoregressive flow-matching baseline compared against in both emotion- and duration-control evaluations.
• CosyVoice 2 — an autoregressive baseline compared against in the emotion-control evaluation.
• Spark-TTS — an autoregressive baseline compared against in both emotion- and duration-control evaluations.
• EmoSteer-TTS — discussed as a related training-free, inference-time emotion-control method (activation steering on diffusion-based TTS) that, unlike TED-TTS, does not jointly address segment-level duration control.