arXiv · 2026 · Preprint
Xingchen Song et al. (Xiaomi (MiLM Plus)) · → Paper · Demo: ✓ · Code: ?
Proposes a framework for “borderless” long-form speech synthesis that pairs a hierarchical natural-language annotation schema with a continuous-tokenizer backbone trained with chain-of-thought reasoning and dimension dropout, positioned as an agent-facing control interface rather than a single-task TTS model.
Problem
Existing TTS pipelines either synthesize utterances one sentence at a time and stitch the results together, or condition on plain-text dialogue turns alone. Both leave the model blind to information that spans beyond a single sentence: how an emotional arc should build over dozens of lines, how multiple speakers interrupt and overlap, and what acoustic environment surrounds the speech. The paper also argues that conventional annotation and filtering pipelines actively discard the most expressive training data, since acoustically “messy” segments (arguments, overlapping talk, background noise) are exactly the ones aggressive quality filters remove, leaving models trained on a narrowed, sanitized distribution.
Method
The system is built as a unified capability set (VoiceDesigner, multi-speaker synthesis, Instruct TTS, and long-form synthesis) rather than a single narrow task, and rests on three components.
Data strategy. Rather than filtering low-quality segments by DNSMOS score, WER, or single-speaker checks (a process the authors note commonly retains only 10-30% of a corpus), the paper adopts a “label, don’t filter” policy: overlapping speech, interruptions, and background sound/music are kept and instead described in natural language, reportedly reaching over 90% data utilization. On top of this retained data, the paper defines a top-down “Global-Sentence-Token” annotation schema with three levels: a Global layer for scene-level metadata (format, speaker profiles, overall emotional trajectory, acoustic environment), a Sentence layer for per-utterance controls (tone, intonation, speed, intent, background state), and a Token layer for phoneme-level detail (stress, polyphone disambiguation, connected-speech rules). Every dimension at every level is expressed as free-form natural-language caption rather than a fixed enum or numeric code.
Model architecture. The backbone uses a continuous tokenizer (the specific codec is not named) that the authors report choosing after controlled comparisons on the same training set. Two training strategies are layered on top. Chain-of-Thought (CoT) reasoning splits the Global-Sentence-Token annotations into an “Instruct” stream (user-supplied hard constraints: scene, speaker identity, environment ratings) and a “Think” stream (the model’s own expressive plan: emotional arc, per-sentence tone/pace/intent, phoneme-level realization). At inference, the model first generates the Think dimensions sentence by sentence as explicit, inspectable output, and only then synthesizes audio, turning the prosody decision into a traceable reasoning chain instead of an implicit internal computation. Dimension Dropout randomly masks a subset of Think dimensions during training (e.g., acoustic-environment description or emotional trajectory) without asking the model to reconstruct them, so the model learns to produce audio from incomplete conditioning; the intended effect is more faithful instruction-following when some dimensions are absent and support for users who specify only a subset of controllable dimensions at inference.
Agentic framing. The paper additionally casts the Global-Sentence-Token schema as a “Structured Semantic Interface” between an upstream LLM agent and the synthesis engine, with the three annotation layers mapped to a session/transport/physical-layer protocol stack analogy. The argument is that this widens the text interface from a narrow lexical-content channel into an information-complete control channel, so a front-end LLM that can understand any input modality (text, video, etc.) can express that understanding as Global-Sentence-Token commands for the synthesis engine to execute, and can compress multi-turn conversation history into a compact schema-conforming state rather than feeding raw history into the network.
Key Results
The paper reports no quantitative benchmark numbers for its system. It explicitly states that it evaluated candidate automated metrics for the target phenomena (CLAP-based audio-text alignment, audio-captioning-based semantic consistency, and signal-level metrics DNSMOS, PESQ, and POLQA) and found each inadequate: cross-modal models trained on short clips lack discriminative power for sustained long-audio phenomena, audio-captioning models operate at too coarse a granularity to capture paralinguistic subtleties like tonal pivots or interruption timing, and signal-level metrics are blind to scene modeling, expressiveness, and instruction adherence. Because of this, the paper states it cannot present quantified capability boundaries in tabular form, and instead points readers to externally hosted demos to judge expressiveness directly.
Novelty Assessment
The paper’s most concrete technical proposals are the CoT-based Instruct/Think decomposition and Dimension Dropout, both training-time mechanisms layered on an existing continuous-tokenizer LM-style backbone rather than a new model architecture in themselves. The hierarchical Global-Sentence-Token annotation schema and the “label, don’t filter” data philosophy are a genuinely distinct framing of how to source and structure training data for expressive long-form speech, though the schema itself is a labeling convention rather than a released dataset or tool. The “Native Agentic” / Structured Semantic Interface framing is primarily conceptual: it reinterprets an existing instruction-conditioned TTS interface as a layered control protocol, but the paper does not report a working end-to-end agent integration or measure the claimed context-efficiency advantage over end-to-end spoken-dialogue models. Overall the contribution reads as an industrial position paper plus two specific training mechanisms, with the mechanisms’ benefits asserted narratively rather than measured.
Field Significance
moderate — The paper articulates a coherent framing for combining long-context, multi-speaker, and acoustic-scene-aware synthesis behind a single hierarchical instruction interface, and proposes two concrete training mechanisms (CoT prosody planning, Dimension Dropout) that other Instruct TTS systems could adopt. Its impact is capped by the complete absence of quantitative evaluation: every improvement claim (data utilization, instruction-following, hallucination reduction) is asserted narratively rather than measured against a baseline, so the paper functions more as a design proposal and provocation for Generation-4 TTS evaluation methodology than as validated evidence.
Claims
- supports: Preserving acoustically “messy” naturalistic speech (interruptions, overlapping speech, background sound) through rich descriptive annotation, rather than discarding it via quality-based filtering, can increase both data utilization and the diversity of expressive training signal available to a speech synthesis model.
Evidence: The paper’s “label, don’t filter” strategy retains and annotates data that conventional DNSMOS/WER/single-speaker filtering pipelines would discard (reported to leave only 10-30% of a corpus per cited prior work), reaching over 90% data utilization, and the authors report that models trained only on clean, simple data were more prone to hallucination at inference. (§2.1)
- supports: Decomposing text-to-speech generation into an explicit reasoning stage over context and prosody dimensions before audio synthesis can make the mapping from context to expressive delivery more inspectable and controllable than an implicit single-pass mapping.
Evidence: The Instruct/Think chain-of-thought decomposition has the model generate global atmosphere, per-sentence tone/intonation/pace/intent, and phoneme-level realization as explicit output before synthesizing audio, described as yielding better precision and global coherence in complex conditioning scenarios. (§3.1)
- complicates: Conditioning a synthesis model on many simultaneous descriptive dimensions (scene, speaker, prosody, background state) risks brittle over-reliance on all dimensions being present at inference, which explicit training-time dimension masking can mitigate.
Evidence: Dimension Dropout randomly masks a subset of Think-stream dimensions during training without requiring reconstruction, intended to let the model follow remaining instructions faithfully when some conditioning dimensions are missing at inference. (§3.2)
- complicates: Expressive, long-form, multi-speaker speech synthesis currently lacks any automated evaluation metric capable of jointly capturing emotional-arc coherence, multi-speaker interaction naturalness, acoustic-scene fidelity, and instruction-following accuracy.
Evidence: The paper tests CLAP-based audio-text alignment, audio-captioning-based consistency checks, and signal-level metrics (DNSMOS, PESQ, POLQA) against its target phenomena and finds each insufficient, leaving the system’s own capabilities unquantified in the paper. (§5)
- refines: A hierarchical, natural-language annotation schema spanning session-level, utterance-level, and phoneme-level descriptions can function as a wider-bandwidth control interface between an upstream instruction source and a downstream synthesis engine than a plain-text-plus-speaker-ID interface.
Evidence: The Global-Sentence-Token schema’s three layers are mapped onto a session/transport/physical-layer control-protocol-stack analogy, argued to let scene, speaker, emotion, and acoustic understanding travel alongside the text rather than being compressed to lexical content alone. (§4, §4.1)
Limitations and Open Questions
The paper reports no quantitative evaluation of its own system: no MOS, no objective metrics, and no comparison table against the Generation-2/3 baselines it discusses (Tacotron, VALL-E, CosyVoice, ChatTTS). All claimed improvements from CoT, Dimension Dropout, and the labeling strategy are narrative rather than measured, and demo access is described only as "via web search" rather than a stable link.
The authors state the system is currently optimized for offline content-creation scenarios (podcasts, audiobooks, film narration) and has not been adapted for real-time interactive settings requiring millisecond-level latency and streaming output. Training data remains speech-centric; the system’s sound-effect and music generation is described as an emergent, unintended byproduct of background audio in training data rather than a purpose-built capability, with an acknowledged quality ceiling relative to dedicated training. The paper also identifies the absence of a reliable automated evaluation framework for expressive long-form speech as an open problem it does not solve, proposing LLM-as-judge and structured human perceptual studies as future directions rather than delivering them here.
Wiki Connections
- Instruction-Conditioned TTS — extends natural-language instruction conditioning from single-sentence style control to a hierarchical schema spanning scene, speaker, and phoneme-level natural-language captions.
- Prosody Control — introduces an explicit chain-of-thought planning stage that generates per-sentence tone, intonation, pace, and phonetic realization before audio synthesis.
- Emotion Synthesis — models emotional tone as an evolving, multi-sentence trajectory rather than a per-utterance label, conditioned through the Global layer’s emotional-arc description.
- Evaluation Metrics — argues that existing automated metrics (CLAP, audio captioning, DNSMOS, PESQ, POLQA) cannot evaluate expressive long-form, multi-speaker speech and leaves this as an open problem.
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — cited as an early “Generation 3” LLM-based TTS system that introduced instruction-style control but only at single-sentence granularity, which this paper’s long-context framing is positioned against.
- CosyVoice 2 — cited alongside VALL-E and ChatTTS as prior LLM-grade Instruct TTS work that this paper’s continuous-tokenizer, chain-of-thought backbone builds beyond.