arXiv · 2026 · Preprint
Kai-Wei Chang et al. (MIT, National Taiwan University) · → Paper · Demo: ? · Code: ?
Introduces TiCo, a spoken dialogue model that follows explicit response-duration instructions by learning to track its own elapsed speaking time during generation, and TiCo-Bench, the first benchmark for evaluating this capability.
Problem
Spoken dialogue models (SDMs) are increasingly deployed in settings where response duration itself is a hard constraint: a voice assistant giving a traffic update while driving, a wearable device limited by battery or bandwidth, or an emergency system that must deliver brief instructions under time pressure. Despite this practical need, time controllability is largely unexplored in SDMs. The paper argues that duration control in speech is fundamentally harder than length control in text LLMs, because word count is only a loose proxy for realized speech duration: a single word may span a variable number of syllables, and speaking rate varies with phonetic composition, linguistic context, prosody, speaker, and communicative condition. As a result, constraining the number of generated words or tokens does not reliably constrain the final spoken duration, and no existing benchmark measures whether SDMs can satisfy explicit duration instructions at all.
Method
TiCo treats a speech-to-speech SDM as a two-stage conditional generator: an intermediate-sequence generator produces a semantic plan z conditioned on the user’s speech query and a textual instruction, and a separate speech generator renders z into the spoken response. TiCo builds on Qwen2.5-Omni-7B’s Thinker-Talker architecture, fine-tuning only the Thinker (the intermediate-representation generator) while keeping the Talker (speech synthesis) frozen.
The core mechanism is the Spoken Time Marker (STM): a textual token inserted into the intermediate sequence (e.g. <6.8 seconds>) that estimates the cumulative speaking duration up to that point, giving the model an explicit, in-context signal of its own realized timing during generation.
Training proceeds in two stages. Stage 1 (time-awareness training) constructs supervision without any human-labeled question-answer pairs: the model freely self-generates an intermediate representation and spoken response for each input, ASR-based alignment (Whisper medium) estimates timestamps at sentence-level punctuation boundaries, and these timestamps are interleaved into the intermediate sequence as STM tokens (on average 13.3 markers per response, mean inter-marker interval 2.7s). The augmented sequence is then modeled autoregressively with a standard supervised fine-tuning objective.
Stage 2 (time-controllable training) augments the instruction with an explicit target duration and applies GRPO reinforcement learning with verifiable rewards to the intermediate-sequence generator. The main reward is a Gaussian function of the gap between the target duration and the duration indicated by the final STM in a sampled sequence. Four auxiliary rewards address specific reward-hacking failure modes observed empirically: a presence reward (generate at least one marker), a monotonicity reward (markers must increase), a repetition penalty (discourage identical repeated markers), and a copy penalty (discourage trivially copying the instructed target as an intermediate marker). CHORD, a dynamically-weighted auxiliary SFT loss that mixes in off-policy expert trajectories from Stage 1, is used alongside GRPO to stabilize training; the authors report that GRPO alone frequently leads to reward hacking without this regularization. At inference time, STMs are stripped via regex before the cleaned semantic plan is passed to the frozen Talker for speech synthesis, so the markers affect only the model’s internal planning, not the final audio content.
Key Results
The paper introduces TiCo-Bench, built from four task categories (question answering, reasoning, creative generation, summarization) drawn from existing speech and text instruction corpora, each instantiated under a Short (10-30s) and Long (30-60s) target-duration regime, for 2,000 evaluation samples. On TiCo-Bench, TiCo reduces overall duration error to 16.2% MAPE, a 2.7x reduction over its Qwen2.5-Omni-7B backbone (43.3%) and a 1.6x reduction over the strongest baseline, a cascaded GPT-5.2 + IndexTTS-2 pipeline (25.2%). TiCo achieves the best MAPE on 7 of 8 task/duration cells; the exception is the Short-Summarization cell (long input, short output), where both the cascade and a 30B open-source SDM outperform TiCo, attributed to this regime being underrepresented in Stage 1 self-generation data. Notably, a 30B model (Qwen3-Omni) is only marginally better than the 7B backbone (42.1% vs. 43.3% MAPE), which the authors use to argue that duration controllability does not emerge from scale alone. Response quality and naturalness are preserved (GPT-score 3.32 vs. backbone 3.31; UTMOS 4.04 vs. backbone 4.09 vs. ground-truth 4.08), and the learned capability generalizes to instructed durations beyond the 41-second cap seen during training and to text-only queries (18.0% MAPE on two text benchmarks, comparable to the 16.2% speech-query result despite training solely on speech).
Novelty Assessment
The core novelty is architectural/training-recipe: inserting a textual, ASR-aligned duration signal directly into the model’s own intermediate generation stream, then using RL with a duration-matching reward computed from that same signal to close the loop between planning and realized output timing. This differs from prior length-control work in text LLMs (which treat token count as a direct proxy for length) and from TTS duration modeling (which aligns text to synthesized speech rather than controlling a dialogue system’s response-level duration). The paper is explicit that the self-generation and ASR-alignment procedure removes the need for paired question-answer supervision, which is itself a data-efficiency contribution. TiCo-Bench is a genuine dataset/evaluation contribution: no prior benchmark isolates time-instruction-following as a distinct axis for SDMs. The underlying components (GRPO, CHORD, LoRA fine-tuning, Qwen2.5-Omni backbone) are all pre-existing; the contribution is the combination and the reward design (particularly the auxiliary rewards, which the ablation shows are necessary to avoid reward hacking) rather than a new base architecture or training algorithm.
Field Significance
moderate — TiCo identifies and addresses a specific, previously unmeasured capability gap in spoken dialogue models (following explicit duration constraints) and provides both a benchmark and an efficient post-training recipe that substantially improves it without degrading response quality. Its scope is narrow (a single backbone, English-only training data, one architectural family) and the practical demand for hard duration constraints in deployed systems is argued for but not independently validated.
Claims
- supports: Exposing an explicit, self-referential duration signal inside a generation model’s own intermediate planning stream enables tighter control over final output duration than approaches that only constrain the amount of content generated.
Evidence: Spoken Time Markers reduce duration MAPE from 43.3% (untrained backbone) to 16.2%, a 2.7x reduction, and outperform a cascaded LLM-planning + TTS baseline (25.2% MAPE) by 1.6x on TiCo-Bench. (§5.1, Table 1)
- complicates: Pipelines that plan output length in one modality (text) and realize it in another (speech) face a structural duration-control ceiling, because the planning stage cannot observe the downstream module’s realized timing.
Evidence: The cascaded baseline’s MAPE rises with target duration on the QA subset (19.9% Short to 24.3% Long) because the text-planning LLM cannot observe the TTS component’s realized speaking rate, while TiCo’s error falls over the same regime (15.4% to 11.7%) since its timing signal is generated inside the same model that produces the final content. (§5.1)
- supports: Reinforcement learning with automatically verifiable, rule-based rewards can be used to train timing/duration behavior in a speech generation system without any paired human-labeled supervision.
Evidence: Stage-2 training uses GRPO with a Gaussian duration-matching reward computed directly from the model’s own generated time markers, requiring no question-answer paired data; reward-component ablation shows MAPE dropping from 32.2% (main reward alone) to 15.4% (full reward set with presence, monotonicity, repetition, and copy penalties). (§3.2, §5.4, Table 2)
- complicates: A duration-control capability learned via post-training on a bounded target range extrapolates to unseen ranges only when generalization is checked explicitly; performance in the regime underrepresented during training remains a residual weakness.
Evidence: Despite training on responses capped at 41 seconds, TiCo maintains low MAE/MAPE up to 60 seconds on InstructS2S and URO-Bench, but underperforms both a cascaded baseline and a larger open-source SDM specifically on the Short-Summarization cell (long input, short output), a regime rare in the Stage-1 self-generation data. (§5.1, §5.2, Figure 3)
Limitations and Open Questions
Warning
All experiments use a single backbone (Qwen2.5-Omni-7B with a Thinker-Talker architecture) and training data drawn solely from English instruction-following queries. Whether the Spoken Time Marker mechanism transfers to SDMs with parallel or interleaved generation patterns, or to other languages and conversational domains, is untested.
The paper also reports two narrower limitations: TiCo underperforms the strongest cascaded baseline specifically on the Short-Summarization subset, where the long-input/short-output regime is underrepresented in Stage 1 training data; and local (intermediate, not just final) marker prediction error remains around 2-3 seconds, which bounds the precision available for fine-grained real-time planning rather than only end-of-response duration matching.
Wiki Connections
- Spoken Language Model — TiCo fine-tunes the Thinker component of an existing speech-in, speech-out language model (Qwen2.5-Omni) to add duration-aware planning without altering its frozen speech-synthesis backend.
- Speech-to-Speech — TiCo is a dialogue-oriented speech-to-speech system; its contribution targets the response-duration axis of spoken dialogue interaction rather than translation or codec-level transforms.
- RLHF Speech — the second training stage applies GRPO with a verifiable, rule-based reward derived from the model’s own duration markers to shape response timing, illustrating RL-based post-training for a non-quality speech generation objective.
- Evaluation Metrics — introduces TiCo-Bench, a benchmark and MAE/MAPE-based evaluation protocol specifically for measuring duration-instruction-following in spoken dialogue models, a capability no prior benchmark isolated.
- Qwen3-Omni Technical Report — used as a 30B open-source SDM baseline in TiCo-Bench; its comparable-to-7B duration error is used to argue scale alone does not confer duration controllability.
- MiMo-Audio — evaluated as an open-source SDM baseline on TiCo-Bench, underperforming TiCo across most task/duration cells.
- URO-Bench — used both as a source of reasoning-category queries for TiCo-Bench and as a generalization testbed for instructed-duration bins beyond the training range.
- Full-Duplex-Bench — cited as part of the broader landscape of SDM benchmarks that evaluate turn-taking and interactivity but not time-controllability, motivating the need for TiCo-Bench.
- Chain-of-Thought Reasoning in Streaming Full-Duplex SDS — cited as related work on intermediate semantic-planning representations in SDMs, the same architectural layer TiCo augments with timing markers.
- Stream RAG — cited as an example of SDMs that extend the intermediate representation with tool-calling capability, part of the broader trend of enriching intermediate planning that TiCo’s timing markers also belong to.
- PersonaPlex — cited as a recent full-duplex conversational speech model representative of the broader SDM landscape TiCo-Bench is positioned against.