arXiv · 2026 · Preprint

Shengfan Shen et al. (Nanjing University / MiLM Plus, Xiaomi Inc. / WeNet Open Source Community) · → Paper · Demo: ✗ · Code: ✗

Recasts expressive style control for voice-assistant TTS as closed-set routing: a small text-only LLM planner reads a prioritized, multi-source context description and picks one prompt clip from a curated single-speaker style library, which an unmodified prompt-based TTS engine then clones.

Problem

Voice assistants need speaking style to follow both explicit user requests and wider context: who the user is, the interaction scenario, time of day, ambient noise. Zero-shot TTS takes its style from one fixed reference clip, so it does not adapt at runtime. Instruction-driven TTS accepts free-form style descriptions, but the authors argue its output is unstable when the description is ambiguous or high-level. Conversational TTS models look only at short dialogue history. The paper’s position is that the bottleneck is not the generative capacity of current TTS engines but the missing decision layer between context understanding and synthesis. Heuristic rules and embedding retrieval, the usual ways to pick a style prompt, fail on implicit preferences and on conflicting signals.

Method

Harness TTS (named after “harness engineering”, the practice of wrapping a model in an external orchestration layer) sits in front of a TTS engine and makes all style decisions outside it. It has three parts.

  • Tool registry (offline). A compact library of expressive prompt clips for one target speaker. Each entry (“tool”) has an audio_id, discrete tags (speed, volume, emotion) and a free-text caption describing its style and suitable scenarios. Tags and captions can come from a speech LM or a human annotator. Tools are of two kinds: axis-based tools that cover combinations of speed, volume and emotion levels, and scenario-preset tools with one dedicated clip per common use case (navigation, bedtime story, safety alert and so on).
  • Observation schema and planner (online). Upstream signals are packed into five fields with a fixed priority order: explicit instruction (5), implicit intent such as time, noise, dialogue history or user emotion (4), scene (3), user profile (2), and system default (1) (Table 1). A text-only LLM planner (Qwen3-0.6B, 1.7B or 4B) receives the registry metadata, the observation and a routing prompt that states the priority policy, plus few-shot examples for explicit, implicit and conflict cases. It outputs a ranked list of five audio_ids. A chain-of-thought (CoT) variant emits a <reason> before each <audio_id>. The planner never hears the audio and is not trained.
  • Executor. The TTS engine receives the target text and the prompt audio bound to the top-ranked tool. The experiments use CosyVoice3 and VoxCPM2 as released; the paper does not report fine-tuning them, although it notes that an ideal executor would be fine-tuned on the target speaker.

Bounding the decision to a finite registry makes every style choice traceable to one audio_id, which is the main design argument (controllability and debuggability) over open-vocabulary instruction control.

Key Results

Routing (RQ1). Labels come from a Gemini-2.5-Pro teacher, so all accuracy figures measure agreement with that teacher (“T-” prefix), not with human judgments. The test cases are also generated by Gemini-2.5-Pro from templates, over a simulated 42-tool registry (27 axis-based, 15 scenario-preset). Qwen3-4B with CoT has the highest T-Exact@1 of all methods on every subset: 0.743 explicit, 0.430 implicit, 0.646 conflict. The larger embedding-retrieval baseline (Qwen3-Embedding-4B) reaches 0.348, 0.259 and 0.199 (Tables 2 to 4); on the conflict subset the smaller 0.6B retriever (0.233) and the keyword baseline (0.286) both score higher than it. The gap is widest on the conflict subset, where retrieval has no way to apply the priority policy. Scale matters a lot: Qwen3-0.6B without CoT scores 0.033 to 0.081 T-Exact@1, far below even the keyword baseline (0.207 to 0.376). CoT helps the 4B model on all three subsets but is not uniformly better: Qwen3-1.7B without CoT beats its CoT variant on the implicit subset (0.333 vs 0.252), and Qwen3-4B without CoT has the highest T-Recall@5 on the explicit (0.559) and implicit (0.507) subsets.

Latency (RQ2). On an RTX 5090 with vLLM, Qwen3-4B without CoT produces its first audio_id at 34.4 ms P50 and 41.2 ms P95, close to the 54.6 to 56.1 ms of Retrieval-4B. The full five-item list takes 179.9 ms P50. CoT raises 4B First-ID latency to 216.3 ms P50 and 323.6 ms P95 (Table 5). The most accurate routing configuration (4B with CoT) is therefore not the configuration that meets the sub-50 ms claim (4B without CoT).

Synthesis (RQ3). Harness (Qwen3-4B planner, 25-clip library) is compared against Instruct, where the engine receives the instruction text with a default reference clip of the same speaker, on 135 LLM-generated test cases. Only the explicit instruction is varied; other observation fields are set to neutral defaults, so this test does not exercise the context-awareness that motivates the method. A Gemini-3.1-Pro pairwise judge prefers Harness in all four conditions (Figure 3): 61.5 vs 38.5 (CosyVoice3, parameter), 66.7 vs 31.1 (CosyVoice3, scenario), 55.4 vs 41.5 (VoxCPM2, parameter) and 57.8 vs 37.8 (VoxCPM2, scenario). Gains are larger on scenario-style instructions than on low-level acoustic ones. UTMOSv2 is higher with Harness in all four conditions, by 0.11 to 0.38 (Figure 4). Speaker stability, the mean pairwise WeSpeaker similarity within each instruction tag, rises with Harness except on CosyVoice3 scenario instructions (90.4 vs 92.9); the authors attribute that exception to Instruct varying style less. There are no human listening tests.

Novelty Assessment

The contribution is engineering integration. Selecting among stored reference prompts for a prompt-based TTS engine is a known deployment practice, which the paper itself describes as pragmatic. What the paper adds is a specific formulation: a structured, prioritized observation schema, dual tag-plus-caption tool metadata, and a small LLM used as a ranked router over that closed set. The planner is prompted, not trained, and the TTS engines are used unchanged. The evaluation leans entirely on LLMs: generated test cases, a teacher LLM as the routing reference, and an LLM judge for instruction following. That makes the routing numbers a measure of how closely small Qwen3 models imitate Gemini-2.5-Pro rather than of correct style choice. The Instruct baseline is also limited: the comparison pits a curated set of 25 style clips against one default clip plus a text instruction, so part of the gain may reflect the curated clips rather than the planner.

Field Significance

Low. The paper gives evidence that choosing among curated single-speaker style prompts can beat direct natural-language instruction on two current instruction-capable TTS engines, under automated judging. It also offers a clear design pattern for keeping style decisions auditable in voice-assistant deployments. Because the evaluation uses only synthetic data, LLM reference labels and LLM judging, it mainly adds a practical data point rather than settled evidence.

Claims

  • supports: Selecting a stored expressive reference prompt can realize requested speaking styles more reliably than passing a free-form style instruction to an instruction-capable TTS engine.

    Evidence: With CosyVoice3 and VoxCPM2 as executors, prompt selection by a Qwen3-4B planner wins pairwise instruction-following judgments over direct instruction in all four parameter/scenario conditions (e.g., 66.7 vs 31.1 on CosyVoice3 scenario instructions) and raises UTMOSv2 by 0.11 to 0.38; judging is by Gemini-3.1-Pro, with no human raters. (§5.3, Figures 3 and 4)

  • refines: Instruction-driven style control struggles more with high-level scenario descriptions than with low-level acoustic attribute requests.

    Evidence: The advantage of prompt routing over direct instruction is larger on scenario instructions than on speed/volume/emotion instructions for both executors (CosyVoice3: 66.7 vs 31.1 against 61.5 vs 38.5; VoxCPM2: 57.8 vs 37.8 against 55.4 vs 41.5). (§5.3, Figure 3)

  • supports: Resolving conflicting context signals for expressive speech requires explicit priority reasoning that embedding-similarity retrieval cannot provide.

    Evidence: On cases where context fields contradict the explicit instruction, Qwen3-4B with CoT reaches 0.646 T-Exact@1 against 0.199 for Qwen3-Embedding-4B retrieval, the widest gap of the three routing subsets. (§5.1, Table 4)

  • complicates: Small LLM planners for speech-style decisions face a sharp accuracy-latency trade-off driven by model scale and reasoning traces.

    Evidence: Qwen3-0.6B without CoT reaches only 0.033 T-Exact@1 on explicit requests, below a keyword baseline (0.376); CoT lifts Qwen3-4B accuracy but raises its P50 first-tool latency from 34.4 ms to 216.3 ms. (§5.1, §5.2, Tables 2 and 5)

  • complicates: Speaker-consistency scores across instructed styles can rise when the system fails to follow the style instruction, so they are not a standalone quality signal.

    Evidence: On CosyVoice3 scenario instructions, direct instruction has higher speaker stability than prompt routing (92.9 vs 90.4) while losing the instruction-following comparison 31.1 to 66.7; the authors read the higher stability as a side effect of less stylistic variation. (§5.3, Figures 3 and 4)

Limitations and Open Questions

Fully automated, LLM-dependent evaluation

All test cases are LLM-generated, routing accuracy is measured against a Gemini-2.5-Pro teacher rather than human labels, and instruction following is scored by a Gemini-3.1-Pro judge. No human listening test is reported, which the authors acknowledge.

  • The synthesis comparison varies only the explicit instruction and sets all other context fields to neutral defaults, so the context-aware behaviour (user profile, scene, implicit cues) is evaluated only at the routing stage, not in synthesized speech.
  • The routing registry is simulated (metadata without audio), and the synthesis library has 25 clips for one speaker. How routing accuracy scales with larger or multi-speaker registries is not tested.
  • The closed-set design limits expressiveness to the styles in the registry; new styles need new recorded or generated clips and annotations.
  • The baseline gives the Instruct condition a single default reference clip, while Harness draws on 25 curated clips. A retrieval or keyword router feeding the same clips to the TTS engine is evaluated only for routing, not for synthesis, so the share of the synthesis gain owed to the LLM planner is not isolated.
  • No code, registry or test data release is mentioned.

Wiki Connections

  • Instruction-Conditioned TTS — Accepts natural-language style instructions but converts them into a choice among stored style prompts, and reports this beats direct instruction conditioning on two instruction-capable engines.
  • CosyVoice 3 — Used unmodified as one of the two TTS executors and as the Instruct baseline it is compared against.
  • VoxCPM2 — Second unmodified TTS executor and Instruct baseline.
  • Qwen3 — Supplies the 0.6B, 1.7B and 4B planner models; the Qwen3-Embedding retrieval baselines come from a related release.
  • Gemini 2.5 — Gemini-2.5-Pro generates the routing test cases and serves as the teacher whose rankings define routing accuracy.
  • OV-InstructTTS — Cited as open-vocabulary instruct TTS that reasons from free-form instructions to acoustic attributes; Harness instead constrains the decision to a closed prompt set.
  • CapTalk — Cited as context-aware conversational TTS that plans prosody turn by turn from dialogue; Harness extends the context to user profile, scene and ambient cues.
  • IndexTTS2 — Cited as a zero-shot TTS with disentangled emotion and timbre control whose reliance on a fixed runtime prompt motivates prompt selection.