arXiv · 2026 · Preprint

Sihang Nie et al. (South China University of Technology / Huya Inc. / Tongji University / The Hong Kong Polytechnic University / Foshan University) · → Paper · Demo: ✓ · Code: ✓

WordVoice inserts an explicit per-word “acoustic planning” step into a CosyVoice3-style LLM TTS: a boundary token triggers prediction (or user override) of five word-level attributes (duration, boundary, energy, pitch, tone) before each word’s speech tokens are generated, backed by a new 4.7k-hour bilingual corpus with word-level annotations.

Problem

LLM-based TTS systems that map text straight to discrete speech tokens produce natural speech but give no handle on individual words. Global style embeddings and natural-language instructions work at utterance level and cannot reliably enforce a specific duration, pause, emphasis or intonation on one word. That precision matters for audiobook narration and for dubbing, where timing must match lip movement. Earlier word-level systems controlled one dimension only: localized emotion in WeSCon, duration and pause in MAGIC-TTS. The authors also point to a data gap: large open corpora carry utterance-level transcripts or instructions, but not reliable word-level acoustic labels.

Method

Data: WordVoice-5A. The authors take audio and transcripts from LEMAS and re-annotate them with a three-stage pipeline (§2). Word timestamps come from both the Montreal Forced Aligner and Qwen3-FA. A loudness-based step refines each word edge. It snaps the edge to a silent “valley” frame when one exists, otherwise to the local loudness minimum, and never moves it past the syllable nucleus. A cross-model consistency check then drops any utterance where the two aligners disagree on overlap, duration or edges; the thresholds keep only the top 8% of aligned data. Five attributes are then extracted per word:

  • Duration from the aligned span.
  • Boundary: following pause binned into five ToBI/C-ToBI-inspired levels, b0 (no pause) to b4 (> 0.4 s).
  • Energy: mean of the top 50% of frame RMS values, normalized to [0, 1].
  • Pitch: mean of the central 80% of the F0 contour, normalized to [-1, 1].
  • Tone: the central F0 contour is fitted with a quadratic, and expert decision trees map it to seven shape categories (flat, rise, strong rise, fall, strong fall, peak, valley) shared by Chinese and English.

The result is about 2,546 h of Chinese and 2,138 h of English, 4,684 h in total (Table 2).

Model. WordVoice is built on the CosyVoice3 two-stage design: an autoregressive LLM emits discrete speech tokens, and a flow-matching (FM) model turns them into acoustic features (§3.1).

WordVoice-LLM (bound-token planning). The speech-token stream is restructured word by word as [s0, ⟨b⟩, v1, s1, ⟨b⟩, v2, s2, …] (§3.2). When the LLM emits the boundary token ⟨b⟩, its hidden state goes to a 2-layer MLP “Word Decoder”. This head predicts the five attributes. Each attribute is embedded, and the embeddings are concatenated into a style token c_i. The token is joined with the word’s text embedding to condition generation of that word’s speech chunk. In free mode the model uses its own predictions. In control mode the user bypasses the Word Decoder for any word and supplies attribute values directly, with no change to the LLM weights. Following pGSLM, continuous attributes are discretized: duration at the 40 ms frame rate, pitch and energy into 20 bins. All five are therefore trained as classification tasks. Their cross-entropy losses are combined with the autoregressive token loss through uncertainty weighting (Eq. 4). The LLM is initialized from Qwen2.5-0.5B (§3.4).

WordVoice-FM (word style modulation). Discrete tokens lose fine acoustic detail, so the word style tokens are also passed to the FM stage (§3.3). Each c_i is repeated over the frames of its word plus the following pause, giving a frame-level sequence. A linear projection turns each frame’s style vector into a scale and shift that modulate the layer-normalized speech-token embedding (FiLM-style). The FM model is trained with the standard flow-matching loss. During training, 30% of input speech tokens are masked so the model relies on the style conditions. At inference, the LLM’s chunk lengths supply the durations for this upsampling, so no external aligner is needed. For zero-shot prompting, the reference audio is aligned with MMS-FA and run through the same attribute pipeline (§3.4.1). Training uses 8 A800 GPUs, 7 epochs for the LLM and 20 for the FM model. The paper does not state whether the FM model starts from CosyVoice3 weights.

Fig. 3: Overall architecture of WordVoice. (a) WordVoice-LLM: During autoregressive decoding, the bound token ⟨b⟩ triggers the prediction of five acoustic attributes. These form a word-level style token to explicitly guide the chunked generation of speech tokens. (b) WordVoice-FM: The style tokens are LLM-based upsampled and injected into the Flow Matching backbone, providing fine-grained word style modulation for high-fidelity waveform synthesis.

Key Results

Evaluation uses held-out WordVoice-5A test sets (about 2,000 Chinese and 1,500 English utterances). The first 30% of each utterance’s audio is the prompt (§4.1). The main baseline is CosyVoice3. Mel-Recon (FM-vocoded ground-truth features) is the upper bound. WordVoice-Control is fed ground-truth word attributes.

Subjective (Table 3; 20 crowd-sourced raters, 80 utterances per system).

  • Among the synthesized systems (excluding Mel-Recon), WordVoice-Control has the highest N-MOS and Ctrl-MOS in both languages.
  • N-MOS: Control 3.689 vs. CosyVoice3 3.550 (zh); 3.773 vs. 3.643 (en).
  • Ctrl-MOS: Control 3.446 vs. 3.028 (zh); 3.645 vs. 3.324 (en). WordVoice-Free also beats CosyVoice3 on Ctrl-MOS (3.383 zh, 3.512 en).
  • Ctrl-MOS gains are significant (Mann-Whitney U, p < 0.01) for both variants in both languages. N-MOS gains are significant for Control in both languages and for Free in Chinese only.
  • Mel-Recon scores above all synthesized systems on every column.
  • Speaker similarity is mixed. In Chinese, CosyVoice3 leads (Spk-MOS 3.527 vs. 3.326 Free and 3.324 Control). In English, the three systems are within the confidence intervals (3.838 CosyVoice3, 3.869 Free, 3.846 Control). The authors’ “speaker-similarity trade-off” therefore shows clearly only in Chinese.

Objective control precision (Table 4). Attributes are re-extracted from generated audio using Qwen3-FA timestamps.

  • WordVoice-Free has slightly lower error than CosyVoice3 on all five attributes in both languages. For example, zh Pit-MAE is 0.1855 vs. 0.203 and en Bnd-ER 42.37% vs. 43.81%.
  • WordVoice-Control cuts errors substantially: zh Eng-MAE 0.0486 vs. 0.103, zh Bnd-ER 12.72% vs. 32.47%, en Pit-MAE 0.0782 vs. 0.1765, en Ton-RER 26.58% vs. 40.92%.
  • The cost is intelligibility. CosyVoice3 has the lowest WER of the synthesized systems: 2.31% zh and 1.06% en, vs. 2.58% / 1.31% for Free and 2.86% / 1.57% for Control.

Against MAGIC-TTS (Table 5, 100-utterance subset). WordVoice has lower duration MAE (0.0383 vs. 0.0438 zh; 0.0397 vs. 0.0593 en) than MAGIC-TTS’s duration variant. It also has lower boundary error (12.82% vs. 16.10% zh; 24.77% vs. 38.77% en) than MAGIC-TTS’s pause variant. MAGIC-TTS controls one attribute per word, so each of its variants is scored only on its own dimension.

Single-attribute control and ablation (Table 6). Constraining one attribute mostly reduces that attribute’s error alone. Dur-only gives zh Dur-MAE 0.0368, with the other errors close to free mode. Pitch and tone interact: Pit-only lowers Ton-RER (zh 27.74%, en 37.31%) compared with free mode (30.97%, 39.98%). The authors acknowledge tone is the least independent dimension. Replacing WordVoice-FM with the original FM (Orig-FM) raises energy and pitch error:

  • zh Eng-MAE 0.0486 → 0.0861, Pit-MAE 0.1100 → 0.1492.
  • en Eng-MAE 0.0475 → 0.0771, Pit-MAE 0.0782 → 0.1161.
  • Duration and boundary barely change.
  • Ton-RER also worsens (zh 20.25% → 25.50%, en 26.58% → 30.12%).

Novelty Assessment

Two architectural contributions sit on an otherwise standard CosyVoice3-style AR-plus-FM pipeline:

  • An in-sequence boundary token with an attribute head that turns per-word prosody into an explicit, overridable intermediate variable.
  • A FiLM-style word-attribute modulation in the flow-matching stage.

Each idea has precedent: explicit variance prediction in FastSpeech-style models, discretized prosody streams in pGSLM, and feature-wise modulation. The new part is placing per-word planning inside the autoregressive token stream so that predicted and user-specified attributes are interchangeable. The FM ablation is clean evidence that the second stage carries the energy and pitch control. The dataset adds value in its own right: a 4.7k-hour bilingual corpus with five word-level labels and a documented alignment-filtering pipeline.

The evaluation is narrower than the claims:

  • There is one conventional baseline, CosyVoice3, which has no word-level control at all.
  • The one word-level competitor, MAGIC-TTS, comes from the same group and is compared on 100 utterances.
  • “Control” results use ground-truth attributes from the test utterance itself, so they measure reconstruction of natural prosody rather than arbitrary user edits.

Field Significance

Moderate — The paper shows that an LLM-based TTS system can expose several word-level prosodic dimensions as explicit, user-replaceable intermediate tokens without retraining per edit. A dedicated modulation path in the acoustic stage is needed for energy and pitch to follow those tokens. It also provides a sizable word-level-annotated bilingual corpus. Its control evidence relies on the same automatic attribute extractors used to build the training labels.

Claims

  • supports: Exposing word-level prosodic attributes as explicit intermediate tokens in an autoregressive speech-token LM enables fine-grained, user-overridable prosody control without retraining.

    Evidence: A boundary token triggers a Word Decoder that predicts duration, boundary, energy, pitch and tone per word; replacing predictions with target values cuts zh Bnd-ER from 32.47% (CosyVoice3) to 12.72% and en Pit-MAE from 0.1765 to 0.0782, with Ctrl-MOS gains significant at p < 0.01. (§3.2, §4.2.2, Table 3, Table 4)

  • refines: In two-stage token-plus-acoustic-model TTS, temporal structure can be controlled at the discrete-token stage, while continuous attributes such as energy and pitch need explicit conditioning in the acoustic stage because quantized tokens discard them.

    Evidence: Removing the word style modulation from the flow-matching stage raises Eng-MAE (zh 0.0486 → 0.0861) and Pit-MAE (zh 0.1100 → 0.1492) while Dur-MAE and Bnd-ER stay nearly unchanged; with ground-truth tokens the same pattern holds. (§4.4, Table 6)

  • complicates: Adding fine-grained structural conditioning to LLM-based TTS can reduce intelligibility.

    Evidence: WER rises from 2.31% (CosyVoice3) to 2.58% (free mode) and 2.86% (control mode) on Chinese, and from 1.06% to 1.31% and 1.57% on English. (§4.2.2, Table 4)

  • complicates: Word-level prosodic attributes are not equally separable; pitch-contour categories remain entangled with pitch level.

    Evidence: Single-attribute control of duration, energy or boundary leaves the other errors near free-mode levels, but pitch-only control also lowers tone error (zh Ton-RER 27.74% vs. 30.97% free), and the authors note tone is the hardest dimension to decouple. (§4.3, Table 6)

  • supports: Multi-dimensional word-level control in one model can match or exceed single-dimension word-level controllers on their own dimension.

    Evidence: WordVoice has lower Dur-MAE than MAGIC-TTS’s duration variant (0.0383 vs. 0.0438 zh, 0.0397 vs. 0.0593 en) and lower Bnd-ER than its pause variant (12.82% vs. 16.10% zh, 24.77% vs. 38.77% en) on a 100-utterance subset. (§4.2.4, Table 5)

Limitations and Open Questions

Warning

Control precision is measured with the same automatic aligner (Qwen3-FA) and attribute-extraction pipeline used to create the training labels. Control-mode inputs are ground-truth attributes of the test utterance, so the results show reconstruction of natural prosody, not success on arbitrary or unusual user edits.

  • The only general TTS baseline is CosyVoice3. The single word-level competitor, MAGIC-TTS, is from the same group and is tested on a 100-utterance subset.
  • Test data are held-out utterances from the same LEMAS-derived corpus. The paper does not state whether test speakers appear in training, so zero-shot cloning of unseen speakers is not established.
  • Speaker similarity drops in Chinese (Spk-MOS 3.324 vs. 3.527 for CosyVoice3). WER rises in both languages.
  • Pitch and energy are quantized into 20 bins and tone into seven categories. Ton-RER treats adjacent tone categories as correct. Both choices limit how fine the measured control really is.
  • “Decoupling” here comes from separate attribute embeddings and is shown empirically (Table 6). There is no disentanglement objective, and tone stays partly entangled with pitch.
  • The paper reports no total parameter count, no inference-speed figures, and no tests on dubbing or audiobook tasks, even though these motivate the work.

Wiki Connections

  • Prosody Control — Provides explicit, per-word control of duration, pause boundary, energy, pitch and tone in an LLM-based TTS system, evaluated objectively and with a control-specific MOS.
  • Autoregressive Codec TTS — Restructures the speech-token sequence of an autoregressive LM TTS into word chunks preceded by a boundary token and a predicted style token.
  • Flow Matching — Adds word-level FiLM-style modulation to the flow-matching token-to-acoustic stage, which ablation shows carries energy and pitch control.
  • Multilingual TTS — Trains and evaluates one bilingual Chinese/English model with a tone taxonomy designed to cover both tonal and intonational languages.
  • Subjective Evaluation — Reports crowd-sourced naturalness, speaker-similarity and a word-style-control MOS (Ctrl-MOS) with significance testing.
  • CosyVoice 3 — Backbone architecture and main baseline; WordVoice keeps its LLM-plus-flow-matching structure and adds word-level planning and modulation.
  • MAGIC-TTS — Single-attribute word-level duration and pause controller from the same group, used as the explicit temporal-control baseline that WordVoice outperforms on its own dimensions.
  • WeSCon — Word-level emotion control system cited as prior work and excluded from baselines because its emotion labels lack deterministic acoustic mappings.
  • LEMAS — Source of the raw audio and transcripts that WordVoice-5A re-annotates with its own alignment pipeline.
  • Qwen2.5 — Provides the 0.5B text LLM used to initialize WordVoice-LLM.
  • CosyVoice 2 — Cited as an instruction-based TTS paradigm offering only utterance-level style control.
  • IndexTTS2 — Cited as a representative cascaded LLM-plus-acoustic-model TTS system lacking word-level structural boundaries.
  • FastSpeech 2 — Cited as an earlier TTS design built on explicit prediction of acoustic attributes, which WordVoice reintroduces inside an LLM pipeline.