arXiv · 2026 · Preprint
Xianhao Zhou et al. (UESTC / Monash University) · → Paper · Demo: ? · Code: ✓
A paired audit of natural-language voice control in three reference-conditioned TTS systems shows that outputs which move in the requested direction usually also change pitch, timing, energy or spectral measures outside the request, and a training-free candidate selector (VoDER-Cal) reduces that collateral change within a fixed three-sample budget.
Problem
Instruction-conditioned TTS (natural-language voice descriptions such as “deep”, “bright” or “rough”) is usually evaluated by asking whether the output matches the description: style preference, naturalness, intelligibility, speaker similarity, or instruction-following benchmarks such as InstructTTSEval and the prompt-adherence metric SPAM. These measures say whether a system responds to a prompt. They do not say whether it changes only the requested attribute while the rest of the voice stays stable. A “deep” request can lower F0 but also slow the speech, make it quieter, darken the spectrum, or move it away from the reference speaker. The paper treats prompt adherence and attribute-specific control as distinct capabilities and asks what changes outside the requested attribute.
Method
Paired audit. Three reference-conditioned systems are audited through their native control interfaces: CosyVoice 3 (Fun-CosyVoice3-0.5B, instruction plus reference speech), VoxCPM2 (inline instruction plus reference) and Fish Audio S2 (the s2-pro checkpoint, called Fish-Speech-S2 in the paper, driven by its native inline voice tag rather than a free-form instruction channel) (§3.1, Table 3). Each system runs the same factorial matrix: 6 reference speakers, 10 English texts, 3 random seeds and 11 conditions (a neutral baseline, six descriptors, four controls), giving 1,980 outputs per system and 5,940 overall. Every descriptor-conditioned output is paired with a neutral output from the same system, speaker, text and nominal seed. The primary claims cover only deep, bright and rough. Warm, young, professional and the identity-preservation control are supplementary.
Measurements. Eight signal-level features are extracted from each output: F0 (in semitones), speaking rate, duration, RMS energy, spectral centroid, 85% roll-off, spectral flatness and zero-crossing rate. Each descriptor gets a pre-registered target set with an expected direction: deep = {F0 down, roll-off down}, bright = {centroid up, roll-off up}, rough = {flatness up, ZCR up} (§3.2). Everything outside the target set counts as off-target. Paired changes are normalized by fixed physical scales shared across systems (for example 3 semitones for F0, 0.8 words/s for rate, 220 Hz for centroid) (Table 4). Content validity comes from normalized ASR transcripts (Whisper is the cited recognizer), and speaker preservation from ECAPA-TDNN cosine similarity to the reference, with x-vector similarity as a cross-check. Both thresholds are fixed on a calibration split (§3.3).
Statistics and controls. Inference is at the request level (system, descriptor, speaker, text). The three seed deltas are averaged within a request, and request blocks are resampled with 10,000 bootstrap replicates. A conditional analysis keeps only outputs whose target score exceeds a one-sided noise threshold derived from baseline seed variation. It then counts outputs with at least one off-target change of at least half its normalization scale (§3.4, §11.3). Four controls are run: a neutral reading instruction, a nonsense descriptor, an identity-preservation instruction and an explicit lower-pitch instruction (a positive control).
VoDER-Cal. For each request, B descriptor-conditioned candidates are compared with the neutral baseline. A direction-aligned target score T averages the normalized target-feature changes. A candidate is feasible if T exceeds both the noise threshold and a fraction ρ of the strongest target score in the pool, and if it passes the content and speaker checks. Among feasible candidates, VoDER-Cal picks the one with the smallest mean absolute normalized change on a selector-side set of non-target features. If no candidate is feasible, it abstains (§5.1). Evaluation uses a separate, non-overlapping set of held-out off-target features. The main operating point is B = 3, ρ = 0.75. The method needs no retraining and no access to model internals.
Key Results
Responses are directional but not isolated. For deep, outputs move in the expected target direction in 84.4% (CosyVoice3), 77.2% (VoxCPM2) and 71.1% (Fish-Speech-S2) of pairs. Bright and rough are weaker and less consistent (44.4% to 71.1%) (§4.1, Table 1). Content is preserved in at least 99.4% of outputs, and mean speaker similarity is 0.821 to 0.902. The off-target movement therefore occurs mostly in outputs that keep both content and speaker identity. In most system-descriptor settings, several non-target features have request-level confidence intervals that exclude zero (for example 5/6 or 6/6 for CosyVoice3, versus 1/6 to 4/6 for VoxCPM2 and Fish-Speech-S2) (Table 1). Systems realize the same descriptor differently. CosyVoice3’s deep response also changes rate, duration, energy, centroid and roll-off. VoxCPM2’s bright response is carried more by energy than by the spectral targets (§4.1, Figure 2).
Conditioning on a clear response does not remove collateral change. In the eight settings with at least ten above-noise responsive outputs, 54.5% (Fish-Speech-S2 bright, the lowest) to 95.8% (VoxCPM2 deep, the highest) of responsive outputs show at least one substantial off-target deviation (§4.2, Table 5). Fish-Speech-S2 deep has only two responsive outputs under the composite threshold, so the authors treat it as descriptive only.
Controls. The explicit lower-pitch instruction lowers mean F0 in all three systems (−7.33 Hz CosyVoice3, −7.17 Hz VoxCPM2, −4.80 Hz Fish-Speech-S2, all intervals below zero). Neutral and nonsense instructions alone shift 5/8 to 7/8 measurements in CosyVoice3 and VoxCPM2, against 0/8 and 2/8 in Fish-Speech-S2 (§4.3, Table 6). Extra natural-language conditioning alone can therefore perturb the voice. This supports comparing against a matched baseline rather than judging conditioned outputs in isolation.
VoDER-Cal. Macro-averaged over nine system-descriptor settings (§5.2, Table 2):
| Policy | Eligible | All checks | Held-out dev. ↓ | Calls |
|---|---|---|---|---|
| Direct | 11.5% | 4.8% | 0.325 | 1 |
| Random | 11.2% | 4.7% | 0.322 | 1 |
| Target-only | 31.3% | 14.1% | 0.344 | 3 |
| VoDER-Cal | 31.3% | 14.3% | 0.276 | 3 |
| Oracle | 31.3% | 14.6% | 0.225 | 3 |
Most of the joint-success gain comes from having three candidates, not from the ranking rule. VoDER-Cal minus Target-only is +0.19 points (95% CI [−0.56, +0.93]), while VoDER-Cal minus Direct is +9.44 points ([+6.85, +12.04]) (Table 7). The matched-budget benefit appears in the continuous measure. Held-out off-target deviation falls from 0.344 (Target-only) to 0.276 (VoDER-Cal), against 0.225 for the Oracle, which has the lowest deviation of the five policies. Target-only selection has higher held-out deviation than single-sample Direct (0.344 vs. 0.325), so picking the strongest target response also picks more collateral change. The ordering of Target-only, VoDER-Cal and Oracle holds for unseen speakers, unseen texts and both together (VoDER-Cal 0.276 to 0.277 vs. Target-only 0.343 to 0.344) (Table 8). Lowering ρ trades target retention for preservation and eligibility. At ρ = 0, joint success is 17.4% and deviation 0.269 (the best values across the four ρ settings), but target retention drops to 87.9% (Table 9).
Listening studies. Ten adult listeners took part in two blinded studies of 45 samples each (§6, §14). In H1, listeners judged the requested attribute changed in 51.1% of pairs and reported a noticeable non-target change in 51.1%, with both in 22.2%. In H2 (VoDER-Cal vs. Target-only, with the neutral output as reference), 63.3% of off-target judgments favored VoDER-Cal as closer to baseline, 9.9% favored Target-only and 26.8% were ties. That is 86.4% of non-tied judgments, consistent across rate, loudness, pitch and voice quality, and robust to A/B position and trial order (Table 10, Table 12). Target expression goes the other way: 29.6% favored Target-only, 20.7% VoDER-Cal and 49.6% were ties. Speaker similarity and naturalness were roughly even. Pairwise exact agreement is low for H1 five-point judgments (22.2% to 40.7%) and moderate for H2 (31.9% to 54.1%) (Table 11).
Novelty Assessment
The main contribution is the evaluation design. It pairs each descriptor-conditioned output with a matched neutral output, uses pre-registered target and off-target feature sets, applies request-clustered bootstrap inference, and adds a conditional analysis that separates “failed to respond” from “responded with collateral change”. The individual measurements are standard librosa/pYIN-style signal features, and the content and speaker checks reuse ASR and ECAPA-TDNN. The novelty is in the paired, preservation-sensitive framing, not in any new measurement model.
VoDER-Cal is a new selection criterion inside the existing best-of-N reranking paradigm, which the paper itself places alongside ASR-verifier reranking such as ASR self-verification. It is a cheap, training-free heuristic. Its binary success gain over Target-only is not significant, and its selector and evaluator features come from the same correlated family, a caveat the authors acknowledge (§7.2). It is therefore not tagged as an architectural contribution. The audit covers three systems, three primary descriptors and English only, and it measures roughness with weak proxies (flatness, ZCR).
Field Significance
Low — The paper gives a concrete, statistically careful demonstration that prompt adherence and attribute-specific control can diverge in current instruction-conditioned TTS. It also offers a reusable paired protocol for measuring off-target change against a matched baseline. The evidence is narrow (three systems, three signal-level descriptors, one language, ten listeners), and the proposed selector gives a modest, inference-time improvement rather than a new modelling direction.
Claims
- complicates: Instruction-conditioned TTS outputs that move in the requested attribute direction frequently also change prosodic, energy and spectral characteristics outside the requested attribute, even when content and speaker identity are preserved.
Evidence: Among outputs whose target score exceeds baseline seed variation, 54.5% to 95.8% show at least one substantial off-target deviation across eight system-descriptor settings in CosyVoice3, VoxCPM2 and Fish-Speech-S2, while content is preserved in at least 99.4% of outputs and speaker similarity stays at 0.821 to 0.902. (§4.1, §4.2, Table 1, Table 5)
- refines: The same natural-language voice descriptor maps to different combinations of acoustic changes in different TTS systems, so descriptor adherence does not correspond to a consistent intervention across models.
Evidence: For “deep”, CosyVoice3 lowers F0 while also changing rate, duration, energy, centroid and roll-off. VoxCPM2’s “bright” response is carried mainly by RMS energy rather than the target spectral measures, and non-target features with intervals excluding zero range from 1/6 to 6/6 across systems. (§4.1, Figure 2, Table 1)
- complicates: Adding natural-language conditioning without a meaningful voice target can by itself shift the acoustic properties of generated speech, so instruction-conditioned outputs need a matched neutral baseline for evaluation.
Evidence: Neutral-reading and nonsense-descriptor instructions shift 5/8 to 7/8 measured features in CosyVoice3 and VoxCPM2 (0/8 and 2/8 in Fish-Speech-S2), while an explicit lower-pitch instruction lowers mean F0 in all three systems. (§4.3, §12.2, Table 6)
- supports: Inference-time reranking of multiple sampled candidates can improve attribute preservation in instruction-conditioned TTS without retraining, but most of the gain in binary success comes from having candidates at all rather than from the ranking rule.
Evidence: With three candidates, joint success rises from 4.8% (Direct) to 14.1% (Target-only), 14.3% (VoDER-Cal) and 14.6% (Oracle). VoDER-Cal minus Target-only is +0.19 points (CI [−0.56, +0.93]), while held-out off-target deviation falls from 0.344 to 0.276, and listeners favor VoDER-Cal’s off-target preservation in 86.4% of non-tied judgments. (§5.2, §6, Table 2, Table 7, Table 10)
- complicates: Selecting the candidate with the strongest target expression tends to increase collateral change, and favoring preservation costs some perceived target strength.
Evidence: Target-only selection has higher held-out off-target deviation (0.344) than single-sample Direct generation (0.325). In H2, target-expression judgments favor Target-only over VoDER-Cal 29.6% to 20.7% with 49.6% ties, despite 99.1% automatic target-score retention. (§5.2, §6, Table 2, Table 10)
Limitations and Open Questions
- Narrow scope. Three systems, three primary descriptors, English texts and six reference speakers. The reference-speech source and the ten texts are not described in the paper itself; they are deferred to the code repository. Higher-level descriptors (warm, professional, emotional) are supplementary and may need richer perceptual definitions (§7.3).
- Signal-level proxies. Target and off-target sets are operational choices, not perceptual decompositions. Several features are correlated (rate with duration, centroid with roll-off), so shifted-feature counts measure breadth, not independent attributes. Spectral flatness and ZCR are weak proxies for roughness (§7.3, §11.1).
- Heterogeneous interfaces. Fish-Speech-S2 is driven through inline voice tags rather than free-form instructions, so cross-system differences mix model behaviour with interface differences (§3.1, §10.2).
- Small listening studies. Ten listeners, 45 samples per study, with low exact agreement on H1 five-point scales. H2 compares selectors only when both return a candidate, not the full abstaining procedure (§7.3, Table 11).
- Selector limits. VoDER-Cal relies on suitable candidates appearing in a small pool and on hand-defined feature sets. Its selector and evaluator features belong to the same correlated measurement family, so the gain in held-out deviation is not evidence of a perceptually optimal ranking. The authors argue that removing systematic coupling requires changes to training objectives or representations (§7.2, §7.3).
Wiki Connections
- Instruction-Conditioned TTS — audits how instruction- and descriptor-driven voice control in three systems changes attributes outside the request, and proposes a selector that acts on those instruction-conditioned outputs.
- Evaluation Metrics — proposes a paired, preservation-sensitive evaluation protocol (target response plus held-out off-target deviation against a matched baseline) and checks its direction with listener judgments.
- Subjective Evaluation — runs two blinded listening studies on system output: perceptibility of target and off-target changes, and selector-level preservation preference.
- CosyVoice 3 — one of the three audited systems (Fun-CosyVoice3-0.5B), with the highest deep target-direction rate (84.4%) and 5/6 to 6/6 non-target features shifting significantly per descriptor.
- VoxCPM2 — audited system, whose bright response is carried mainly by energy rather than the spectral targets.
- Fish Audio S2 — audited as Fish-Speech-S2 (
s2-pro) through its native inline voice tags. - InstructTTSEval — prompt-adherence benchmark that the paper positions its preservation-sensitive audit as complementing.
- SPAM — human-aligned style prompt adherence metric, cited as an example of evaluation that measures agreement with the prompt but not off-target stability.
- ASR self-verification for codec TTS — prior inference-time candidate filtering that VoDER-Cal extends from content reliability to attribute preservation.
- ControlSpeech — earlier system combining reference-based cloning with language-guided style control, cited as part of the controllable-TTS line the audit targets.
- ECAPA-TDNN — speaker-embedding model used for the speaker-preservation validity check.