arXiv · 2025 · Preprint (accepted at SSW13)
Kanagawa et al. (NTT) · → Paper · Demo: ✓ · Code: ?
A TTS method with iterative multi-step interaction that allows users to progressively refine synthesized speech style through natural language directions, emulating the voice-director/voice-actor feedback loop of professional recording sessions.
Problem
Expressive TTS systems can generate diverse speaking styles via text prompts or speech prompts at initial synthesis time, but they provide no mechanism for users to give feedback and iteratively improve a generated output. In professional voice recording, directors routinely refine performances across multiple takes, providing nuanced natural language directions. Prior work on text-prompted TTS (e.g., PromptTTS) supports only simple descriptions of speaking rate or pitch at single inference, and cannot incorporate feedback conditioned on previously generated speech. This gap means synthesized speech that deviates from the intended style cannot be corrected through natural interaction.
Method
The system is built around a FastSpeech2 backbone conditioned on a speech embedding derived from a reference audio clip. The speech encoder uses a frozen HuBERT BASE model (pre-trained on ReazonSpeech) to extract frame-level representations, which are aggregated by a bidirectional LSTM, attention mechanism, and weighted sum into a fixed-length vector. A style token layer post-processes this vector to produce the final speech embedding, which conditions the acoustic model. The output mel-spectrogram is converted to waveform by HiFi-GAN V1.
The key new component is a style refiner that maps a pre-refined speech embedding to a refined embedding using the text content of a director’s instruction. The refiner consists of: (1) cross-attention between the LSTM-intermediate speech representations and the text direction tokens; (2) an aggregation module that collapses the cross-attended representations to a fixed-dimensional vector; (3) a FiLM layer that applies feature-wise linear modulation to the original embedding conditioned on the aggregated representation. A fine-tuned Gemma2 LLM processes the text direction and provides the key/value inputs to the cross-attention. The refiner is trained separately from the backbone using paired (pre-refined embedding, direction text, refined embedding) triples extracted from the interactive dataset, with an L1 loss.
At inference time, each interaction cycle takes the previous synthesis output as the new speech prompt and applies the style refiner using the next direction, feeding the updated embedding back into the backbone. This allows multiple rounds of iterative correction.
Training proceeds in four stages: (1) backbone pre-training on 8K hours of in-house Japanese data for 600K steps; (2) GAN fine-tuning for 200K steps to improve naturalness; (3) fine-tuning on the combined interactive and non-interactive datasets for 30K steps; (4) style refiner training on the interactive dataset for 10K steps. Different LLMs are used for direction generation (ChatGPT-4o), data augmentation (Llama-3.1-Swallow-8B-Instruct v0.2), and the refiner itself (Gemma2) to reduce bias from any single model’s style.
A new interactive dataset was purpose-built for this task: 13.7 hours, 1,606 direction cycles, two voice actors, 52 speaking styles following an industry guideline for TTS style classification. Directions were LLM-generated rather than produced on the spot by human directors, and were augmented with four paraphrase templates (Simple, Easy, Hard, Medium Style) yielding seven variants per original direction.
Key Results
Subjective evaluation across three experiments measured style refinement quality and naturalness:
In the iterative style refinement evaluation (5-point MOS, 82 participants), Iterative (ours) significantly outperformed the Identical baseline across all three direction cycles, confirming that the style refiner introduces real changes aligned with directions rather than no-op modifications. Iterative (ours) slightly but significantly underperformed the Actor-Guided oracle (actual voice actor recordings) in overall results and in most style groups (Fear, Anger, Sadness), with no significant gap in Joy and Surprise.
In the style refinement accuracy evaluation (7-point MOS, -3 to 3, 110 participants), the Matched condition (direction seen by listener matches the one used for refinement) significantly outperformed the Random (Dissimilar) condition across all style groups. Random (Similar) scored comparably to Matched, showing that semantic proximity of direction text, rather than exact wording, governs perceived alignment.
Naturalness MOS (5-point scale, 168 participants) showed no significant difference between Iterative (ours), Single-shot (ours), and Actor-Guided (oracle), indicating that iterative multi-step refinement does not accumulate naturalness degradation.
Specific MOS values are reported only in figures (not extractable from the paper text).
Novelty Assessment
The core novelty is the task framing: modeling TTS style refinement as a multi-turn voice-director/voice-actor interaction rather than a single-shot conditioning problem. No prior TTS work addresses this feedback loop. The interactive dataset (with LLM-generated multi-step directions covering both paralinguistic and linguistic modifications) is a practical contribution that makes this task trainable at scale.
The technical components, however, are entirely standard: FastSpeech2 acoustic model, HuBERT speech encoder, style token layer, cross-attention, FiLM conditioning, and fine-tuned LLM are all established techniques applied to this new interaction setting. The contribution is engineering integration plus a new training data paradigm, not architectural innovation. The evaluation is crowdsourced subjective scoring with custom MOS scales, which is reasonable for a new task but limits comparison to prior work.
Scope is narrow: Japanese only, two voice actors, proprietary data, speaker-dependent model. The authors acknowledge these limitations and position this as a first step.
Field Significance
Moderate — This paper introduces a new interaction paradigm for expressive TTS that mirrors professional recording workflows, filling a gap that prior text-prompted TTS methods leave open. Its contribution is primarily conceptual (new task framing) and data-driven (purpose-built interactive dataset) rather than architectural. The evaluation demonstrates that iterative LLM-guided style refinement is feasible without naturalness loss, providing an initial proof of concept. Generalization beyond Japanese, proprietary data, and two speakers remains undemonstrated.
Claims
-
supports: Iterative natural language feedback can progressively refine the speaking style of synthesized speech without accumulating naturalness degradation.
Evidence: Over three interaction sessions, Iterative (ours) significantly outperformed the Identical baseline on a 5-point style refinement MOS, and naturalness MOS showed no significant difference between Iterative and the oracle condition across all style groups. (§4.2, §4.4, Figure 4, Figure 6)
-
complicates: Global speech embedding-based conditioning cannot accurately reflect fine-grained positional or linguistic instructions in expressive TTS.
Evidence: Low-scoring examples in the test set contained directions targeting specific word positions (“at the beginning”, “at the end”, “for the part of…”) or linguistic modifications (“hold your breath”, “place just a slight pause between words”), which the speech embedding manipulation approach could not handle. (§5.1, §5.2, Table 4)
-
supports: Semantic similarity of style direction text, not exact wording, governs how well listeners perceive style refinement as aligned with the instruction.
Evidence: In the style refinement accuracy evaluation, directions semantically similar to the one used for refinement (Random Similar) yielded scores comparable to the Matched condition, while semantically dissimilar directions scored significantly lower across all style groups. (§4.3, Figure 5)
-
complicates: Holistic subjective evaluation scales may not adequately capture fine-grained stylistic alignment in iterative TTS refinement tasks.
Evidence: Even the Actor-Guided oracle condition scored around 3 out of 5 on the iterative style refinement MOS, which the authors attribute to the evaluation task not fully discriminating subtle style differences; similar evaluation difficulties have been noted in text-to-image/video generation research. (§4.2, §5.3)
Limitations and Open Questions
Warning
All training and evaluation data is proprietary in-house Japanese speech from two voice actors. No public dataset is used, and no results are reported outside this setup. Reproducibility and generalization are untested.
The style refiner is speaker-dependent; the authors plan to extend to speaker-independent operation as future work. The current model refines only paralinguistic information (speaking style via global embeddings), not linguistic content, so instructions involving pauses, stress, or pitch accent placement cannot be followed. The directions cover only two of four practical categories from actual recording sessions (paralinguistic and text-expressible linguistic instructions), omitting demonstrative and gestural instructions entirely. The evaluation task design is also noted as an open problem: the relatively low absolute scores even under oracle conditions suggest that existing MOS paradigms do not cleanly measure this type of fine-grained iterative alignment, and more sensitive evaluation methods are needed.
Wiki Connections
- Instruction-Conditioned TTS — this paper demonstrates iterative instruction conditioning, extending the paradigm beyond single-shot text-prompted TTS to multi-turn feedback loops.
- Emotion Synthesis — the system is evaluated across 52 Japanese speaking styles grouped into five emotional categories (fear, joy, anger, sadness, surprise), providing evidence on which emotional style clusters are more and less amenable to iterative refinement.
- Self-Supervised Speech — HuBERT BASE is used as the core speech encoder in both the backbone TTS model and the style refiner, making SSL representations central to the style embedding architecture.
- Subjective Evaluation — the paper introduces custom MOS protocols for a new task type (iterative style alignment), including a 7-point direction-alignment scale, and highlights the inadequacy of existing evaluation methods for fine-grained iterative TTS assessment.
- Speaker Adaptation — the current model is speaker-dependent, fine-tuned per voice actor; extending to speaker-independent operation is identified as the primary future work direction.