TTS-1 Technical Report

arXiv · 2025 · Preprint

Atamanenko et al. (Inworld AI) · → Paper · Demo: ✓ · Code: ✓

Inworld TTS-1 and TTS-1-Max are autoregressive LLaMA-backed TTS models of 1.6B and 8.8B parameters trained with a three-stage pipeline (pre-training, SFT, GRPO RL alignment) that generates 48kHz speech with in-context voice cloning, 11-language support, and fine-grained emotion control via text-embedded audio markup tags.

Problem

Large-scale autoregressive TTS systems built on LLM backbones frequently fall short in production settings: they lack high-fidelity output (most stop at 16 or 24 kHz), exhibit inconsistent volume in streaming concatenation, suffer from hallucination artifacts in long generations, and provide coarse or absent mechanisms for style and emotion control that are independent of the reference audio. Existing models also rarely demonstrate a complete training methodology covering RL alignment, which is becoming central to LLM quality but has seen limited systematic adoption in TTS.

Method

TTS-1 and TTS-1-Max share the same three-component architecture: an audio encoder, a SpeechLM, and an audio decoder. The SpeechLM backbone is LLaMA-3.2-1B for TTS-1 and LLaMA-3.1-8B for TTS-1-Max, with vocabulary expanded from 128,256 to 193,856 tokens to accommodate 65,536 audio tokens. At inference, a reference audio clip is tokenized by the encoder, concatenated with the reference transcript and target text, and fed to the SpeechLM as a prompt. The SpeechLM autoregressively generates audio tokens, which the decoder converts to a 48kHz waveform.

The audio codec builds on the X-codec2 architecture with a single 65,536-token codebook operating at 50 tokens per second. A super-resolution module consisting of interleaved strided 1D transposed convolutions and ResNet blocks is added to the decoder to lift output from 16kHz to 48kHz. An RMS loudness loss term is added to the decoder training objective to prevent volume inconsistencies between reconstructed audio segments, which are particularly audible during streaming concatenation.

Training proceeds in three stages. Pre-training exposes the SpeechLM to roughly 1 million hours of raw multilingual audio mixed with approximately 10% text tokens from RedPajama-v2 and instruction data from LAION OIG. Supervised fine-tuning (SFT) then trains on approximately 200,000 hours of high-quality transcribed audio filtered by DNSMOS score and speaking rate, using the final pre-training learning rate as initialisation. RL alignment applies Group Relative Policy Optimization (GRPO) with a composite reward combining WER (evaluated via Whisper-large-v3), speaker similarity (WavLM-large speaker verification embeddings), and DNSMOS perceptual quality. Each reward is normalised to [0, 1] and combined with equal weights, with reward components conditionally activated only on samples containing the relevant annotations.

Emotion and style control is achieved through audio markup tags embedded in the input text (8 speaking styles, 7 non-verbal vocalizations). Direct tag prepending during SFT failed because the codec’s single-codebook design entangles acoustic and semantic information, preventing the model from isolating style from speaker identity. The effective solution pairs neutral and stylized utterances from the same speaker and fine-tunes the SpeechLM with LoRA on these 180-hour, 340-speaker paired examples.

Streaming inference uses two mechanisms to prevent audible discontinuities: concatenation is restricted to non-voicing regions (silence gaps between words), and the audio decoder receives an extended context window to stabilise volume across adjacent segments. These additions are reported to have negligible latency overhead.

Key Results

On an internal English evaluation spanning short (avg 7 chars), medium (avg 87 chars), and long (avg 153 chars) inputs, TTS-1-Max achieves 5.1% overall WER and TTS-1 achieves 6.3% WER. RL alignment consistently reduces WER over the SFT-only checkpoints: TTS-1 SFT scores 7.9% and TTS-1-Max SFT scores 6.9% on the same set (Table 8). The 48kHz decoder achieves DNSMOS 4.195 versus 4.178 (24kHz) and 4.110 (16kHz) on the Seed-TTS English eval (Table 2).

In an internal TTS arena with approximately 400 blind preference votes across 20 annotators, TTS-1-Max beats 11LABS Multilingual V2 (59.1%), Cartesia Sonic 2 (60.9%), and OpenAI TTS-1-HD (60.7%) in head-to-head win rates (Table 9). These comparisons use built-in speaker voices for all systems and are limited to English, with a relatively small pool of annotators per pair.

Speaker similarity comparisons show that decoding the generated audio with the prompt audio tokens in context raises SIM from 0.495 to 0.535 on the Seed-TTS English eval (Table 10).

Novelty Assessment

The primary novelty lies in two areas: (1) adapting GRPO with a composite, conditionally-activated reward function for TTS, and (2) extending X-codec2 to 48kHz via a super-resolution decoder module with an RMS loudness training term. The three-stage training methodology (pre-training → SFT → RL) is consistent with what Seed-TTS and CosyVoice have demonstrated, but TTS-1 provides a detailed account of GRPO formulation, reward design, and training infrastructure. The LoRA-based paired neutral/stylized data strategy for audio markup training is a practical engineering contribution to emotion control in codec TTS, though it builds on established LoRA fine-tuning and implicit conditioning ideas from the field. The overall system is an engineering integration of LLaMA backbones, codec tokenization, and GRPO alignment rather than a structural departure from current practice. Model weights are not released, which limits reproducibility and independent benchmarking.

Field Significance

Moderate — TTS-1 provides a thorough account of scaling autoregressive codec TTS to a production system at 8.8B parameters with full RL alignment and 48kHz output, contributing a useful reference for the GRPO reward design and streaming inference engineering. Its primary value is demonstrating that the pre-training → SFT → GRPO pipeline transfers cleanly to TTS at this scale, and providing concrete ablation evidence that pre-training and RL alignment each add measurable WER and speaker similarity gains over SFT alone.

Claims

• supports: RL alignment with composite perceptual rewards improves multiple speech quality dimensions simultaneously over SFT-only baselines in autoregressive codec TTS.

  Evidence: GRPO with combined WER + speaker similarity + DNSMOS rewards achieves gains in all three metrics over the SFT baseline, while single-reward models improve only their target dimension; TTS-1 WER drops from 7.9% (SFT) to 6.3% (RL-aligned). (§3.5, Table 8)

• supports: Audio pre-training on large-scale raw speech substantially improves subsequent SFT quality in LLM-based TTS beyond what instruction-tuned LLM initialisation provides.

  Evidence: Initialising SFT from an audio pre-trained LLaMA-3.2-1B checkpoint yields lower SFT loss, approximately 15% lower WER, and approximately 3% higher speaker similarity than initialising from the base LLaMA-3.2-1B-Instruct checkpoint. (§3.4, Figure 5)

• supports: Scaling SpeechLM parameter count in autoregressive codec TTS consistently improves intelligibility and speaker fidelity across languages.

  Evidence: TTS-1-Max (8.8B) achieves 5.1% overall WER and higher SIM across all 11 evaluated languages compared to TTS-1 (1.6B) at 6.3% WER, with performance correlating with pre-training loss differences. (§4, Figure 8, Table 8)

• complicates: Style conditioning via discrete text tags conflicts with speaker identity preservation in single-codebook codec TTS architectures.

  Evidence: Direct prepending of style markup tags during SFT produced no effect on synthesized speech because the single-codebook design entangles acoustic and semantic information; successful style control required constructing paired neutral/stylized utterances from the same speaker and applying LoRA fine-tuning. (§3.6)

• complicates: Streaming audio delivery in autoregressive TTS introduces audible artifacts and volume inconsistencies at segment boundaries that require specific engineering mitigations independent of the generative model’s quality.

  Evidence: Without concatenation restricted to non-voicing regions and context-aware decoder decoding with extended audio prompt context, segment boundaries introduce clicks and volume drops; these are engineering-layer problems independent of SpeechLM quality. (§5.1)

Limitations and Open Questions

Warning

Model weights are not publicly released, making independent benchmarking and replication impossible. All evaluations use proprietary or internal test sets; the internal TTS arena covers only English and uses approximately 20 annotators with a modest vote count per pair.

The training data is drawn from a proprietary mixture of public and licensed sources whose exact composition is not disclosed, limiting reproducibility. The evaluation framework does not include standard public TTS benchmarks (e.g., LibriTTS or VCTK test sets), making direct numerical comparison with published systems that do report on these benchmarks difficult.

The paper notes that speaker similarity metrics fluctuate with emotionally expressive speech, suggesting that current automated evaluation protocols may not fully capture perceptual quality in dynamic scenarios. The audio markup system, while effective for English, shows “reduced fidelity” when generalizing style control to non-English languages. Prompt audio caching can cause prosodic bleed from the reference audio into generated speech, and longer sequences generated from short prompts may degrade in quality.

Wiki Connections

• Autoregressive Codec TTS — TTS-1 and TTS-1-Max are large-scale examples of this paradigm, using LLaMA backbones and a single-codebook codec, contributing GRPO RL alignment and 48kHz output as extensions to the established autoregressive codec recipe.
• Zero-Shot TTS — Both models perform voice cloning purely through in-context learning from a short reference audio clip, without speaker embedding models or speaker-ID-based conditioning.
• RLHF Speech — The paper provides a detailed treatment of GRPO adaptation for TTS with a composite reward function combining WER, speaker similarity, and DNSMOS, including conditional reward activation for emotion and non-verbal markup samples.
• Multilingual TTS — The system supports 11 languages with joint pre-training and SFT, with per-language WER and SIM evaluation, and emergent style generalization to non-English languages via audio markups.
• Emotion Synthesis — The audio markup system introduces 8 speaking styles and 7 non-verbal vocalizations via text-embedded tags, with a novel LoRA-based paired-utterance training strategy to disentangle style from speaker identity.
• Neural Audio Codec — TTS-1 introduces a custom codec extending X-codec2 to 48kHz via a super-resolution decoder module with an RMS loudness loss, enabling high-fidelity streaming synthesis.
• VALL-E (VALL-E) — TTS-1 builds on the neural codec language model paradigm VALL-E established, extending it with RL alignment and higher-resolution audio.
• Seed-TTS (Seed-TTS) — The paper adopts Seed-TTS’s evaluation set for codec and decoder quality assessment, and cites its RL alignment work as a motivating precedent for GRPO application to TTS.
• CosyVoice (CosyVoice) — TTS-1 notes CosyVoice’s use of a speaker embedding model as an alternative approach to style/speaker disentanglement that TTS-1 avoids but acknowledges has complementary advantages.
• CosyVoice2 (CosyVoice 2) — Cited alongside CosyVoice as a reference for the scalable multilingual autoregressive codec TTS design space within which TTS-1 positions itself.