Field Overview

Abstract

Modern speech generation is about much more than making a voice sound natural. A useful system must also get the words right, preserve the intended speaker, express the right style, respond at the right time, and behave well outside a clean test set. This overview brings together all 23 integrated concept areas to explain where those goals reinforce one another—and where they pull in different directions.

Warning

Retrospective Q3 boundary. This overview was assessed on 2026-09-12 using reviewed evidence published through 2025-09-30. It describes what that bounded evidence supports under the later review, not what researchers necessarily knew at the end of Q3 2025.

What Matters Across the Field

A good score is not the same as good speech

Speech quality has several parts. A system can pronounce the words correctly and still sound unnatural. It can sound natural while losing the target speaker’s identity. It can preserve the voice but flatten the emotion or respond awkwardly in conversation. That is why a transcript score or a single overall quality score is rarely enough. Evaluations need to measure the dimensions that matter for the task, especially any quality that a new method might put at risk (E2 TTS, Codec-LM Co-design, Full-Duplex-Bench v1.5, Neural-codec metric assessment).

Automatic evaluators still have an important role. Learned quality predictors usually follow human ratings better than older signal-distortion measures when the test data resembles their training data. But their rankings can change with the language, recording conditions, synthesis architecture, or level of analysis. A metric that works across whole systems may still be poor at judging individual utterances (Speech Quality Assessment review, Neural Audio Codec Embedding Distances).

The same caution applies to LLM and audio-language-model judges. They can often separate systems that are clearly different, but become less dependable for close comparisons, fine-grained prosody or emotion, non-verbal sounds, and unfamiliar domains. Before replacing listeners, an automatic judge should be checked against people on the same task, domain, attribute, and scoring scale (MTalk-Bench, Instruction-Perception Gap, Prosody Diversity Benchmark).

This shared lesson appears in the work on evaluation metrics, subjective evaluation, instruction-conditioned TTS, emotion synthesis, speech-to-speech systems, spoken language models, and neural codecs. The practical message is simple: measure each important quality separately, and be clear about what an automatic score has actually been validated to measure.

Speech tokens have to serve two different purposes

Language models want speech tokens that make words and sentence structure easy to predict. Audio decoders want tokens that keep the details of the voice: rhythm, pitch, timbre, and fine waveform structure. Improving tokens for one purpose often makes them less useful for the other. Tokens trained mainly for linguistic discrimination tend to support clearer, more coherent generation; tokens trained mainly for reconstruction tend to preserve richer sound detail (AudioLM, SpeechTokenizer, Codec-LM Co-design).

Many systems handle this by splitting the job. A semantic representation plans the words and longer-range structure, then acoustic tokens or a continuous generator rebuild the voice and waveform. This is the basic idea behind several hierarchical systems, from codec language models to zero-shot TTS and spoken agents (VALL-E, SoundStorm, NaturalSpeech 3).

The split helps, but it does not make the problem disappear. If the semantic stage makes an error, the acoustic stage usually cannot repair the missing meaning. If semantic supervision is too strong, the representation may throw away useful speaker or prosodic information. Long-form generation adds another challenge: both the linguistic plan and the acoustic realization must remain consistent over time (Long-Form Speech Generation with Spoken Language Models, Optimizing Speech Language Models for Acoustic Consistency).

This trade-off ties together autoregressive codec TTS, neural codecs, self-supervised speech representations, zero-shot TTS, speech-to-speech systems, and spoken language models. There is no universally best tokenizer. Frame rate, codebook structure, training objective, and decoder capacity all change where a representation sits between easy language modeling and faithful audio reconstruction.

Choosing an architecture means choosing a bottleneck

There is no single generator that wins on every requirement. Autoregressive codec models are good at step-by-step semantic planning and flexible prompting, but token-by-token decoding costs time and can accumulate alignment errors. Parallel transformer encoder–decoder systems remove much of that serial work, but then need explicit duration, pitch, and variation models to avoid skipped, repeated, or flat speech (Tacotron, FastSpeech 2).

Diffusion models are flexible and can refine an output over several steps, but those steps add latency. Flow matching often reaches useful quality with fewer steps. Even so, a headline step count is not a fair comparison unless the systems use similar data, representations, solvers, hardware, and latency definitions (DiTTo-TTS, F5-TTS).

The same choice appears at the waveform decoder. GAN vocoders remain attractive when very fast generation matters. Their quality depends heavily on discriminator design, periodic structure, and whether they were trained on the imperfect features they will see at deployment (HiFi-GAN, BigVGAN). Flow and diffusion vocoders can improve some parts of reconstruction through iterative refinement, while distillation can narrow the speed gap (PeriodWave, WaveFM).

For streaming systems, it also helps to separate time to first audio from total generation time. A model can begin speaking quickly and still have expensive sequential decoding afterward. Causal codecs, vocoders, and voice-conversion systems can approach offline quality, but only when their representations and training are adapted for limited future context (FocalCodec-Stream, Conan, Low-Latency Vocoder Analysis).

The most useful question is therefore not “Which architecture is best?” It is “Where should this system spend its sequential reasoning, parallel acoustic generation, and perceptual reconstruction budget?” The answer depends on whether the main constraint is latency, long-form stability, control, compute, or final waveform quality.

Control works because factors are partly separable

It is easier to control a system when content, speaker identity, prosody, emotion, and style do not all share one undifferentiated representation. Separate conditioning paths let a user change one attribute while trying to preserve the others. This pattern appears across TTS, voice conversion, neural codecs, and expressive generation (ControlSpeech, NaturalSpeech 3, Vevo2).

But “disentangled” does not mean perfectly independent. Accent, timing, pitch, identity, and habitual expression naturally overlap. Removing every trace of one factor can also remove information needed for another, reducing intelligibility, naturalness, or speaker similarity. Direct pitch mechanisms and targeted preservation losses often help precisely because a generic latent split does not capture every dependency (PeriodCodec, DiEmo-TTS, ZSDEVC).

Natural-language instructions make control more flexible than fixed labels, but they also make success harder to measure. A sample may sound good while ignoring the requested style, or follow the instruction while harming the voice. Fine-grained emotion work shows the same issue: a global score can miss changes within an utterance (Fine-Grained Emotional Speech Synthesis, Instruction-Perception Gap). In this overview, disentanglement means reduced interference under a tested intervention—not proof that the latent factors are statistically independent.

Zero-shot voice transfer works, but not equally well everywhere

A short reference recording can let TTS or voice-conversion systems speak in the voice of someone who was not in the training set. That is a real and well-supported capability. It does not mean that every reference works equally well. Duration, spoken content, noise, channel, language, accent, emotion, and the evaluation method can all change how much identity is preserved (VALL-E X, OpenVoice, NaturalSpeech 3).

Training diversity matters here. Across speaker adaptation, voice conversion, and zero-shot TTS, systems trained on a broader range of speakers and recording conditions tend to generalize better to unseen voices. Still, total hours alone do not explain the gain. Speaker, language, and accent coverage; filtering; transcription quality; pretraining; architecture; and model capacity usually change together.

Multilingual TTS makes these limits easy to see. Shared training and common phonetic representations can help low-resource languages, but an imbalanced corpus can pull quality toward high-resource languages. Average scores then hide pronunciation or speaker-identity problems in individual languages (Multilingual synthesis for Ojibwe, Mi’kmaq, and Maliseet, Vietnamese Zero-Shot TTS, MahaTTS).

Language and dialect also need explicit care. A common phonetic alphabet can reduce spelling-to- sound ambiguity, while language identifiers or routing can prevent a shared model from averaging distinct accents and speaking styles (OpenVoice, DiaMoE-TTS). Continued pretraining and lightweight adapters can extend coverage, but the dedicated fine-tuning evidence is still small. Its single paper found that adapters preserved an existing multilingual GAN-TTS system better than full fine-tuning during new-speaker and new-language adaptation; that result is promising, not yet architecture-independent (Unseen Speaker and Language Adaptation with Adapters).

A full-duplex model must manage the conversation

Letting a model listen while it speaks is only the first step. A useful full-duplex agent must also decide what incoming speech means. Is the user genuinely interrupting? Is it a brief overlap? Is it a listener saying “mm-hm” to encourage the speaker to continue? The agent then has to keep speaking, yield, or respond without losing the meaning of its current turn (Freeze-Omni, OmniFlatten).

Responding sooner is not always better. Systems that react quickly to any user sound are more likely to stop unnecessarily when they hear a short overlap or backchannel. Systems that protect their current response tend to react more slowly to genuine interruptions. In the reviewed evaluations, no system handles both sides strongly across interaction scenarios (Full-Duplex-Bench, FD-Bench, Full-Duplex-Bench v1.5, FLEXI).

This is the concrete meaning of the responsiveness-versus-floor-holding trade-off: the system is balancing quick interruption response against its ability to continue speaking when it still has the conversational turn. The right balance depends on the setting, interruption threshold, and errors or delays in the underlying speech model. Time to first audio, total decoding time, interruption response, backchannel behavior, and turn-taking accuracy should be measured separately.

Post-training improves what the feedback can see

Preference and reward training can improve the qualities that appear in the feedback signal. Across several speech architectures, studies report better intelligibility, speaker similarity, naturalness, emotion, or instruction following than their supervised or metric-only baselines (Preference Alignment for Zero-Shot TTS, Align-SLM, Emotion-Aligned Diffusion TTS).

The catch is that the optimizer cannot protect a quality it cannot see. A reward focused on word accuracy may flatten prosody. A preference model trained in one domain may favor the wrong cues in another. Poorly constructed preference pairs or overly long optimization can also reduce held-out quality (RLHF for Diffusion TTS, No Verifiable Reward for Prosody, Speech DPO).

Multi-objective rewards help, but the weights still express a choice about which failures matter most. Post-training also cannot reliably create a capability missing from the base model. The evidence across RLHF for speech, subjective evaluation, emotion synthesis, flow matching, diffusion TTS, autoregressive codec TTS, and spoken language models therefore supports a bounded conclusion: feedback optimization is useful, but its gains are only as broad as the signal and validation used to guide it.

Singing is a useful stress test

The singing evidence is much smaller—ten papers, with all conclusions still emerging—but it tests assumptions that are easy to miss in ordinary speech. Speech references can guide singer identity only if the system handles the mismatch between speaking and singing. Codecs need training examples that cover high pitches. Identity, style, naturalness, vibrato, and musical timing are difficult to improve together (Bridging Speech and Singing, VibE-SVC, Singing Voice Conversion Challenge 2025).

Explicit pitch or energy controls make some aspects easier to steer, and synchronized video can provide timing information when phoneme durations are unavailable. These are useful early results, not general solutions (Phoneme-Level Energy Control, PerformSinger). Singing reminds us that a tokenizer, identity metric, or adaptation recipe that works on speech may not transfer unchanged to long pitch movements and tightly fixed musical timing.

The Recurring Trade-offs

• Easy language modeling versus rich audio detail. More abstract tokens simplify word and sentence prediction but usually discard some information about voice and prosody.
• Parallel speed versus explicit structure. Removing token-by-token decoding saves time, but duration, alignment, pitch, and variation then need to be modeled elsewhere.
• Independent control versus preserved identity. Stronger factor separation makes targeted changes easier until it also removes cues that carry accent, timing, expression, or identity.
• Quick interruption response versus continuing the current turn. A more sensitive agent reacts faster to genuine barge-ins but is also more likely to stop for harmless overlap.
• Strong optimization versus broad quality. Reward or metric optimization improves what it measures while leaving other qualities exposed.
• Shared multilingual capacity versus even coverage. Joint training helps transfer, but weak languages and dialects can disappear inside an acceptable average score.

Questions Worth Following

1. Can automatic evaluators remain reliable across languages, architectures, domains, and close system comparisons—and where are human listeners still essential?
2. Which tokenizer properties predict both easy language modeling and good reconstructed speech when data, decoder, and compute are controlled?
3. How should attribute leakage be measured directly, rather than inferred from one downstream score or classifier?
4. Which parts of unseen-voice generalization come from more hours, and which come from speaker, language, accent, and recording diversity?
5. Can spoken agents handle overlap, backchannels, interruption, and long conversations without trading away semantic coherence?
6. Can preference training protect qualities that are absent from its reward and validation data?
7. Which lessons from ordinary speech survive in low-resource languages, atypical voices, and singing?

Read by Concept

Concept	Overview	In Depth
Autoregressive Codec TTS	Overview	In Depth
Diffusion Text-to-Speech	Overview	In Depth
Disentanglement	Overview	In Depth
Emotional and Expressive Speech Synthesis	Overview	In Depth
Evaluation Metrics	Overview	In Depth
Fine-Tuning Foundation Models	Concept page	Render pending
Flow Matching	Overview	In Depth
GAN Vocoders	Overview	In Depth
Instruction-Conditioned Text-to-Speech	Overview	In Depth
Multilingual Text-to-Speech	Overview	In Depth
Neural Audio Codecs	Overview	In Depth
Prosody Control	Overview	In Depth
RLHF for Speech	Overview	In Depth
Self-Supervised Speech Representations	Overview	In Depth
Singing Voice Synthesis and Conversion	Concept page	Render pending
Speaker Adaptation	Overview	In Depth
Speech-to-Speech Systems	Overview	In Depth
Spoken Language Models	Overview	In Depth
Streaming Text-to-Speech	Overview	In Depth
Subjective Evaluation	Overview	In Depth
Transformer Encoder–Decoder TTS	Overview	In Depth
Voice Conversion	Overview	In Depth
Zero-Shot Text-to-Speech	Overview	In Depth

Scope and Traceability

The 23 concept graphs contain 2,226 paper memberships but 497 unique papers. The difference—1,729 memberships—comes from papers that inform more than one concept. The graphs contain 406 local claim clusters: 280 strongly supported, 112 emerging, and 14 contested. This overview presents the 11 human-approved broader claims once and uses the 49 approved direct relationships to interpret narrower connections. The 15 preserved agent proposals are not used as evidence.

Evidence is limited by canonical publication dates through 2025-09-30. The retrospective assessment was completed on 2026-09-12 and is recorded in the published 2025-Q3 snapshot. This overview remains a replaceable current-state rendering of the living concept graphs and reviewed registry; the snapshot is the authority for the separate historical quarterly report.

Every conclusion traces from a human-approved broader claim or a local claim cluster to paper-level evidence, then to the paper page and cited source section. Citations here are representative rather than exhaustive. Complete provenance remains in wiki/_claims/*.yaml and wiki/_claims/_reconciliation/registry.yaml.