arXiv · 2026 · Preprint
Ahmet Erdem Pamuk et al. (Independent Researcher) · → Paper · Demo: ? · Code: ✓
A 183.2M-parameter non-autoregressive flow-matching Diffusion Transformer, trained from scratch on Turkish inside the frozen continuous latent space of VoxCPM2’s AudioVAE2 and fine-tuned into a single production voice, reaches lower band-matched WER than the larger XTTS-v2 and F5-TTS on a released Turkish benchmark.
Problem
Large open TTS systems treat Turkish as one language among dozens and reach their quality with multi-billion-parameter backbones that rule out single-GPU or on-device serving. Turkish also strains hand-built text frontends: agglutinative morphology, vowel harmony, and the spoken form of acronyms make grapheme-to-phoneme (G2P) and normalization rules brittle. The authors also argue that autoregressive latent decoders, such as VoxCPM2, accumulate errors over number-dense conversational utterances, where a misread digit changes meaning.
Method
FreyaTTS keeps only the AudioVAE2 codec of VoxCPM2, frozen and never trained. Its encoder maps 16 kHz audio to 64-dimensional continuous latents at 25 Hz, and its decoder outputs a 48 kHz waveform. The autoregressive VoxCPM2 stack (text-semantic LM, FSQ bottleneck, residual acoustic LM, local diffusion head) is discarded. The authors stress that the 48 kHz rate fixes the output sample rate, not the bandwidth: the usable band is still limited by the 16 kHz encoder input and the training audio.
Text enters as raw characters from a 92-symbol Turkish vocabulary (29-letter alphabet, digits, punctuation, whitespace). There is no BPE, phonemizer, G2P step, or discrete speech tokenizer. Four ConvNeXt-1d blocks refine the character embeddings into a feature sequence. A small MLP over the masked mean of these features predicts the log total latent length, so the noise tensor can be sized before denoising. Digit strings are the one exception to end-to-end text handling: they are expanded to spoken form at the frontend, because the duration predictor under-allocates frames for compact digit strings.
The generator is a 16-layer DiT (width 640, 10 heads, SwiGLU feed-forward with hidden size 2048). Each block has RoPE self-attention over frames, cross-attention from frame queries to the character features, and a feed-forward layer, all modulated by the timestep through adaLN-zero. Cross-attention is the only path from text to acoustics. This differs from the filler-padded implicit alignment of E2 TTS and F5-TTS. Training uses a linear-path (rectified-flow) conditional flow-matching loss over masked latent frames, plus a log-duration regression term with weight 0.1. There is no adversarial, autoregressive, or token cross-entropy loss. Inference runs a plain 32-step Euler ODE solver with no classifier-free guidance or sway sampling.
Pretraining runs 150k steps (batch 64, AdamW) from random initialization on an internal multi-speaker Turkish corpus of undisclosed size. The corpus has no speaker labels and the network has no speaker conditioning, so each noise draw produces a different speaker. Two supervised fine-tuning stages follow. SFTv1, the “voice lock”, fine-tunes all parameters on one consented professional speaker recorded at 16 kHz over a narrowband channel; the authors keep step 1000 because longer training destabilizes the voice. SFTv2 continues on the same corpus plus mined one-word and two-word segments of the same speaker (from MMS forced alignment), to fix collapses on isolated acknowledgments, bare numbers, and lone acronyms. A thin inference wrapper adds clause-level chunking, a duration floor, and a voicing-based retry.
Key Results
All quality numbers come from the authors’ own Freya-TR-Eval benchmark: 495 conversational Turkish sentences drawn from Common Voice 17-tr and CoVoST2-tr texts plus Gemini-generated lines. WER and CER use Whisper-large-v3 after downsampling every system’s output to 8 kHz, so the narrowband FreyaTTS voice is not penalized against wideband baselines (§4.1, Table 2):
| System | Params | WER (%) | CER (%) | MOS |
|---|---|---|---|---|
| Piper (tr) | 16M | 4.4 | 1.1 | 3.47 ± 0.22 |
| Coqui GlowTTS (tr) | 28M | 12.1 | 3.3 | 2.53 ± 0.19 |
| MMS-TTS (tr) | 36M | 6.8 | 1.7 | 3.58 ± 0.20 |
| SpeechT5 (tr) | 144M | 83.4 | 45.5 | 1.59 ± 0.14 |
| FreyaTTS | 183.2M | 8.0 | 3.0 | 3.68 ± 0.22 |
| F5-TTS (tr) | 336M | 24.3 | 10.9 | 3.63 ± 0.23 |
| XTTS-v2 | 470M | 11.1 | 3.9 | 3.82 ± 0.19 |
- Intelligibility. FreyaTTS has lower WER and CER than the two larger systems, XTTS-v2 and the Turkish fine-tuned F5-TTS. It does not have the lowest error overall: the phonemizer-driven VITS models Piper (WER 4.4%) and MMS-TTS (6.8%) are lower. A sentence-level bootstrap on an independent re-synthesis gives WER 7.9% [6.8, 9.1]. As calibration anchors, real FLEURS-tr recordings score 9.7% WER under the same pipeline, and re-encoding the target speaker’s real recordings through AudioVAE2 alone costs 2.6% WER (§4.2).
- Naturalness. 24 native Turkish raters gave 2566 blind ratings. FreyaTTS has the second-highest MOS point estimate (3.68), behind XTTS-v2 (3.82) and the highest among the five systems at or below 183.2M parameters. The authors note that the 95% intervals of the top four systems overlap (§4.2, Table 2).
- Voice lock. Across 114 regenerations of a fixed text, the pretrained model’s F0 standard deviation is 74.9 Hz, with a 60% rate of crossing the 165 Hz male/female boundary between draws. After SFTv1 it falls to 5.0 Hz, and ECAPA-TDNN similarity to the target speaker centroid is 0.873 ± 0.029. This is close to the 0.883 ± 0.026 ceiling measured on real recordings of the same speaker (§4.3, Figure 2).
- Inference layer. Removing the whole inference wrapper changes WER by at most 0.3 points (paired-bootstrap CI [-0.3, 0.0]). The voicing retry fires on 1 of 495 sentences (§4.2).
- Efficiency. On an RTX 4090 with released library stacks, FreyaTTS has the lowest medium-bucket RTF (0.11, vs. 0.13 F5-TTS, 0.33 XTTS-v2, 0.37 VoxCPM2, 1.08 Spark-TTS) and the highest throughput (9.4 audio-s/s at 4 engines, vs. 8.4 for F5-TTS). Its VRAM use is the second smallest (1.5 GB, vs. 0.8 GB for F5-TTS) (§4.4, Table 3). F5-TTS has the lower long-bucket RTF (0.05 vs. 0.10). XTTS-v2’s native streaming has the lowest library-stack TTFT (0.30 s vs. 0.52 s). A static-batching ODE server for FreyaTTS reaches 65.2 audio-s/s at batch 8, vs. 24.8 for VoxCPM2 through the nanovllm serving engine. On an Apple M3 CPU the model runs at RTF 0.70 to 1.00 depending on length (§4.4).
The comparisons have limits. The benchmark is the authors’ own. FreyaTTS synthesizes in its trained voice, while XTTS-v2 and F5-TTS must clone one fixed shared reference clip, and the authors note that cloner scores can depend on that choice. FreyaTTS’s TTFT counts only the first clause, while offline systems report full-utterance latency. The human-speech anchor is on a different corpus.
Novelty Assessment
The contribution is engineering integration. Every component is established: a frozen pretrained continuous latent codec (from VoxCPM2), a DiT with adaLN-zero and a ConvNeXt text refiner (as in F5-TTS), cross-attention text conditioning with an explicit total-length predictor, a rectified-flow objective, and single-speaker fine-tuning. The new part is the combination. The model trains a compact NAR generator from scratch on top of a codec borrowed from an autoregressive foundation model, for one mid-resource language. The pretrain-then-voice-lock recipe and the mined short-utterance stage are practical recipes, not new methods.
The paper is a single-language technical report with in-house data of undisclosed size. Its main evaluation is on a benchmark the authors built, and one fixed reference clip conditions both cloning baselines. Its strengths are a human listening study with confidence intervals, bootstrap intervals, calibration anchors, an ablation of the inference wrapper, a broad efficiency comparison, and the release of weights, code, and benchmark.
Field Significance
Low — FreyaTTS shows that a compact NAR flow-matching model, trained from scratch inside a frozen borrowed latent space, can be competitive for one mid-resource language at a fraction of foundation-model size. It is a narrow, Turkish-only application of known components. Its reusable evidence is mainly the voice-lock measurement, which shows how a speaker-agnostic flow-matching prior becomes a stable single voice, and the released Turkish benchmark and harness.
Claims
- supports: A non-autoregressive flow-matching generator trained in a frozen, pretrained continuous latent space can reach competitive intelligibility for a mid-resource language at a fraction of the parameter count of multilingual zero-shot systems.
Evidence: The 183.2M FreyaTTS model reaches 8.0% WER and 3.0% CER on Freya-TR-Eval, versus 11.1%/3.9% for the 470M XTTS-v2 and 24.3%/10.9% for the 336M Turkish F5-TTS. Only the DiT and text encoder are trained; AudioVAE2 stays frozen. (§3.2, §4.2, Table 2)
- complicates: A learned character-level neural voice does not automatically beat compact phonemizer-driven systems on recognizer-based intelligibility; ASR error and listener naturalness can rank systems differently.
Evidence: Piper (16M, WER 4.4%) and MMS-TTS (36M, WER 6.8%) have lower WER than FreyaTTS (8.0%) but lower MOS (3.47 and 3.58 vs. 3.68). XTTS-v2 has higher WER (11.1%) but the highest MOS (3.82), and the top four MOS intervals overlap. (§4.2, Table 2)
- supports: Without speaker conditioning, a flow-matching prior trained on unlabeled multi-speaker data samples a different speaker per noise draw. Brief full-parameter fine-tuning on one speaker fixes the identity in the weights, but longer fine-tuning destabilizes it again.
Evidence: Cross-generation F0 standard deviation falls from 74.9 Hz (pretrained, 60% male/female boundary crossings) to 5.0 Hz after 1000 SFT steps, and ECAPA-TDNN similarity to the target reaches 0.873 against a 0.883 real-speech ceiling. By 3k steps the standard deviation drifts back toward 11 Hz. (§3.5, §4.3, Figure 2)
- refines: In character-level non-autoregressive TTS with a single global length prediction, digit-string and short-input failures are driven by duration allocation and training-data coverage, not by decoding order.
Evidence: The duration head under-allocates frames for compact digit strings, so digits are expanded to spoken form at the frontend. Collapses on isolated acknowledgments and bare tokens trace to a corpus with no clips of two words or fewer, and are repaired by adding mined one- and two-word segments in SFTv2. Error still rises with input length, and clause chunking only mitigates it. (§3.3, §3.5, §4.2, Figure 2)
- complicates: Comparing narrowband-trained and wideband systems with ASR-based intelligibility requires bandwidth matching and calibration anchors before absolute WER values can be read.
Evidence: All outputs are downsampled to 8 kHz before Whisper-large-v3 transcription. Under this protocol, real FLEURS-tr speech scores 9.7% WER and re-encoding target-speaker recordings through the frozen codec alone gives 2.6%. The authors caution that the corpora differ and Whisper WER is sensitive to text distribution. (§4.1, §4.2)
Limitations and Open Questions
Warning
Training data are internal and their size is not reported, and the main quality comparison uses the authors’ own benchmark. The two cloning baselines are each conditioned on one fixed shared reference clip, which the authors acknowledge can affect cloner scores.
- The shipped model has one voice: there is no speaker conditioning or prompt pathway, so it cannot clone or switch voices. Multi-voice extension is future work.
- The voice comes from a narrowband telephony-channel recording, so the 48 kHz output has a telephony-band fidelity ceiling. Bandwidth extension is proposed but not done.
- Digit strings need spoken-form expansion at the frontend, so the “rule-free” design is not fully end-to-end for numbers. Isolated acronyms and bare tokens remain a documented failure mode handled by post-training and the inference wrapper.
- Long-horizon drift persists: WER rises with input length, and clause chunking only mitigates it. The authors attribute residual word skips to the single global length prediction and are prototyping a CTC-monotonic aligner.
- Evaluation covers Turkish only. Speaker consistency is measured on in-distribution sentence-length text.
- Efficiency comparisons mix FreyaTTS in fp32 eager PyTorch with baselines’ released stacks. TTFT definitions differ across systems (first clause, first streamed chunk, or full utterance).
Wiki Connections
- Flow Matching — FreyaTTS trains a 16-layer DiT with a linear-path conditional flow-matching objective over frozen continuous latents and samples with a plain 32-step Euler solver, without guidance or sway sampling.
- Speaker Adaptation — The voice-lock stage adapts a speaker-agnostic multi-speaker prior to one consented speaker by full-parameter fine-tuning, with F0-consistency and ECAPA similarity evidence and an over-fitting point at longer training.
- Subjective Evaluation — The paper runs a blind MOS study with 24 native Turkish raters (2566 ratings, 95% confidence intervals) across seven systems and notes that the top intervals overlap.
- VoxCPM2 — FreyaTTS reuses VoxCPM2’s frozen AudioVAE2 as its latent space, replaces its autoregressive backbone, and compares against it on inference efficiency.
- VoxCPM — The continuous-latent, tokenizer-free predecessor whose autoregressive design FreyaTTS explicitly departs from.
- F5-TTS — A Turkish fine-tuned F5-TTS is a quality and efficiency baseline; FreyaTTS borrows its ConvNeXt text refiner but replaces filler-padded implicit alignment with cross-attention and a duration head.
- XTTS — XTTS-v2 is the larger multilingual zero-shot baseline, with higher MOS but higher band-matched WER than FreyaTTS.
- Spark-TTS — The autoregressive baseline in the efficiency comparison, about 10 times slower in RTF than FreyaTTS.
- E2 TTS — The filler-padded NAR flow-matching formulation that FreyaTTS positions against by using explicit cross-attention conditioning.
- ZipVoice — Cited as a compact flow-matching precedent for small, fast NAR TTS.
- Whisper — Whisper-large-v3 is the recognizer behind every band-matched WER and CER figure and the content-similarity metric.