arXiv · 2026 · Preprint

Ho-Lam Chung et al. (National Taiwan University) · → Paper · Demo: ✗ · Code: ✗

Replaces the text-semantic frontend of a pretrained VoxCPM2 TTS with a Traditional-Chinese language model built on a Taiwan-context tokenizer, and shows in a controlled comparison that this frontend swap, not voice fine-tuning alone, lowers CER on Taiwanese Mandarin sentences with embedded English.

Problem

Off-the-shelf TTS systems handle Taiwanese Mandarin poorly. Their accent drifts toward other Mandarin variants, their text tokenizers fragment common Taiwan terms into many tokens, and their pronunciation breaks down at intra-sentential Mandarin-English switch points (e.g. an English word such as “Transformer” or an acronym such as “LLM” inside a Mandarin sentence). The authors locate all three failures on the text side: the tokenizer and the text-semantic language model that plans prosody were never adapted to Taiwan-context text. Prior code-switching TTS work retrains the acoustic model for the language pair rather than adapting the text frontend.

Method

The system is built bottom-up in three parts.

PangolinTokenizer is a byte-level BPE tokenizer (114,822 entries) whose merges are learned on a stratified Taiwan-context corpus of about 20 billion tokens covering formal Traditional Chinese, Taiwanese Mandarin speech transcripts, named entities, romanized Taigi and Hakka, Bopomofo, code-switching technical text and ASR transcripts. A custom streaming, deterministic BPE trainer keeps memory bounded. The byte-level base guarantees lossless round-trip (100% on every PangolinBench subset).

Barbet is a 1.09B-parameter decoder-only language model trained on PangolinTokenizer: 28 layers, hidden size 1536, grouped-query attention (16 query / 2 key-value heads), RoPE, tied embeddings, and a four-layer cycle that interleaves global attention, sliding-window attention (8,192 tokens) and a Mamba state-space mixer. It trains in three stages over about 150B tokens (general multilingual and code mix; mid-training with more Traditional-Chinese and Taiwan text; long-context extension to 262,144 tokens). It is a base model, not instruction-tuned, because it serves as the text-semantic frontend that plans “what to say and how to say it”.

BlueMagpie-TTS attaches Barbet to the reused VoxCPM2 acoustic stack, which renders the frontend’s plan into 48 kHz audio with a diffusion-based residual acoustic model. A bridge module (RMSNorm, a linear projection, and one residual SwiGLU block with a zero-initialized output projection) maps Barbet’s 1536-dimensional hidden states into the input space the acoustic stack expects; it is the only component trained from scratch. Training has two stages: (1) with Barbet and the acoustic stack frozen, the bridge is trained by hidden-space distillation to match the hidden states the original VoxCPM2 frontend produces for the same text; (2) Barbet, bridge and acoustic stack are jointly fine-tuned on target-voice speech (one consenting author’s voice; amount not reported). At inference the guidance strength is raised from 2.0 to 2.8 (to help commit to English tokens at switch points), diffusion steps are set to 9 instead of 10, and a bad-case retry re-samples detected anomalous outputs; the same settings are applied to every compared system. The interface also exposes voice cloning, speaker-vector control and streaming, but these modes are not evaluated.

Key Results

Intelligibility (Table IX). CER is computed by a synthesize-then-recognize loop with the Taiwanese-Mandarin code-switching ASR Breeze-ASR-25 on an internal 1,000-sentence Taiwan-localized test set rich in English words, acronyms, numbers and proper nouns.

SystemCER (%)
VoxCPM2, zero-shot with reference speech11.45
VoxCPM2, acoustic stack fine-tuned to target voice6.43
BlueMagpie-TTS4.81

BlueMagpie-TTS has the lowest CER of the three systems: 58.0% relative reduction from the zero-shot base and 25.2% from the voice-fine-tuned base. The fine-tuned baseline shares the acoustic stack, the target-voice data and the fine-tuning hyperparameters, so the 6.43 to 4.81 gap isolates the frontend (tokenizer, Barbet and bridge together). The remaining failures concentrate on short acronyms: “LLM” is articulated unclearly and transcribed as “LOL and” (§VII-E, Table VIII). The paper reports no WER, speaker similarity or MOS.

Human preference (Table X). Ten native Taiwanese Mandarin listeners compared BlueMagpie-TTS against the zero-shot VoxCPM2 base on 500 sentences, three ratings per sentence, majority vote. BlueMagpie-TTS is preferred on 328 sentences (65.6%), VoxCPM2 on 85 (17.0%), and 87 (17.4%) show no clear preference; among 413 decisive sentences BlueMagpie-TTS wins 79.4% (two-sided sign test, p < 10^-30). The listening study does not include the voice-fine-tuned baseline.

Tokenizer (Tables I-IV). Among eight tokenizers on PangolinBench, PangolinTokenizer has the lowest overall rate (0.485 tokens/character; TAIDE 0.486, the others 0.526 to 0.610) with the smallest vocabulary (114,822; the next smallest is the VoxCPM2 tokenizer at 122,753), and the highest vocabulary-efficiency score (1.795 vs. 1.408 for LLaMA 3.1). It does not win every subset: TAIDE (318K vocabulary) is lower on formal Traditional Chinese, colloquial Taiwanese Mandarin, named entities and the Taiwan lexicon weighted cost (1.70 vs. 2.48), and on the code-switching subset PangolinTokenizer has the highest rate of all eight (0.478; TAIDE and Gemma 3/4 reach 0.380). Against the replaced VoxCPM2 tokenizer it uses 30.0% fewer tokens on 1,000 colloquial sentences (19,082 vs. 27,245; Table IV).

Frontend LM (Table VI). On TAIDE-14, Barbet has the lowest bits per byte of four ~1B models (0.7488 vs. 0.7577 for MiniCPM5-1B, 0.9190 for LLaMA-3.2-1B and 2.0082 for the instruction-tuned LFM2.5-1.2B) and the lowest tokens per byte (0.2164).

Speed. Median inverse real-time factor is 4.75 (maximum 5.29) on unspecified hardware (§VII-E).

Novelty Assessment

The contribution is primarily engineering integration. Byte-level BPE, a hybrid attention/Mamba decoder, a projection bridge trained by hidden-state distillation, and joint fine-tuning are all established techniques; the new element is assembling a region-specific tokenizer and frontend LM and grafting them onto a strong pretrained acoustic stack. The most useful piece of evidence is the controlled comparison against a voice-fine-tuned baseline, which separates frontend effects from voice adaptation. That separation is only partial: the tokenizer, LM and bridge change together (the paper acknowledges that tokenizer and frontend contributions are not separated), and the comparison is measured with a single ASR on an internal test set. The tokenizer results are text-side and largely orthogonal to speech quality; notably, PangolinTokenizer is not the most efficient tokenizer on the code-switching subset itself.

Field Significance

Low — a narrow localization study for Taiwanese Mandarin with embedded English, evaluated on an internal test set, with no public release stated in the paper. Its transferable result is that replacing only the text-semantic frontend of a frozen-then-jointly-tuned pretrained acoustic stack can reduce code-switching intelligibility errors beyond what voice fine-tuning achieves, which supports treating the frontend LM as a swappable component for regional or low-resource adaptation.

Claims

  • supports: In decoupled semantic-acoustic TTS, adapting the text-semantic frontend to the target language variety improves intelligibility beyond what fine-tuning the acoustic model to the target voice achieves.

    Evidence: With the same VoxCPM2 acoustic stack, target-voice data and fine-tuning hyperparameters, swapping in the Taiwan-context Barbet frontend through a bridge lowers CER from 6.43% to 4.81% (25.2% relative); voice fine-tuning alone lowers it from 11.45% to 6.43%. (§VII-C, Table IX)

  • supports: A pretrained acoustic stack can be reused under a new, separately pretrained text language model by training a small bridge with hidden-state distillation before joint fine-tuning.

    Evidence: The bridge (RMSNorm, linear projection, zero-initialized residual SwiGLU) is the only module trained from scratch; it is first distilled to match the original VoxCPM2 frontend’s hidden states with both pretrained parts frozen, then the assembled model is jointly fine-tuned. (§VI-A)

  • refines: In code-switching TTS, short acronyms embedded in a matrix language remain the hardest spans, while longer English words and frequent proper nouns are rendered reliably.

    Evidence: The main residual error is acronym articulation (“LLM” transcribed as “LOL and”); “OpenAI” and “Transformer” are recovered cleanly, and numbers and units fall in between. (§VII-E, Table VIII)

  • complicates: Text-tokenizer compression gains do not transfer uniformly to code-switched text.

    Evidence: PangolinTokenizer has the lowest overall tokens/character (0.485) but the highest rate of eight tokenizers on the code-switching subset (0.478 vs. 0.380 to 0.457), even though code-switching is the system’s target use case. (§IV-E, Table I)

  • supports: Pairwise preference tests with native listeners can confirm ASR-loop intelligibility gains for accent and code-switching quality.

    Evidence: On 500 test sentences rated by ten native Taiwanese Mandarin listeners, BlueMagpie-TTS is preferred on 65.6% of sentences against 17.0% for zero-shot VoxCPM2, matching the CER ordering; the study does not include the voice-fine-tuned baseline. (§VII-D, Table X)

Limitations and Open Questions

Warning

All speech results come from one internal, unreleased 1,000-sentence test set scored by a single ASR (Breeze-ASR-25), with one target voice and no MOS, WER or speaker-similarity metric. The listening study compares only against the zero-shot base, so the human preference result does not separate the frontend effect from voice fine-tuning.

  • The tokenizer, Barbet and bridge change together, so their individual contributions to the CER gain are not separated (acknowledged in §VIII).
  • The amount of target-voice fine-tuning data and the full system parameter count are not reported, and no code, weights or demo link is given in the paper.
  • The bad-case retry is enabled for all systems and is a source of run-to-run variation; CER is reported from a single run with no confidence interval.
  • Test sentences are short (one or two clauses), so Barbet’s 262K-token context and long-form synthesis are untested.
  • Voice cloning, speaker-vector control and streaming are exposed by the interface but not evaluated.
  • Acronyms spelled letter by letter remain a failure mode; the authors propose more acronym-rich code-switching frontend data rather than acoustic changes.

Wiki Connections

  • Multilingual TTS — adapts a TTS frontend to Taiwanese Mandarin with embedded English and evaluates code-switched synthesis at the switch points.
  • Flow Matching — reuses and jointly fine-tunes VoxCPM2’s residual acoustic stack, a diffusion transformer trained with a flow-matching objective, and tunes its guidance strength and sampling steps for code-switching.
  • Subjective Evaluation — reports a pairwise preference test with ten native listeners and a sign test on majority votes.
  • VoxCPM2 — supplies the acoustic stack, the zero-shot and fine-tuned baselines, and the replaced tokenizer and frontend.
  • VoxCPM — introduces the tokenizer-free text-semantic / residual-acoustic split that makes the frontend replaceable.
  • Llama 3 — its tokenizer (LLaMA 3.1) is a PangolinBench baseline with a vocabulary size similar to PangolinTokenizer.
  • Qwen2.5 — its tokenizer (shared with Qwen3) is a PangolinBench baseline.
  • Gemma 3 — its tokenizer (shared with Gemma 4) is a PangolinBench baseline and the strongest with TAIDE on the code-switching subset.