SwissText · 2026 · Workshop
Kakooee et al. (University of Applied Sciences and Arts Northwestern Switzerland) · → Paper · Demo: ? · Code: ?
A controlled comparison of three High German to Swiss German speech pipelines (direct, via Swiss German text, via audio-derived phoneme strings) on two fine-tuned backbones, finding that listeners prefer the Swiss German text intermediate while transcript-overlap metrics prefer direct synthesis.
Problem
Swiss German is a spoken dialect continuum with no standardized orthography, so text labels are inconsistent and paired data is scarce. In deployed Swiss applications users usually type High German, so a TTS system has to map High German text to Swiss German speech. The open design question is which intermediate representation to use: none at all, written Swiss German (whose spelling variability may confuse the synthesizer), or phonemes (which could remove orthographic ambiguity but need a phoneme supervision source that SwissDial does not provide).
Method
All three pipelines start from High German text and produce Swiss German speech, and all are trained on SwissDial, a parallel multi-dialect corpus with Swiss German audio aligned to Swiss German and High German text (eight dialects):
- DE-TTS synthesizes Swiss German speech directly from High German text.
- CH-TTS first translates High German to written Swiss German with a fine-tuned T5 model, then synthesizes from that text.
- PH-TTS maps High German directly to fused Swiss German phoneme strings with a second fine-tuned T5 model, then synthesizes from the phonemes.
During TTS training the translators are not used, since SwissDial already supplies the aligned text; they are only needed at inference.
Phoneme supervision for PH-TTS is built from the audio. A wav2vec 2.0-based phoneme recognizer (Wav2Vec2Phoneme) transcribes the recordings into discrete, whitespace-separated phoneme sequences, which are then fused into compact word-like units (“continuous” phoneme strings). Three fusion strategies are compared on a 100-example manually fused gold set, scoring space insertion as a binary decision between consecutive phonemes: LLM-prompted fusion guided by the Swiss German reference text, alignment of eSpeak-NG text phonemes to the audio phonemes, and dual CTC-segmentation alignment. The LLM strategy is selected and used to produce the PH-TTS training targets. The same gold set was also used to tune each strategy’s parameters and prompts.
Two synthesis backbones are fine-tuned on each branch. SpeechT5 (a unified encoder-decoder pre-trained across speech and text) needs tokenizer workarounds: unsupported phoneme symbols are mapped to rarely used spare tokens, and German umlauts and the Eszett are deterministically remapped (for example ä to æ) for all text-conditioned components. Orpheus, an open-source Llama-based speech LLM TTS system, uses its original tokenizer unchanged; the paper does not describe its audio tokenizer, model size, or training hyperparameters. Orpheus is trained on the full SwissDial split used for SpeechT5 and additionally on half of it.
Evaluation uses a closed-loop protocol (reference text to TTS to audio to STT to transcript): an internal Whisper model fine-tuned on Swiss German transcribes the synthesized Swiss German speech into High German, scored with WER and SacreBLEU against the High German reference. Human MOS (5-point naturalness/quality) compares original recordings with each branch, with 6 raters for SpeechT5 and 4 for Orpheus.
Key Results
Fusion strategy (Table 1). LLM-based fusion reaches 94.9 precision, 93.8 recall and 98.1 accuracy on boundary placement, against 30.6 / 30.0 / 76.3 for eSpeak phoneme alignment and 10.7 / 10.6 / 69.4 for dual wav2vec2 alignment; it is the highest of the three on all three measures.
Closed-loop objective metrics. For SpeechT5 (Table 2, 316 samples), DE-TTS has the lowest WER (0.214) and highest SacreBLEU (0.657) of all rows, including the original recordings (0.235 / 0.607). CH-TTS matches the original recordings almost exactly (0.237 / 0.607), and PH-TTS is clearly worse (0.355 / 0.471). For Orpheus (Table 3, 312 samples), DE-TTS has the lowest synthesized WER (0.254) and highest synthesized SacreBLEU (0.603) with full data, ahead of CH-TTS (0.284 / 0.556); with half data the order flips, and CH-TTS has the lowest synthesized WER (0.285 vs. 0.316 for DE-TTS) and the highest synthesized SacreBLEU (0.554 vs. 0.540). PH-TTS is the weakest branch in every objective column (0.469 to 0.496 WER).
MOS. For SpeechT5 (Table 4), CH-TTS (4.00) is close to the original recordings (4.04), followed by DE-TTS (3.53) and PH-TTS (3.10). For Orpheus (Table 5), CH-TTS is the highest synthesized condition with full data (4.67, vs. 4.86 for originals), ahead of DE-TTS (3.80) and PH-TTS (3.43). Halving the training data changes MOS by -0.11 for CH-TTS (4.56), -0.54 for DE-TTS (3.26) and -0.18 for PH-TTS (3.25), so the DE-TTS minus PH-TTS gap shrinks from 0.37 to 0.01 and the two become indistinguishable. Originals were not re-rated in the half-data test.
Read evenhandedly, each intermediate wins somewhere. Direct High German conditioning wins the transcript-overlap metrics in three of four objective settings (both SpeechT5 columns and Orpheus full data), plausibly because the reference is High German text and DE-TTS preserves that surface form. Swiss German text wins every MOS comparison and the Orpheus half-data objective columns. The phoneme intermediate loses on every metric and backbone, but under half data its MOS ties direct synthesis. The listening tests are small (4 to 6 raters, roughly 66 to 127 rated samples per condition), no confidence intervals or significance tests are reported, and SpeechT5 and Orpheus were rated in separate tests with different rater pools, so the cross-backbone MOS comparison (Orpheus higher overall) is not a controlled one.
Novelty Assessment
The contribution is empirical. Every component (SpeechT5, Orpheus, T5 translators, Wav2Vec2Phoneme, LLM-prompted fusion, a fine-tuned Whisper scorer) is existing technology; the novelty lies in the controlled three-way comparison of intermediate representations for a non-standardized dialect and in the practical recipe for building phoneme supervision from audio when no lexicon exists. The phoneme branch is not a fair test of phonemes as such: its labels are automatically derived from audio and then LLM-fused, so it measures this specific noisy pipeline rather than an oracle phoneme condition, as the authors acknowledge. The observation that transcript-overlap metrics against a High German reference disagree with listeners on DE-TTS versus CH-TTS is the most transferable finding, though it rests on small rater pools.
Field Significance
Low. The paper adds a narrow but practical data point for dialect TTS without a standard orthography: a written-dialect intermediate is preferred by listeners over direct standard-language conditioning, and an audio-derived phoneme intermediate is not yet competitive. It also gives a concrete case where closed-loop ASR scoring against a cross-variety reference misranks systems, which is relevant to evaluation of any TTS pipeline that changes lexical content before synthesis.
Claims
- supports: For dialects without a standardized orthography, conditioning TTS on a translated written-dialect intermediate can yield more natural output than conditioning directly on the standard-language text.
Evidence: CH-TTS has the highest synthesized MOS on both backbones (SpeechT5 4.00 vs. 3.53 for DE-TTS; Orpheus 4.67 vs. 3.80 with full data and 4.56 vs. 3.26 with half data). (§4.2, Tables 4 and 5)
- complicates: Closed-loop ASR metrics scored against a reference in a different language variety can misrank TTS pipelines whose intermediate step legitimately changes wording.
Evidence: With High German references, DE-TTS gets lower WER and higher SacreBLEU than CH-TTS for SpeechT5 (0.214 vs. 0.237 WER) and full-data Orpheus (0.254 vs. 0.284), the opposite of the MOS ranking; for SpeechT5, DE-TTS even beats the original recordings (0.235 WER). (§4.1, §5, Tables 2 to 4)
- complicates: Phoneme intermediates derived automatically from audio (phoneme recognition plus fusion) can underperform text intermediates in low-resource dialect TTS, because recognition and fusion errors propagate into the conditioning signal.
Evidence: PH-TTS is the weakest branch on every objective column and every MOS comparison across SpeechT5 and Orpheus, even though the selected LLM fusion scores 98.1 boundary accuracy on the gold set; the authors attribute the gap to noisy supervision and tokenizer mismatch. (§4, §5, Tables 1 to 5)
- refines: Sensitivity to reduced training data depends on how close the conditioning representation is to the target dialect.
Evidence: Halving Orpheus training data lowers MOS by 0.11 for CH-TTS but by 0.54 for DE-TTS (and 0.18 for PH-TTS), and leaves the half-data DE-TTS and PH-TTS MOS separated by only 0.01. (§4.2.2, Table 5)
Limitations and Open Questions
Small, uncontrolled listening tests
MOS comes from 6 raters (SpeechT5) and 4 raters (Orpheus) with no confidence intervals or significance tests, and the two backbones were rated in separate tests. Narrow MOS differences (for example PH-TTS vs. DE-TTS under half data) and the cross-backbone comparison cannot be treated as established.
- The PH-TTS comparison conflates the phoneme representation with the quality of its automatic supervision; there is no oracle or lexicon-based phoneme condition to separate the two.
- The fusion gold set (100 examples) was used both to tune strategy parameters and prompts and to select the strategy, so the Table 1 scores are optimistic.
- Results are not broken down by dialect, so it is unknown whether the ranking holds across all eight SwissDial dialects, and speaker identity is not analysed.
- The STT scorer is an internal fine-tuned Whisper model, and SpeechT5 required ad hoc character and phoneme remapping into spare tokens, which may handicap SpeechT5’s PH-TTS branch specifically.
- Orpheus’s model size, codec, and training settings are not reported; code and demo availability are not stated.
- Translation quality of the T5 DE to CH and DE to phoneme models is not evaluated on its own, so its contribution to end-to-end quality is not isolated.
Wiki Connections
- Transformer Encoder-Decoder TTS — SpeechT5, a pre-trained encoder-decoder, is fine-tuned on all three intermediates as the lightweight backbone.
- Autoregressive Codec TTS — Orpheus, a Llama-based speech LLM TTS system, is fine-tuned on all three intermediates and on half the data, giving the highest synthesized MOS in this study.
- Self-Supervised Speech — PH-TTS training labels come from a wav2vec 2.0-based phoneme recognizer applied to SwissDial audio, and its errors are identified as a bottleneck.
- Subjective Evaluation — MOS with small human rater pools is used as the primary selection signal and reverses the objective ranking between DE-TTS and CH-TTS.
- Evaluation Metrics — Demonstrates a failure mode of closed-loop WER/SacreBLEU scoring against a reference in a different language variety.
- Whisper — A Whisper model fine-tuned on Swiss German serves as the STT component of the closed-loop objective evaluation.