ACL Findings · 2025 · Conference
Zhang et al. · → Paper · Demo: ? · Code: ?
Textless speech-to-speech translation (S2ST) is advanced by constructing a “unit language” from n-gram language modeling over discrete speech units, providing text-free cross-modal and cross-lingual guidance that reaches performance comparable to models trained with recognized text.
Problem
Textless S2ST must simultaneously bridge two gaps without text supervision: extracting structured linguistic features from continuous, variable audio (cross-modal modeling) and aligning sequences across languages when parallel text pairs are unavailable (cross-lingual modeling). Existing methods address these challenges separately through unit quantisation or masked language model objectives, but the lack of text-level structure leaves both cross-modal and cross-lingual learning underspecified. This is especially limiting for low-resource or unwritten languages where transcribed training data does not exist.
Method
The system builds on the standard textless S2ST pipeline from Lee et al. (2022): an acoustic encoder (A-Enc), a 12-layer Transformer textual encoder (T-Enc), unit decoders for source and target, and a unit-based HiFi-GAN vocoder that synthesises speech from mHubert discrete units (1000-unit vocabulary). Source and target speech are first mapped to mHubert discrete units, which serve as pseudo-text targets.
The central contribution is the “unit language,” a compressed sequence representation built by merging contiguous units into “unit words” via dynamic programming over n-gram language model probabilities. Given a unit sequence, the algorithm finds the maximum-likelihood segmentation into unit words of at most K units each (K=3 in practice), using 2-gram conditional probabilities estimated from corpus frequencies. The resulting unit language is consistently shorter than raw unit sequences across all four evaluated language directions, falling between character-level text and raw units in length, which reduces the alignment difficulty that plagues long discrete-unit sequences. A SentencePiece vocabulary of 10k types is applied to the merged sequences.
Two auxiliary decoders are added to the base architecture: a source text decoder (S-Dec) that predicts source-side unit language from an intermediate T-Enc layer (the 2nd layer, controlled by hyperparameter r), supporting cross-modal learning, and a target text decoder (T-Dec) that predicts target-side unit language from the full encoder output, supporting cross-lingual learning. Both decoders use 2-layer Transformers trained with cross-entropy losses (L_CM and L_CL), combined with the primary unit prediction objectives in a weighted multi-task objective.
An initially problematic finding is that applying L_CM and L_CL simultaneously produces less improvement than either alone. Analysis via attention sparseness and localness metrics shows that CM guidance, applied at middle encoder layers, disrupts the cross-lingual representations learned in upper layers. To resolve this, the paper introduces task prompt modeling: two learnable vectors (b_CM and b_CL, each of hidden dimension 512) are prepended to the encoder input. The b_CM prompt is active through the first r layers (guiding cross-modal processing), then replaced by b_CL for the remaining layers (guiding cross-lingual processing). A negative MSE loss between the two prompts (weight -3.0) encourages distinct representations, improving both tasks when combined.
Key Results
On the VoxPopuli benchmark across four language pairs (Es-En, Fr-En, En-Es, En-Fr), the full proposed system (unit language + task prompt) achieves an average ASR-BLEU of 21.5, compared to 20.3 for the strong baseline (Table 2). This represents a +1.2 BLEU average improvement. Cross-lingual training alone (L_CL) contributes more than cross-modal training (L_CM), with average gains of +0.8 and +0.4 BLEU respectively, suggesting that cross-lingual alignment is the more acute bottleneck in textless S2ST.
The unit language matches the performance of models trained with recognised text for cross-lingual guidance specifically, and the full system reaches parity with recognised-text training overall (avg 21.5 BLEU in both cases). Compared to BPE-based pseudo-language construction (Shen et al., 2024), the unit language achieves +0.5 BLEU average improvement (21.2 vs 20.7), attributed to n-gram context information that BPE merges do not use (Table 6). Hyperparameter analysis confirms that CM auxiliary training must be applied at an encoder layer well separated from CL training to avoid gradient interference (Table 4).
Novelty Assessment
The unit language construction is a novel representation technique: applying n-gram language modelling to compress discrete speech unit sequences into text-like forms without any labelled data. The resulting sequences reliably fall in a length regime that facilitates alignment, and the n-gram context makes them more linguistically structured than BPE-merged units. However, the underlying architecture (Transformer enc-dec), the feature extractor (mHubert), and the vocoder (unit-based HiFi-GAN) are all existing components from prior work. Task prompt modelling as a conflict mitigation strategy is a known technique in multi-task NLP that is applied here to speech translation without structural modification.
The empirical finding that CM and CL training conflict due to layer-level competition is a genuine diagnostic contribution, supported by sparseness and localness analyses across multiple language pairs. However, the evaluation is restricted to four European language directions (Spanish, French, English), which share broadly similar phonological characteristics, and the method’s behaviour on typologically distant languages is untested. The paper also lacks human evaluation of output speech quality; improvements are measured only by ASR-BLEU, which captures translation accuracy but not naturalness or fluency.
Field Significance
Moderate — this paper provides a practical and well-analysed method for bridging the gap between raw discrete unit sequences and text-like representations in textless S2ST, reaching text-supervised performance without any labelled transcriptions. The task prompt conflict mitigation offers a transferable technique for multi-task speech models where cross-modal and cross-lingual objectives compete. Its impact is primarily within the textless S2ST sub-area; the contribution to the broader speech generation field is incremental.
Claims
-
supports: Discrete speech unit sequences can be compressed into text-like representations via n-gram language modelling, providing effective text-free guidance for both cross-modal and cross-lingual learning in speech translation.
Evidence: Unit language (2-gram merged mHubert units, K=3) achieves average ASR-BLEU of 21.5 on VoxPopuli across four language directions, matching recognised-text training and yielding +1.2 BLEU over the baseline that uses raw discrete units alone. (§2.1, §3.2, Table 2)
-
complicates: Combining cross-modal and cross-lingual auxiliary objectives in multi-task sequence-to-sequence speech training can produce gradient interference that limits the gains of each individual loss.
Evidence: Simultaneously applying source (L_CM) and target (L_CL) unit language losses without task prompts yields combined gains no greater than either loss alone, due to CM guidance at middle encoder layers disrupting CL representation learning in upper layers, as confirmed by attention localness analysis. (§2.3, §3.2, §4.3)
-
supports: Learnable task-specific prompt vectors can separate competing auxiliary objectives within a shared Transformer encoder by enforcing distinct intermediate representations for each task.
Evidence: Two prompt vectors (b_CM and b_CL) prepended at different encoder layer boundaries, with a negative MSE diversification loss, fully recover the +1.2 BLEU combined gain from unit language guidance that is otherwise lost to task interference. (§2.4, §3.2, Table 2)
-
refines: Sequence length compression is a critical enabler of cross-lingual alignment in textless speech translation, and n-gram-merged unit sequences provide substantially better compression than BPE pseudo-languages.
Evidence: Unit language lengths fall consistently between raw units and character-level text across all four VoxPopuli language pairs; unit language CL training outperforms BPE pseudo-language by +0.5 BLEU average, attributed to n-gram contextual information absent in BPE merges. (§4.1, §4.6, Figure 4, Table 6)
Limitations and Open Questions
Warning
The evaluation uses ASR-BLEU only: output speech is recognised by ASR and scored with BLEU against text references. Naturalness, fluency, and speaker consistency of the synthesised speech are not assessed. The paper’s own Limitations section acknowledges this gap.
The method is validated on four European language directions (Spanish, French, English) from VoxPopuli. Generalisation to typologically distant or low-resource languages, where phonological unit distributions may differ substantially, is unexamined. Unit language construction with K=3 requires approximately 12 hours of preprocessing per 160M-unit corpus, which may limit applicability at scale. The task prompt mitigates but does not fully eliminate the CM/CL conflict; the remaining interaction between the two objectives is not resolved.
Wiki Connections
- Speech-to-Speech — this paper directly advances textless S2ST by introducing unit language guidance and task prompt modelling to close the gap with text-supervised methods.
- Self-Supervised Speech — mHubert (a multilingual self-supervised model) provides the discrete unit vocabulary that the unit language construction operates on, making SSL representations a prerequisite for the method.
- Multilingual TTS — the system handles four language pairs and demonstrates that text-free unit representations can match recognised-text performance for cross-lingual speech generation.
- Spoken Language Model — the paper treats mHubert discrete units as pseudo-text tokens for sequence modelling, contributing to the design space of unit-based spoken language systems.