arXiv · 2025 · Preprint
Cheng et al. (ByteDance) · → Paper · Demo: ? · Code: ?
An end-to-end simultaneous speech-to-speech translation system that unifies cross-lingual voice cloning with real-time speech synthesis in a multimodal LLM, trained with a novel two-stage RL framework combining per-chunk and sequence-level rewards, reducing cloned-speech output latency from ~10 seconds to ~3 seconds while surpassing commercial simultaneous interpretation systems in translation quality.
Problem
Simultaneous interpretation (SI) systems face four persistent challenges: cascaded architectures accumulate errors across ASR, MT, and TTS stages; existing end-to-end approaches support only speech-to-text output and cannot produce translated speech directly; multi-speaker confusion in long-form audio degrades quality; and translated speech tends to inflate in duration relative to source audio. No product-ready system combines low-latency speech-to-speech output with voice cloning. Evaluation methodology has compounded the problem, with prior SI benchmarks using less stringent standards than real-world conditions, causing systematic overestimation of system quality.
Method
Seed LiveInterpret 2.0 builds on the Seed LLM family by extending a pre-trained language model with a pre-trained audio encoder that processes streaming audio input. The resulting multimodal LLM autoregressively generates interleaved text tokens and audio tokens, producing simultaneous translation with real-time speech synthesis and cross-lingual voice cloning within a single forward pass. The duplex processing architecture segments input audio into sequential chunks; at each chunk, the model uses both the current audio segment and accumulated prior context to generate a partial translation.
Training proceeds in three stages. First, multitask multimodal continual training on approximately 100 billion tokens spanning audio-to-text transcription, text-to-audio synthesis, and text-only tasks establishes cross-modal alignment. Second, supervised fine-tuning on human-annotated data activates the model’s data-driven read-write policy, multi-speaker discrimination, speech translation, and voice cloning capabilities. Third, a two-stage RL phase fine-tunes the quality-latency balance using PPO with an adaptive KL penalty controller.
The RL framework addresses a fundamental tension in simultaneous translation: process-level feedback at each chunk ensures intra-segment consistency, while sequence-level feedback ensures inter-segment coherence. Stage 1 trains exclusively on five single-turn rewards: detection accuracy (penalising premature translation), translation initiative (rewarding timely output), translation quality (comparing output to reference), time compliance (matching source speech duration), and format consistency. Stage 2 introduces complementary multi-turn rewards: a lagging reward that penalises cumulative wait time and a sequence-level translation quality reward. This staged approach prevents the model from over-optimising fast-to-learn process rewards at the expense of global coherence.
Key Results
On the RealSI longform benchmark (zh-en), Seed LiveInterpret 2.0 achieves a VIP score of 79.5 versus 53.2 for the best commercial competitor. In the speech-to-speech setting, it achieves SVIP 67.8 at Average Lagging 5.18 seconds; the only commercial system supporting speech-to-speech output reaches SVIP 3.0 at AL 48.21 (Table 1). For en-zh, VIP reaches 70.1 (versus 42.0 best commercial) and SVIP 64.7 at AL 4.75.
At the sentence level, the model achieves BLEURT 64.9 and COMET 84.1 on zh-en speech-to-text, outperforming SeamlessStreaming (BLEURT 55.8, COMET 76.4) and all commercial baselines (Table 2). It is the only system in the comparison that supports voice cloning throughout.
The ablation comparing RL against the SFT baseline is informative: on RealSI, VIP improves from 75.1 to 79.5 while FLAL drops sharply from 3.90 to 2.37 (Table 3), indicating that RL tightens latency more substantially than it improves translation quality. The two-stage RL scheme outperforms both single-turn-only and multi-turn-only variants: multi-turn-only achieves lower latency but reduced VIP, while single-turn-only shows high latency with comparable quality (Table 6).
Novelty Assessment
The integration of end-to-end speech-to-speech translation with cross-lingual voice cloning in a single autoregressive model is a genuine advance. Prior end-to-end SI systems either produce text only or use cascaded synthesis without speaker preservation. The two-stage RL framework with complementary process and outcome rewards is a new training procedure for simultaneous translation; the specific five-dimensional single-turn reward design and the staged warm-up scheme have not appeared in prior speech translation work.
The base architecture follows established patterns from multimodal speech LMs rather than introducing a new model structure: combining a pre-trained LLM with an audio encoder and audio token generation is a known approach. The RL reward components individually draw from established RLVR literature. Reproducibility is limited: training data, model weights, and all commercial baseline identities are proprietary, making independent verification of the results impossible.
Field Significance
Moderate. This paper demonstrates that product-level simultaneous speech-to-speech interpretation with voice cloning is achievable in a unified end-to-end model, pushing beyond the speech-to-text frontier that prior end-to-end SI work occupied. The two-stage RL framework provides a concrete recipe for balancing process and outcome rewards in streaming generation tasks with paired latency-quality objectives. The new SVIP metric introduces a more operationally complete evaluation standard for spoken interpretation quality. The proprietary setting limits direct replication, but the design principles and evaluation methodology can serve as a template for future open research in simultaneous speech translation.
Claims
-
supports: End-to-end speech-to-speech translation trained with reinforcement learning can achieve substantially lower latency than cascaded commercial systems while surpassing them in translation quality.
Evidence: On RealSI (zh-en), the system achieves VIP 79.5 and SVIP 67.8 at AL 5.18 seconds, versus the best commercial competitor at VIP 53.2 and SVIP 3.0 at AL 48.21. (§3.3, Table 1)
-
supports: Two-stage RL training that separates fine-grained process rewards from holistic sequence-level outcome rewards outperforms applying either strategy in isolation for streaming speech generation.
Evidence: The combined two-stage scheme achieves higher VIP and lower AL than both single-turn-only and multi-turn-only variants on internal benchmarks; multi-turn-only trades translation quality for latency, while single-turn-only cannot exploit global latency constraints. (§4.3, Table 6)
-
complicates: RL reward design for speech generation is prone to exploitation when individual reward signals are applied without complementary constraints.
Evidence: Training with the Time Compliance Reward alone caused the model to reduce audio duration by ~35% and text token count by ~15%, yielding an 8-point BLEURT drop while satisfying the temporal constraint; adding an adversarial quality reward restored translation fidelity. (§4.2, Table 5)
-
supports: Reinforcement learning applied after SFT improves latency more substantially than translation quality in simultaneous speech translation.
Evidence: RL reduces FLAL from 3.90 to 2.37 and AL from 2.82 to 2.58 on RealSI, while VIP improves more modestly from 75.1 to 79.5 compared to the SFT baseline. (§4.1, Tables 3, 4)
-
complicates: Automated MT metrics such as BLEURT and COMET do not fully capture the quality of simultaneous speech interpretation in realistic long-form scenarios.
Evidence: The paper introduces SVIP as a composite human evaluation metric combining translation quality, delivery latency, speech rate, pronunciation, and fluency, arguing that automated metrics fail to assess the pragmatic requirements of live spoken interpretation. (§3.2, Appendix A)
Limitations and Open Questions
Warning
Training data, model weights, and all commercial baseline identities are proprietary. The headline results cannot be independently reproduced or verified.
Evaluation covers only Chinese-to-English and English-to-Chinese translation; generalisation to other language pairs is untested. The paper does not isolate the contribution of voice cloning quality from translation quality within the SVIP metric, making it difficult to assess how each component affects the composite score. Future work identified by the authors includes broader language coverage, improved voice personalization, and enhanced speech stability and expressiveness.
Wiki Connections
- Speech-to-Speech - this paper extends the speech-to-speech paradigm to simultaneous interpretation with cross-lingual voice cloning in a unified autoregressive model.
- Zero-Shot TTS - voice cloning preserves source speaker identity in the translated output without per-speaker fine-tuning.
- Multilingual TTS - the system targets bidirectional Chinese-English simultaneous translation with speaker-aware output.
- RLHF Speech - the two-stage RL framework with single-turn and multi-turn verifiable rewards is the paper’s core training contribution.
- Streaming TTS - duplex chunk-by-chunk processing achieves near-real-time simultaneous output with average latency around 3 seconds.
- Evaluation Metrics - introduces SVIP, a composite human evaluation metric for speech interpretation quality covering latency, speech rate, pronunciation, and fluency.
- Seed-TTS (Seed-TTS) - Seed LiveInterpret 2.0 builds on the Seed-TTS audio generation framework for producing audio tokens and is trained using the Seed LLM continual training methodology.