arXiv · 2026 · Preprint

Yuxiang Zhao et al. (Shanghai Jiao Tong University / Shanghai Innovation Institute / Microsoft / AISpeech Co., Ltd.) · → Paper · Demo: ✓ · Code: ✓

X-Translator is an open, real-time cascaded speech-to-speech translation system (streaming ASR, LLM machine translation, prompt-conditioned TTS). A session controller commits only stable ASR segments for translation and routes each segment to a per-speaker voice prompt, which gives the highest short-form speaker similarity among the evaluated systems at the cost of the highest latency.

Problem

Real-time speech-to-speech translation (S2ST) has to balance translation quality, latency, speech naturalness and speaker consistency together. Research systems such as Translatotron, the Seamless family and Seed LiveInterpret improve the models themselves. Proprietary products (Doubao AST 2.0, Qwen LiveTranslate, GPT Realtime Translate) show the application is practical, but they are black boxes, so their segmentation, streaming and speaker-handling choices cannot be inspected. Long-form, multi-speaker sessions raise runtime problems that model-level work leaves open. Partial ASR hypotheses keep changing, turn boundaries are unclear, and each translated segment needs a prompt from the correct speaker. Short-form benchmarks also miss failures that only show up over long sessions, such as voice drift, speaker collapse and accumulated latency.

Method

X-Translator is a modular cascade. A browser client streams audio over WebSocket to a server-side session controller. The controller holds the ASR hypothesis buffer, committed segments, translation and TTS requests, per-speaker prompt buffers and playback metadata. The reported configuration uses frozen, off-the-shelf backends: Qwen3-ASR (1.7B) for streaming recognition, LMT-60-8B for text translation, and X-Voice (0.4B) for prompt-conditioned synthesis with 48 kHz output (§4.3, Table 1, Appendix C). The paper trains no model. Alternative supported backends include X-ASR, Paraformer-zh, SenseVoice-Small, LMT-60-4B, HY-MT1.5 and IndexTTS/IndexTTS2 (Table 1). The stack can run on a single RTX 3090 (§1).

Incremental segment commitment (§3.2). Partial ASR hypotheses go to the browser for display only and never reach MT or TTS. A commitment layer cuts stable prefixes from the growing hypothesis buffer using three triggers: punctuation boundaries, silence or no-update timeouts, and speaker-boundary flushes raised by the prompt manager. Each committed segment becomes a structured event with source text, start and end times, a speaker ID and diagnostic metadata. MT and TTS then run serially on each committed segment.

Speaker-aware prompt management (§3.3). An online speaker tracker uses CAM++ embeddings (1.0 s window, 0.5 s hop) to label speaker intervals on the input timeline. Each committed ASR span is assigned to the speaker whose intervals overlap it the most. Every speaker has its own prompt bucket with two 6-second views. The fixed prompt is the first 6 s of usable speech and does not change after that. The rolling prompt is the most recent 6 s. A configured synthesis mode chooses between them. If no complete prompt exists yet, the system uses whatever speaker audio is available or falls back to a default voice. The paper does not state which prompt mode was used in the reported experiments.

The controller logs ASR commit time, MT and TTS request and response times, and playback time, so end-to-end delay can be broken down by module (§3.4).

Long-form evaluation protocol (§4.4.2-4.4.3). For single-speaker long audio, VAD removes non-speech and the timelines are normalized by cumulative speech duration. Resemblyzer embeddings are computed on 10 s windows with a 5 s hop. Three metrics follow. Source-Source Long-Gap similarity (SSLG) averages source window pairs at least half the session apart (γ = 0.5). Target-Target Long-Gap similarity (TTLG) applies the same computation to the target speech. Global Source Similarity (GSS) compares each target window with the mean source embedding. For multi-speaker audio, VibeVoice-ASR provides speaker-attributed transcripts. Source and target segments are matched by multilingual text-embedding similarity within a ±0.10 relative-position window. The paper then reports duration-weighted Spk. SIM, Target Purity (resistance to speaker collapse) and Source Purity (resistance to speaker fragmentation), both computed from a duration-weighted speaker correspondence matrix.

Key Results

Short-form (OpenSTBench, Table 2). Translation-quality scores are computed on an ASR transcript of the generated speech. No system wins every metric.

  • Speaker preservation is X-Translator’s clearest strength. Its Spk. SIM is 0.87 for En→Zh (Doubao 0.84, GPT Realtime 0.69, Qwen3-LiveTranslate 0.60) and 0.88 for Zh→En (Doubao 0.84, GPT 0.66, Qwen 0.57).
  • Intelligibility depends on the direction. For Zh→En, X-Translator has the lowest target WER (3.04% vs. 3.70%, 6.91%, 8.30%) and ties Doubao on Emo. SIM (0.86). For En→Zh, its CER of 5.88% is second-worst; only GPT Realtime is higher (9.66%).
  • Naturalness is lowest. X-Translator has the lowest UTMOS in both directions: 2.66 vs. 2.83-3.58 for En→Zh, and 3.39 vs. 3.43-4.20 for Zh→En.
  • Translation quality trails. S2ST COMET is 0.78 (En→Zh) and 0.74 (Zh→En). The best systems reach 0.81 (Qwen3-LiveTranslate, En→Zh) and 0.77 (Doubao, Zh→En).
  • Latency is highest. Start offsets are 2,584 ms and 4,056 ms. GPT Realtime starts at 814 ms and 800 ms. X-Translator also has the highest ATD and cATD in both directions.

S2TT vs. S2ST (Table 3 vs. Table 2). Scored on the returned translation text, X-Translator reaches COMET 0.83 for En→Zh. After synthesis and target ASR it drops to 0.78, a loss of 0.05. For Zh→En it stays at 0.74 under both protocols. On text, Qwen3-LiveTranslate and Doubao tie for the top COMET (0.85 En→Zh, 0.78 Zh→En).

Coverage (Table 12). X-Translator succeeds on 98.92-100% of calls across the OpenSTBench subsets. The proprietary baselines range from 93.50% to 100%. Quality metrics are computed on successful calls only.

Long-form single-speaker voice stability (Table 4, about 2.5 h per direction). X-Translator’s TTLG is 0.91 for En→Zh (TEDLIUM-3) and 0.90 for Zh→En (WenetSpeech), against a source SSLG of 0.89. Its GSS is 0.83 and 0.79, tied with Doubao and above GPT Realtime (0.69, 0.66) and Qwen3.5-LiveTranslate (0.53, 0.50). Qwen3.5-LiveTranslate has the highest TTLG (0.94 in both directions) but the lowest GSS. Its target voice is very stable, but it matches the source speaker poorly.

Long-form multi-speaker (Table 5).

  • En→Zh (VoxConverse). X-Translator’s Spk. SIM is 0.77, tied with Doubao. Target and source purity are 0.85 and 0.93, against 0.89 and 0.94 for Doubao.
  • Zh→En (Bilibili interviews). Spk. SIM is 0.77 against Doubao’s 0.78. Purities are 0.84 and 0.88, against 0.91 and 0.90 for Doubao.
  • Collapse and fragmentation. Qwen3.5-LiveTranslate has near-perfect Source Purity (1.00 and 0.99) but Target Purity of only 0.39 and 0.38. This pattern fits several source speakers collapsing into a few target voices. GPT Realtime’s Source Purity falls to 0.74 for Zh→En, which indicates fragmentation.

Multilingual (FLEURS, Table 6). The evaluation covers 70 directions: 19 from Zh, 16 into Zh, 19 from En and 16 into En. Macro-averaged COMET on Whisper-medium transcripts of the target speech is 0.69 overall. The group means range from 0.68 (Zh→X) to 0.70 (X→En). X→En has the largest spread (std 0.09). The paper releases direction-level scores as artifacts but does not print them.

Fairness of the comparison. Each proprietary baseline is run twice and averaged. These systems are behavioral baselines only, and the Qwen service changed between the short-form and long-form collections (Qwen3 vs. Qwen3.5-LiveTranslate). No open-source S2ST system, such as Seamless or StreamSpeech, is compared. The paper has no human listening test.

Novelty Assessment

The contribution is mainly engineering integration. ASR, MT and TTS are all existing frozen models. The authors themselves say they do not propose a new S2ST model. The new parts are runtime policies. The commitment layer applies familiar simultaneous-translation ideas (stable-prefix selection, silence and punctuation endpointing) at the system level. The prompt manager adds overlap-based speaker binding and per-speaker fixed and rolling 6 s prompt buckets. Both are sensible and clearly specified, but simple, and the paper never ablates them. No experiment varies the commitment triggers, compares fixed with rolling prompts, or disables speaker routing. The reported gains therefore cannot be separated from the strong zero-shot cloning of the X-Voice backend.

The more reusable contribution is the evaluation. The long-gap metrics (SSLG, TTLG, GSS) separate target-voice stability from faithfulness to the source speaker. The bidirectional Target and Source Purity metrics separate speaker collapse from speaker fragmentation. Both pairs expose failure modes that a single speaker-similarity score hides. The released open stack and logs are a practical baseline for deployment-oriented S2ST research.

Field Significance

Moderate. X-Translator shows that an open, low-cost cascade built from strong zero-shot TTS and per-speaker prompt routing can match or beat proprietary real-time translators on speaker preservation. It still trails them on latency, naturalness and translation quality. Its long-form stability and speaker-purity metrics offer a reusable way to separate voice stability, source fidelity, speaker collapse and speaker fragmentation in streaming S2ST.

Claims

  • supports: Cascaded speech-to-speech translation with prompt-conditioned zero-shot TTS can preserve source-speaker identity better than end-to-end proprietary real-time translators.

    Evidence: The cascade with X-Voice and per-speaker prompts reaches the highest short-form Spk. SIM in both directions (0.87 En→Zh, 0.88 Zh→En vs. at most 0.84 for any proprietary system) and ties the strongest baseline on long-form GSS (0.83, 0.79). (§5.1, Table 2; §5.2, Table 4)

  • complicates: In cascaded real-time speech translation, gains in speaker preservation come with higher latency and lower naturalness than tightly integrated systems.

    Evidence: X-Translator has the slowest start offset (2,584 and 4,056 ms vs. about 800 ms for GPT Realtime Translate), the highest ATD and cATD, and the lowest UTMOS in both directions, while trailing the best COMET by 0.03 in both directions. (§5.1, Table 2)

  • refines: Target-voice consistency over a long session does not imply preservation of the source speaker’s identity, so long-form voice evaluation needs separate stability and source-fidelity measures.

    Evidence: Qwen3.5-LiveTranslate has the highest target-target long-gap similarity (0.94) but the lowest global source similarity (0.53 and 0.50), whereas X-Translator pairs TTLG of 0.91 and 0.90 with GSS of 0.83 and 0.79. (§5.2, Table 4)

  • refines: Speaker similarity alone hides multi-speaker failures in speech translation; collapse of several source speakers into one target voice and fragmentation of one speaker across voices need separate purity measures.

    Evidence: Qwen3.5-LiveTranslate scores near-perfect Source Purity (1.00 and 0.99) with Target Purity of only 0.39 and 0.38, which indicates speaker collapse. GPT Realtime Translate’s Source Purity falls to 0.74 for Zh→En, which indicates fragmentation. (§4.4.3; §5.3, Table 5)

  • complicates: Scoring speech-to-speech translation from an ASR transcript of the synthesized speech mixes synthesis and recognition errors into translation-quality scores.

    Evidence: X-Translator’s En→Zh COMET is 0.83 on its returned translation text but 0.78 after synthesis and target ASR, while Zh→En stays at 0.74 under both protocols. (§5.1, Tables 2-3)

Limitations and Open Questions

Warning

The paper has no ablation of its two runtime contributions, segment commitment and speaker prompt routing, and does not say which prompt mode (fixed or rolling) was used. Its speaker-preservation results therefore cannot be attributed to the controller rather than to the frozen X-Voice backend.

  • All evaluation is automatic: Resemblyzer similarity, UTMOS, ASR-based COMET and BLEU. There is no human listening test for naturalness, speaker similarity or translation adequacy.
  • The proprietary baselines are black boxes and change over time. Qwen3-LiveTranslate (short-form) and Qwen3.5-LiveTranslate (long-form) are different service versions, so their results cannot be compared across tables. No open research S2ST system is included as a baseline.
  • The long-form and multi-speaker sets are small: 10 recordings per direction, about 1.7-2.9 h each. One of them is a self-collected Bilibili interview set that is not redistributed.
  • Multilingual results are reported only as group macro-averages. Direction-level scores, which the paper describes as varying non-negligibly, are released as artifacts but not shown.
  • The cascade inherits ASR error propagation and cannot jointly optimize translation, prosody, speaker similarity and latency. Overlapping speech, rapid turn-taking and similar-sounding speakers make speaker assignment ambiguous, and an early prompt error can affect many later segments (§6).
  • Latency is reported end to end only. The per-module timing logs are described, but no per-module breakdown is reported.

Wiki Connections

  • Speech-to-Speech — An open cascaded S2ST system (ASR, MT, prompt-conditioned TTS) with explicit runtime commitment and speaker routing, compared against proprietary real-time translators.
  • Speaker Adaptation — Per-speaker fixed and rolling 6 s prompt buckets, routed by speaker-overlap binding, adapt a frozen zero-shot TTS to each live speaker in multi-speaker sessions.
  • OpenSTBench — Supplies the short-form evaluation protocol (translation, speech quality, latency, speaker and emotion similarity); several authors overlap.
  • X-Voice — The frozen prompt-conditioned TTS backend used in all reported experiments, which carries the speaker-cloning burden.
  • IndexTTS — A supported alternative TTS backend in X-Translator’s modular stack; not used in the reported results.
  • IndexTTS2 — Listed together with IndexTTS as a supported prompt-conditioned TTS backend option.
  • Seed LiveInterpret 2.0 — An end-to-end simultaneous voice-preserving S2ST system that the paper positions as complementary model-level work.
  • Seamless — Cited as model-level expressive and streaming S2ST work; it identifies voice cloning and speech inflation as deployment concerns.
  • SeamlessM4T — A unified multilingual S2ST model that the paper contrasts with its replaceable-component cascade.
  • UniSS — Unified expressive, voice-preserving S2ST cited as complementary direct-model work.