arXiv · 2025 · Preprint
Zhou et al. (JD Health International Inc.) · → Paper · Demo: ? · Code: ✓
JoyTTS integrates MiniCPM-o’s Qwen-7B LLM with CosyVoice2 TTS, trained on 2,000 hours of conversational audio, to produce an open-source spoken chatbot with zero-shot voice cloning.
Problem
End-to-end spoken chatbot systems such as Qwen2.5-Omni and LLaMA-Omni2 offer strong conversational capabilities but do not support voice cloning and do not release training code, limiting adaptability for personalised applications. MiniCPM-o is open-source but ships with a GPT-SoVITS TTS module whose voice cloning quality is insufficient for realistic personalised interactions. JoyTTS addresses both gaps by replacing the TTS component with CosyVoice2 and releasing the full training pipeline.
Method
JoyTTS is structured as a four-module pipeline. A Tokenizer module converts both text and audio inputs into tokens. An LLM-Chat module (Qwen-7B, approximately 7B parameters) processes these tokens to produce text output along with intermediate hidden states. An LLM-TTS module receives a combination of text embeddings and projected LLM-Chat hidden states to generate speech tokens. A Generator module (CosyVoice2-based) converts those speech tokens into mel-spectrograms and final audio.
The coupling between LLM-Chat and LLM-TTS is the central design choice. Hidden states of dimension 3584 from Qwen-7B are projected via an MLP layer to 768 dimensions and then summed with the text embeddings of the corresponding output tokens, forming a TTS embedding (TTS_embed = Emb(y_i) + MLP(h_i)). This gives the TTS module access to contextual semantic information from the LLM. For voice cloning at inference, both the prompt text and a reference audio waveform are first processed by LLM-Chat to produce TTS embeddings that capture the target speaker’s characteristics before speech generation begins.
Training proceeds in two stages. Stage 1 trains LLM-Chat and LLM-TTS independently so that each module can specialise: LLM-Chat is trained until its hidden states align well with text labels, and LLM-TTS is trained for high-quality speech synthesis. Stage 2 jointly fine-tunes both with a combined loss (L_LLM-Chat + L_LLM-TTS) to optimise end-to-end coherence. The training corpus consists of approximately 400,000 multi-turn dialogue samples (~2,000 hours total) derived from RedGPT and GeneratedChat0.4M text dialogues synthesised to audio using CosyVoice2, with prompt text-audio pairs drawn from WenetSpeech4TTS to reinforce voice cloning.
Key Results
On seed-tts-zh, JoyTTS achieves speaker similarity (SS) of 0.73 and WER of 5.09. Against baselines on the same benchmark: GPT-SoVITS scores SS 0.55 and WER 5.13, and standalone CosyVoice2 scores SS 0.748 and WER 1.45. JoyTTS substantially outperforms GPT-SoVITS on speaker similarity and closely matches it on WER, but its WER is more than three times worse than standalone CosyVoice2, indicating that joint LLM-TTS training introduces intelligibility costs. End-to-end latency is reported at 1.8 seconds on a single NVIDIA 4090D without any inference-time engineering optimisations.
Novelty Assessment
The contribution is primarily engineering integration: MiniCPM-o and CosyVoice2 are both established open-source systems, and the coupling mechanism (projecting LLM hidden states into the TTS embedding space via an MLP) is a straightforward architectural bridge rather than a structural innovation. The two-stage training procedure and combined loss are standard practice for joint LLM-TTS development. The genuine value is the open-source release of a working voice-cloning chatbot trained on 2,000 hours of conversational data, which lowers the barrier for downstream experimentation. Performance is mixed: voice similarity improves over the GPT-SoVITS baseline, but intelligibility regresses sharply relative to running CosyVoice2 in isolation, and no explanation is offered for the gap.
Field Significance
Low — JoyTTS is an open-source engineering integration of two existing models that provides a reproducible starting point for spoken chatbot development with voice cloning. It does not introduce new architectural ideas, and its empirical results reveal a clear intelligibility trade-off compared to the TTS component used in isolation.
Claims
-
supports: Routing LLM hidden states into the TTS module’s embedding space can enable voice cloning in an end-to-end spoken chatbot without a dedicated speaker encoder.
Evidence: JoyTTS projects Qwen-7B hidden states via an MLP into the CosyVoice2-based LLM-TTS embedding, achieving SS of 0.73 on seed-tts-zh using prompt audio as the only speaker reference. (§1.2)
-
complicates: Jointly training an LLM-Chat module with a TTS module in a chatbot pipeline can degrade intelligibility relative to running the TTS component standalone, even when speaker similarity improves.
Evidence: JoyTTS achieves WER of 5.09 compared to 1.45 for standalone CosyVoice2 on seed-tts-zh, despite closing the speaker similarity gap (JoyTTS SS 0.73 vs. CosyVoice2 SS 0.748). (§4, Table 1)
-
supports: End-to-end spoken chatbot systems pairing a 7B-parameter LLM with an autoregressive TTS module can achieve sub-2-second response latency on a single consumer GPU without specialised inference optimisations.
Evidence: JoyTTS reports 1.8-second end-to-end latency on a single NVIDIA 4090D with no engineering optimisation applied. (§4)
Limitations and Open Questions
The evaluation is limited to a single Chinese benchmark (seed-tts-zh), leaving performance on English, multilingual, or spontaneous conversational speech uncharacterised. The WER gap between JoyTTS (5.09) and standalone CosyVoice2 (1.45) is large and unexplained; the paper does not ablate whether the regression originates from the joint training procedure, the conversational data distribution, or the MLP projection coupling. No subjective listening tests are reported, making it impossible to assess naturalness or perceived quality beyond intelligibility and speaker similarity metrics. Emotion control, identified as a target for future work, is not implemented in the current system.
Wiki Connections
- Zero-Shot TTS — JoyTTS uses audio prompt pairs drawn from WenetSpeech4TTS to enable zero-shot voice cloning at inference, with no speaker enrolment required.
- Spoken Language Model — the LLM-TTS module generates discrete speech tokens directly from LLM-Chat hidden states, placing JoyTTS within the spoken language model paradigm for end-to-end dialogue systems.
- Autoregressive Codec TTS — speech token generation in JoyTTS follows an autoregressive codec TTS pattern, building on the approach established by CosyVoice2.
- Neural Audio Codec — speech tokenisation in the Tokenizer and Generator modules relies on CosyVoice2’s internal speech token representation.
- Speaker Adaptation — zero-shot voice cloning via audio prompt conditioning is the central personalisation mechanism in JoyTTS.
- CosyVoice2 — JoyTTS replaces MiniCPM-o’s original GPT-SoVITS TTS module with CosyVoice2, directly building on its streaming speech synthesis and voice cloning capabilities.