IEEE TBME · 2026 · Journal
Ding Ma et al. (Nagoya University / Beihang University / TARVO, Inc.) · → Paper · Demo: ✓ · Code: ?
Uses phoneme text as a training-only auxiliary modality for sequence-to-sequence voice conversion of electrolaryngeal speech into normal speech: a joint speech-text network learns fused representations, which are then transferred into a speech-only seq2seq VC model of unchanged size, so inference needs no text.
Problem
Laryngectomees who speak with an electrolarynx produce electrolaryngeal (EL) speech: a mechanical excitation with no natural F0 contour, strong radiated device noise, and a slower, atypical speaking rate. EL-speech-to-normal-speech conversion (EL2SP) with seq2seq VC can fix the duration mismatch that frame-aligned statistical VC cannot, but it needs parallel data, and patient recordings are tiny (minutes per speaker). Prior seq2seq EL2SP systems from the same group rely on TTS pretraining followed by fine-tuning (following Yen et al., ASRU 2021) and on TTS-generated synthetic data (SD) with two-stage fine-tuning (Ma et al., IEEE TASLP 2025). Both still use speech-only representations, and the authors argue that the large acoustic gap between EL and normal speech makes the encoder’s intermediate representations noisy, so errors accumulate in the decoder. Earlier ways of using text in VC (Cotatron, joint TTS/VC training with multi-source Tacotron) either need text at inference or add model and training complexity, and none had been tested on EL speech.
Method
All models are Transformer encoder-decoders built in ESPnet (6 encoder and 6 decoder layers, 4 heads, attention dimension 384, feed-forward 1,536, reduction factor 3), operating on 80-dimensional mel-filterbanks at 24 kHz. Waveforms come from speaker-dependent Parallel WaveGAN vocoders trained with the default recipe (eight vocoders across the datasets). Training runs in three parts.
Part 1, pretrained modules. A Transformer TTS model is pretrained on JSUT (single-speaker Japanese, about 10 h). Two branches come from it: (a) a one-to-one seq2seq VC model in the style of the Voice Transformer Network, obtained by freezing the TTS decoder and training a new speech encoder to reconstruct JSUT speech so it learns to produce text-like linguistic representations from audio; (b) a target-normal TTS model fine-tuned on the target normal speech and transcripts of each EL2SP dataset.
Part 2, joint speech-text network. The speech encoder from (a) and the frozen text encoder plus speech decoder from (b) form a joint network trained on EL speech and its phoneme text as dual inputs, with target normal speech as output. Three fusion strategies use cross-attention (speech as query, text encodings as key and value) with residual connections: middle-level fusion (MF) applies it after the speech encoder; input-level fusion (IF) applies it to the input feature sequence before the speech encoder; hybrid-level fusion (HF) does both.
Part 3, reconstruction training. The text encoder is removed and the decoder is frozen. Only the speech encoder is updated, from EL speech alone, so that its output matches what the frozen decoder learned to decode in Part 2. The loss is the standard seq2seq VC objective (L1 on mel outputs, weighted BCE on the stop token, guided attention loss), optionally extended with an L1 reconstruction loss between the speech-only encoder output and the Part 2 fused representations, weighted by λ = 0.01. This is effectively a teacher-student transfer of the fused representation into a speech-only encoder (mapped here to distillation; the authors call it “autoencoder-style reconstruction”). The final model is the same 30.4M-parameter seq2seq VC network as the baselines and needs no text at inference.

The systems form a 3x3 grid: each fusion strategy in a “-1” variant (original EL2SP data only), a “-2” variant (adds 7,296 TTS-generated parallel SD utterances plus text in Part 2, with Part 3 run on original data only) and a “-3” variant (“-2” plus the extra reconstruction loss). Baseline 1 is plain pretraining plus fine-tuning; Baseline 2 is the SD-based two-stage fine-tuning of the authors’ TASLP 2025 system. All systems share the same seq2seq VC pretraining.
Key Results
Evaluation covers five small in-house Japanese datasets: three from real laryngectomees (Patient-1, -2, -3, roughly 10 to 30 minutes of EL speech each) and two with simulated EL speech from healthy speakers (Pseudo-Patient-1/2). Each test set has 40 utterances. Objective metrics are MCD, CER from a pretrained ASR model, log-F0 RMSE and log-F0 correlation (Tables II to VI).
- Best proposed system (P-HF-3, hybrid fusion + SD + reconstruction loss). It has the lowest MCD and lowest CER of all eleven systems on all five datasets. Examples: on Patient-1, MCD 5.74 and CER 18.4 versus 6.24 and 23.3 for Baseline 2 (Table II); on Patient-2, 6.09 and 19.7 versus 6.38 and 26.2 (Table III); on Patient-3, CER 27.0 versus 37.0 (Table IV). The margin over P-HF-2 on MCD is small (5.74 vs 5.75 on Patient-1, 7.20 vs 7.21 on Patient-3).
- Pitch metrics are mixed. P-HF-3 is alone at the top on F0 CORR for Patient-1 (0.73) and Patient-3 (0.92), and on F0 RMSE for Patient-2 (0.38) and Pseudo-Patient-1 (0.22). It ties other systems on F0 RMSE for Patient-1 (0.22) and Patient-3 (0.32). It trails on F0 CORR for Patient-2 (0.87 vs 0.88 for P-HF-2), Pseudo-Patient-1 (0.71 vs 0.73 for four other systems) and Pseudo-Patient-2 (0.82 vs 0.83), and on F0 RMSE for Pseudo-Patient-2 (0.37 vs 0.36). The authors say text integration favours linguistic content at a small cost in pitch reconstruction. Their own count is 16 of 20 comparable results “best”, which only works if ties are counted.
- Speech-only data budget. With only the original data, the “-1” systems beat Baseline 1 almost everywhere, but Baseline 2 (which adds SD) usually beats them. Baseline 2 has lower CER than every “-1” system on Patient-1, -2, -3 and Pseudo-Patient-1, and lower MCD on Patient-2, Patient-3 and Pseudo-Patient-1. The “-1” systems match or beat Baseline 2 on MCD and F0 RMSE for Patient-1. On Pseudo-Patient-2 all three are ahead on MCD and F0 CORR, two of three are ahead on CER (P-IF-1 and P-HF-1), and they match or beat it on F0 RMSE. Text integration helps, but it does not replace the extra data.
- Fusion hierarchy. Within matched variants, input-level fusion generally beats middle-level fusion, and hybrid fusion generally beats input-level fusion, mainly on MCD and CER. The reconstruction loss (“-3”) adds small further gains over “-2”.
- Subjective naturalness MOS (20 native Japanese listeners, 15 samples per system, Patient-1/2 only, Fig. 5): P-HF-2 scores 3.54 and 3.34, P-IF-2 3.43 and 3.21, P-MF-2 3.07 and 3.10, Baseline 2 2.90 and 2.98, Baseline 1 2.63 and 2.19. For reference, raw EL speech scores 1.09 and 1.07 and target normal speech 4.53 and 4.05. All three proposed systems are ahead of both baselines, but the P-MF-2 versus Baseline 2 gap is small on Patient-2 (3.10 vs 2.98). The “-3” systems were not included in the listening tests.
- Subjective intelligibility MOS (Fig. 6) was run only for P-HF-2 against the anchors: 3.84 vs 1.97 for EL speech on Patient-1 and 3.75 vs 1.80 on Patient-2, against normal-speech ceilings of 4.75 and 4.47. Baselines were not rated, so this test shows improvement over the raw input, not over prior systems.
- Sensitivity (Table VII, P-MF-3 on Patient-1): λ = 0.01 and 4 heads give the best MCD (6.00) and CER (20.7). λ = 0.001 gives a slightly worse CER (23.1) than no auxiliary loss (22.1). All settings fall within 0.04 dB MCD of each other.
- Speed: about 0.91 s for a 5-second utterance on one GPU (RTF ≈ 0.18, of which 0.17 is the autoregressive seq2seq model).
All comparisons are against the authors’ own baselines with the same pretraining, so they are well controlled. However, no external EL2SP or VC system is compared, and statistical significance is reported only for Patient-1 and Patient-2.
Novelty Assessment
The contribution is a training procedure, not a new inference architecture. Text is injected through cross-attention into a joint network during training, and the result is then transferred back into a speech-only encoder with a frozen decoder and a feature-matching loss, so the deployed model is unchanged. The ingredients (Voice Transformer Network-style TTS pretraining, cross-attention fusion, synthetic parallel data, freezing the decoder while training the encoder) are each known. The novelty lies in combining them so that text helps EL2SP without being needed at inference, and in systematically comparing three fusion positions. The method extends the authors’ EMBC 2025 conference version (“Improving Electrolaryngeal Speech Enhancement via a Representation Learning Method Based on Integrated Text and Speech Representations”), adding input- and hybrid-level fusion, the auxiliary reconstruction loss and more datasets. Gains over the strongest baseline are consistent but moderate, and most of the improvement over the weaker baseline comes from synthetic data rather than from text integration.
Field Significance
moderate: The paper shows that transcripts available at training time can improve a speech-only seq2seq VC model in a severe, very low-resource source-domain mismatch, with no added inference cost, and it shows an ordering of fusion positions (input-level fusion ahead of middle-level, hybrid ahead of both). Evidence is limited to Japanese, one model family (autoregressive Transformer seq2seq VC with mel-spectrograms) and small in-house datasets. It is best read as a careful clinical-VC engineering result and a reusable “train with text, infer without it” recipe, not a general advance in VC modelling.
Claims
- supports: Text transcripts used only during training can improve speech-only sequence-to-sequence voice conversion under severe source-target acoustic mismatch, without adding inference inputs or parameters.
Evidence: After training with joint speech-text fusion and transferring to a speech-only encoder, the strongest final 30.4M-parameter EL2SP model (P-HF-3) beats the two-stage synthetic-data baseline of the same size on MCD and CER across all five datasets (Patient-1 CER 18.4 vs 23.3; Patient-3 CER 27.0 vs 37.0). In the listening test, which covered the “-2” systems, P-HF-2 also gets higher naturalness MOS than that baseline on both rated patient datasets (3.54 vs 2.90, 3.34 vs 2.98). (§IV-A, Tables II–VI; §IV-B, Fig. 5)
- refines: Where auxiliary text is fused relative to the speech encoder matters: early fusion that lets the encoder internalise text-aligned cues transfers better to a speech-only model than fusing at the encoder output.
Evidence: Within matched data and loss settings, input-level fusion generally beats middle-level fusion and hybrid fusion beats both on MCD and CER. Naturalness MOS rises from P-MF-2 (3.07) to P-IF-2 (3.43) to P-HF-2 (3.54) on Patient-1. (§IV-A2, Tables II–VI; §IV-B, Fig. 5)
- complicates: In data-starved voice conversion, extra synthetic parallel data contributes more to quality than a better training-time representation does.
Evidence: Without synthetic data, the text-integrated “-1” systems usually trail Baseline 2 (speech-only plus synthetic data) on CER, for example Patient-3 CER 40.8 to 46.5 vs 37.0. Adding synthetic data (“-2”) gives large gains, significant in 19 of 24 comparisons on Patient-1/2. (§IV-A1, §IV-A2, Tables II–VI)
- complicates: Adding linguistic supervision to conversion models can improve intelligibility and spectral fidelity while leaving pitch-contour reconstruction unchanged or slightly worse.
Evidence: The best-overall system leads on MCD and CER on every dataset but trails other systems on log-F0 correlation for Patient-2, Pseudo-Patient-1 and Pseudo-Patient-2. The authors report that significance is clearer for MCD and CER than for pitch metrics. (§IV-A2, Tables III, V, VI)
Limitations and Open Questions
Small, single-language, in-house evaluation
All five datasets are in-house Japanese corpora with about 10 to 30 minutes of EL speech and 40 test utterances each, and two of them use simulated EL speech from healthy speakers. No external EL2SP or modern VC baseline is compared. Results may not carry over to other languages, other electrolarynx devices or larger patient populations.
- The listening tests cover only the “-2” systems on two datasets. The best objective system (P-HF-3) has no subjective scores, and intelligibility MOS was not collected for the baselines.
- Training needs transcripts of the patient’s EL recordings and a target-normal TTS model per dataset. That burden is small for read speech but matters for real clinical recording.
- The model is autoregressive and non-causal, so conversion starts only after the full utterance. The reported RTF of about 0.18 is not interaction latency, and streaming EL2SP is left to future work.
- Target speech is a healthy stand-in speaker for Patient-1 but the patient’s own preoperative voice for Patient-2/3, so speaker identity preservation is not evaluated (no speaker-similarity metric).
- CER relies on a single pretrained ASR model that is not specified in detail. Pitch metrics move little, and the gains there are often not significant.
- Code is not mentioned; audio samples are online.
Wiki Connections
- Voice Conversion — A dedicated seq2seq VC system for electrolaryngeal-to-normal speech, evaluated with MCD, CER, log-F0 metrics and listener MOS.
- Transformer Encoder-Decoder TTS — Trains a Transformer TTS model whose text encoder and speech decoder seed the joint network, and builds the VC model on the same TTS-style Transformer encoder-decoder with TTS pretraining.
- Subjective Evaluation — Runs separate naturalness and intelligibility MOS tests with 20 native Japanese listeners, using raw EL speech and target normal speech as lower and upper anchors.
- Conan — Cited only as an example of the chunkwise streaming VC direction the authors name as future work for low-latency EL2SP. It is not used or compared.