arXiv · 2026 · Preprint
Aoduo Li et al. (Guangdong University of Technology / South China University of Technology / University of Macau / Huizhou University) · → Paper · Demo: ✗ · Code: ✗
Distills the persona-aware emotional reasoning of a 14B LLM into an 11.8M-parameter adapter that predicts pitch, energy, duration, and pause targets to steer a frozen GPT-SoVITS v4 backbone for character-consistent anime voice synthesis. The paper reports acceptance to ACM ICMR 2026.
Problem
Character voice applications (anime avatars, VTubers, game companions) need speech that is both emotionally expressive and consistent with a specific character’s persona. The authors argue that existing TTS either produces natural but emotionally flat speech or expressive speech that drifts away from the character, and that emotional TTS treats emotions as isolated labels rather than as expressions shaped by personality (the same “angry” line should sound different for a reserved and an impulsive character). They frame three obstacles: a semantic-acoustic gap between textual persona descriptions and prosody, emotion-persona entanglement, and the cost of running a billion-parameter LLM at synthesis time.
Method
ATRIE has an offline distillation phase and an online inference phase. Offline, Qwen 2.5 14B acts as a “Teacher Persona-LLM”: given input text and a persona configuration (2-3 personality adjectives, a speech-pattern register, and an emotional volatility score authored by experts), it emits a chain-of-thought rationale plus structured JSON targets for valence, arousal, dominance (VAD) and relative F0 and energy. The rationale is embedded with a frozen Sentence-BERT into a 768-d vector.
A student “P2P” (persona-to-prosody) adapter (4 Transformer layers, hidden size 512, 8 heads, 11.8M parameters) uses cross-attention to align semantic tokens with phoneme-level frames and has four parallel heads predicting pitch, energy, duration, and pause. It is trained with an MSE loss to the teacher’s prosody targets, a semantic alignment loss to the rationale embedding (λ_sem = 0.5), and an InfoNCE-style contrastive persona loss (λ_con = 0.3, τ = 0.07) that pulls generated prosody embeddings toward the character’s anchor embedding and away from other characters. Training uses AdamW at 1e-4 with cosine annealing for 100 epochs.
The paper describes the conditioning path as a “Persona-Prosody Dual-Track” (P2-DT) design: a Timbre Track takes a global embedding from a pretrained speaker verification model and scalar-quantizes it with a 512-entry codebook as a time-invariant identity anchor, while a Prosody Track uses a hierarchical flow-matching predictor (8 flow steps, CFG scale 2.0) to produce time-varying pitch, energy, and rhythm conditioned on the persona context. Both are fused at the variance adaptor of the backbone. The backbone is GPT-SoVITS v4 (official v4.0.1 release), which combines a 300M-parameter autoregressive semantic token predictor, a 200M-parameter VITS-based acoustic decoder described as using conditional flow matching for mel generation, and a CNHuBERT reference encoder; it was pretrained on roughly 1,000 hours of Chinese audiobook data. The prosody scalars are injected after the duration predictor and before the flow decoder, so backbone weights are not modified. At inference, a reference clip is selected from a 2,154-clip emotion-labelled library by nearest VAD distance to the predicted emotional target.
Key Results
On AnimeTTS-Bench (Table 3), ATRIE reports UTMOS 4.28, character consistency score (CCS, ECAPA-TDNN cosine similarity) 0.86, emotional expression accuracy (EEA, emotion2vec classification accuracy) 0.84, MCD 4.10, and RTF 0.18. The strongest baseline, CosyVoice 2 retrained on the same data, has higher naturalness (UTMOS 4.38) and lower MCD (3.90) but lower CCS (0.76) and EEA (0.72), with RTF 0.65. FastSpeech 2, VITS, and VALL-E trail on every quality metric. The gain is therefore on persona-centric metrics that the authors define, at a small cost in automatic naturalness relative to the best baseline.
Cross-character results (Table 4) give CCS 0.82 and 0.84 and EEA 0.79 and 0.81 for the two secondary characters, and per-emotion results (Table 5) show the lowest EEA on “Tsundere” (0.76) and the highest F0-RMSE on that category (88.2). On an extended 50-character benchmark (30 seen, 20 unseen), the paper reports speaker verification EER 0.04, and text-to-persona retrieval on the unseen characters reaches mAP 0.75 and R@1 0.62, against 0.55 mAP for LAION-CLAP and 0.52 for MuLan (Table 6).
Ablations (Table 7) show that replacing the LLM teacher with a VAD regressor lowers CCS from 0.86 to 0.80 and EEA from 0.84 to 0.70; removing chain-of-thought lowers EEA to 0.72; removing the contrastive loss lowers CCS to 0.79; random reference selection drops CCS to 0.76 and EEA to 0.62; and shuffling persona configurations gives the largest drop (CCS 0.71, EEA 0.58). Latency on an RTX 4090 (FP16) is 180 ms per second of audio (RTF 0.18), and 0.25 on an RTX 3090. A user study with 15 participants experienced in anime voice evaluation, rating 20 samples (10 baseline, 10 ATRIE) on a 1-5 scale, gives Voice Consistency 4.2 vs. 3.4 (p < 0.01) and Emotional Authenticity 4.0 vs. 2.9 (p < 0.001), with 78% overall preference for ATRIE.
Several reporting details conflict within the paper: CCS is defined as mean ECAPA-TDNN cosine similarity in §4.2 but as 1 − EER in §4.4 (where EER 0.04 would imply CCS 0.96, not the reported 0.86). The base AnimeTTS-Bench is described as 2,154 utterances from 3 characters (§4.1.1), while the retrieval gallery is 2,154 clips from 50 characters (§4.6). Characters B and C are part of the base 3-character dataset with character-stratified splits, yet Table 4 presents them as zero-shot unseen characters. The GPT-SoVITS v4 baseline that the per-character and per-emotion Δ CCS columns are measured against is never reported numerically.
Novelty Assessment
The pieces are established: GPT-SoVITS supplies the synthesis, an off-the-shelf LLM supplies emotion labels and prosody targets, and the adapter is a small Transformer with variance-style prediction heads and a standard contrastive loss. The new element is the recipe: using an LLM’s chain-of-thought about a character persona as a distillation target for explicit prosody controls, so the LLM is not needed at synthesis time, combined with VAD-based reference-clip selection. This is mainly engineering integration with a modest architectural component (the adapter and the flow-matching prosody predictor). The ablations support each part of the recipe, but every comparison uses the authors’ own benchmark and their own CCS and EEA metrics. Baselines are retrained on a small single-domain corpus, and the external-system comparison uses only 20 samples. The retrieval experiment reuses the same embedding space for a cross-modal persona retrieval task, which is mostly an argument for the venue rather than a speech-generation contribution.
Field Significance
Low — The paper provides one data point that LLM-derived, persona-aware prosody targets distilled into a lightweight adapter can raise emotion classifier accuracy and speaker-embedding consistency for a frozen expressive TTS backbone. Its evidence comes from an in-house anime benchmark with no released code or demo and several internally inconsistent descriptions of metrics, splits, and figures, so it is weak confirmatory evidence rather than a reliable reference point.
Claims
- supports: Distilling an LLM’s reasoning about character persona and context into explicit prosody targets can improve emotional expressiveness of a frozen TTS backbone without running the LLM at synthesis time.
Evidence: Replacing the 14B LLM teacher with a plain VAD regressor drops EEA from 0.84 to 0.70 and CCS from 0.86 to 0.80; a 7B teacher sits in between (EEA 0.78); the 11.8M adapter runs at RTF 0.18 on an RTX 4090. (§5, Table 7; §4.5)
- supports: Prompting the teacher LLM to reason step by step before emitting emotion targets yields better emotion targets than direct emotion prediction.
Evidence: Removing chain-of-thought prompting lowers EEA from 0.84 to 0.72 and raises F0-RMSE from 62.1 to 75.8. (§5, Table 7)
- supports: A contrastive loss that separates characters in prosody-embedding space helps stop speaker identity from drifting as emotional expressiveness increases.
Evidence: Removing the contrastive persona loss lowers CCS from 0.86 to 0.79, the largest drop among the control-mechanism ablations, while EEA falls less (0.84 to 0.80). (§3.3; §5, Table 7)
- complicates: Gains in persona consistency and emotion accuracy from explicit prosody control can come with lower automatic naturalness than strong LLM-based TTS.
Evidence: ATRIE leads on CCS (0.86 vs. 0.76) and EEA (0.84 vs. 0.72) but trails retrained CosyVoice 2 on UTMOS (4.28 vs. 4.38) and MCD (4.10 vs. 3.90). (§4.3.1, Table 3)
- supports: In reference-prompted expressive TTS, how the reference clip is chosen strongly affects both identity consistency and emotional accuracy.
Evidence: Random reference selection drops CCS to 0.76, EEA to 0.62, and raises F0-RMSE to 105.2. The paper also reports that about 8% of high-arousal samples get an inappropriate reference when emotions share similar VAD profiles (e.g., excited vs. angry). (§5, Table 7; §6.3)
Limitations and Open Questions
Warning
All results come from the authors’ own AnimeTTS-Bench, with no code, model, demo, or dataset link, and with persona metrics (CCS, EEA) defined by the authors. Descriptions of the metrics, splits, and figures contradict each other (see the note under Key Results), so the reported numbers cannot be independently reproduced or reliably compared with other work.
- The role of the 14B LLM at inference is unclear. §1.2 and §4.5 say it is used only offline, but Algorithm 1 calls the Persona-LLM at inference, and §6.2 lists the LLM’s ~500 ms first-token latency as a limitation for interactive use. The streaming claim in §3.8 is not backed by any latency-to-first-audio measurement.
- The image shipped as Figure 1 depicts a retrieval system (query encoder, ANN index, re-ranking) rather than the distillation pipeline its caption describes; no Teacher LLM, P2P adapter, or GPT-SoVITS backbone appears in it.
- The text claims “Emotion Prompting has the largest impact (-4.7% CCS)” among control components, but no variant with that name appears in Table 7. The -4.7% figure belongs to “Only Latent (no Prosody)”, and the contrastive-loss ablation drops CCS more (-8.1%). §5.1 attributes a hyperparameter sensitivity analysis to Figure 6, which is the ablation heatmap.
- LLM-generated VAD targets are not validated against human annotations; validation relies on downstream classifier scores and on prompt-stability correlations (Pearson > 0.92).
- Evaluation is limited to Japanese anime character speech, and the backbone is pretrained on Chinese audiobook data. The authors note that other languages would need language-specific prosodic modeling.
- The user study is small (15 raters, 20 samples) and compares only against an unspecified baseline.
- Quality depends on reference-library coverage; characters with little voice data may not reach the same consistency, and long sentences (over 50 characters) can lose consistent emotional intensity with a single reference.
Wiki Connections
- Emotion Synthesis — Generates persona-dependent emotional speech and evaluates it with an emotion2vec classifier and human emotional-authenticity ratings.
- Prosody Control — Predicts explicit pitch, energy, duration, and pause targets that are injected into the backbone’s variance adaptor.
- Zero-Shot TTS — Claims generalization to 20 unseen characters using reference-clip prompting with a frozen backbone, though the split descriptions are inconsistent.
- Flow Matching — Uses a hierarchical flow-matching predictor (8 steps) for the time-varying prosody track.
- Subjective Evaluation — Includes a 15-rater study of voice consistency and emotional authenticity on a 1-5 scale.
- CosyVoice 2 — Not cited in the bibliography but used as the strongest baseline (the bibliography entry [8] points to CosyVoice); it beats ATRIE on UTMOS and MCD but trails on CCS, EEA, and RTF.
- VALL-E — Retrained neural codec LM baseline that ATRIE outperforms on all reported quality metrics and on RTF.
- FastSpeech 2 — Retrained non-autoregressive baseline; fastest on RTF but lowest on UTMOS, CCS, and EEA.
- P2VA — Closest prior persona-to-voice work, which the authors contrast as needing per-character fine-tuning (the bibliography entry is garbled but names P2VA).
- emotion2vec — Pretrained emotion classifier used to compute the EEA metric.
- ECAPA-TDNN — VoxCeleb-pretrained speaker encoder used to compute CCS and the speaker verification EER.
- NaturalSpeech 3 — Factorized-codec approach that the authors contrast as relying on reference audio rather than semantic persona descriptions.