arXiv · 2025 · Preprint
Wang et al. · → Paper · Demo: ? · Code: ?
Fed-PISA disentangles speaker timbre from expressive style via private per-client ID-LoRA adapters and shared federated Style-LoRA adapters, using a collaborative filtering-inspired aggregation strategy to enable privacy-preserving cross-client style learning for voice cloning with substantially lower communication cost than prior federated TTS systems. Accepted at ICASSP 2026.
Problem
Personalised voice cloning requires capturing both a speaker’s vocal timbre and their expressive style, but each client’s local data is typically too sparse to learn rich stylistic variation in isolation. Federated learning allows clients to collaborate without sharing raw audio, yet prior federated TTS systems (FedSpeech, Federated Dynamic Transformer) handle this poorly on two fronts: they incur large communication costs by transmitting full or near-full model parameter sets, and they suppress stylistic heterogeneity across clients in order to preserve individual speaker timbre, leaving the collaborative benefit largely unrealised. The core tension is that speaker identity and expressive style are competing optimisation objectives when trained through a single shared adapter.
Method
Fed-PISA builds on GPT-SOVITS-V4 as a frozen backbone, a two-stage hybrid system comprising a GPT-style autoregressive semantic predictor (modelling discrete acoustic/semantic unit sequences conditioned on text and reference audio) followed by a SoVITS Conditional Flow Matching (CFM) decoder that renders waveforms from those units. The total backbone size is 410M parameters.
The framework introduces two distinct LoRA adapters (rank 8, scaling factor 16), each with a separate role. The ID-LoRA is a private, client-specific adapter trained once on neutral speech samples from a single client, using cosine similarity between predicted and reference speaker embeddings as the objective. It captures vocal timbre and channel coloration and is permanently frozen after local training, never transmitted to the server. The Style-LoRA is a federated adapter trained on expressive speech samples via token-level cross-entropy, with gradients blocked from flowing into the ID-LoRA. Only Style-LoRA weights are uploaded to the server each round.
On the server side, rather than averaging Style-LoRA updates uniformly (as in FedAvg), the aggregation is personalized using attention scores derived from pairwise cosine similarity between clients’ LoRA matrices, scaled by a temperature of 0.5. Separate attention scores are computed for the A and B matrices of the LoRA decomposition to stabilise the low-rank space. Each client receives a custom aggregated style model that weights updates from stylistically similar peers most heavily, an approach inspired by collaborative filtering in recommendation systems.
Training runs for 50 communication rounds with 20% client participation per round. Each round, selected clients first perform 80 steps of timbre cloning (updating ID-LoRA) and then 20 steps of stylization (updating Style-LoRA). The 80/20 split is validated on a held-out set to balance speaker identity preservation against expressive style learning. Four datasets with emotion annotations are pooled and mapped to 10 unified style categories using an emotion2vec classifier.
Key Results
FED-PISA achieves nMOS 4.08, WER 2.7%, speaker similarity (SS) 0.645, and style expressivity (SE) 0.704 averaged across ESD, EmoV-DB, RAVDESS, and CREMA-D test splits (Table 2). Against the zero-shot in-context learning baselines, fine-tuning provides large gains: COSYVOICE2 zero-shot achieves SS 0.619 and nMOS 3.84, while GPT-SOVITS-V4 zero-shot gives SS 0.464 and nMOS 3.39.
Against federated baselines, the gains are substantial. FedSpeech (FastSpeech2-XL backbone) achieves SE 0.416 and SS 0.556 at a communication cost of 145.28 GiB; Federated Dynamic Transformer achieves SE 0.463 and SS 0.602 at 456.35 GiB. FED-PISA reaches SE 0.704 and SS 0.645 at 45.8 GiB, a 3-fold communication reduction versus FedSpeech and a 10-fold reduction versus Federated Dynamic Transformer. These prior systems also use substantially weaker backbones than GPT-SOVITS-V4, which limits the directness of comparison.
Ablations confirm the necessity of both components. Removing ID-LoRA drops SS from 0.645 to 0.507; removing Style-LoRA drops SE from 0.704 to 0.588. Replacing the personalized aggregation with FedAvg (under the same LoRA/backbone configuration) degrades SE to 0.476 and SS to 0.523, demonstrating that naive averaging causes a style-homogenisation effect.
Novelty Assessment
The disentangled LoRA design, splitting speaker identity from style into separately optimised, separately routed adapters, is a genuinely novel structural choice for federated TTS, rather than a straightforward application of existing components. The collaborative filtering-inspired personalized aggregation is also new in this context, providing a principled alternative to FedAvg when clients exhibit heterogeneous styles. However, all the building blocks (LoRA, GPT-SOVITS-V4, collaborative filtering, federated optimisation) are individually established; the contribution is in recognising that this particular decomposition solves a known failure mode of federated TTS. The backbone comparison limitation is real: FedSpeech and Federated Dynamic Transformer run on older, weaker architectures, so the observed gains partially reflect modern backbone quality rather than the federation strategy alone.
Field Significance
Moderate - Fed-PISA addresses a practical gap in federated TTS by showing that disentangling timbre and style into separate adapter modules allows collaborative style learning without degrading speaker identity, while meaningfully reducing communication overhead. The personalized aggregation mechanism can serve as a template for style-heterogeneous federated learning beyond voice cloning. The field is niche (federated TTS has only a handful of prior systems), so direct influence depends on whether federated deployment scenarios gain traction in privacy-sensitive applications.
Claims
-
supports: Disentangling speaker identity from expressive style into separate adapter modules improves both speaker similarity and style expressivity in personalised voice cloning.
Evidence: Ablations on GPT-SOVITS-V4 show removing the private ID-LoRA drops SS from 0.645 to 0.507, while removing the Style-LoRA drops SE from 0.704 to 0.588; a single unified adapter cannot optimise both objectives without mutual degradation. (§3.3, Table 2)
-
supports: Collaborative style learning across clients in a federated setting can overcome per-client data scarcity and produce higher speaker similarity than purely local fine-tuning.
Evidence: Fed-PISA exceeds Local FT (LoRA) in both SS (0.645 vs 0.529) and nMOS (4.08 vs 3.36), demonstrating that sharing expressive style information across clients compensates for limited per-client data. (§3.2, Table 2)
-
supports: Personalized aggregation weighted by stylistic similarity between clients outperforms uniform averaging for federated style learning.
Evidence: Under identical GPT-SOVITS-V4 backbone and LoRA configuration, attention-based personalized aggregation achieves SE 0.704 and SS 0.645 versus FedAvg’s SE 0.476 and SS 0.523, confirming that style-averaged updates homogenise client models and harm expressivity. (§3.3, Table 2)
-
complicates: In speaker adaptation, stylisation steps beyond a small fraction of total training degrade speaker identity and peak naturalness, establishing a ceiling on achievable style transfer without identity loss.
Evidence: Holding total steps at 100, speaker similarity decreases monotonically as stylisation steps increase, while naturalness peaks at 20% stylisation (m=20); excessive stylisation corrupts identity and degrades nMOS. (§3.4, Figure 2)
-
complicates: Fair cross-framework comparison in federated TTS is obstructed when competing methods are architecturally coupled to legacy backbones and cannot be reimplemented on modern foundations.
Evidence: FedSpeech and Federated Dynamic Transformer use FastSpeech2-XL and Transformer-TTS respectively and are incompatible with LoRA; the evaluation must compare them on their original configurations, making it impossible to isolate the contribution of the federation strategy from the backbone quality. (§3.1.2)
Limitations and Open Questions
Warning
The federated baselines (FedSpeech, Federated Dynamic Transformer) run on architecturally distinct and weaker backbones; the performance gap reflects backbone modernization in addition to the proposed federation strategy. No ablation on GPT-SOVITS-V4 with a standard FedAvg baseline using the same LoRA parameter budget is reported to cleanly isolate the aggregation contribution.
Evaluation uses 60 speakers treated as 60 independent clients from four relatively small emotion datasets, which may not represent the statistical diversity or scale of real-world federated deployments. Naturalness is evaluated by 22 in-lab researchers rather than crowd-sourced listeners, which may limit generalisability of the nMOS scores. The paper does not address communication round latency or asynchronous client availability, both of which matter in practical on-device deployment. How the framework scales to many more clients or to cross-lingual speaker sets remains untested.
Wiki Connections
- Voice Conversion - Fed-PISA frames voice cloning as a federated VC/TTS task, targeting both speaker identity transfer and stylistic expressivity from limited per-speaker data.
- Speaker Adaptation - The private ID-LoRA component provides a parameter-efficient mechanism for adapting a shared backbone to individual speaker timbre in a privacy-preserving setting.
- Disentanglement - Fed-PISA explicitly separates speaker identity and expressive style into distinct LoRA modules trained with non-overlapping objectives, providing a federated instantiation of attribute disentanglement.
- Emotion Synthesis - Style expressivity (SE), measured by an emotion2vec classifier, is a primary evaluation target; the stylisation training phase optimises the Style-LoRA on emotionally labelled expressive speech.
- Subjective Evaluation - Naturalness is assessed through a blind nMOS listening test conducted with 22 in-laboratory raters across all four evaluation datasets.