Domain Sensitive Federated Learning with Fisher-Informed Pruning

Chenchen Lin, Wenhao Yuan, Zhengji Xu, Xuehe Wang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 17535-17544

Federated Learning (FL) serves as a prominent distributed training paradigm, enabling devices to collaboratively train a shared model with local data. However, in practice, clients generally possess data from multiple distributional domains, posing significant challenges to efficiency and generalization. In this paper, we propose a domain-sensitive federated pruning framework, FedFIP, that preserves domain-invariant structures while retaining domain-specific representations. The Domain-Sensitive Fisher Pruning (DSFP) module estimates channel importance per domain via Fisher information, and uploads this signal to the server to obtain a globally shared pruning mask. Given local domain heterogeneity, each client reuses its Fisher information to selectively reactivate domain-specific channels, yielding personalized sparse models that remain structurally aligned yet adapt to local heterogeneity. To further enhance performance, we adopt a Domain-Sensitive Regularization (DSR) module: the server builds domain prototypes from uploaded importance signals and broadcasts them back. Guided by the domain prototypes, we adopt a structure-contrastive loss to strengthen intra-domain consistency and inter-domain discriminability. We propose a structure-aware aggregation algorithm that fuses heterogeneous personalized architectures into a domain-generalized global model. Experiments on multi-domain benchmarks demonstrate that FedFIP surpasses state-of-the-art FL baselines while substantially shrinking model size.