Differentially Private Federated Learning With Local Regularization and Sparsification

Anda Cheng, Peisong Wang, Xi Sheryl Zhang, Jian Cheng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10122-10131

Abstract


User-level differential privacy (DP) provides certifiable privacy guarantees to the information that is specific to any user's data in federated learning. Existing methods that ensure user-level DP come at the cost of severe accuracy decrease. In this paper, we study the cause of model performance degradation in federated learning with user-level DP guarantee. We find the key to solving this issue is to naturally restrict the norm of local updates before executing operations that guarantee DP. To this end, we propose two techniques, Bounded Local Update Regularization and Local Update Sparsification, to increase model quality without sacrificing privacy. We provide theoretical analysis on the convergence of our framework and give rigorous privacy guarantees. Extensive experiments show that our framework significantly improves the privacy-utility trade-off over the state-of-the-arts for federated learning with user-level DP guarantee.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Cheng_2022_CVPR, author = {Cheng, Anda and Wang, Peisong and Zhang, Xi Sheryl and Cheng, Jian}, title = {Differentially Private Federated Learning With Local Regularization and Sparsification}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {10122-10131} }