Differentially Private Machine Learning: Implementation and Analysis of Gradient and Dataset Perturbation Techniques

Abstract

The increasing use of machine learning poses significant privacy risks, especially when sensitive data is used, and conventional anonymization methods have proven insufficient. Differential privacy is a rigorous framework for data privacy providing strong mathematical guarantees. The possibility of applying this framework to machine learning solves the privacy problem. We will present the fundamental basis of these concepts to empirically investigate, implement, and analyse two techniques for integrating differential privacy into machine learning pipelines. The first technique, dataset perturbation, involves adding calibrated Gaussian noise directly to the training data and then using any standard machine learning pipeline. The second, gradient perturbation, centers on differentially private stochastic gradient descent, is an approach that injects noise into the gradients during the training phase. For the comparative study, we developed a multi-class classification architecture using a real-world, sensitive medical dataset derived from the MIMIC-IV database. Model performance was evaluated against a non-private baseline, using the appropriate metrics considering our class imbalance, such as Macro F1-score and Macro OVO AUC. The results confirm the trade-off between privacy and utility in the models developed, where higher privacy guarantees consistently result in reduced model utility. For the specific context of this study, gradient perturbation provided a slightly more advantageous model in overall balance of utility and privacy. Ultimately, the thesis provides strong evidence for the feasibility of training useful and formally private machine learning models on real-world medical data, successfully demonstrating a practical "sweet spot" between privacy and performance can be found.