Document type
Master thesisPublication date
Publication license
Please use this identifier to cite or link to this item: https://hdl.handle.net/2445/186003
Machine learning copies as a means for black box model evaluation
Journal Title
Authors
Director/Tutor
Journal ISSN
Volume Title
Related resource
Abstract
[en] The use of propietary black-box machine learning models and APIs in the form of Machine Learning as a Service, makes it very difficult to control and mitigate their potential harmful effects (such as lack of transparency, privacy safeguards, robustness, reusability or fairness). The state-of-the-art technique of Machine Learning Classifier Copying allows us to build a new model that replicates the decision behaviour of an existing one without the need of knowing its architecture nor having access to the original training data.
PRESC (Performance and Robustness Evaluation for Statistical Classifiers) is an existing free software tool for the evaluation of machine learning classifiers, which is maintained by Mozilla’s Data Science team. It aims to provide a toolkit to analyze model performance beyond the standard accuracy-based methods and into areas
which tend to be underexplored in data science practice.
Among the multiple applications of Machine Learning Classifier Copying, a systematic construction and examination of model copies has the potential to be an universally accessible and inexpensive approach to study and evaluate a rich variety of original models, and to help understand its behavior. In this work, an implementation of Machine Learning Classifier Copying has been contributed to the PRESC project, so that this tool becomes readily accessible to researchers and practitioners, and its applicability and performance in a synthetic problem has been explored to understand the copying process. The solution provides a model agnostic sampling strategy and an automated copy process for a number of fundamentally different
hypothesis spaces, so that the set of achievable copy-model-fidelity measures can be used as a diagnostic measure of the original model characteristics.
Description
Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2020-2021. Tutor: David Zeber i Oriol Pujol Vila
Subject (English)
Citation
Citation
ROVIRA ESTEVA, Muriel. Machine learning copies as a means for black box model evaluation. [consulted: 17 of August of 2026]. Available at: https://hdl.handle.net/2445/186003