Please use this identifier to cite or link to this item:
http://hdl.handle.net/2445/186003
Title: | Machine learning copies as a means for black box model evaluation |
Author: | Rovira Esteva, Muriel |
Director/Tutor: | Zeber, David Pujol Vila, Oriol |
Keywords: | Aprenentatge automàtic Sistemes classificadors (Intel·ligència artificial) Treballs de fi de màster Machine learning Learning classifier systems Master's theses |
Issue Date: | 2-Sep-2021 |
Abstract: | [en] The use of propietary black-box machine learning models and APIs in the form of Machine Learning as a Service, makes it very difficult to control and mitigate their potential harmful effects (such as lack of transparency, privacy safeguards, robustness, reusability or fairness). The state-of-the-art technique of Machine Learning Classifier Copying allows us to build a new model that replicates the decision behaviour of an existing one without the need of knowing its architecture nor having access to the original training data. PRESC (Performance and Robustness Evaluation for Statistical Classifiers) is an existing free software tool for the evaluation of machine learning classifiers, which is maintained by Mozilla’s Data Science team. It aims to provide a toolkit to analyze model performance beyond the standard accuracy-based methods and into areas which tend to be underexplored in data science practice. Among the multiple applications of Machine Learning Classifier Copying, a systematic construction and examination of model copies has the potential to be an universally accessible and inexpensive approach to study and evaluate a rich variety of original models, and to help understand its behavior. In this work, an implementation of Machine Learning Classifier Copying has been contributed to the PRESC project, so that this tool becomes readily accessible to researchers and practitioners, and its applicability and performance in a synthetic problem has been explored to understand the copying process. The solution provides a model agnostic sampling strategy and an automated copy process for a number of fundamentally different hypothesis spaces, so that the set of achievable copy-model-fidelity measures can be used as a diagnostic measure of the original model characteristics. |
Note: | Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2020-2021. Tutor: David Zeber i Oriol Pujol Vila |
URI: | http://hdl.handle.net/2445/186003 |
Appears in Collections: | Màster Oficial - Fonaments de la Ciència de Dades Programari - Treballs de l'alumnat |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
tfg_rovira_esteva_muriel.pdf | Memòria | 5.48 MB | Adobe PDF | View/Open |
PRESC-master_thesis.zip | Codi font | 32.85 MB | zip | View/Open |
This item is licensed under a Creative Commons License