Please use this identifier to cite or link to this item:
Title: Building uncertainty models on top of black-box predictive APIs
Author: Brando Guillaumes, Axel
Torres, Damià
Rodriguez-Serrano, José A.
Vitrià i Marca, Jordi
Keywords: Aprenentatge automàtic
Xarxes neuronals (Informàtica)
Intel·ligència artificial
Machine learning
Neural networks (Computer science)
Artificial intelligence
Issue Date: 2-Jul-2020
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Abstract: With the commoditization of machine learning, more and more off-the-shelf models are available as part of code libraries or cloud services. Typically, data scientists and other users apply these models as ''black boxes'' within larger projects. In the case of regressing a scalar quantity, such APIs typically offer a predict() function, which outputs the estimated target variable (often referred to as y¿ or, in code, y_hat). However, many real-world problems may require some sort of deviation interval or uncertainty score rather than a single point-wise estimate. In other words, a mechanism is needed with which to answer the question ''How confident is the system about that prediction?'' Motivated by the lack of this characteristic in most predictive APIs designed for regression purposes, we propose a method that adds an uncertainty score to every black-box prediction. Since the underlying model is not accessible, and therefore standard Bayesian approaches are not applicable, we adopt an empirical approach and fit an uncertainty model using a labelled dataset (x, y) and the outputs y¿ of the black box. In order to be able to use any predictive system as a black box and adapt to its complex behaviours, we propose three variants of an uncertainty model based on deep networks. The first adds a heteroscedastic noise component to the black-box output, the second predicts the residuals of the black box, and the third performs quantile regression using deep networks. Experiments using real financial data that contain an in-production black-box system and two public datasets (energy forecasting and biology responses) illustrate and quantify how uncertainty scores can be added to black-box outputs.
Note: Reproducció del document publicat a:
It is part of: IEEE Access, 2020, vol. 8, p. 121344 -121356
Related resource:
ISSN: 2169-3536
Appears in Collections:Articles publicats en revistes (Matemàtiques i Informàtica)

Files in This Item:
File Description SizeFormat 
702678.pdf7.05 MBAdobe PDFView/Open

This item is licensed under a Creative Commons License Creative Commons