Multinomial logistic regression and stochastic natural gradient descent

Abstract

[en] Function optimization is a widely faced problem nowadays. Its interest, in particular, lies in every learning algorithm in AI, whose achievements are measured by a Loss-Function. On one hand, Multinomial Logistic Regression is a commonly applied model to engage and simplify the problem of predicting a categorical distributed variable which depends on a set of distinct categorical distributed variables. On the other hand, Gradient Descent allows us to reach local extrema of a smooth function. Moreover, large datasets force the use of online optimization. Improving the convergence speed and reducing the computational cost of gradient based online learning algorithms will automatically translate into a significant enhancement on many machine learning processes. In this text, we present a Stochastic Gradient Descent algorithm variant, specifically designed for Multinomial Logistic Regression learning problems by taking advantage of the geometry and the intrinsic metric of the space. We compare it to current most advanced stochastic algorithms, and we provide the favorable experimental results obtained.