Estudi dels mètodes d'optimització al deep reinforcement learning aplicat als videojocs

Abstract

[en] This report studies the Reinforcement Learning problem based on the Sonic the Hedgehog™ video game franchise. Fundamental concepts of RL are introduced in order to formalize the problem and present some state of the art Deep Reinforcement Learning (DRL) algorithms to solve it. It is also studied the core optimization problem from deep learning. In addition, some mathematical background is presented behind the optimization problem and behind the solutions based on Gradient Descent optimization algorithm. The aim is to empirically show that Adam is the best choice of optimizer for DRL methods in the Sonic benchmark. In the course of achieving it, first it has been explained the main elements of reinforcement learning and how this kind of problems can be formulated as Markov Decision Processes. Then it has been examined its combination with deep learning, also called Deep Reinforcement Learning, and different algorithms are provided to exemplify how they are combined. The second part of this work studies the mathematical background behind optimization in deep learning. This work also considers the optimization problem that appears when wanting to update the parameters of a neural network and a basic solution, the Gradient Descent optimization algorithm. Different variants of Gradient Descent have been looked at, along with the analysis of its challenges and the most common optimization algorithms that improve the basic Gradient Descent. During the experimentation phase, a set of Sonic levels has been used, splitted in train and test levels, to define each environment and common metrics to evaluate the experiments score and time. Each experiment consist in a DRL algorithm, between Joint PPO and Rainbow, and an optimization method between Adam, Adadelta, AdaGrad, Nesterov Accelerated Gradient and Momentum. Results show that Adam is the best choice of optimizer for both DRL algorithms, making the agents achieve better scores and learn to generalize between different levels. That empirically proves that Adam is the best optimizer for Joint PPO and Rainbow on the Sonic benchmark.