Value-Based Reinforcement Learning algorithms in Sparse Distributed Memories to solve the Mountain-Car Problem

Abstract

In the framework of digital electronics optimization of the memory resources used is a crucial issue. Therefore many Control algorithms are studied in order to improve the trade-off between computational power and memory requirements. In this work we explore some possibilities to improve current state-of-the-art Temporal-Difference (TD) Reinforcement Learning (RL) strategies. We made use of a type of local function approximation structures known as Sparse Distributed Memories (SDMs). The interest of this investigation underlies on the belief that SDMs architectures can help to avoid the exponential increase of memory sizes due to a linear increase in the state’s variables. Because RL doesn´t rely in prior information of the environment this is a frequent problem for these algorithms, as a lot of different features can appear to play a role when in fact only few of them are really relevant for the agent; a sampling of the states along with a method to generalize unseen states’ values becomes a must.The main achievement has been a method capable to distribute the memory locations which ensured that regions in the state space more needed had a more intense coverage, with the purpose to improve approximations’ resolution while keeping low memory requirements and high-dimensional scalability. We gave attention also to another issues as the reduction in the number of parameters.