Proposta per a una arquitectura escalable d'ingesta de missatges

Abstract

Our world is becoming more interconnected with everything. As time passes, more electronic components emit information to the Internet. Mobile devices, intelligent wearables, beacons, any kind of machine with enough electricity to connect to the Internet may send information of its surroundings. At Beabloo, we strive to achieve a higher understanding of the environment these devices describe us by listening them. As the amount of these devices increases, harnessing such high amount of information becomes difficult, and we believe it is necessary to study new ways that allow us scale up horizontally, and allows us to bring such information as fast as possible to our data processing pipelines, which then are being used to calculate Key Performance Indicators that will be used for decision making at higher level. This paper describes a design of a data ingression architecture, keeping in mind previous design flaws that slowed down ingestion capacity. Its objective is, by keeping a small and minimalistic architecture design, to allow attaining an almost real-time data processing performance. The project uses Apache Kafka, and Apache Storm to achieve a new real-time execution environment, and to be able to evaluate its performance, it uses Prometheus to store performance metrics and Grafana to visualize them with ease.