Implementación de una arquitectura end-to-end para la recogida y explotación de datos desde una API pública. Caso de uso - consumo y visualización de datos del COVID-19

Abstract

[en] This project shows the design and development of an architecture based on Docker technology with the aim of studying its performance through metrics and being able to generate the visualization of the data in real time to extract possible conclusions. To test the tool, data on the global incidence of COVID-19 have been used. The architecture consists of different parts that configure it and for its implementation the most current and innovative tools have been used. For the data collection layer, a Kafka bus has been used in streaming based on an API (Application Programming Interface) of data, for data storage MongoDB has been used as a database, for data processing and its subsequent Visualization of the metrics, Prometheus has been used together with Grafana, allowing a User Friendly visualization through graphs, and finally, for the visualization of the data, MongoDB Charts has been used, a tool that has allowed the visualization of the data through heat maps. It can be concluded that the architecture has been designed in such a way that it is capable of managing data from any type of data source in a generic and fully modular and scalable way.