Monitorización de datos en tiempo real usando ELK y desarrollo de una web de visualización y gestión

Abstract

[en] In medium to large projects, monitoring and organizing services begins to be important, since it increases the scalability and contributes to the detection of issues and the taking of measures quickly. My work has consisted, overall, in developing a monitoring project. This implies, on the one hand, monitoring different aspects of the logs that are generated and sent in the calls to services that my company offers to the client, such as the total volume, the volume of errors, the average time, or the timeouts. We also had to create alerts based on this monitoring, whose purpose is to send a webhook to a website every time an anomaly is detected or a desired condition is met. On the other hand, I have also contributed to the development of this website, which aims to allow users to quickly view key aspects of the health of the services and notify them by SMS or email when the anomalies are generated. The monitoring part has been done using an ELK cluster (acronym for Elasticsearch, Logstash and Kibana), technologies that allow us to search and analyze our logs easily (Elasticsearch), ingest data from different sources simultaneously, transform it and send it to a warehouse (Logstash), visualize and manage the cluster from a GUI (Kibana). Once we have the data in indexs, we can create machine learning jobs, which will be fed of selected and transformed data from these indexs, and will learn to detect and generate anomalies using, among oth- ers, unsupervised learning. In addition, ELK allows the creation of watches, which are scheduled querys which verify if their result meets a condition, and if so, perform an action (in our case send a webhook).On the other hand, the website is being developed using Spring for the backend and Angular for the frontend, as well as an Oracle database. This web renovates and unifies the two websites that were used in this project before we began to use ML jobs of ELK: one web of monitoring, which created graphs of the data of ELK (obtained using its API) and associate them with a service; the other one was an administration website, which allowed grouping users, grouping services, linking groups of users with a service, creating rules that a service must meet, notify users associated to the service in case that a rule is not meeting, etc.