Exploring transformers for localizing moments of actions

Abstract

[en] The field of machine learning is being applied to all aspects of our daily lives. From chatbots that speak like a human to artificial intelligence able to generate art, there are many neural networks capable of doing a better job than humans. In 2022, the Ego4D dataset was published, a set of large-scale first-person annotated videos as never seen before, which opened the door to new branches of research on video analysis. The publication of the dataset was accompanied by several challenges, in particular, one of them is the one we will face in this memory: the Moment Queries, which consists of the temporal localization of concrete actions in a video. This is a highly complex problem that needs very powerful image analysis techniques! Transformers are a type of neural network based on the concept of attention that revolutionized the field of Deep Learning. Since their appearance in 2017, they have proven their usefulness in various artificial intelligence applications such as natural text processing or computer vision, being able to outperform all previous results of Neural networks. In this work, we have made an in-depth study of Transformers and we have tested their performance in the field of image classification. With the knowledge obtained, we have analyzed how they can be applied to the Moment Queries problem. We have developed our proposal - a model that uses a pyramid of attention mechanisms to refine the data and provide the prediction modules with the best possible information. With this implementation, we have managed to locate and catalog 28% of the actions with a tIoU value better than 0.5.