Article similarities using transformers

Abstract

[en] The field of natural language processing is essential in today’s data-driven world. In 2017 the Tranformers architecture was introduced based on the concept of attention from 2014. The effects of this new structure were already changing the paradigm when the language processing model BERT marked an inflection point, in 2018. BERT makes use of the Transformers’ parallelization to achieve a network that can be pretrained. In that pretraining, the model is able to learn how a language works on its own: by only feeding it with texts. An improved version came out shortly after, RoBERTa, after which most of the models were based. In this thesis, we will focus on studying BERTa (a RoBERTa-based Catalan language model) with a dataset from the Gran Enciclopèdia Catalana. That analysis will include tasks to assess how does the model perform with real-world data. The study aims to validate the quality of the resulting embeddings produced by the model in order to further use them to build an article retrieval platform. There, each article query could be related to those with similar information. The semantic textual similarity describes how alike a pair of sentences are and this will be a fundamental target for the designed experiments and development. Finally, the results will be visualized and interpreted by using a simple front- end tool also created in this work.