Please use this identifier to cite or link to this item: http://hdl.handle.net/2445/102724
Title: Probabilistic topic modeling with latent Dirichlet allocation on Apache Spark
Author: Cortés Hinojosa, Carlos Omar
Director: Cerquides Bueno, Jesús
Keywords: Mètodes estadístics
Tractament del llenguatge natural (Informàtica)
Programari
Tesis
Algorismes computacionals
Dades massives
Statistical methods
Natural language processing (Computer science)
Computer software
Theses
Computer algorithms
Big data
Issue Date: 5-Jun-2016
Abstract: In a world in which we have access to vast amount of data, it is important to develop new tools that allow us to navigate through it. Probabilistic topic models are statistical methods to analyse text corpora and discover themes that best explain its documents. In this work, we introduce probabilistic topic models with special focus on one of the most common models called Latent Dirichlet Allocation (LDA). To learn LDA model from data, we present two variational inference algorithms for batch and online learning. Both algorithms are implemented on a popular Big Data computing framework known as Apache Spark. We introduce this framework and We study the algorithm scalability and topic coherence in two different news data sets from New York Times and BBC News. The results point out to the need to tune up Apache Spark in order to boost its performance and to the goodness of the resulting topics in the BBC News dataset.
Note: Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2016, Director: Jesús Cerquides Bueno
URI: http://hdl.handle.net/2445/102724
Appears in Collections:Treballs Finals de Grau (TFG) - Enginyeria Informàtica
Programari - Treballs de l'alumnat

Files in This Item:
File Description SizeFormat 
memoria.pdfMemòria2.65 MBAdobe PDFView/Open
codi_font.zipCodi font2.21 MBzipView/Open


This item is licensed under a Creative Commons License Creative Commons