Amb motiu del tancament d'estiu, la validació de documents es reprendrà a partir del 28 d'agost de 2026. Disculpeu les molèsties.
Con motivo del cierre de verano, la validación de documentos se reanudará a partir del 28 de agosto de 2026. Disculpad las molestias
Due to the summer closure, document validation will resume starting August 28, 2026. We apologize for any inconvenience.

Files

memoria.pdf (2.58 MB)
codi_font.zip (2.15 MB)

Document type

Bachelor thesis

Publication date

2016-06-05

Publication license

memòria: cc-by-nc-sa (c) Carlos Omar Cortés Hinojosa, 2016
Please use this identifier to cite or link to this item: https://hdl.handle.net/2445/102724

Probabilistic topic modeling with latent Dirichlet allocation on Apache Spark

Journal Title

Authors

Director/Tutor

Journal ISSN

Volume Title

Related resource

Abstract

In a world in which we have access to vast amount of data, it is important to develop new tools that allow us to navigate through it. Probabilistic topic models are statistical methods to analyse text corpora and discover themes that best explain its documents. In this work, we introduce probabilistic topic models with special focus on one of the most common models called Latent Dirichlet Allocation (LDA). To learn LDA model from data, we present two variational inference algorithms for batch and online learning. Both algorithms are implemented on a popular Big Data computing framework known as Apache Spark. We introduce this framework and We study the algorithm scalability and topic coherence in two different news data sets from New York Times and BBC News. The results point out to the need to tune up Apache Spark in order to boost its performance and to the goodness of the resulting topics in the BBC News dataset.

Description

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2016, Director: Jesús Cerquides Bueno

Subject

Mètodes estadístics, Tractament del llenguatge natural (Informàtica), Programari, Treballs de fi de grau, Algorismes computacionals, Dades massives

Subject (English)

Statistical methods, Natural language processing (Computer science), Computer software, Bachelor's theses, Computer algorithms, Big data

Citation

Collections

Treballs Finals de Grau (TFG) - Enginyeria Informàtica
Programari - Treballs de l'alumnat

Citation

CORTÉS HINOJOSA, Carlos Omar. Probabilistic topic modeling with latent Dirichlet allocation on Apache Spark. [consulted: 16 of August of 2026]. Available at: https://hdl.handle.net/2445/102724

Export metadata

JSON - METS

Share record