Carregant...
Tipus de document
Treball de fi de màsterData de publicació
Llicència de publicació
Si us plau utilitzeu sempre aquest identificador per citar o enllaçar aquest document: https://hdl.handle.net/2445/212885
Can large language models replace human in speech analysis?
Títol de la revista
Autors
Director/Tutor
ISSN de la revista
Títol del volum
Recurs relacionat
Resum
[en] This thesis delves into the rapidly growing domain of Large Language Models (LLMs) and examines their relevance in the insurance sector, specifically focusing on their use in speech analysis to evaluate service quality. With the rapid escalation in the popularity of LLMs, we have the opportunity to analyze their practical use,
focusing on Generali Seguros’ customer service operations. This research is based on a partnership with Generali Seguros, which provided valuable access to audio recordings of their customer service calls and the associated evaluation templates used for assessing their teleoperators.
The core objective is to investigate the potential and real-world applications of LLMs in analyzing and evaluating the quality of service provided by Generali’s teleoperators. To facilitate this, the study utilizes a secure and confidential environment provided by AWS, selecting commercially available models for analysis. The approach begins with converting the audio calls into Spanish text through an audioto-text model, followed by improvements to this transcription method. Next, the study evaluates a baseline LLM that supports multiple languages and allows for fine tuning.
A significant aspect of this research includes addressing the challenges inherent in LLMs, such as their tendency towards ’inventing’ responses and providing vague answers. Efforts to mitigate these issues involve both employing the baseline model in English —anticipating better performance due to its primarily English training—and implements strategies to enhance its effectiveness. Additionally, fine-tuning of the model is conducted, with the objective of specializing the model to required task.
Despite efforts to enhance the LLM, a notable finding of this study is the model’s consistent failure to predict the minority group in the data, underscoring the limitations of current commercial models in fulfilling this specific evaluative function.
The thesis concludes that, while LLMs show promise, they are yet to fully meet the demands of specialized tasks such as nuanced speech analysis in customer service settings. For transparency and further research, all codes used in this study are made available in a GitHub repository (Gareta, 2023).
Descripció
Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Curs: 2023-2024. Tutor: Santi Seguí Mesquida i Carolina Martínez Pérez
Citació
Citació
GARETA CASAS, Pol. Can large language models replace human in speech analysis?. [consulta: 21 de gener de 2026]. [Disponible a: https://hdl.handle.net/2445/212885]