Can large language models replace human in speech analysis?

Abstract

[en] This thesis delves into the rapidly growing domain of Large Language Models (LLMs) and examines their relevance in the insurance sector, specifically focusing on their use in speech analysis to evaluate service quality. With the rapid escalation in the popularity of LLMs, we have the opportunity to analyze their practical use, focusing on Generali Seguros’ customer service operations. This research is based on a partnership with Generali Seguros, which provided valuable access to audio recordings of their customer service calls and the associated evaluation templates used for assessing their teleoperators. The core objective is to investigate the potential and real-world applications of LLMs in analyzing and evaluating the quality of service provided by Generali’s teleoperators. To facilitate this, the study utilizes a secure and confidential environment provided by AWS, selecting commercially available models for analysis. The approach begins with converting the audio calls into Spanish text through an audioto-text model, followed by improvements to this transcription method. Next, the study evaluates a baseline LLM that supports multiple languages and allows for fine tuning. A significant aspect of this research includes addressing the challenges inherent in LLMs, such as their tendency towards ’inventing’ responses and providing vague answers. Efforts to mitigate these issues involve both employing the baseline model in English —anticipating better performance due to its primarily English training—and implements strategies to enhance its effectiveness. Additionally, fine-tuning of the model is conducted, with the objective of specializing the model to required task. Despite efforts to enhance the LLM, a notable finding of this study is the model’s consistent failure to predict the minority group in the data, underscoring the limitations of current commercial models in fulfilling this specific evaluative function. The thesis concludes that, while LLMs show promise, they are yet to fully meet the demands of specialized tasks such as nuanced speech analysis in customer service settings. For transparency and further research, all codes used in this study are made available in a GitHub repository (Gareta, 2023).