Education with language models: analyzing uncertainty estimation techniques

Abstract

The widespread adoption of Large Language Models (LLMs) underscores the significance of recognizing both their capabilities and constraints. This study aims to delve into understanding the functioning of Large Language Models (LLMs), with a specific focus on GPT models (Sai, 2023), such as GPT-3.5 (Koubaa, 2023) and GPT-4 (OpenAI, 2023). Additionally, it will demonstrate the development of a Chatbot tailored for educational purposes, employing a diverse array of tools. Through systematic examination, this study seeks to determine whether the utilization of LLMs and GenAI can be deemed trustworthy for educational purposes. Moreover, this research will address the challenge of uncertainty estimation, particularly in black-box models, highlighting the need for reliable methods to evaluate model confidence. The investigation will incorporate various experiments de- signed to evaluate the stability and accuracy of these models. Through comprehensive experimentation, this study seeks to contribute to a deeper understanding of LLMs’ behavior, their potential applications in education, and the challenges associated with uncertainty estimation in black-box models. The corresponding notebooks and datasets for this thesis, can be found in the following GitHub repository, https://github.com/DaphneDjiakouri/MasterThesis.