Comparative analysis of open source large language models

Abstract

[en] This study investigates the potential of using smaller, locally hosted language models (LLMs) to perform specific tasks traditionally handled by large LLMs, such as OpenAI’s Chat-GPT 3.5. With the growing integration of LLMs in corporate environments, concerns over costs, data privacy, and security have become prominent. By focusing on question answering and text summarization tasks, we compare the performance of several smaller models, including Flan T5 XXL, Phi 3 Mini, and Yi 1.5, against Chat-GPT 3.5. As the two experiments show, one on question answering and the second one on text summarization, this tasks can be done by the tested models at the same level than the state of the art Chat-GPT 3.5. Concluding that depending the use intended for the LLM one of the different models could best fit as the variety in the response structure and verbosity highly depends on the model selected.