From Prompting to Fine-Tuning: LLM Strategies for CEFR-Based Readability Assessment

Abstract

This paper explores the use of generative open-source large language models (LLMs) to identify the CEFR level of texts for learners of English, a critical step in readability-controlled text adaptation for pedagogical purposes. To enable a systematic and cost-effective evaluation, we assess model performance on a manually annotated CEFR classification dataset, rather than relying on the more complex and less stable evaluation of generated adaptations. We evaluate several prompting strategies, including prompts enriched with CEFR descriptors or vocabulary lists, across multiple LLM families, and compare their performance to traditional readability tools and proprietary LLMs. Results show that prompting alone yields moderate performance and high variability, with models below 8B parameters behaving inconsistently, while fine-tuning markedly improves classification accuracy. Our experiments also show the importance of balanced, manually annotated training data, particularly for low-resource CEFR levels such as A1.