Tipus de document
Treball de fi de màsterData de publicació
Llicència de publicació
Si us plau utilitzeu sempre aquest identificador per citar o enllaçar aquest document: https://hdl.handle.net/2445/232409
Training Small Language Model Rankers with Supervised and Reinforcement Learning
Títol de la revista
Autors
Director/Tutor
Contribució addicional
ISSN de la revista
Títol del volum
Resum
Large language models make it possible to express recommendation as a constrained generation problem: given a user history and a candidate set, the model writes an ordered list rather than scoring each item separately. This interface is flexible, but relying on frontier-scale models is not always compatible with local deployment, cost, latency, privacy, or energy constraints. Small language models therefore offer an important test case: can a compact ranker be aligned well enough for a structured recommendation task?
This thesis investigates when supervised fine-tuning is sufficient and when reinforcement learning adds useful signal for such a small language model ranker. The study uses Qwen3-4B in a fixed local-identifier interface and compares imitation learning with two reward-based training strategies: one gives credit to the generated list as a whole, while the other tries to distribute credit according to the positions in the ranking. Because generated rankings can be parsed, validated, and scored automatically, recommendation provides a natural setting for studying reinforcement learning from verifiable rewards. The evaluation also includes simple classical candidate-set baselines, so the generative rankers can be interpreted against lightweight score-and-sort methods rather than only against each other. The experiments show that method suitability depends on the information structure of the task. In a binary setting, supervised fine-tuning is already well matched to the objective, while position-specific credit has limited signal to exploit. In a graded setting, reinforcement learning becomes more meaningful because rewards express gradual, position-sensitive utility; both reward-based variants improve the head of the generated ranking relative to the supervised baseline and substantially outperform the strongest simple candidate-set baseline considered. The thesis therefore argues for choosing alignment methods according to binary relevance, graded relevance, position sensitivity, and reward information density, rather than treating reinforcement learning as a generic improvement over imitation.
Descripció
Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Jordi Vitrià
Recurs relacionat
Matèries (anglès)
Citació
Citació
VIRGILI MATEU, Júlia. Training Small Language Model Rankers with Supervised and Reinforcement Learning. [consulted: 17 of November of 2026]. Available at: https://hdl.handle.net/2445/232409