Tipus de document

Treball de fi de màster

Data de publicació

Llicència de publicació

memòria: cc-by-nc-nd (c) Virgili Mateu, Júlia, 2026
Si us plau utilitzeu sempre aquest identificador per citar o enllaçar aquest document: https://hdl.handle.net/2445/232409

Training Small Language Model Rankers with Supervised and Reinforcement Learning

Títol de la revista

Contribució addicional

ISSN de la revista

Títol del volum

Resum

Large language models make it possible to express recommendation as a constrained generation problem: given a user history and a candidate set, the model writes an ordered list rather than scoring each item separately. This interface is flexible, but relying on frontier-scale models is not always compatible with local deployment, cost, latency, privacy, or energy constraints. Small language models therefore offer an important test case: can a compact ranker be aligned well enough for a structured recommendation task? This thesis investigates when supervised fine-tuning is sufficient and when reinforcement learning adds useful signal for such a small language model ranker. The study uses Qwen3-4B in a fixed local-identifier interface and compares imitation learning with two reward-based training strategies: one gives credit to the generated list as a whole, while the other tries to distribute credit according to the positions in the ranking. Because generated rankings can be parsed, validated, and scored automatically, recommendation provides a natural setting for studying reinforcement learning from verifiable rewards. The evaluation also includes simple classical candidate-set baselines, so the generative rankers can be interpreted against lightweight score-and-sort methods rather than only against each other. The experiments show that method suitability depends on the information structure of the task. In a binary setting, supervised fine-tuning is already well matched to the objective, while position-specific credit has limited signal to exploit. In a graded setting, reinforcement learning becomes more meaningful because rewards express gradual, position-sensitive utility; both reward-based variants improve the head of the generated ranking relative to the supervised baseline and substantially outperform the strongest simple candidate-set baseline considered. The thesis therefore argues for choosing alignment methods according to binary relevance, graded relevance, position sensitivity, and reward information density, rather than treating reinforcement learning as a generic improvement over imitation.

Descripció

Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Jordi Vitrià

Recurs relacionat

Citació

Citació

VIRGILI MATEU, Júlia. Training Small Language Model Rankers with Supervised and Reinforcement Learning. [consulted: 17 of November of 2026]. Available at: https://hdl.handle.net/2445/232409

Exportar metadades

JSON - METS

Compartir registre