Training Small Language Model Rankers with Supervised and Reinforcement Learning

dc.contributor.advisorVitrià i Marca, Jordi
dc.contributor.authorVirgili Mateu, Júlia
dc.date.accessioned2026-10-28T15:43:25Z
dc.date.available2026-10-28T15:43:25Z
dc.date.issued2026-06-01
dc.descriptionTreballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Jordi Vitrià
dc.description.abstractLarge language models make it possible to express recommendation as a constrained generation problem: given a user history and a candidate set, the model writes an ordered list rather than scoring each item separately. This interface is flexible, but relying on frontier-scale models is not always compatible with local deployment, cost, latency, privacy, or energy constraints. Small language models therefore offer an important test case: can a compact ranker be aligned well enough for a structured recommendation task? This thesis investigates when supervised fine-tuning is sufficient and when reinforcement learning adds useful signal for such a small language model ranker. The study uses Qwen3-4B in a fixed local-identifier interface and compares imitation learning with two reward-based training strategies: one gives credit to the generated list as a whole, while the other tries to distribute credit according to the positions in the ranking. Because generated rankings can be parsed, validated, and scored automatically, recommendation provides a natural setting for studying reinforcement learning from verifiable rewards. The evaluation also includes simple classical candidate-set baselines, so the generative rankers can be interpreted against lightweight score-and-sort methods rather than only against each other. The experiments show that method suitability depends on the information structure of the task. In a binary setting, supervised fine-tuning is already well matched to the objective, while position-specific credit has limited signal to exploit. In a graded setting, reinforcement learning becomes more meaningful because rewards express gradual, position-sensitive utility; both reward-based variants improve the head of the generated ranking relative to the supervised baseline and substantially outperform the strongest simple candidate-set baseline considered. The thesis therefore argues for choosing alignment methods according to binary relevance, graded relevance, position sensitivity, and reward information density, rather than treating reinforcement learning as a generic improvement over imitation.
dc.format.extent26 p.
dc.format.mimetypeapplication/pdf
dc.identifier.urihttps://hdl.handle.net/2445/232409
dc.language.isoeng
dc.rightsmemòria: cc-by-nc-nd (c) Virgili Mateu, Júlia, 2026
dc.rights.accessRightsinfo:eu-repo/semantics/openAccess
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/3.0/es/
dc.subject.classificationAprenentatge per reforç (Intel·ligència artificial)
dc.subject.classificationClassificadors (Lingüística)
dc.subject.classificationSistemes classificadors (Intel·ligència artificial)
dc.subject.classificationTreballs de fi de màster
dc.subject.otherReinforcement learning
dc.subject.otherClassifiers (Linguistics)
dc.subject.otherLearning classifier systems
dc.subject.otherMaster's thesis
dc.titleTraining Small Language Model Rankers with Supervised and Reinforcement Learning
dc.typeinfo:eu-repo/semantics/masterThesis

Fitxers

Paquet original

Mostrant 1 - 1 de 1
Carregant...
Miniatura
Nom:
TFM_Virgili_Mateu_Julia.pdf
Mida:
817.96 KB
Format:
Adobe Portable Document Format