Training Small Language Model Rankers with Supervised and Reinforcement Learning
| dc.contributor.advisor | Vitrià i Marca, Jordi | |
| dc.contributor.author | Virgili Mateu, Júlia | |
| dc.date.accessioned | 2026-10-28T15:43:25Z | |
| dc.date.available | 2026-10-28T15:43:25Z | |
| dc.date.issued | 2026-06-01 | |
| dc.description | Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Jordi Vitrià | |
| dc.description.abstract | Large language models make it possible to express recommendation as a constrained generation problem: given a user history and a candidate set, the model writes an ordered list rather than scoring each item separately. This interface is flexible, but relying on frontier-scale models is not always compatible with local deployment, cost, latency, privacy, or energy constraints. Small language models therefore offer an important test case: can a compact ranker be aligned well enough for a structured recommendation task? This thesis investigates when supervised fine-tuning is sufficient and when reinforcement learning adds useful signal for such a small language model ranker. The study uses Qwen3-4B in a fixed local-identifier interface and compares imitation learning with two reward-based training strategies: one gives credit to the generated list as a whole, while the other tries to distribute credit according to the positions in the ranking. Because generated rankings can be parsed, validated, and scored automatically, recommendation provides a natural setting for studying reinforcement learning from verifiable rewards. The evaluation also includes simple classical candidate-set baselines, so the generative rankers can be interpreted against lightweight score-and-sort methods rather than only against each other. The experiments show that method suitability depends on the information structure of the task. In a binary setting, supervised fine-tuning is already well matched to the objective, while position-specific credit has limited signal to exploit. In a graded setting, reinforcement learning becomes more meaningful because rewards express gradual, position-sensitive utility; both reward-based variants improve the head of the generated ranking relative to the supervised baseline and substantially outperform the strongest simple candidate-set baseline considered. The thesis therefore argues for choosing alignment methods according to binary relevance, graded relevance, position sensitivity, and reward information density, rather than treating reinforcement learning as a generic improvement over imitation. | |
| dc.format.extent | 26 p. | |
| dc.format.mimetype | application/pdf | |
| dc.identifier.uri | https://hdl.handle.net/2445/232409 | |
| dc.language.iso | eng | |
| dc.rights | memòria: cc-by-nc-nd (c) Virgili Mateu, Júlia, 2026 | |
| dc.rights.accessRights | info:eu-repo/semantics/openAccess | |
| dc.rights.uri | http://creativecommons.org/licenses/by-nc-nd/3.0/es/ | |
| dc.subject.classification | Aprenentatge per reforç (Intel·ligència artificial) | |
| dc.subject.classification | Classificadors (Lingüística) | |
| dc.subject.classification | Sistemes classificadors (Intel·ligència artificial) | |
| dc.subject.classification | Treballs de fi de màster | |
| dc.subject.other | Reinforcement learning | |
| dc.subject.other | Classifiers (Linguistics) | |
| dc.subject.other | Learning classifier systems | |
| dc.subject.other | Master's thesis | |
| dc.title | Training Small Language Model Rankers with Supervised and Reinforcement Learning | |
| dc.type | info:eu-repo/semantics/masterThesis |
Fitxers
Paquet original
1 - 1 de 1
Carregant...
- Nom:
- TFM_Virgili_Mateu_Julia.pdf
- Mida:
- 817.96 KB
- Format:
- Adobe Portable Document Format