Tipus de document
Treball de fi de màsterData de publicació
Llicència de publicació
Si us plau utilitzeu sempre aquest identificador per citar o enllaçar aquest document: https://hdl.handle.net/2445/232387
Analyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection
Títol de la revista
Autors
Director/Tutor
Contribució addicional
ISSN de la revista
Títol del volum
Resum
Large language models can be trained as recommenders that optimize ranking quality directly, but training them with reinforcement learning over a full dataset is costly. This motivates data selection: choosing a small, high-value subset to train on. This thesis studies MiniRec (L. Wang et al., 2026), a method that scores each example for learnability, representativeness, and diversity using a small proxy model, and is presented as a general, model-agnostic approach to data-efficient RL-based recommendation. We first reproduce it on Amazon CD/Vinyl, where a 256-sample subset recovers 93.5% of full-data NDCG@10 at roughly a tenth of the training cost, matching the original paper.
We then transfer the unmodified method to two different settings, MIND news and Yelp business ranking (Wu et al., 2020; Yelp Inc., 2023), where it scores below random selection on both. A signal-by-signal analysis explains why: on MIND, the reward signal is almost entirely explained by two unrelated factors, the number of candidates and the position of the correct answer, while the other two signals carry no information at all; Yelp shows the same pattern, even though an independent encoder finds real differences between the same items, which places the fault in the proxy rather than in the data. Every method we test on these two datasets, including MiniRec itself, ties random selection. This is not because the choice of examples is irrelevant: a selector built only from information outside the proxy, using simple facts about user history and business category, performs slightly better than random. The contribution of this thesis is this diagnosis: because all three of MiniRec’s signals are read from a single frozen proxy, its reported generality does not survive a genuine change of task, and the usable signal, where it exists, lies outside the proxy rather than within it.
Descripció
Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Santi Seguí Mesquida
Recurs relacionat
Matèries (anglès)
Citació
Citació
SOLER SALES, Arnau. Analyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection. [consulted: 17 of November of 2026]. Available at: https://hdl.handle.net/2445/232387