Analyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection

dc.contributor.advisorSeguí Mesquida, Santi
dc.contributor.authorSoler Sales, Arnau
dc.date.accessioned2026-10-27T13:11:43Z
dc.date.available2026-10-27T13:11:43Z
dc.date.issued2026-06-28
dc.descriptionTreballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Santi Seguí Mesquida
dc.description.abstractLarge language models can be trained as recommenders that optimize ranking quality directly, but training them with reinforcement learning over a full dataset is costly. This motivates data selection: choosing a small, high-value subset to train on. This thesis studies MiniRec (L. Wang et al., 2026), a method that scores each example for learnability, representativeness, and diversity using a small proxy model, and is presented as a general, model-agnostic approach to data-efficient RL-based recommendation. We first reproduce it on Amazon CD/Vinyl, where a 256-sample subset recovers 93.5% of full-data NDCG@10 at roughly a tenth of the training cost, matching the original paper. We then transfer the unmodified method to two different settings, MIND news and Yelp business ranking (Wu et al., 2020; Yelp Inc., 2023), where it scores below random selection on both. A signal-by-signal analysis explains why: on MIND, the reward signal is almost entirely explained by two unrelated factors, the number of candidates and the position of the correct answer, while the other two signals carry no information at all; Yelp shows the same pattern, even though an independent encoder finds real differences between the same items, which places the fault in the proxy rather than in the data. Every method we test on these two datasets, including MiniRec itself, ties random selection. This is not because the choice of examples is irrelevant: a selector built only from information outside the proxy, using simple facts about user history and business category, performs slightly better than random. The contribution of this thesis is this diagnosis: because all three of MiniRec’s signals are read from a single frozen proxy, its reported generality does not survive a genuine change of task, and the usable signal, where it exists, lies outside the proxy rather than within it.
dc.format.extent29 p.
dc.format.mimetypeapplication/pdf
dc.identifier.urihttps://hdl.handle.net/2445/232387
dc.language.isoeng
dc.rightscc-by-nc-nd (c) Soler Sales, Arnau, 2026
dc.rights.accessRightsinfo:eu-repo/semantics/openAccess
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/3.0/es/
dc.subject.classificationAprenentatge per reforç (Intel·ligència artificial)
dc.subject.classificationAnalítiques d'aprenentatge
dc.subject.classificationSistemes classificadors (Intel·ligència artificial)
dc.subject.classificationTreballs de fi de màster
dc.subject.otherReinforcement learning
dc.subject.otherLearning analytics
dc.subject.otherLearning classifier systems
dc.subject.otherMaster's thesis
dc.titleAnalyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection
dc.typeinfo:eu-repo/semantics/masterThesis

Fitxers

Paquet original

Mostrant 1 - 1 de 1
Carregant...
Miniatura
Nom:
TFM_Soler_Sales_Arnau.pdf
Mida:
318.34 KB
Format:
Adobe Portable Document Format