Analyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection
| dc.contributor.advisor | Seguí Mesquida, Santi | |
| dc.contributor.author | Soler Sales, Arnau | |
| dc.date.accessioned | 2026-10-27T13:11:43Z | |
| dc.date.available | 2026-10-27T13:11:43Z | |
| dc.date.issued | 2026-06-28 | |
| dc.description | Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona. Any: 2026. Tutor: Santi Seguí Mesquida | |
| dc.description.abstract | Large language models can be trained as recommenders that optimize ranking quality directly, but training them with reinforcement learning over a full dataset is costly. This motivates data selection: choosing a small, high-value subset to train on. This thesis studies MiniRec (L. Wang et al., 2026), a method that scores each example for learnability, representativeness, and diversity using a small proxy model, and is presented as a general, model-agnostic approach to data-efficient RL-based recommendation. We first reproduce it on Amazon CD/Vinyl, where a 256-sample subset recovers 93.5% of full-data NDCG@10 at roughly a tenth of the training cost, matching the original paper. We then transfer the unmodified method to two different settings, MIND news and Yelp business ranking (Wu et al., 2020; Yelp Inc., 2023), where it scores below random selection on both. A signal-by-signal analysis explains why: on MIND, the reward signal is almost entirely explained by two unrelated factors, the number of candidates and the position of the correct answer, while the other two signals carry no information at all; Yelp shows the same pattern, even though an independent encoder finds real differences between the same items, which places the fault in the proxy rather than in the data. Every method we test on these two datasets, including MiniRec itself, ties random selection. This is not because the choice of examples is irrelevant: a selector built only from information outside the proxy, using simple facts about user history and business category, performs slightly better than random. The contribution of this thesis is this diagnosis: because all three of MiniRec’s signals are read from a single frozen proxy, its reported generality does not survive a genuine change of task, and the usable signal, where it exists, lies outside the proxy rather than within it. | |
| dc.format.extent | 29 p. | |
| dc.format.mimetype | application/pdf | |
| dc.identifier.uri | https://hdl.handle.net/2445/232387 | |
| dc.language.iso | eng | |
| dc.rights | cc-by-nc-nd (c) Soler Sales, Arnau, 2026 | |
| dc.rights.accessRights | info:eu-repo/semantics/openAccess | |
| dc.rights.uri | http://creativecommons.org/licenses/by-nc-nd/3.0/es/ | |
| dc.subject.classification | Aprenentatge per reforç (Intel·ligència artificial) | |
| dc.subject.classification | Analítiques d'aprenentatge | |
| dc.subject.classification | Sistemes classificadors (Intel·ligència artificial) | |
| dc.subject.classification | Treballs de fi de màster | |
| dc.subject.other | Reinforcement learning | |
| dc.subject.other | Learning analytics | |
| dc.subject.other | Learning classifier systems | |
| dc.subject.other | Master's thesis | |
| dc.title | Analyzing Data-Efficient Reinforcement Learning for LLM-Based Recommendation through Data Selection | |
| dc.type | info:eu-repo/semantics/masterThesis |
Fitxers
Paquet original
1 - 1 de 1
Carregant...
- Nom:
- TFM_Soler_Sales_Arnau.pdf
- Mida:
- 318.34 KB
- Format:
- Adobe Portable Document Format