Do LLMs Understand User Preferences? Evaluating LLMs on User Rating Prediction

arxiv.org

2 points·PaulHoule··2 comments

I've often pondered the idea of semantic recommendation but it is a dynamic problem with an aversion for repetition. I am looking forward to digging into this paper. A big question for me is if you can successfully avoid repeating different coverage while maximizing affinity.

My content-based recommender shows me one piece of content at a time, so like Tik Tok or Tinder, I get a clean signal. (Unlike the Youtube problem which is more like predicting which distractor from the image on https://tvtropes.org/pmwiki/pmwiki.php/Funny/Idiocracy you will click on)

My main evaluation metric is area under curve for predicting “will I like the item?” but that metric doesn’t necessarily correlate to satisfaction. I’ve done a round of improving the model to add another percentage point to my AUC but right now that’s a distraction from targeting annoyances such getting 10 articles about the same news event.

The real frontier is “sequential recommendation” where all the literature seems to come from China and India and I fear there will be a “missile gap” for e-commerce in the next few years. (e.g. the state of the art in e-commerce in the US is “You’ve subscribed to Prime for 15 years and they figure you’ll subscribe to Prime for 15 years even if two day shipping is downgraded to five day shipping”)

Reply on news.ycombinator.com