ML is not that good at predicting consumers' choices
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
So the best experience was this internal event that we had, where the raters would say that certain Ad would not fare well (long term), while the initial metrics (automated) were showing the opposite (short temr). So then we'll gather into this event, and people would "debug" these and try to find where the differences are coming through.
Then we had to help another group, where ML failed miserably detecting ads that should've not been shown on specific media, and raters came to help giving the correct answers.
The one thing that I've learned is that humans are not going to be replaced any time soon by AI, and I've been telling my folks, friends or anyone (new-born luddities) - that automation is not going to fully replace us. We'll still be needed as teachers, evaluators, fixers, tweakers/hackers - e.g. someone saying - this is right, and this is not, this needs adjustment, etc. (to the machine, ai, etc.).
Maybe machines are going to take over us one day, but until then, I'm not worried...
(I've also understood I knew nothing about staticics, and how valuable linguists are when comes to forming clear, concise and non-confusing (no double meaning) questions)
To this day, if I go to our airport in Sofia (Bulgaria), and my baggage is over the limit of 20 or was it 25kg I have to go to another place, pay for it and come back (why? bureaucracy - not only I have to do it, but I'm slowing anyone waiting for me to this - it's like 25-50 meters one place to the other)
Unlike Frakfurt, Munich or Heathrow airport where I can pay that fine right there.
Some jobs simply should not exist.
Even first class passengers can't turn up with unlimited luggage but, in my experience relating to families flying between London and the middle East, have household staff who make arrangements and coordinate with carriers.
When I fly between London and East Asia I shop around between different carriers and, in economy class, I'm normally offered up to two checked bags with a total weight of between 30 and 35kg depending on carrier and promotional offers.
And not just any people. Lower to middle class, blue collar people.
The type who are the least able to travel for work, most likely to have families, least able to transition between careers etc. And crucially in many countries the people who often decide elections.
Post-2008 did that to a lot of workers, as well, causing some of them to leave the workforce entirely. I know more people than I'd like to who are over 60 now, but were made redundant in the wake of the financial crisis and economic restructuring. They never worked in their fields again, some of them were forced to retire and others are working dead end jobs for $10 an hour despite having been middle to upper middle class a decade or so prior.
Sure, maybe their kids or their grandkids might be able to dig themselves out of the hole their parents were forced to drag them down into, but that doesn't do much for people who lost their livelihoods and have died/will die in destitution.
The recession didn't make anyone redundant. The people that were laid off were laid off because the people hiring them ran out of money to pay all of their employees. Yes, it is true "many people are laid off because they are redundant", but in the wake of the recession, companies ran out of money and they had to stop doing things that cost money, and then lay off people that did those things.
Also, to be clear, automating work didn't cause those people to lose their jobs - because their jobs were stopped.
Also, I'm very sorry for all of the people that were impacted by that recession. It was big amd a lot of people hurt because of it.
I know people don't use it properly, but it's worth knowing.
The Centuria was a roman military unit of 100 men (The centuria size/meaning actually varied over time) but when the decimation punishment was applied that would mean that (by draw) every 10th men in the centuria would be killed, here is the catch, the people of their own centuria had to kill their own mates in decimation.
You say that the AI gave a bad answer, but it did give an answer right? Really fast? And it was cheaper than convening the panel of experts?
That's the fear of AI replacing humans. It's not that it works so well, it's that it works poorly (but fast and cheap).
The main selling point for Nest is having a "learning thermostat". Perhaps my schedule is just not predictable enough, but the auto-generated temperature schedules it generates after its "learning" period is not even close to what I would manually set up on a normal thermostat.
Maybe I'm just an "edge case" or part of the "long tail"
- Use a python script to switch heating from my heat pump to my gas furnace when energy prices invert. - Have live monitoring of my HVAC pressure health so that my servo controlled actuators don't bork my system.
And then after a few instances, I just turn off all the automation and set up a schedule like normal.
Same with the "away from home" which seems to randomly think I'm away and I have no idea why.
Oh, and the app doesn't show me filter reminders, only the actual device, which I never touch all the way downstairs. There's not even any status to let me know if it's accepted a new dialed-in temperature, as I've had it fail to capture a request, and then I go back, and see it never updated/saved the new temp. Just zero feedback to confirm that the thermostat has responded to any input, and zero notification from the app if this happens.
Just thoroughly unimpressed.
Thankfully I didn't buy this junk, as it was pre-installed by the owner of my rental. Can't imagine actually paying for something that's only real feature is being able to remotely control my temperature once in a while.
In the end because most of the house was empty, I let the Nest do its thing and installed a separate mini-split AC in my office I kept set at 72 year-round because that's a sane and reasonable temperature for an office. Don't try to "nudge me into tolerating higher temps", respect my agency and choice about what is a comfortable environment for me to work in.
As a side note, I will never again buy a Nest product.
If you have a fairly regular life I would think a schedule would outdo ML pretty much all the time, because you know exactly what that schedule should be. ML might be useful for a secret agent whose life is so erratic that a schedule would be useless.
That is to say ML is maybe better than falling back to nothing.
Sounds like in that case it's better to just control things manually.
ML is pattern recognition. Anything outside of that is still AI, but it isn't ML. I can think of very few feature sets we could supply to help predict someone will be deployed to East Asia for a few days other than scraping calendars and mail for religious and military organizations.
From a design perspective, Nest and others are either additively learning in situ to enhance a base model or they are working from a base model that doesn't directly learn, just classifies workflow to categorize observations on a base model. I doubt heavy training is occurring where the Nest and similar is treated as the central compute node.
If the task is clear enough, ML can take it on by itself, but this requires clear rules and an absolutely unambiguous definition of what winning means. For example, the best chess players in the world are machines, and are FAR better than the best human players. Same for Go (the game, not the programming language).
So I gotta ask HN...what the heck was so popular about Nests?! It's one thing to be go after shiny lures like new iPhone apps or luxury items...but a Thermostat?!
Mind boggling...
Generally this is an algorithm that might work for one person:
* Is it equal or higher than the upper temperature outside? Run COOLing function (if it exists) until the LOWER inside temperature bound is reached.
* ELSE Is it equal or lower than the lower temperature outside? Run HEATing function (if it exists) until the UPPER inside temperature bound is reached.
In both cases DO NOT continue to run the circulation fans, at least above a whisper quite slow circulation, past the thermal shift operation. Luke-warm airflow makes the system perceptibly anemic and the results dis-satisfactory.
There are a million and one cases where a pdocut could be 10x better with 10% more effort from me,and where that tradeoff is more than worth it (hell, it's why I use Linux). But the average consumer is aggressively turned off by having to do any "work", and the type of person hanging out on Hacker News is very high-percentile for wanting to put a nonzero amount of work into optimizing this type of thing.
Away from home should be an easy problem to solve assuming Nest can talk to your phone(which is almost 100% true in real life). IN my experience there are several easy heuristics that can achieve ~ 90% precision and recall in home detection, like are you connected to a wifi or even combining it with some IMU data to be more confident.
Then it became clear the thermostat wasn't getting enough power from my 2-wire thermostat transformer and that made it even flakier. I finally threw it away and replaced it with a $20 dumb thermostat, which will still be working fine after the zombie apocalypse. No more Nest products for me.
Having to make an additional click because I receive something I have never searched for is unnerving.
The only learning feature it has is figuring out how long it takes to heat or cool the house given the current weather. Before a schedule change, can heat or cool the house so it hits next target temperature on time. This seems to work extremely well.
Everything else like schedule and away settings are configured by the user.
Once nice feature is it is fully programmable from the thermostat, without internet. You only need the app for setting a geofence for automatic home/away.
Example:
Setting is 72, reading is 73. AC is not on, I guess the thermostat is trying to save energy. I lower setting to 71, reading instantly drops to 72! I don’t think it’s a coincidence, this has happened several times.
This can be explained by hysteresis [0], which all thermostats use to avoid cycling the A/C too fast.
But the second part where the reading drops instantly is strange. Sounds like some kind of software heuristic where they're trying to make the user feel more comfortable about the hysteresis interval. Or something.
If I wanted those kind of savings, I could have just turned down my thermostat myself. Jeesh.
Matt Risinger (youtube expert builder guy) mentioned these are not anywhere near as valuable as they seem, and I'm inclined to agree. It's nice to be able to flick it on vacation mode when you're away I suppose.
I'd still buy it again, nice geeky metrics, and it's a quality company, but it doesn't save me anywhere close to 30% (or whatever the claim was).
Another "If you buy a hammer you might also want to buy " -> "a nail". Ill forgive the singular.
Just to be clear those are not cherry picked - they were my first two attempts.
The most natural interpretation there is that Sarah bought a washing machine and a dryer simultaneously, not that, after buying a washing machine the month prior, she was finally ready to buy a dryer.
Utter nightmare (unnatural obsolescence, systemic perversity, pollution...) but. I have met R'n'D who admitted the goal was just to have something new to have people want to replace the old, on unsubstantial grounds.
Actually, there is one method they use to make suggestions that works very well.
If you have a wishlist, sometimes they'll recommend items from there.
Both of these result in a higher fraction of conversions in this kind of targeting vs other targeting criteria.
It’s embarrassing, when associates glance at my screen.
2. We are discussing how ads are selected for presentation.
The ad targeting knows you looked at washing machines, but doesn’t know you purchased.
However, they've been like this for over a decade so it's likely there intentionally. here's one way that could be possible:
There could be some popular third party service that's integrated on many e-commerce sites that sells this information and doesn't actually give a damn if you bought the refrigerator or not. They're selling you, not the refrigerator.
That's the problem with data brokers, it's mostly low quality data.
Yep, totally a 2hr task for an engineer who works on homedepot.com to “check” that you bought a fridge from lowes.com after you first price shopped the other site. Also a two hour task for a Google engineer to know you bought one in person at Best Buy after researching online first.
Yes, there are basic cases (buying from same merchant as who’s suggesting) that should be handled, but let’s not foolishly pretend that’s the average case, let alone majority / all.
The Amazon case could be the same problem I discussed before. Third party sellers can pay fees to promote/boost their listings on Amazon so ultimately the same incentive structure holds if there's fees for impressions and not just sales.
You can throw your hands up and yell impossible by looking for outliers with just about anything
Show me where I did that?
You know your ROI, but your ad company doesn‘t.
I heard multiple stories from amazon, that hey still let the ad campaign or targeting running even if you bought the product for random times so external companies could not get insight in your businesses.
If you got the washing machine you will not click to buy another. If you pay per click, it makes no difference how long you let the campaign running if it is targeted.
I work in the ad industry.
I had just preordered novel 9 of The Expanse, and I got an email recommending something else from the same authors: novel 8 of the Expanse. A more sensible recommendation engine might have assumed that someone who preorders part n+1 of a series may already have part n. Not to mention that Amazon should have known that I already had novel 8 on my Kindle.
I guess generating personalized recommendations at scale is still too expensive. We just get recommendations based on what other customers with vaguely similar tastes were interested in.
Amazon doesn't seem to understand many things surrounding the Kindle. For example, it calculates the progress reading through a book by the last page I looked at. That means if I finished a book and jumped to the introduction it'll now be convinced I only read 1% of the book. This is so dumb, and I don't know why they even do it that way - the Kindle hardware should easily be capable of precisely keeping track of what pages I looked at.
Literally to the point where YouTube never pulls me down into the rabbit hole anymore, I watch one video because it was linked from somewhere else, then I bounce.
Sometimes I want to watch videos about people doing programming, but usually I don't. When I do though, I would like to easily get into a mode to do just that. Right now that essentially involves switching accounts or hoping random search recommendations are good enough.
I don't think that matters at all. People don't complain that they're getting recommendations that would have been great if they had come in an hour/day earlier or later. When you get a recommendation like that, you consider it a good recommendation.
Instead, they complain that they're getting recommendations for awful content that they wouldn't choose to watch under any circumstances.
They mostly offer stuff I’ve already watched or stuff on my watch list.
edit: My bad - it was just suggesting products I had recently previously viewed
- the more of the same thing algorithm. You clicked this thing, would you like to click it again. And again. And some more.
- the ever popular "we've shown you this thing a hundred times now and you never clicked it; we'll just assume we are right and you are wrong" algorithm.
- the ooooh we've detected that your ip address is in Germany and predict that you are now fluent in German. Would you like some Schnitzel with you Schlager music? This one in particular drives me nuts. I have user profiles with these companies for many years, browser settings that specify a preferred language, etc. I consistently never do anything in German with them. And they'll go ... here's some German content for you. Completely useless and obviously the only criteria they use for these recommendation is location. Worse, if I travel they'll unhelpfully suggest things for those locations as well. Basically, most of their top recommendations are the same generic stuff that they serve to everybody else in the same location.
Recommendation engines are hard and these companies gave up years ago and instead routinely by pass their own AI with some simple if .. else logic. I know how this stuff works. There's a little corner in the UI for the cute AI kids to do their thing but essentially all the prime real estate in their UIs is reserved for good old if else logic, basic profiling, and whatever their marketing department wants to promote to everybody.
If user in Germany recommend generic German stuff. There's no logical explanation other than that for the absolute garbage recommended by default. If you clicked a thing, here's some more things from the same source in random order. Amazon has a notion of books being in a certain order ... so why recommend I start with part 21 of a 50 book series by an author I've never bothered to read? Maybe book 1 would be a better start ... Book series are great value for them because if I get hooked, I consume the whole thing.
Most recommendations are just variations of simple profiling (age, sex, location) that consistently trump actual recommendations combined with very rudimentary similarity algorithms. You don't need AI for any of that. I work with search engine technology, it's not that hard.
It often finds a video I would like and throws it on my front page. I avoid it for awhile thinking it wouldn't be a good fit (I don't recognize the creator, bad thumbnail/title, unclear why the content would be of interest to me, etc) but find it was great and I should have watched it days ago.
If I log out of my account, the front page of the site is just awful, makes me want to throw up.
If ML was benefiting me, it would know that 90% of the time I fire up Hulu I plan to watch the next episode of what I was watching last time. And it would make that a one click action. Instead I have to scroll past promotional garbage...every single time. Assholes.
If being so reductive, that's also the scientific method. Form a model on some existing data, with the goal of it being predictive on new unseen data. Key is in favoring the more predictive models.
> they called it magic, we call it math, but both seem to have about the same outcome
Find me some sheep entrails that can do this: https://imagen.research.google/
Just trying two things at random and picking the one that makes some arbitrary metric go up, is not the scientific method. It’s gradient descent.
I do also think that ML as a field progresses through the scientific method ("I theorise that this network with residual connections will converge faster, lets see if there's a significant difference") - but maybe not to the full extent it could.
> Just trying two things at random and picking the one that makes some arbitrary metric go up, is not the scientific method. It’s gradient descent.
I'd say that's closer to evolutionary algorithms. GD finds (locally) the direction to tweak the weights to improve predictions on a given batch.
The problem is that they don't have a way of collecting "annoying" metrics as easily as they do those two. So it's a big blind spot in terms of "should we tweak more in favor of one or the other."
Translation: Computers can't read minds.
A bigger generalization is that, whenever a software feature becomes essentially mind reading; someone's either feeding a hype engine or letting their imagination run away.
The best things to do in that case is to pop the bubble if you can, or walk away. I will often clearly state, "Computers can't read minds. You're making a lot of assumptions that will most likely prove false."
It reminds me a lot of other populist folk-science belief, like vaccine hesitancy. Despite overwhelming data to the contrary, a huge portion of the US population believes that they are somehow better off contracting COVID-19 naturally versus getting the vaccine. I think when effect sizes per individual are small and only build up across large populations, people tend to believe whatever aligns best with their identity.
I don't think you know who andrew gelman is. Additionally, that's not the conclusion derived from this study.
That is, to maximally understand, and therefore predict, consumer preferences is likely to require information outside of data on choices and behavior, but also on what it is like to be human.
http://www.stat.columbia.edu/~gelman/
https://en.wikipedia.org/wiki/Andrew_Gelman
which would give strong prior evidence that the author's writing is not absurd.
Note also that he has strong connections to a leading psychologists meaning he has some background on human choices and behavior.
https://en.wikipedia.org/wiki/Susan_Gelman
Please be careful with attack statements such as "so absurd that it's not worth engaging with seriously" which some feel are out of character with HN.
Have you ever put this data up against something similar to the peer review system in academia, where several experts from a competing deparment (or ideally competing company) try to pick your results apart, disprove your hypothesis?
Let's say I go to the store to buy milk. The store has a perfect ML model, so they're able to predict that I'm about to do that. I walk into the store and buy the milk as planned. So how does the ML help drive revenue? The store could make my life easier by having it ready for me at the door, but I was going to buy it anyway, so the extra work just makes the store less profitable.
Maybe they know I'm driving to a different store, so they could send me an ad telling me to come to their store instead. But I'm already on my way, so I'll probably just keep going.
Revenue comes from changing consumer behavior, not predicting it. The ideal ML model would identify people who need milk, and predict that they won't buy it.
The simplest: Predict what features a user is most interested in, drive them to that page (increasing their predicted conversion rate) -> purchases that occur now that would not have occurred before.
Similarly: Predict products a user is likely to purchase given they made a different purchase. The user may not have seen these incremental products. For example, users buys orange couch, show them brown pillows.
Like above, the same actually works for entirely unrelated product views. If users views x,y,z products we can predict they will be interested in product w and we can advertise it.
Or we predict a user was very likely to have made a purchase, but hasn’t yet. Then we can take action to advertise to them (or not advertise to them).
The reason to raise those questions is that for many people, the word prediction has connotations of surveillance and control, so it is best not to use it loosely.
The meaning of the word "predict" is to indicate a future event, so it doesn't make grammatical sense to put a present tense verb after it, as you have done in "Predict what features a user is most interested in." Aside from the verb being in the present tense, being interested in something is not an event.
You can't predict a present state of affairs. If I look out the window and see that it is raining, no one would say that I've predicted the weather. If I come to that conclusion indirectly (e.g. a wet umbrella by the door), that would not be considered a prediction either because it's in the present. The accurate term for this is "inference", not "prediction".
The usage of the word predict is also incorrect from the point of view of an A/B test. If your ML model has truly predicted that your users will purchase a particular product, they will purchase it regardless of which condition they are in. But this is the null hypothesis, and the ML model is being introduced in the treatment group to disprove this.
I predict the weather in NYC is 100F. I don’t know whether or not that is true.
Really a pedantic argument, but to appease your phrasing you can reword my comment with “We predict an increase in conversion rate if we assume the user is interested in feature x more than feature y”
In ordinary language, you are making inferences about what users are interested in, then making inferences about what products are relevant to that interest. The prediction is that putting relevant products in front of users will make them buy more - but that is a trivial prediction.
Philosophically -- personally -- I think this is just another way big data erodes our autonomy and humanity while _also_ providing new forms of convenience. We have no way of knowing where suggestions come from, or which options are concealed. Evolution provides no defense against this form of manipulation. It's a double edged sword, an invisible one.
I regularly buy the same brand of toilet paper, socks, and sneakers. Machine learning can predict that.
But, machine learning can't predict that I spent the night at my parents house, really liked the fancy pillow they put on the guest bed, and then had to buy one for myself. (This is essentially the conclusion in the abstract.)
Such a prediction requires mind reading, which is impossible.
Also ML can predict that, BTW. Facebook knows you are connected to your parents. If the pillow seller tells Facebook that your parents bought the pillow, then Facebook knows and may choose to show you an ad for that pillow.
I think you're letting your imagination run away, and I think you're trying to exceed the limits of the kind of information that you can collect and act upon.
What you're trying to do is mind reading, and computers physically cannot do that. (Nor can people)
You are friends with your parents on Facebook. Your parents buy the pillow. The pillow seller tells facebook that your parents bought the pillow.
Now Facebook knows that somebody who is your friend recently bought the pillow. Facebook may decide to show you an ad for that pillow because somebody who is your friend recently bought the pillow.
The result may look like "mind reading", but it's actually very simple in terms of actual prediction.
Clearly, this doesn't generalise to cases where you have highly specific data (e.g. if you're Google).
However, cases with large societal impact are more likely to be the latter? They may perhaps better be viewed as "conditioned on data that is so valuable that nobody is going to publish or explain it", which kind of is in the complement of the review?
The worst part of big data is the data itself. Used to be common will be shared on Facebook webs about "what is your political compass". There results were used to create political profiles of users and targeted propaganda.
You don't need ML to predict the data that there user already has given.
Of course when an AI does that, we then say its just doing statistics, not reasoning.
Until you have built a recommendation engine from scratch, it is hard to appreciate the complexity. I don't mean the complexity of the code or algorithm (ALS and Spark are straightforward enough) but the contextual problem. Models end up being large collections of models in a complex hierarchy, with hyperparams to tune higher level concepts such as "surprise" or business targets such as "revenue", "engagement" etc. TikTok have nailed this, as has Spotify.
no, AI simply doesn't do that. Even Demis Hassabis of Deepmind fame in a recent interview pointed this out. Machine learning is great on averaging out a large amount of data, which is often useful, but it doesn't generate true novelty in any human sense. AI can play Go, it can't invent Go.
In the same way today's recommender systems are great at averaging out my last 50 shopping items or spotify playlist but they can't take a real guess at what truly new thing I'd like based on a genuine understanding of say, my personality. Which is reflected in the quality of recommendations which is mostly "the thing you just bought/watched", which is ironically often incredibly uninteresting.
Few humans do either, even great artists create from, or react to, the art they experienced in their life. At the end of the day this is a rather metaphysical question. A Go model is trained to play Go, not create new games.
> on a genuine understanding of say, my personality
That is exactly what they are trying to do. The purpose of many recommendation systems is to uncover the latent variables and categories that you might like. They are not just averaging what you listened to. It is feasible (but very unwise) to predict certain users like to listen to happy upbeat tracks on Fridays, and sad songs sung in Romance languages on Mondays.
Each time you skip a track or relisten to a track (implicit feedback) or like a track (explicit feedback) you are giving it information, which might be more than just the interaction (the timestamp, the IP address location, how quick you were to skip the track etc.).
On the other hand, the purpose of those systems is simply to optimize what generates the most short term income, in which case, you get 'boring'/simple shopping recommendations that on average make more profit.
Systems have tried to predict model the human experience, but in the past there have been epic fails so nobody is trying to explicitly do that anymore. The classic example would be stores deciding teens were pregnant and inadvertently alerting their parents. Nobody wants to see ads for Amazon's in-house toilet paper brand just as they are about to get up to go to the toilet.
This is still entirely superficial. There's no recommender system that could even tell what a 'sad track' is other than by human labelling, or distinguish say, a satirically sad track from the genuine thing, or accurately tell what any individual even perceives as happy or sad which is highly subjective.
It's not about metaphysics but about a very practical fact that these statistical systems fail at. Making genuinely novel human recommendations requires and understanding of human nature in general, individual nature of the person in question, and understanding of context and meaning. In short a system like this needs to be able to reason about what it is that it's recommending and who it's talking to.
There is no system that could invent a game like Go because it has nothing to do with data in any direct sense. It'd require being able to reason about what makes games compelling and aesthetically pleasing to human beings and that's not a question of mapping one button click to another.
The side benefit is that it is mathematically repeatable while also being tweakable with variable inclusion or exclusion.
https://scholar.google.com/scholar?q=%22multiple+regression%...
I frequently get targeted ads for exactly the item I'm interested in, e.g. a pair a shoes that actually look very similar to a pair I was considering but from a different brand.
My YouTube recommendations bring me a ton of new videos on topics I'm interested in. I do a tiny amount of curation, but overall rarely get objectively bad videos recommended.
Amazon recommends consumable products for me at approximately the rate I consume them. When my pet died and I stopped purchasing treats, they seem to have figured that out and stopped recommending them.
The "you just bought a car, would you like to buy another car?" does happen from time to time, but not all that often to be honest.
I find Amazon loves to tell me to buy ... the thing they know I just bought and you don't need more than one of ...
I hardly ever get ads or offers for things I want.
How do you mess that up?
Ironically I find the "dumb" ads on cable tv news to be a lot more effective since they have to target by interests.
Are you sure you want to watch this without your partner ?
Yes ? We recommend the following service for finding temporary accommodation on short notice
For example, if your water consumption is log-Cauchy, I will have a very hard time predicting it because the variance is infinite.
Every other online ad has been a dumpster fire, from gross images, things aimed at people decades older or younger, or large items I just bought and aren’t buying again in years.
Second, this is only saying that right now, ML's performance is "not that good." It says nothing about future technical advances. If you look at the track record of ML in the past three decades, it's amazing, and if that performance is repeated in the next three decades, who even knows what things might look like. (Machine sentience? Maybe.)
Sometimes I'd settle for some vanilla common sense. Watch Episode 1 of something on Prime. You'd think next time it would suggest Episode 2, right? It does...its thumbnail #25 in the grid & below the fold.
I believe this was a conclusion of the Netflix Prize contest, getting much more accurate at predictions was hard because people would not reliably rate a movie the same.
>Several years ago a conversation about a similar topic prompted me to look at the ad targeting data Facebook had on me. At the time I'd had a Facebook account for 12 years with lots of posts, group memberships and ~500 friends. Their cutting edge data collection and complex ad targeting algorithms had identified my "Hobbies and activities" as: "Mosquito", "Hobby", "Leaf" and "Species": https://imgur.com/nWCWn63. Whatever that means.
Similar to the thesis of How Brands Grow by Byron Sharp
Both e-commerce and social media are really not good at gathering express feedback for what people want and valuing that expressly. Please, let me tell you that I did spend time looking at this thread about the latest reality TV scandal but I don‘t want to hear about it ever again! Please, let me tag options as “maybe” or let me tell you what you’d need to change for me to buy that shirt. Public, performative Likes and Favourite lists that are instantly reactivation spam-fodder… Come on, you know better.
I used to work for a big e-commerce site (the leading site for 18-25 y.o. females). We had millions of references (really) and it was a problem. The search team had layers upon layers of ranking algos, incredible papers at conference… but still, low impact on conversion. It was more than anything else that we could do, but nowhere as transformative as it could be. Instead, I suggested copying the Tinder interaction in a companion app:
* left, never see that item again;
* right, add it to a long list of stuff you might want to revisit. We probably would have to separate that from the Favourite list to avoid clutter, but maybe not, to make that selection worthwhile.
The learning you could get from that dataset, even with a basic RL algo to queue suggestions… People thought it was “too much” which I’m still bitter about.
The entire internet: "I heard you like comics. Would a Marvel Unlimited subscription interest you?"
Our main conclusion is that for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard. In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante. Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise—so much so that it is often a waste of effort.
My initial reaction on skimming it is "savvy consumers are becoming increasingly desensitized to a sea of Facebook ads, Amazon fake reviews and rigged star ratings, undisclosed compensated influencers", aka the "unprecedented information environment" as the author describes things. I wouldn't call an Amazon review or FB ad/influencer post/affiliate link "information" as distinct to "this laptop has a rated battery life of 18h" or "this toaster comes in the following 5 color choices"; it's simply an influencing attempt; whether those contain any information (/misinformation), and whether users trust that they contain accurate information, seems to be something the study doesn't want to look into. Really the authors seem to be giving a very-judgment-free pass to anything calling itself "information".
Consider "unprecedented information environment" could also characterize the 2016 and 2020 US elections, 2016 UK Brexit referendum and 2022 Philippines election: "it is important to first understand how consumers make choices, particularly in the current information environment in which they have access to an unprecedented amount of information at the time they are making decisions." [obviously third-party political advertising is far less trustworthy than consumer advertising, but still].
What if the authors merely succeeded in proving that the rise in targeted influencing attempts has rendered the public more wary of targeted influencing attempts?
What a complete waste of time, money and CO2 being burned up in the data centers.
It's just that as soon as you start out on every problem with 'ML will solve this!' that you're going to end up with a bunch of crap. The right tool for the problem wins every time.
Then it got taken over by ads and SEO and corrupting influences and it's just not that good anymore. IMO, the problem with DL isn't the tech. It's the way its being used. The reality is: For 99% of things advertised to me, I don't want to buy the goddamn product, and no amount of advertising will make me want to buy it. It's gotten to the point where if I see an ad for a product I think I'm more likely to buy a competitor whose ad I haven't seen because I assume the competitor is investing more in the product than the marketing.
And everyone seems to have forgotten about hybrid approaches of ML and human beings that, IMO, are really good. But alas, "they don't scale".
But at the same time, it's really interesting. For as much data as facebook should have about me, their ad rec's really suck and always have. (Perhaps it's because my only ad clicks ever are accidental ones?) I'm kind of astounded at how poor that result is. That said, I'm always very impressed by spotify's recommender system. I think it's one of the best on the net.
Another thing I find interesting is that non-vote-based social media feed systems all really suck. Once they ditched chronological ordering it stopped appealing to me, and I don't know exactly why that is. Evidently I'm on some tail of the curve they don't care about.