>Plus, these experiences were shitty for the user but they must have worked

I spent years as a data scientist in the online display advertising space, and I was shocked at how much of a disconnect there was between advertiser ROI and clicks/impressions/whatever metric is being sold to the advertiser. There are very few parties whose best interests involved clearing up that connection. I heard one story of an ad executive getting fired for introducing A/B testing to their advertisers and proving the negative ROI.

Big flashy ads certainly improve clicks, but it's not clear if they make a customer want to buy a product.