When the internet came in and destroyed print subscriptions and tech giants like FB/Google started controlling online advertising, getting clicks became the way to make money. We really only have ourselves to blame for choosing ad-driven media content at the expense of new dark patterns that help to prop the system up.
I hate clickbait headlines too, but they work. A/B testing shows publishers that they work.
What should a publisher do to create less sensational content without sacrificing revenue?
Don't forget people rejecting subscriptions and demanding the journalism for free.
But it wasn't the internet and Google alone. It was greed, and the elevation of media properties as shareholder golden geese. They saw exponential returns in the late 90's. When revenue slowed, they lost favour with their new corporate owners and were alike choked that way.