The problem begins and ends with the conflict of interest that Google both sells ads and selects search results. If they didn't have a vested interest in people visiting sites with their ads on them, they could decimate the number of spam results.
The only thing that influences Google search results is Google's desire to keep as many people using Search as often as possible, since nearly all of Google's money comes from showing those text ads at the top of the Search results. This is all public information, you can read it in the 10K etc.
So if Search sucks, it's not because Google has the wrong incentives but because they can't solve the problems Search faces.
Admittedly more in some areas than others.
If a site is hosting ads from the Google ad network and is barely above being spam then Google will prioritize it over other results in order to maintain its quarterly revenue predictions.
I use your crazy search engine all the time, and it's fucking great. Last time I checked, though, you're getting like $30 a month from Patreon and that's it, haha.
You're too much of a well-adjusted normal human being, that's your problem. You need that touch of Zuckerwellian sociopathy to get to the next level
To be serious, it's notoriously difficult to actually make a profit off search. I guess we'll see if Kagi will be able to eke out a market for itself, that may change things. Right now, I think investors will look at Bing and go "yikes".
Doesn't go to say I couldn't build a fantastic search engine with more resources than my current shoestring budget, the hard question remains: Why isn't this just a waste of money? Would take one hell of a sales pitch to gloss over that concern.