The issue is that they don't want to. They'd rather be a middleman offering you "useful recommendations" (that they or may not sell to the highest bidder) instead of offering you value.
I can assure you that force is strong with me.
I was using searx instances with reasonable results but many of them started failing recently.
Anyway, I hope everyone finds a good one. I fear things will only get worse though.
You remember the pagerank paper? It described how Google classifies each page on a scale from "something that links to good pages" to "informative page". Google and other search engines produces links to the latter. And since then, web site operators have had strong incentives to be on the "informative page" end of the range. Today I don't think the 2000s code can find a lot of pages of the former kind (well, outside Facebook).
It produced very nice results back then. It was/is good code, but good results need more than just good code, it needs good input too.
It makes no financial sense for Google to give you good search results and get you off Google as soon as possible.
Instead, if they make you spend more time on Google due to having to go through more crappy results, they can sell more ads.
Most people won't change search engine and will stomach it.
Until ChatGPT happened and can save you the pain of having to use Googles search engine.
Google operates the largest display ads network. They literally *pay* websites for SEO spam, and take a very healthy cut off the top.
I wish people would stop acting like Google has been in a noble battle against the spam sites, when those sites generate Google billions of dollars a year in revenue.
The obvious question is, why would they ruin search for display? The answer is greed combined with hubris. They were able to double dip for years, but they killed the golden goose.
Everybody with a brain knew this would happen when they bought Doubleclick, and it took longer than expected, but here we are.