That's the big problem here. All the new search engines rely on the infrastructure of the old ones. It would take very little collusion between Google and Microsoft to crush something like Kagi. And the moment any new competitor gets big, they'll do it.
What's needed is an anti-trust breakup. There's simply too big a conflict of interest between Google, the search engine product, and Google, the ad-tech giant earning money off the websites.
Google is using their search engine monopoly to control the wider web, and fill it with spammy SEO garbage to juice their advertising revenues.
That's the dumbest part of it all; Kagi's not an entirely new search engine. It's Google's index but without the ad-tech juicing, and some clever use of a few other search indices to specialize results.
(I.e. Google could unshit itself at any moment, it just doesn't want to.)
The problem being the aforementioned conflict of interest. The moment you're a competitor to Google Search (or any of Google's other products), they can kill your API access and ruin you.
"In a way, the brief flicker of Neeva’s existence tells everything you need to know about the last 20 years of search-engine supremacy. Building a search engine is hard. Building one better than Google is even harder. But if you want to beat Google, a better search engine is only the very beginning. And it only gets harder from there." [1]
[1] https://www.theverge.com/23802382/search-engine-google-neeva...
Not to mention Kagi's continued sustainability issues (28,071 users at the time of writing) [2] and their plans for their own AI products [3] [4] that would inevitably and eventually doom them.
[2] https://kagi.com/stats [3] https://d-shoot.net/kagi.html [4] https://news.ycombinator.com/item?id=40011314
Also, ADeerAppeared is right that government intervention aka a major breakup needs to happen for competition to thrive and I don't see it happening (although I hope I'm wrong).
And if you want to know our plans, just check our own blog [1].
Any new entrant sort-of has to prove itself also as crawling sites can be hard, e.g. Cloudflare which fronts an appreciable portion of the web. Also large sites like Reddit you need to crawl them at 100s of pages a second to keep up to date with content, not something they're generally going to appreciate without a recognisable benefit to them.
It's very hard, getting harder as websites grow more hostile to crawlers. (With more and more putting up login walls to keep poorly written AI crawlers out.)
But the real problem is that tech is culturally unused to dealing with monopolists "on even ground". The wider tech community operates on the implied worldview that they can take on anyone by just asking daddy A16Z for a duovigintillion dollars. For 25 nearly-continuous years, you could just outspend to get into even the most hostile markets.
That economic climate is just ... gone now. No more free money. You can't just buy your way to the top anymore, the business has to financially work.
And most importantly: You can't outspend the incumbents anymore, but they still can outspend you.
We'd be better with 5 players doing things slightly differently. Nobody would get completely wiped by a single change.