Unless they mitigate those risks they will only exist for as long as google or bing wants them to. The only ways they survive are: - Mitigating those risks and costs (e.g., building/using own index, well designed caching could help) - Staying small enough in terms of searches and users to be under the radar for Google and Microsoft - Pray for the mercy of two of the most ruthlessly anticompetitive companies in existence (laughable) - Convincing Google or Microsoft that they are worthwhile to acquire (but this kills the service for me anyways)
Price hiking +150% for the stated reason that my direct competitor increased my costs certainly shows the pressure is on and working as intended. On the off chance that kagi devs or management reads this, PLEASE find a way to isolate yourself from being totally reliant on google,bing,etc. Unless you are going for an acquisition exit from Google or Microsoft, it will kill your company eventually.
Is there a legal issue with spoofing user agent to be the google crawler? Spoofing is certainly enough to get rid of article paywalls for 99% of sites Ive encountered. At least last I heard you can also work around cloudflare captcha by just routing requests through a worker on their service.
Fascinating; cnn.com reports 47 on the front page, npr.org is at 16, developer.hashicorp.com is at 9. I don't think that metric is doing what they think it is, or rather maybe they're trying to target only savanna.gnu.org style sites or something