The whole point of the API access change was to charge AI model-makers. I'd be ironic if the API change made destroyed their product and made their data unsellable.
Recently, they came around looking to recruit me. I told them fire Spez or fuck off. (17 year Reddit user here)
I think if the business model had been thought in the sense of the communities and involving mods and users in, it would have been genius, a lot of smaller companies would kill to have people genuinely recommending their products/tools that are hidden behind the biggest wall of them all...
Alas they completely ignored this as a viable avenue and went for the quick buck.
But this won't last and at some point people are going to move on from mass scraping, either because they already got what they want or because garbage goes in garbage comes out or because most of the content will be bot generated and require too much filtering to be useful.
Of course this is the opinion and rambling of a moderately educated individual and I might be totally wrong.
Change can often be for the best, pretty sure it wasn't in this case...
Aaron is probably turning once again in his grave...
I have no idea how intensive these are, as I most of these over bar drinks when I was traveling. Could've just been someone making stuff up as well. But my marketing friends have confirmed that they use Reddit very heavily, as it's a great sales funnel if you play it right.
https://www.reddit.com/robots.txt
Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.
Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.
https://help.kagi.com/kagi/search-details/search-sources.htm...
[0] Gradually for several years already.