iFixit CEO Kyle Wiens calls out Anthropic for disruptive crawling

> I feel the blame mostly falls on Anthropic not being considerate when they scrape other people's content.

That's certainly a fair assumption. I haven't looked into it in any detail. Is IFixit sending appropriate cache headers or is Anthropic just ignoring them? I don't know. But at some point, your caching strategy and your ability to respond to spikes in demand is your own problem. We live in reality, and in reality you can't really control who is hitting your website when. Sure, it would be nice if they respected your cache control policies, but what are you doing when you know they won't? I'll refer back to my previous statement. It's totally understandable when someone who didn't expect ever HN levels of attentions to be overwhelmed. But if you're a site as large and respected as IFixit, you've got to do better than the "unwashed masses".

I believe you. But at the same time it's insane to me we have these standards that aren't upheld on one end or the other. I'm fulling willing to acknowledge that Anthropic is the problem here. But at some level, and at some point, don't you lose the ability to say the site is down because it's too popular? I'm totally open to the arguments that the traffic overloaded the Cloudflare limits or whatever. But the frequency of outages caused by not just traffic at Anthropic levels, but at random HN post levels are enough to take down websites. I guess this is just my cognitive dissonance between being told on HN "you don't need to scale! You're not Facebook!" and "OMG our went down because we have a few thousand concurrent visitors!"