Comment by bastawhiz | Hacker News Reader

Comment by bastawhiz | Hacker News Reader

Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or a binary file that they surely don't care about. I see them in my logs for my podcast hosting service, hitting the same 404s for months and months. They end up hitting audio files and downloading (or attempting to download) many gigabytes of content each day. They don't do anything with audio! As soon as they get the HTTP headers, they could say "oops, I don't care about this" and disconnect. And in many cases, they got those URLs from the enclosure tags of RSS feeds... I'm not sure what they were expecting!

In the cloud, not in the cloud, frankly it seems to me like their business would be far more efficient if they tuned what their crawler actually crawled.

Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.

Why not use Semrush?

You don't have to "use" ahrefs for them to crawl your website

I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.

Why not just block any requests with Ahrefs user-agent?

I cannot express emphatically enough how powerful the concept of classes of service can be for simplifying your production issues.

I think people see the notion of a network neutrality as an all or nothing thing. Network Neutrality between networks, yes, but within your racks you’re trying to balance a large equation instead of two or three smaller ones.

If they reclassified sketchy pages they wouldn’t have nearly as much of the sorts of trouble you mention.

No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O.

My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s

Reply on news.ycombinator.com