Comment by AshamedCaptain | Hacker News Reader

Comment by AshamedCaptain | Hacker News Reader

> The enemy possesses every advantage. Personal website owners are nearly always just ordinary individuals. They often have glacially slow upload bandwidths, [...]. The enemy has seemingly unlimited funds, massive resources, acres of server farms,

One example of this is Googlebot "attacks". I have a personal, mostly static website that I've been serving from a home connection for over 25 years, upgrading the hardware and moving it with me as I relocated. Out of these 25 years, Google has been number #1 user of bandwidth, CPU time, and as a consequence the #1 generator of headaches.

During the mid 2000s I had numerous problems keeping the site online since the Google crawler would just effectively DDoS it, sending tens of concurrents requests completely ignoring the Crawl-Delay in robots.txt (and most of robots.txt, fwiw). Mind you, this was not the most powerful hardware you could find, since it had to be quite low power to be left 24h on, but it was perfectly able to serve the handful of connections per hour I expect out of my website.

Obviously your only alternative is to just ban the entire Google IP range and as a consequence disappear from the Internet. Nowadays the problem is less noticeable because my 1-2W server can handle the onslaught brought by Googlebot just fine, but they still ignore the Crawl-Delay.

From what I've been told, the Google Bot absolutely tries to respect the robots.txt. The advice I got was to double check that my syntax and expressions were really correct. A lot of people screw it up so it has to be basically perfect otherwise it won't be parsed correctly. Are you willing to share your robots.txt file?

Googlebot supports their set of robots.txt instructions https://developers.google.com/search/docs/advanced/robots/cr...

Crawl Delay is not part of it.

The technical solution is to apply a rate limit on your web server. Then Google bots will receive a HTTP status code 429 "too many requests" that will force them to slow down.

Nginx has `limit_req_zone` that allows fine-tuning the rate limits.

Why not use something like Cloudflare as a caching layer between you and Google?

Because Google is the one that needs to change.

While that may very well be true, it's also wildly impractical. I would imagine the outcome of that stance will be the skeleton meme "still waiting and google still DDoSes my site."

That company did not exist until several years after the events in the story occurred.

Quote from the article:

"From time to time, I consider shielding my website with Cloudflare's services. I resist the temptation, because I strongly suspect that, despite Cloudflare's current good intentions, in the long-run, this is playing into the hands of the corporate-controlled web. Relying on commercial web services limits the freedom we have as personal website owners. Once they have us relying on them, we are forced to play by their rules, and at that point the battle is over. We may not have realized it yet, but we have been captured. Unless we manage to free ourselves, in a short time, we will be working for the enemy. [...]"

Just right within the last few days Cloudflare's DDoS protection has started blocking older and/or niche browsers, so all I can say about them is #*'@~?!§%$§#@!!

What are we blocking? Can you email details to me? (jgc@cloudflare.com)

Thank you, reply sent.

Unrelated, but that is a hilarious username.

Google does not support crawl delay.

You can set your crawl rate lower every 90 days here https://www.google.com/webmasters/tools/settings

other possibilities are here https://support.google.com/webmasters/answer/48620?hl=en there is a way to lower crawling without total blocking (returning HTTP 503 if there is too much load)

I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.

crawl delay was an extension of /robots.txt and never a part of the original "recommendation"

and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...

I really wonder if this is an area that would be best served by creating a cooperative, allowing you to spread some of this knowledge and troubleshooting across a larger number of humans.

But that's fundamentally a social problem, and the field of software is nothing if not consistent in its resolve to try every other solution before resorting to emotional work. Co-ops that try to come up with a set of rules to handle all situations end up failing, because rules-lawyering turns into a game to be won by some people, at the expense of everyone else (see also, HOAs).

Odd, I have always seen the google bot respect the robots.txt. If I disallowed it, it disappeared from my traffic.

Of course, generally people do not want to do this, as they want to appear in google.

Reply on news.ycombinator.com