Hacker News Reader

Comment by DocG | Hacker News Reader

We had a client who contacted with an issue: site goes down every 15 minutes. After couple of hours of debugging we found the culprit. Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be. If you decide to scrape a site dont bring it down or assume its a big server. Keep it civil and without impact on performance and nobody minds.

View on HN

Correct call on keeping it civil. I know for all the scraping I do I put my name and contact info in the header so if people try to look and find an issue they know how to get in contact. But I also make sure I'm not making requests that would ever affect their servers. Gotta stay nice.

On this note, does anyone know of a drip/gradual scrape process for not overloading a host's servers, such that the process could run over the span of a day/week? I'm not worried about the content changing rapidly.

Uhhh, have a brain and common sense when you write your scraper?

The simplest thing to do is NOT parallelize your scraper, and sleep for a few seconds between requests.

Why have you not reported it to them or/and police?

You don't need police in environments where you can defend yourself.

Ok, so for them it is no longer a problem, but what about other people? It is their ethical duty to report them to the police, otherwise they were an accessory in the crime (In some countries there are laws to enforce this)

If you make a shitty site that can't handle a few concurrent requests this is your problem. It's not a DDoS. Not even a DoS. Just trying to request some pages you make available, and you failing at doing so.

Or do you also plan on suing Google once they scrape your site?

Looks like somebody didn't read the parent.

> "scraping" site every 15 minutes with massive ddos

Google doesn't scrape every 15 minutes. It was implied that they scraped the site often enough to cause a DDoS.

Looks like someone has never ran a website with even a slight amount of traffic. A few requests every 15 minutes is nothing and your site should be able to handle it.

Additionally, it clearly wasn't a DDoS. Quoting from the parent:

> Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be

If there is only one server ip, it is by definition not distributed. Just a misuse of the term DDoS.

> Google doesn't scrape every 15 minutes.

Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the site responds slow.

Additionally, Google respects `/robots.txt` (which allows the site owner to define a crawl delay) and uses sitemaps for hints, so it doesn't have to re-crawl every single HTTP object "every 15 minutes".

We were contacted as outsiders to come in and have a look. We managed to stop site from crashing with configuring but basically the site was bombarded with too many dynamic requests and having no cache it took its toll. Server did not wait for responses, it just sent us bunch of requests all at once.

Any decent webserver can throttle requests per source ip.

Yes, agreed though you must also take into account the idea of the "straw that breaks the camels back".

We gave the info to the client. We were in no position to report it.

How is a single server a ddos?

Did you mean "We couldn't handle the traffic caused by their scraping which lead to a denial of service"?

You are correct. Dos was correct term

Reply on news.ycombinator.com