Hacker News Reader

This is what made me laugh:

The Administrative Office of the U.S. Courts reported that the PACER system was being inundated with requests. One request was being made every three seconds.

Inundated by one request every three seconds? That's pretty pathetic if you ask me. They must not have planned for the PACER system to be used very much by very many people.

View on HN

That's fair, but remember that if you scrape HN with a 'sleep 3' your IP gets throttled quickly and then eventually blocked ...

What would be an appropriate delay for HN?

It depends on what you're using it for. Unless you need the bleeding edge of new stories, 30-45 minutes between each scrape is probably fine. If you're looking to stay on top of 'new', your best bet is about every 10 minutes.

More frequent than that, and you're just causing trouble for the server with no real benefit for yourself.

Paul already provides data dumps. Searchyc for it.

I wonder if that is pretty standard? Do most services throttle at 3 seconds or is that being overly strict?

Most are only throttled by the bandwidth at either end.

Reply on news.ycombinator.com