Hacker News Reader

Comment by JNRowe | Hacker News Reader

Slightly tangential to this post, but related to the series.

Have you had any pushback from the sources you've been scraping with actions? I can't imagine how lightly hitting GitHub from this action is a problem, but a few of the other scrapers you've discussed are hitting resources in ways we've definitely seen website operators get upset about before.

I ask because I've been playing with a couple of scrapers for info I'm interested in, but I'm a little reluctant to toggle the repo visibility switch. Partly, because I don't want the grief and selfishly because I don't want the source to disappear.

View on HN

Nothing at all so far.

I've been pretty careful to be polite though - the majority of my scrapers work no more than once an hour, and the few I've done that are more frequent than that don't run more than once every ten minutes (GitHub Actions can't be scheduled more often than that).

Those tend to be sites which I would expect to be able to handle the traffic very easily.

When I'm operating sites myself I don't tend to even notice a scraper that only hits a single page every few minutes. The things that hurt me are crawlers that hit every page on a site, potentially more than a request per second and with traffic patterns that are certain to hit un-cached sources.

Excellent, thanks!

I'm purposely being careful too. Big backoff on any error, curl's --time-cond from git commit timestamp, etag where possible, etc.

I'm not particularly concerned about an actual DoS'ing issue, more being on the end of one of those stories that pops up every so often("website owner sics lawyers on F12").

Oh interesting - I hadn't considered using --time-cond.

I'm always a bit suspicious of If-Modified-Since because I'm never sure which timezones will be in effect - so I tend to use ETags instead, which achieve the same goal without involving any clock time.

I actually wrote a tiny Python CLI tool for doing those, designed to be used in CI: https://pypi.org/project/conditional-get/

Reply on news.ycombinator.com