Crawly – Never write another web scraper

0. Arrived at site

1. Entered HN URL

2. Entered email reluctantly

3. Checked email, nothing received

4. Started writing comment on HN, frustrated

5. Checked email again, still nothing

6. Posted comment, disappointed

Link to more of an overview / documentation without having to fork over an email address? Silly.

I got a reply within 3 minutes.

The email said, "We set our crawler loose on <site>, and WOW did we find some interesting results."

The resulting CSV file? 0 bytes. I guess that's "interesting" in its own way.

This won't work when I need to scrape 100,000 pages in an hour.

CloudScrape works

www.ncbi.nlm.nih.gov };)