Ask HN: Who uses web scraping?

6 points·AlexMcP··3 comments

A few survey questions about collecting data via the web:

Do you have experience with people using web scraping commercially?

What field or industry was it being used in?

Do you think it fell in a 'grey' area, or was it explicitly disallowed or allowed?

Thanks for any info!

Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments. Although I am happy that I did. :)

Eventbrite used scraping in the early days. They first argued it was a publicly available website. "Public" in the sense that the site was browse-able without restriction.

The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them."

[1] - Yes

[2] - Lead generation, selling data, consumer + market research

[3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.

Reply on news.ycombinator.com