Of Spiders and Scrapers: Decomposing Web Pages 101 | Hacker News Reader

Of Spiders and Scrapers: Decomposing Web Pages 101

linux-mag.com

6 pointslinuxmag9 comments

There are plenty of legitimate sites without data feeds where scraping becomes necessary such as some .gov sites.

"Over the years, I’ve likely scraped tens of Web sites. In fact, in 2005, I scraped the entirety of the then-current Linux Magazine site to port its content to a new content management system."

Society does not benefit by allowing these people such freedom. So, shouldn't this guy be prosecuted and sent to jail?

Huh? The guy wrote this article for the same magazine he scraped articles from back in 2005 so I'm assuming he has, at the very least, some sort of affiliation with them. It sounds like it was part of a data migration strategy rather than theft.

Yes, I realised that after reading the quote another three times. The article itself however seems to be about scrapping, or theft, so I thought that quote describes what he does well, but of course failed to see that he works for Linux Magazine.

Scraping itself is not theft. It could be used to commit theft, but there are far more legitimate uses.

Scraping is theft. Alone downloading and storing a copyrighted article is forbidden by law in certain countries.

It doesn't matter if you make the article public then or not.

Somebody should tell Google.

Sorry but this just isn't how the web works. Scraping is identical to crawling is identical to browsing is identical to caching.

Republishing someone else's work without their permission is probably copyright infringement but just downloading and parsing it is not.

That's not true.

The european legislation has explicitely made an exception taht you are able to download and cache the document, as long as it is necessary for something to work (eg browser). Or if you are using it for personal use only.

But otherwise, storing a copyrighted document is illegal (interstingly twitter messages are not considered copyrighted).

Reply on news.ycombinator.com