Stopping web scrapers with HTML character codes?

2 points·icu··7 comments

How robust is the idea to protect my website content from web scrapers by converting all my text into HTML character codes like this: <xmp>Hello World</xmp>

View on HN

Its not effective vs any scraping; how do you imagine scraping works? In the most basic case, a program makes a HTTP request, gets the source of your page, and processes it via a HTML parser.

In some cases, the same HTML source will be run through a headless browser to parse it, in which case javascript trickery can't protect it either.

The basic answer is: if an end user can see it in a browser, you can't hide it from a scraper.

I run scraping with Batch, CURL, SED, BCP and SQL Server 2012 or MySQL. My scripts basically download the source code into a text file, run a regex with SED to get content and then put the info into a DB with BCP. I don't take into account any HTML codes for text so such a technique would break my scraping technique.

If the scrapers are targeting your site specifically, then it's not even remotely robust.

Yes this seems to be the case, thanks for your reply.

You can't stop scraping. The only thing you can do is detect them. If detected, there are several fun stuff to do. Like showing different content. BUT: Be aware that you might block a "legal" SEO scraper!

I believe this could negatively affect SEO

Thank you I didn't even consider this.

Reply on news.ycombinator.com