Hacker News Reader

Comment by vasco | Hacker News Reader

Not one of the methods listed here would detract a decent scraper. Moreover you would either screw with your users or with SEO if you wanted to make this or that technique more aggressive.

If your database has really great content it's not because some kid has copy of your website online that you'll lose users. Stackoverflow has been scraped to death and nobody goes to the other sites to check out answers.

View on HN

I don't think there is even any need to scrape stackoverflow. They make all their data available for free.

Right, but other sites scrape that free info and republish it with the goal of drawing traffic. Apparently, Stack Overflow has an API. I don't know the functionality of their API, but it doesn't matter: via API or web scraping, the content of the SO database is all over the web, yet SO is thriving.

You said "other sites scrape that free". I agree that info is all over the web but it isn't scraped, and I don't think you should keep calling it scraping.

Yet there seem to be people who use their data to make a bit of money on the side. I haven't had time to do so, but I wanted to ask SO if the translated SO pages on humbug.in/stackoverflow are against any rules. He punches their pages trough googles translate and stuffs them full with banners.

Nearly drove me nuts because a friend who I was working on a project with kept searching in german. As a result he got the terribly translated pages first followed by the actual SO pages.

They wouldn't indeed, but, it makes it more difficult in the hopes that they scrape a different data source.

If they can easily find another source to scrape the same data, said data probably isn't worth spending this much time and effort protecting anyways.

Do you really think a bunch of scrapers' time is more valuable then yours? I'm assuming that you have the ability to create this great piece of content that people want to have so I think your time would be better spent either creating more of said content or pursuing other endeavors other then making some kids waste an additional 15 minutes figuring out a way to solve an interesting puzzle.

people will not go there, however the other website will get accidental traffic which might be 2% of stackoverflow's traffic, this is a little bit of money for doing nothing but running an automated script.

Reply on news.ycombinator.com