Hacker News Reader

Comment by hartleybrody | Hacker News Reader

This is one of the best arguments for scraping data, if you can manage it. Generally businesses invest much more time and resources into maintaining an up-to-date, usable website than they would on an API, especially if they don't use a service-oriented architecture internally (ie, consume their own API)

View on HN

What type of business can afford being down every time the site they're scrapping is tweaked?

How often do websites really change the layout of their DOM? Sure, some sites that are actively developed (ie social networks) but for a lot of business websites where information doesn't change that often, like restaurants, reviews or clothing, websites are pretty consistent.

Plus, building a parser isn't always that much more work than trying to build a client for a convoluted API.

It's obviously case-by-case, but I think a lot of people dismiss scraping outright when there are a lot of useful applications.

1. Any business that scrapes many sites.

2. Who says it goes down? Might just have stale or incorrect data before the tweak is patched.

OMG, what?

Scraping data for offline processing, but certainly should not be for user-facing traffic. Cosmetic tweaks can blow up the scraper's parser, UI technologies are built on a taller stack and are prone to higher latencies and points of failure, yada yada.

Indeed. Very risky. Scraping is not really a sustainable model when you need predictability. Too many variables.

An example comes to mind:

The Yelp API doesn't expose a given user's public bookmarks, reviews, etc., but they can be easily scraped from the user-facing website.

http://www.yelp.com/developers/documentation/v2/overview

Reply on news.ycombinator.com