Hacker News Reader

Comment by rmbyrro | Hacker News Reader

I have sympathy for your operational issues and costs, but isn't this kind of complaint the same as a shopping mall/center complaining of people who go in, check some info and go out without buying?

I understand that bots have leverage and automation, but so does you to reach a larger audience. Should we continue to benefit from one side of the leverage, while complaining about the other side?

View on HN

It's more like a mall complaining that while they're trying to serve 1000 customers, someone has gone and dumped 10000000 roombas throughout the stores which are going around scanning all the price tags.

No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side.

My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)

I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.

You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.

Just advertise the API in the headers. Or better yet, set the buttons/links only to be accessible via .usetheapi-dammit selector. Lastly, provide an API and a “developers.whatever.com” domain to report issues with the API, get API keys, and pay for more requests. It should be pretty easy to setup, especially if there’s an internal API available behind the frontend already. I’d venture a dev team could devote 20% to a few sprints and have an MVP thing up and running.

For the 2nd part, I have done scraping and would always opt for an API if the price is reasonable over paying nosebleed amounts for residential proxies

I think lots of website owners know exactly where the value in their content exists. Whether or not they want to share that in a convenient way, especially to competitors etc is another story.

That said if scraping is inevitable, it’s immensely wasteful effort to both the scraper and the content owner that’s often avoidable.

Writing a scraper for a webpage is typically far more development effort than writing an API wrapper

Yes, but the scraper in this context is already built. A bird in hand and all that.

In this case, yes, obviously. But as far as "it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API", that presupposes no existing integration.

Yes, exactly. Nobody is standing up and saying "we're the ones doing this, and here's what we wish you'd put in an API".

Also, I'm a big Jenson Button fan.

No, because those are people going to the mall. Not robots 100x the quantity of real people.

Unlike the mall a website can scale up to serve more users at relatively low cost. Also those robots may bring more people to the website. Potentially a lot more even.

Reply on news.ycombinator.com