Comment by Doctor_Fegg | Hacker News Reader

Comment by Doctor_Fegg | Hacker News Reader

Parent Full thread

But what when there are separate interfaces specifically designed for real users and machines?

Case in point: OpenStreetMap (run by volunteer sysadmins) provides completely open, machine-readable data dumps. You can use these to set up your own geocoding, rendering or similar service. There is copious documentation, several post-processed dumps for particular purposes, etc. etc.

OSM also provides a friendly, human-scale, user-facing interface for map browsing and search. There are clearly documented limitations/TOUs for automated use of this interface.

Does that prevent people from setting up massive scrapers to scrape the human-facing interface, rather than using the machine-facing data dumps? No, it does not; and the volunteer sysadmins have to spend an inordinate amount of time protecting the service against these people.

DataHen's proudly admitted practices ("No need to worry about IP bans, we auto rotate IPs on any requests that are made."; "our massive pool of auto-rotating proxies, user agents, and 'secret-sauce' helps get around them") is directly antithetical to this sort of scenario. I find this incredibly irresponsible and unethical.

Almost always the API is nerfed relative to the web site.

Almost all web sites that authenticate use "submit a form with username and password and respect cookies"; often sites that don't authenticate to use the web site require authentication for the API. Every API uses a different authentication scheme and requires custom programming: for web sites you have the URL of the form and the name of the username and password field and you are done.

If you feed most HTML pages through a DOM parser like BeautifulSoup you can extract the links and interpret them through regexes. You might be done right then and there. If you need more usually you can use CSS classes and id(s) and... done!

I wrote a metadata extractor for Wikimedia HTML that I had working provisionally on Flickr HTML immediately and had working at 100% in 20 minutes. No way I could have gotten the authentication working for an API in 20 minutes.

Of course if there is another, unlimited way of getting the data, then it's perfectly fine to "redirect" people to another avenue.

What this conversation is about (since we're talking about web scrapers here, not "data downloaders" or whatever you would call it) is when there is no other avenue to get the data, you should be able to access the same data via a machine as you could when you're a person.

Sure, but Till/DataHen appears to have a host of measures in place to ignore that "redirect".

Yes, how are they supposed to know if the "redirect" is good or not? Facebook would helpfully redirect you to their Terms and Conditions, something you definitely should be able to ignore and scrape your data til your heart is content anyways.

Similarly to HTTP, the tool does not decide if usage is "nice" or "evil", only the user with their use case decides that.

>you should be able to access the same data via a machine as you could when you're a person

I generally agree with this, but I do see the problem for certain spaces.

Concert tickets, for example. People write scrapers to get the best seats and sit on them so they can scalp them later at inflated prices. Or other first-come/first-serve situations, online auction "sniping", etc.

Same thing applies here as I said in another comment (https://news.ycombinator.com/item?id=28060567), some people (me included) also write scrapers for getting the best seats in concerts/shows as I'm a big fan of the artist and really want to go there with good seats.

Perhaps it would be better to not gate by whether you happen to have visited a website fast enough relative to other people. For auctions, at least, there is a simple solution in the form of extending the auction timer whenever someone bids.

Technically, this is the same rabbit hole that leads to DRM and device intent superceding user intent.

Because that's the only way to be sure.

It's unfortunate that it also allows havoc and burdens good services. But device-over-user is not a future I want to live in.

Surely you must have a way to throttle human-originated abuse (such as someone spamming F5)? If so then this would work equally well for the machine.

Determine a reasonable rate limit and apply it to the human-facing version, with maybe a link to the machine-readable version in the error message?

Reply on news.ycombinator.com