Avoiding Webscraping Throttling Using Python and Tor as a Proxy
boredhacking.com
boredhacking.com
I'm particularly sensitive to this because I'm constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.
(note: it is not a captcha solving service, its a collaborative project by cloudflare, google and others)
5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services;
And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms.
Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively.
If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
It's what I'd say, except said more clearly.
It may be naive to assume people will honor the desires of a website owner, but that doesn't change the moral argument - you shouldn't try to convince yourself that you're in the right just because its difficult for the website owner to enforce their wishes for how people use their system, or because everyone does it - "that's just the way it is" doesn't make it right.
I don't think we're talking about discrimination or equal access or even right to back up issues which might apply some moral right to access a site in ways not desired by a service owner - the vast majority of cases for scraping are where the scraper goes on to use that information in ways that are for the commercial benefit of the scraper and usually to the detriment of the site owner.
The arguments about data access to publicly funded information has more merit but I don't see them applying in any way to private services.
Am I missing some other fundamental right that applies here?
Yes, and they extract that benefit because in the process they add a value to that information for end-user.
> Am I missing some other fundamental right that applies here?
Yes. When it's not prohibited by law, it's allowed.
Frankly, I'm glad I don't live in your community.
I just got back from grocery shopping. Before I went in, I stomped off some mud I had on my feet. As I walked down the aisles, I accidentally knocked a bag of something off a shelf, so I turned around, picked it up, and returned it to its proper spot. When I got to the ice cream coolers, I first looked through the glass to decide what I wanted, and only opened the freezer door when I'd made my selection, sealing the door again once I was done.
There aren't any laws forcing me to do those things. But I'm happy to do them, because (a) when we all cooperate in this manner, it makes life better for all of us; and (b) the additional costs of cleaning and shrinkage would make the groceries more expensive for us if we didn't do it.
Have you ever been to a part of the world that doesn't have the same kind of "high trust" environment that most of us in the West enjoy? For example, in America we can safely assume that at a bus stop, people will of their own accord form up an orderly line when the bus comes, each person waiting their turn. My experience in some other nations is that people will swarm around the bus door, each jockeying to be the next one in. There aren't any laws forcing people to act as they do here, but I think we're all better off with the societal norm that drives people to do it our way.
You assumed that people who don't want their data being scraped are the nice guys and people who want to use that data are the baddies. Nope, my point was that it's the other way around and if you're truly concerned with community wellbeing you actually have to be on my side of this argument.
I support every point in your response expect that I don't get your motivation to fight for folks who either don't understand how internet works or consciously use dirty tricks to block information access just to protect their shaky profits.
Look it up, my first comment in this thread was about making it illegal because I think everybody would benefit from making it so. If you publish your data and it's accessible by browser and not copyrighted, we shouldn't make it hard to collect that data for automated processing.
Look up to the top of this sub-thread, I started it. In that comment I specifically addressed that in the systems I'm responsible for:
- The traffic that (apparently) comes from competitors scraping our prices exceeds the traffic coming from legit customers. We're paying more to supply data to our competitors than to our customers!
- There are sometimes actual reasons beyond wanting "shaky profits" for wanting to limit what site users can do, including development resources to built features and APIs, as well as the actual cost of the computations.
- And I have no idea where your assertion about "dirty tricks" comes from. I'm having trouble finding anything "dirty" in trying to detect people abusing the system, and temporarily blocking their access.
And like I said above, I'm still glad I don't live in your neighborhood. Because not only don't you have any interest in being a positive member of society in commerce (you ignored all my comments on that topic), but I now see that on a personal level (e.g., "You assumed that...") you're also condescending.
Make an API, put your site behind CDN. Couldn't be more simple. And there's more they could do.
> - There are sometimes actual reasons beyond wanting "shaky profits" for wanting to limit what site users can do, including development resources to built features and APIs, as well as the actual cost of the computations.
Already answered that.
> - And I have no idea where your assertion about "dirty tricks" comes from. I'm having trouble finding anything "dirty" in trying to detect people abusing the system, and temporarily blocking their access.
There was an attempt to make a startup to compare prices in local stores, that caused an outrage among shop owners. They too claimed they were "abused". If you dive into how all those standards like html, http and etc were designed in a first place, you'd find that they were made with an idea that data is expected to be easily digestible by machines. Fighting it is futile and postpones us from having nicer things.
You could just export your prices in some CSV form on regular basis if making a proper API is too hard and redirect incoming scraping traffic to some README page instead of fighting a battle you can't possibly win? That's of course only valid if that business doesn't mostly rely on depriving customers and competitors from information about prices. In that case you have my compassion, but it's clear that pro-community-social-bla-bla-bla rhetoric is nothing more that a disguise. That I understand, but oppose.
Or should I finally respond to that lame remark about neighborhood you're trying to push? Meh.
Look, luddism is bad for everybody in a long term. We saw riots against Uber in some country where people turn cars upside down or burn them. It hurts their earnings and they feel it's unfair. And you may say it's immoral for Uber to do so. But what they did is simple. They identified multiple huge inefficiencies in that market: pricing, negotiation, checkout, reputation. Solved it and found a way to profit from it. And that will eventually happen at every corner where huge inefficiency exists due to mere lack of communication and price negotiation. And yes, prices in most cases will have to go down and become less disperse and some won't like it at all. But that's just competition reenabled by technology.
Is it immoral? I don't think so, because net effect is positive. Is that your right to scam tourist 10x more for airport-hotel trip or take 2x longer route just to earn a bit more? Technically yes, but who would indorse such behaviour? You don't like the price? Don't take that client. Platform is systemically lowering prices or violates existing regulations? Vote for better regulations, vote for enforcing those that already exist. Platform is fundamentally broken? Well, make a better one. Technically, it's not that hard, Uber is one of the most replicated business ideas at the moment.
And on that guy complaining that oh those competitors who are scraping their prices. So they're scraping each other and protecting their websites from being scraped by each other? So at the end they all have competitors information, but pumped lots of resources in scraping, protection and trying to serve content to both bots and clients? Wow, what a tragedy. What a horrible person would want it to stop. If that's it, they could just as simply pick up a phone and agree to share that data between them, because in the end outcome will be the same, minus resources wasted on arms race.
Or better, from the beginning make your data machine-friendly. Because eventually, they'll do that. Eventually somebody like Google or Amazon or some other big company will find an incentive to make them gradually and willingly share and structure that data. And somebody will find a way and resources to integrate that data into reusable knowledge graphs and somewhere along the way create a positive feedback loop. And somebody will profit from that huge. Consumers will surely benefit, that somebody will, data-donor companies that adapt will do.
And don't forget there's some progress in ML, automated decision making and all that. I personally as a customer would love to have best prices, objective products comparison with zero interaction with multiple whacky small vendors websites. I'd better have smth like Siri do that for me.
One way or another, it's happening. Small businesses have little to say here if anything. My unpopular opinion was to recognise that process, do something to stop wasting resources on war between scrapers and anti-scrapers and hopefully avoid appearance of another single monopolist from solving the problem if we don't. Because if we don't we'll just have another few years of HN headlines about how bad that X unicorn company is to somebody.
Now what's wrong with that?
I get it, really. World's change quick, people don't. They have life, it hurts, they're sad. We're empathetic, we're sad too. We don't want to be sad, so we don't want them to be sad. See? Easy. Except the fact that it doesn't help anybody other than mild therapeutic effect. And is completely irrelevant to this thread.
If anybody cares to explain and expand what is that moral issue I'm missing here or how is it relevant, I'm all ears.
I feel like I've put much more effort in making that social interaction fruitful and got only lazy "I'm sorry for you" in return. Ouch.
What ethical qualms are those? The fact that someone else tells you that by using something you agree with a bunch of agreements that they made up on the spot?
However, I do agree that people should care more about the ethics of web scrapping.
Tell that to Aaron Swartz.
LOL! Never read 'em, never going to, couldn't care less about any of them.
If it was that important they'd make people create and account and sign in and limit access that way. Don't expect people to take all the blah blah blah seriously. It's just a web page.
Abusing free and open leads to people locking down free and open.
Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!
I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them.
Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.
Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind:
- Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article)
- Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
So not even a tiny bit sleazy then?
Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
We offered an API for just two customers, with images and product information for video games. One of them preferred scrapping our page anyway, so we throttled them.
The person who started this reply thread repeated part of it twice for good reason. Bots, unchecked, can and do take up a majority of the resources on small to medium sites.
I'd have been much happier with the article if the author had taken a moment to implement a bit of throttling on his end to at least go easy on the remote server.
T&C's have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&C's.
[1] https://www.eff.org/deeplinks/2018/04/scraping-just-automate...
[2] https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
[3] https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
Where would this be significantly different? For T&C to be valid, you'll have to agree to them before using the service. Since you generally can't know, much less agree to, the T&C of a website before using that website, you have a chicken & egg problem where the law tends to come down on "these T&C aren't binding" imho.
You're right about the re-publishing though. Storing is likely a different issue, even browser-caching is storing; iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy.
My point exactly.
https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scr...
Today everything open enough can be considered "free to do whatever people want to do with it". Before doing anything in this world I think you'll have to face this and think about how to deal with it if you have to.
OT: I don't think it's great that small competitors have problems because of this but on the other side I'm not one of those people who repeat the idea of "competition as being good for innovation & customers" that we all heard a lot in some economic lectures.
I think in an ideal world there'd be no need for more than 1 basic model of smartphone, notebook or car. In this disney world we'd have global efforts to innovate, develop an manufacture what we need and build that stuff in the best possible quality with fairness, modularity and reparability in mind.
Yes you'd call this monopolies but in the end the freedom from this endless competition makes another way to work possible. Also innovation doesn't have to come from competition as we've seen from many examples (including what the first intelligent "homo XYZ" invented).
Then of course there'd be no need to scrape for battling your competition. We'd have APIs for everything we'd like to know. Wouldn't this be nice?
> After seeking brief details of his employment, gross income, and savings, and whether he was a property owner, [the online bookmaker] asked him to specify a password, a memorable question and answer, and a bank account number.[...]
> In seeking to identify such a pre-existing contract counsel for the claimant faced significant difficulties. In particular he was unable to point to any promise or commitment by the claimant which might form part of such a contract and provide the consideration necessary to make it legally binding.[...]
> The provision of an on-line interactive platform is in effect simply a more modern equivalent of the expressed readiness of a potential contracting party [...] to enter into contracts by receiving and responding orally to telephone calls.
> Accordingly, the claimant has failed to establish the existence of any legal contract to host and give binding effect to [the clause]
Also of note:
> Many, one might suspect most, would have passed up on that invitation [to read the documents] and proceeded directly to click on "Agree", even though it was suggested that they should do so only when they had read and understood the documents. Even if, exceptionally, the defendant in fact chose to look at the documents, he would have been faced [with a long and complex agreement]. It would have come close to a miracle if he had read [the clause], let alone appreciated its purport or implications, and it would have been quite irrational for the claimant to assume that he had. This was an entirely inadequate way to seek to make the customer liable [...], and is a further factor rendering the [clause] an unfair term.
Spreadex Ltd v Cochrane [2012] EWHC 1290 (Comm) http://www.bailii.org/ew/cases/EWHC/Comm/2012/1290.html
Note that this is not necessarily a full appraisal of the law, but merely meant to illustrate that forming a binding contract is more difficult than having terms available. Furthermore, even if there is no contract, there are IP considerations (copyright, database rights depending on the data) before you can use the data.
If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.
If you block the small guys from scraping you're just helping already huge monopolies tighten their grip on the content market. Even "nice" scrapers like Google blatantly steal your content, blocking everyone else just makes their monopoly worse. They steal pricing data for "Google Shopping" regularly.
If it annoys you that much just build an API and charge tiny amount to use it.
Better, I think, would be using HTTPS proxies. But not free ones, which tend to get burned down pretty quickly. There are sites that lease private proxies, and guarantee that they work.
> https://smartproxy.com/scraping/proxies
> Global Residential Proxy Network for Web Scraping. Never Get Blocked Again. High Reliability, Easy to Use. Choose Your Plan Now!
Sessions are the only way to use crawlera with libraries like cloudflare-scrape, which pin your authentication to a specific IP.
<span class="pull-right" id="ipv4">2a0b:f4c1::7</span>