Web Scraping in 2016
franciskim.co
franciskim.co
OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi...
Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.
(Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.)
For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good reason - it's reflective of how the the demand in the market. People want cheap leads to spam, and that's bad use of technology.
Generally I tend to focus more on words and phases like 'automation' and 'scripting a bot'. I'm just automating my life, I'm writing a bot to replace what I would have to do on a daily basis - like looking on Facebook for some gifs and videos then manually posting them to my site. Would I spend an hour each and every day doing this? No, I'm much more lazier than that.
Who is anyone to tell me what I can and can't automate in my life?
It's baloney.
You can try telling the judge it's baloney, but if he's going by current precedent, he probably won't agree with you.
You are exactly right. But although a site can deny you access for any arbitrary reason (it's their website, after all) obviously government think they are the ones to enforce this crap.
What if the ToS say you can only access a site while jumping hoops? Only read the ToS after a while and wasn't hooping? Well too bad, now you are being sued for reading the main page _and_ the ToS page without jumping around.
This comment Terms of Service: If you read any of this text you owe lerpa $1.000.000 to be paid up until 09/01/2016.
If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic.
Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.
This is a double standard plain and simple, and a very dangerous one at that.
Even normal trespassing laws are way too overreaching (see how it is handled in the UK for a saner example) but now you have the amazing possibility of remote trespassing.
The fun part is that it's just a matter of someone hiding something that says you cannot access the site in a place that you have to access the site in order to read -- the ToS. Suing people over this is idiotic.
The real problem is the involvement of Govt, and this kind of absurdity regarding ToS, EULAs and so on, is something that has been going on for decades. If you have the money you can make Govt your personal watch dogs.
If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.
1. Google can come in
2. Other Americans can't come in
3. Chinese people can come in (or anywhere else where US laws don't apply)
It might not be unfair, but it is certainly pointless and arbitrary.
And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively.
The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case.
I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.
your next argument may very well be a very racist one with the very same excuse you used above.
They could scrape your website and then they prevent you form scraping your own data back.
The whole process is silly; it reflects the duct tape and chicken wire nature of the www.
No one should have to "scrape" or "crawl".
Data should be put into a open universal format (no tags) and submitted when necessary (rsynced) to a public access archive, mirrored around the world.
This to bridge the gap until we reach a more content addressable system (cf. location based).
Clients (text readers, media players, whatever) can download and transform the universally formatted data into markup, binary, etc. -- whatever they wish, but all the design creativity and complexity of "web pages" or "web apps" can be handled at the network edge, client-side.
"Crawling" should not be necessary.
No one should have to store HTML tags and other window dressing for data.
Dream on.
To give an example, there is a lot of free open source software mirrored all over the internet, mostly on ftp servers, but also on http, rsync, etc.
If you use Linux or BSD you probably are using some of this software. If you use the www, then you are probably accessing computers that use this software. If you drive a new Mercedes you are probably using some of this software. There are a lot of copies of this code in a lot of places.
Is that centralized? Does anyone hosting a mirror ("repository") "own" the software? Is it the same person or entity hosting every mirror?
Compare Google's copies of everyone else's data, also replicated in a lot of places around the world. Who "owns" this data?
In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.
Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
Not really.
Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info.
If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them.
In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine.
tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.
And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too.
Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser.
Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect.
Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser.
It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.
Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images.
Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors, and that one very popular user's post are all stored in cache.
Then along comes a scraper. He thinks, "Hey, this site is serving up a million page impressions a day. It can definitely handle me scraping the site".
But when he runs the scraper, he fills up the cache with a ton of data that it doesn't need, causing cache evictions and general performance degradation for everyone else.
Bandwidth is certainly part of it, but there's also also database and app-server load (which may be the actual bottleneck) that a scraper isn't necessarily bypassing.
I'd consider that a bug not a feature but I still think it's incumbent on me, the guy scraping the website, not to trigger it.
And as a webmaster, how can I tell the difference before it's too late?
Analyzing data that you're not allowed to access gives you/your company a competitive advantage, which is affecting their service/business even if it's not posted/distributed publically.
These are the same websites and companies that are loading evercookies and doing browser fingerprinting, that break as much as possible the anonymity citizens should enjoy, with Real Name policies, using network analysis to find who your friends are and what your politics and buying habbits are, that routinely rip private information from you cell phone and share it with oppressive regimes.
You're not in Kansas anymore Toto.
Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part.
We respect TOS, we respect robots.txt and so on. Just because you study scraping techniques doesn't mean you intend to break the law.
> Breaking captchas and the like is basically blackhat work
Um, captchas only work if they work. If breaking them is trivial, they shouldn't exist. Don't shoot the messenger for pointing out the front door is unlocked.
If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth).
It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it?
[1]: https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-d... (french)
Yes, this defense is being petty abotu details, but I find businesses using post-hoc discoverable limitations to limit people rights annoying.
Being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them (even though, all the scraping in the world won't have any impact).
How so? I send a web request, they send me the content in a response. If they aren't happy with that then they should refuse my request.
If I can modify my web browser to view a site, but skip the ads, that should be my right. If the site owner codes their site to detect this and then blocks my request to see their site, that should be their right. If I modify my ad-blocker to get around their ad-blocker-block, that should be my right, and so on. As long as we don't get into something like DDOS territory where a reasonable web site has no good technological way of avoiding the problem caused by a user, this isn't something for government to get involved in.
We need updated legislation that covers malicious actors that issue DDoS attacks but leaves normal people that scrape consciously and carefully alone.
Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right
If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright.
You may access said database (via the web), but as soon as you start reconstituting the database from scraping... you're in breach.
It's a law, it is illegal in the UK, I'm sure most countries have some equivalent law on their books, all of the EU does. The law looks recent, but UK copyright and patent used to cover it, the 1997 date is just a separate statute to clarify the position.
This isn't even true metaphorically. It's like a shop front: there may be public access, but it is NOT public property.
Taking the store metaphor further, it would be more like you knocking on the front door of a clothing store and the store owners open the door and throw every possible piece of clothing at you, shirts, shorts, underwear, including coupons to "partner" stores, when all you wanted was a pair of pants.
Upon knocking, if the store owner hands you instructions on how to enter their store and interact with their products in a personalized shopping experience, that would be one thing. But when the clothing owner throws everything at you at once, what they flung at you is for all practical purposes public property.
This is called "clickwrap". There is usually a notice in the footer of each page that says something like "By using this site, you agree to our Terms of Service." Typically, this kind of notice has been held enforceable. More recently, judges have been demanding that such notices be placed more prominently before they're held enforceable (e.g., somewhere above the fold), but that's it.
>Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
The reasonable laws that exist in meatspace are not applicable online, because once you hit someone else's server, you're considered to be on their property and they have the right to control what you do there. There is no "public property" from which to safely stand and take photographs in the internet.
Also, photographs of structures may not be free to use. Architectural copyrights went into effect in the early 90s and have a term of either 90 or 120 years. Thus, if you take a photograph of a building built in 1991 and the year is not yet 2111, there is a chance that the architect can claim infringement.
1. Total privacy, they will not track me activity on their website, including any logs.
2. They will send me a cashier's check for $1,000 for each byte that they send to me.
3. They will provide me with Mana Sakura's cell phone number.
I'm still waiting for checks and a phone number.
It is ridiculous. Something like "pagewrap" can't trump the consumer protections that apply to a physical good like a book, it would be laughed off. But the law doesn't contemplate network access so reasonably.
The architect can claim infringement all they want, they don't have a case. From https://www.law.cornell.edu/uscode/text/17/120 :
The copyright in an architectural work that has been constructed does not include the right to prevent the making, distributing, or public display of pictures, paintings, photographs, or other pictorial representations of the work, if the building in which the work is embodied is located in or ordinarily visible from a public place.
This is an important caveat to architectural copyright, however, so thanks for clarifying.
See also https://mentalhealthcop.wordpress.com/2013/09/20/place-to-wh... which shows the same ambiguity exists in the UK.
IANAL but this seems perverse. In no meaningful sense am I on corporate property when my computer in my house sends signals to another computer, formatted so that they will be re-sent in turn to a series of other computers, the last of which decides on its own based entirely on the signal it receives from the penultimate host to send a "response" to a different series of other computers, the last of which is my computer in my house.
Surely there are better ways to enforce IP restrictions than this tortured analogy of networked computing to physical location?
Even if we entertain a distinction between browsewrap and clickwrap, browsewrap is generally enforceable, especially after minor modifications to placement and/or font size.
Even for sighted people, the notice is often easy to miss - and this is by design.
https://en.wikipedia.org/wiki/EBay_v._Bidder%27s_Edge
The courts have generally disagreed with that interpretation.
No, its not. It may be in public view, but that's a different issue.
This is a gross misunderstanding of how the internet works.
So I'm not so sure that police will escort you out of a Walmart because they caught you taking a picture of the parking lot with your smartphone.
If you're entering a country, do its laws not apply to you until you've seen a copy of them? "Oh, sorry, no one told me theft is illegal here. Where does it say that? Oh, I see. Okay. I'll stop now. Thanks for letting me know."
If you cross the border without necessary documents, does that country have no right to detain you, simply because you haven't checked the laws?
Just because a website is visible and public doesn't mean its content is public domain. It just means that your first order of business as a user should be to check the terms of service. Sure, most people using a website probably don't need to--same as not needing to check a country's stance on murder--and so can just use the website as intended without violating the terms. But when you plan on using it in a way that might not be intended, and you don't check the terms of service, well, that's on you.
I don't need to check your terms of services if I'm doing something that I'm allowed to do by law anyway; the TOS cannot deny me those rights (they might, of course, grant me additional rights provided that I follow certain conditions).
The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be out of jobs.
As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database.
This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much like BitTorrent, they'd get into the protocol and send subpoenas to the ISPs behind the IPs that serve their pages, and use that info to sue the customer.
When my company was shut down by a legal threat related to scraping, I did suggest to my lawyer that we create something like a browser extension that would grab the data we needed out of normal client-side browsing sessions. This wouldn't be as nice as controlling the flow of information ourselves but it would've worked OK. My lawyer strongly suggested avoiding that as it could've been construed as conspiratorial conduct that would've made criminal prosecution more likely.
I think imbuing technical protocols with legal implications would be even worse than the current situation since then changing anything on a protocol would require changing the law and getting a protocol implementation slightly wrong would carry real-world legal repercussions on the order of licensing your work in the public domain rather than retaining copyright. Let the lawyers make the law and check the human terms of service before using the data. Trying to out-lawyer the lawyers is like challenging a hedgehog to a butt-kicking brawl.
"The User-Agent request-header field contains information about the user agent originating the request. This is for [...] the tracing of protocol violations [...]. User agents SHOULD include this field with requests"
Many scrapers disregard this part of the protocol. Of course, whether a headless browser should send a different UA is an interesting question.
The CFAA is a really bad law and creates the network effect lock-in that we all considered a natural part of the web. It doesn't have to be that way -- users should be free to use any browsing appliance they want, including so-called "scrapers".
Big companies like Google not only got their start by flagrantly violating the CFAA, copyright, and privacy laws, but they continue to do so. The moral of the story is hurry up and get big before you get sued or arrested.
There's a long history of ridiculous web scraping rulings based on technical misunderstandings by neophyte judges, including Ticketmaster v. RMG, where infringement was found because the company scraped data out of a page with the Ticketmaster logo on it.
Facebook sued a company called Power Ventures which read out only the user's own data. The founder was found personally liable for $3 million in damages. Facebook did this because they don't want it to be easy for their users to move between social media services. If it's easy, Facebook has to compete on merit instead of just keeping switching costs high. Facebook doesn't like that, so they sue people who make it possible -- and the law says they should win.
We badly need a revised law, but the powers-that-be will strongly oppose it because it would threaten their monopoly over web properties. They continue to flaunt their strategic ignorance of these laws and then take shelter behind them to stop risk from small innovators (i.e., having to compete fair and square).
In the real world, we have a lot of laws that mostly prevent this kind of bad behavior. In cyberspace, the structure is such that most of those laws are not applicable. We need to update and port the pro-small-business logic we have for meatspace companies so that it counts online too. The state of affairs online is really bad.
I want to get a law called the "Consumer Data Freedom Act" passed, which would allow users to access any web property with any non-disruptive browsing device, including custom scrapers that don't impose much more load than a typical user browsing session would.
Judge: George Jung, you stand accused of possession of six hundred and sixty pounds of marijuana with intent to distribute. How do you plead? George: Your honor, I'd like to say a few words to the court if I may. Judge: Well, you're gonna have to stop slouching and stand up to address this court, sir. George: [stands] Alright. Well, in all honesty, I don't feel that what I've done is a crime. And I think it's illogical and irresponsible for you to sentence me to prison. Because, when you think about it, what did I really do? I crossed an imaginary line with a bunch of plants. I mean, you say I'm an outlaw, you say I'm a thief, but where's the Christmas dinner for the people on relief? Huh? You say you're looking for someone who's never weak but always strong, to gather flowers constantly whether you are right or wrong, someone to open each and every door, but it ain't me, babe, huh? No, no, no, it ain't me, babe. It ain't me you're looking for, babe. You follow? Judge: Yeah... Gosh, you know, your concepts are really interesting, Mister Jung. George: Thank you. Judge: Unfortunately for you, the line you crossed was real and the plants you brought with you were illegal, so your bail is twenty thousand dollars.
I’d assume a lot of HN users are from such locales.
We don’t always have to assume US laws apply globally – they don’t.
anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in the EU (since unlike in the US, databases as pure collections of facts have their own copyright protections)
The legally safest locations probably are outside the western world if you are targeting western sites.
Every case I've seen wrt Ryanair (they sue a lot of people) has resulted in a win for Ryanair. Do you have details on the case you're describing?
>anti-scraping: [...]
Scraping purely factual data is one of my points of defense in the US. I don't want to give it away.
>It's still risky though, the safest locations probably are outside the western world if you are targeting western sites.
Yeah, this was ultimately the conclusion I had to come to. However, outside the West, the Western companies will just send someone with a briefcase full of $100 bills and pay them off. Corrupt government officials in these locations want the goodwill of a big American company a lot more than they care about any particular random guy.
There is only one workable solution: run the service totally anonymously and maintain good opsec so that your cover isn't blown. All under the table. This has its own issues, like making it difficult to receive payment and putting one at much greater legal risk than a mere CFAA dispute, but it's the only option if you don't plan to get shut down.
I edited my original comment to reflect that.
Also, since this is somewhat untouched territory, don't be so sure that you'll get a judge who is as well-versed in web scarping and infrastructure as you, or shares your opinions on the subject. (And given that precedents are so important in US laws, you better hope someone else before you didn't get such a judge.)
If you're not going to run it totally anonymously, you should be prepared to jettison and repackage it when you get found it (so that you appear to be complying with the C&D).
Scraping is a huge part of the web, and everyone does it. It sucks that it has to live underground because only big companies can duke it out in court.
I played with the idea of creating some social aggregation type service with some friends (as a business). The more I read about FB's past behavior with regard to this, and how essential they are to any sort of service like, that, I canned the project. Regardless of what their TOS say, if you get on their radar and they send you a cease-and-desist, it's game over. Facebook is not in the business of subverting their revenue stream, so if you are making money off them and it's preventing them from capitalizing on their users, don't expect to last long if you exist by their grace.
Really, there's an interesting space between so small nobody cares and large enough that getting shut down is a real problem. A lot of projects start small and end up (relatively) large, but without a good way to pay for the service itself. While not every service needs to be a business and make money, once you reach the level where you risk either being shut out of your data source or you need to somehow work out an understanding with that source, how do you approach that when being able to pay is off the table? Not to mention the problem approaching before you have to and forcing the situation, or waiting too long and risking the wrath of the source because you've abused their service as long as you have. Has anyone else been in this situation and found an approach that works?
I understand the use of ToS clauses to prevent scraping but I do kind of wonder to what extent they have authority here.
IANAL, but surely this would fall under copyright law? While re-publishing copyright-protected data without consent is probably unlawful in your region (like scraping an art site and re-posting the images), I wouldn't think just scraping data points for a different purpose (like scraping amazon for the purposes of price comparison) is nearly so clear cut (or enforceable), but maybe I'm just naive.
Companies like PriceZombie are forced to stop because the CFAA says that Amazon can prevent them from accessing their servers by decree alone. A ToS isn't even really necessary for this, but it helps them pin down their argument.
PriceZombie could try to get the data from third-party caches, but it only solves part of the problem, because copyright and trademarks come back into the picture once you have a replica of the target page. In Ticketmaster v. RMG Technologies, the judge found RMG infringing on Ticketmaster's trademarks and copyrights because the page they were scraping included Ticketmaster's logo. The judge said the copy of the full page that existed momentarily in RAM while the scraper extracted the non-copyrightable data constituted a copy that infringed on Ticketmaster's rights, even though the logo was never used by the application in any way, it just happened to be on the page.
If you were referring to a different decision I'd love to read it. I follow this stuff (and at one time explored what legal action our startup could take against scrapers). In our case we also offered a paid API so it was fairly easy to establish damages.
"The panel held that the defendant, a former employee whose computer access credentials were revoked, acted “without authorization” in violation of the CFAA when he or his former employee co-conspirators used the login credentials of a current employee to gain access to computer data owned by the former employer and to circumvent the revocation of access. "
I think that case is unambiguous - this guy was using someone else's credentials to access secured systems after having been explicitly told that he could not. I was referring to the MySpace case.
I don't think these two cases are in conflict; IMO they are very different. Additionally, for our purposes in this comment thread, we're talking about scraping of publicly available websites by outside parties, not by former employees whose access has been explicitly revoked. That is different than either of these cases.
The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the server's owner would go a long way to establishing your rights to access, but no one is going to give that to a small player.
They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume personal integrity when you are surfing the internet?
I have high hopes that the judicial system some time not too far from now will realize that since the law should be a reflection of the current moral standings it will always be behind, trying to catch up with us and that those who break the law while not breaking the current moral standings are still "good citizens" unworthy of prison or fines.
I guess Google won this iteration of the internet because of the double-standars site owners stand by, to allow Google to scrape anything while hindering any competitors from doing the same. There will only be a true competitor to Google when we in the next iteration of the internet realize that searching vast amounts of data (the internet) is a solved problem, that anyone can do as good a job as Google, and move on to the next quirk, around wich there will be competition, and in the end that quirk will be solved, we'll have a winner, signaling that is it time to move on to the next iteration.
Call my cynical if you will, but I'd leave "while abiding the law" out of that, or at least replace it with "while hoping they aren't breaking the law". Due diligence on these matters is often sadly lacking. They'll take the information first and only consider any such implications when/if they come up later.
Large organisations like Google probably will make the up-front effort to remain legal, because they are in the public eye enough for lack of doing so to attract a lot of unwanted press, but you don't have to get a lot smaller than that to start finding companies who are a lot less careful (or in some cases wilfully negligent).
For instance the browser choice script that came with Windows imposed by the EU never worked. It was a "bug". Somehow they must have omitted to test the feature...
Until last year Microsoft started playing nice, and I think Google and Facebook have become the new corporate villains. But recently the Windows team seems to be minded to challenge them in that position.
I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.
If that doesn't hold up in court, in future on your first visit to Google it will simply display some text and require that you click 'I agree' to continue.
Either way, it seems reasonable to me that you should agree to their terms in order to use their service.
I'm honestly wondering about the double standard. There is a rational way to discuss morality/ethics and subsequent laws regarding most technical aspects, that often mirrors real world (read: offline/analog) scenarios. It's unfortunate that the legal system has instead been appropriated by lawyers.
omg, really?
It's unfortunate that the internet has instead been appropriated by hackers. It's unfortunate that the stock market has instead been appropriated by traders. It's unfortunate that the asylum has instead been appropriated by inmates.
I'm not sure why you dislike this 'appropriated by lawyers' outcome: For web crawling look at robots.txt, for other uses look at the Terms link on the homepage. If you don't agree to the terms then stop accessing the website. Seems straightforward and fair to me.
This statement is demonstrably false, as shown by all the places in the world where this type of TOS-nonsense actually does not hold up in court.
And in the USA, it's (as usual) even slightly more absurd: The only reason it does hold up in court is because Google can afford justice.
If Google's actions were illegal, I'm sure that they would have been sued even if their scraping and indexing usually is helpful for the website owner
I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt.
At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres.
Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition.
At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts.
---
Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I've actually had to do some hacking on the RabbitMQ library to let it handle the various ways I've seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could reliably handle large (>100K) messages without eventually wedging.
There are other fun problems too, like the fact that I have a postgres database that's ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet).
---
It's all on github, FWIW:
Manager: https://github.com/fake-name/ReadableWebProxy
Agent and salt scheduler: https://github.com/fake-name/AutoTriever
- https://github.com/fake-name/ExHentai-Archival
- https://github.com/fake-name/PatreonArchiver
- https://github.com/fake-name/xA-Scraper
- https://github.com/fake-name/DanbooruScraper
Or... well, 4 separate projects. Whoops?
At one point, a friend and I were looking at trying to basically replicate the google deep-dream neural net thing, only with a training set of porn. It turns out getting a well tagged dataset for training is somewhat challenging.
Well-tagged hentai is trivially accessible, though. I think there's probably a paper or two in there about the demographics of the two fan groups. People are fascinating.
Next up, automate the consumption too!
And what is served through their website is resized. So web-scraping is an inferior approach.
I'm not scraping high value sites like that (I mostly target amateur original content). It's not really of interest to other businesses. As such, I tend to just run into things like normal cloud-flare wrapped sites, and one place that tried to detect bots and return intentionally garbled data.
If I run into that sort of thing, I guess we'll see.
But if the end justifies the means... http://luminati.io/
As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use.
Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with databases a bunch, and actually experienced deploying a complex multi-component application across multiple disparate data centers.
Regarding costs, I really have no idea. It depends on how rapidly you cycle the UA, and how fast whatever you're scraping is.
$.ajaxSetup({
dataFilter: function (data, type) {
if (this.url === 'some url that you want to watch!') {
// Do anything with the data here
awesomeMethod(this.data)
}
return data
}
})
With a selector it's easy to grab data, here's a linux command that gets every user that posted in this thread:
lynx -base -source 'https://news.ycombinator.com/item?id=12345693' | hxnormalize -x | \
hxselect -c -s '\n' "td > table > tbody > tr > td.default > div:nth-child(1) > span > a.hnuser"
27 cookiecaper
22 franciskim
6 fake-name
4 niftich
4 flukus
4 elmigranto
4 downandout
3 tedunangst
3 siegecraft
3 muglug
3 minimaxir
3 madamelicHere is an example of injecting a jQuery script into a page with jQuery loaded and getting nicely formatted information returned. [1]
[1]https://github.com/adam-s/playboy-fm/blob/master/server/scra...
(Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that's another issue).
I've been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX/app-like sites, if the trend at the federal level means anything.
That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML.
But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It's been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back
[0] https://github.com/wgetsnaps
[1] https://github.com/wgetsnaps/tdcj-state-tx-us--death_row
On a blog post by Paul Kinlan ('Open Web Advocate' at Google and Chromium) [1], I lamented that we ended up here instead of the semantic web because the semantic web was hard to execute. Instead, every web page is a black-box, only navigable by an intelligent and/or sufficiently persuadable human.
But this is also why I don't buy ethical arguments against scraping. Sure, legally any company can unilaterally set any TOS prohibition against behavior they don't want, and these terms may be tested in court. But navigating a page in an automated manner that's designed to resemble interactions of humans (ie. through Selenium) is in my opinion ethical, because it merely time-shifts a user's activity.
I'm not saying you're one of these people, but it's frustrating when companies do this to potential employees and the potential is told by friends and other management type people, "well that's the company you just have to deal with it".
When someone flips it on the company then it's immature.
I applied somewhere recently and they invited me out to a pre-interview lunch. That went well so they called me in for an interview. That went well and the VP told me he would call me back to set up a second (third?) interview.
I never heard back from him. An ex-coworker there went to the VP to find out what was going on and the VP said he decided he wanted someone with more experience in the specific area they're working in.
But last he told me was he liked me and would schedule another interview, then when he changed his mind he never let me know.
I think people on both sides should be courteous and respectful through the process, but if employers are treating interviewees poorly then they shouldn't be surprised when they start getting treated poorly.
First, it's hard to know when companies are doing this intentionally versus when things just get lost in the shuffle. (Never attribute to malice what can be explained by incompetence, and all that.) Meanwhile, the author was clearly ignoring the interviewer intentionally.
Second, the fact that Company A treated you rudely doesn't give you license to treat unrelated Company B rudely. For that matter, I'm not sure that the fact that Employee 1 at Company A treated you rudely gives you moral license to treat Employee 2 at Company A rudely. Show a little compassion for someone trapped in a dead-end job trying to put food on their family's table, for crying out loud.
> but all it explained was how to make a few API calls in order to solve a very specific problem.
Yeah, the very specific problems everyone runs into time after time. He presents specific solutions, and reasonable context. If I was googling for one of these problems, I'd be very happy to run into this page.
> Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails "
Your arrogance is my matter-of-fact.
In reply to XCSme - no I am not new to Node and my point of the post is to illustrate some of the techniques that I haven't seen published anywhere to HN and the community. My focus is quite different from what you think it is, so maybe it is my bad for bad writing skills, I'm still new to writing and learning.
https://contently.com/strategist/2015/01/28/this-surprising-...
I believe you should treat others how you want to be treated. FYI, recruiters do not usually follow up with rejected candidates and many are unresponsive. It's their way of telling candidates they are no longer interested.
I guess the distinction is between whether one wants to just "toy around" or run the spider for-real.
However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn't really that slow, relatively speaking, but I had a few hundred gigs of HTML to chew through every time I made a correction, so it was important to keep the turnaround as fast as possible.
I eventually managed to manually partition the task so I could launch separate Node scripts to handle different parts of it, but it wasn't a perfect split, and there was a fair bit of duplicated work, where a shared cache would have helped a great deal.
In retrospect, I should have thrown my JS away and started again in something with easy threading like Java or C#. But -- familiar story -- I'd underestimated the complexity of the task to begin with, and by the time I understood, I'd sunk a lot of time into writing my JS parsing code and didn't fancy converting it all to another language, particular when it always seemed like "just one more" correction to the parsing would make everything work right. In the end, what was supposed to take a weekend took about three months of work, off and on, to finish.
You can invoke as many lambdas from your application as you want in parallel and you're not going to be bottlenecked by your CPU :)
I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get thread pools and synchronized thread-safe collections out of the box.
And Lambda would have been totally impractical. As I said, I had hundred of gigs of data to process. If I'd been uploading this over my puny ADSL upstream every time, I'd still be waiting for a single run to complete.
I'm not trashing Node. I like it. There's a reason I used in the first place, after all. But for this particular use-case, I didn't find it was very good fit.
That aside, hitting Insta like this is playing with fire, because you're really dealing with Facebook and their legal team.
I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and will likely retire that bot soon.
It kinda sucks that I'm just grabbing publicly-available content in a very low-bandwidth way, but I really can't convince myself that what I'm doing is very ethical.
My to-do list includes making my crawler into a more well-behaved bot and that will have to go.
Google Cache link: http://webcache.googleusercontent.com/search?q=cache:https:/...
Archive.is link: http://archive.is/DQccs
I do a good bit of scraping, and made RubyRetriever[1] to make my life easier but it seems like I'm getting roadblocked on occasion, probably due to some of the things you mention in your article.
Is there any way for a site to verify that only their JS and CSS files are linked? Like preventing injection?
Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans.
The random wait settings specified in the post are better than nothing, but still too flimsy. You would need to put hours between requests, only request during a certain 15 hour periods, take days off, and eventually you aren't scraping regularly enough to do much good.
Scraping is not an API, and I should know- I used to do it for a living. Its unreliable. It requires constant maintenance. APIs can break too, but they are meant for the sort of consumption you are trying for.
If you scrape for a living, only do it as a side job.
I've noticed that most sites actually don't change that often. I deal with changes once or twice every 3 months.
"If you scrape for a living, only do it as a side job."
This is true if you are scraping the low hanging fruit. I scrape 40+ sources (I do have access to a few APIs as well) and then have to extract the patterns/data I need to then integrate it into my business model. This is all automatic now and I only work on upgrading for speed and efficiency.
If you have to scan millions of urls daily from 1 site, it's probably not going to work out. You need to figure out clever ways of getting the data and using it without breaking any laws or pissing off the site owner.
I guess it's part password manager (it stores passwords encrypted in browser storage, not remotely) and part automation wizard :)
It's almost impossible for a website to reliably detect that a client web browser is being automated, and I find I can make Selenium scripts much more adaptable to breaking changes in websites when they occur than I can when hooking up my code directly.
I actually disagree with the contention that Selenium is slower than directly scraping though. The Firefox driver has always been lightning fast for me and the bottleneck is almost always server requests that would have been necessary either way.
The difficulties are invariably in "post-processing"; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout/URL/data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non-deterministic nature, and general problems associated with long-running background processes in web stacks.
Et cetera.
In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression.
It would be quixotic to simply retort that sites should provide "proper" APIs but I would love to see more attempts at solutions that go beyond the superficial.
I can agree with this after having written a scraper as part of core business functionality (we paid a company for access, but access was just to bare HTML blobs and CVS and not an actual API).
However, to what degree you want to do all this is negotiable whereas the 'core' of screen-scraping is not---all scrapers have to first figure out how to get text, parse it, then stick it back in their system.
An example of what I mean when I say 'negotiable' is....
> working around incomplete data on the page
Deciding how to do this depends on your problem domain. Sometimes, we'd get bad computed data from our source but not care because it just meant more work putting more work in calculating it from a more raw source.
> not hitting your target site too often
If they publish how often you are allowed to scrape, this isn't too difficult. If not, then trial and error is the only solution. On occasion, a site simply just doesn't know/care. For example, in my case, the site was static content behind a CDN, so that if we were anywhere under 200 req/second then no flags would ever be raised.
For most smaller sites, that you are unofficially scraping, you may be limited to 1 request every 2 seconds.
He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie protected search feature, so that my painstaking effort (I'm not a dev, more a DB guy) could be reproducible more easily by other users of this service.
But this shouldn't appen in the first place because all post of this service are stored in a cleanly organized MySQl DB. Yet as no method is provided the only way to get back structured data is by scrapping (as the webmaster told me that no, he won't run custom SQL because "he don't want to mess his DB").
So even if all the data is publicly available through the internet forum only a geek can download a personal archive... or google because google scrape and store everything.
https://github.com/kingkool68/zadieheimlich/blob/master/func...
I don't think it's any of the regular meanings: http://www.ldoceonline.com/search/?q=Lead
But it doesn't seem to be any of these slang terms either: http://www.urbandictionary.com/define.php?term=lead
The few that I've seen just 'ban' your IP for a few minutes. If you hit Wikipedia too much too quickly, they will essentially refuse to serve you for a while. It was a number of years ago I was doing it, but basically you would be scraping then you would just stop getting info (Maybe I wasn't reading response codes and could've realized quicker what was happening)
No need to scrap it when you can readily download a nicely formatted .xml.zip file containing all knowledge written by mankind.
But there seems to be little demand for these kinds of systems and just throttling/blocking/CAPTCHA solutions are much simpler.
Of course it can! You won't be able to defeat even the simplest attempt on anti-scraping based on statistical data. Like even keeping a list of individual rate-limits for /16 subnets of actual visiting users and you are in trouble.
Anyways I added your stuff here along with other data mining resource:
https://github.com/kevindeasis/awesome-fullstack#web-scrapin...
And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors.
You can't stop them, but you can make their lives painful.
You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well.
> And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames.
Legitimate browser plugins, etc. I'd just use electron or selenium with `nth-child`, `:visible`, `[class*="…"]`, etc.
What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games.
Can you be so sure? The Union blockade of the Confederacy had plenty of holes, and smugglers / privateers / blockade-runners made good money getting through (when they survived) ... but that doesn't mean the blockade wasn't effective all the same at weakening the Confederate military and economy.
Sure, xpath and css selector experts can figure it out, but that's not everyone
And traffic from good scrapers is of course pretty much impossible to measure so you don't know how big percentage of scrapers you got rid of in total.
Of the sweatshops that must have been setup to deliver this service. That, is to me the true horror of this story.
Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.
It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that.
On the other hand, Tor is likely to be 90% evil. When in doubt, just block it. (That makes me think, I should run some proper stats and maybe publish a blog post about that. )
The traffic from the site itself, if it's hosted there, would come from the intranet IP address, right? Not the public facing one.
> It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that.
Agreed, but it's fairly easy to block the AWS IP traffic on web endpoints and not on the API endpoints.
- rails application
- scraping with nokogiri gem on Ruby
- simple models doing the scraping in rails app
- some scraping is parsed with CSS selectors - nokogiri
- some scraping is parsed with regex - nokogiri
- persisting to DB, Text, even Google docs
- presentation on web, text, pdf, xls
Boom
browser.findElement(webdriverio.By.id('#Next')).click();