Hacker News Reader

Comment by minimaxir | Hacker News Reader

I wrote and posted a statistical analysis of the OKCupid dataset, but took it down immediately once the information about how the data was obtained came to light.

The problem here is that the data included in the release (answers to questions) can only be accessed by logged-in users, and the scraper they used utilizes a login, in which case the researchers blatantly violated the ToS and a DMCA takedown of the dataset was a valid response by OKC.

However, almost everything else in an OKC profile, including username, city, and sexual orientation, is public to logged out users by default. (see this analysis of mass-scraped OKC data: http://yakamo.org/?p=112). That makes things a grayer area than usual, and serves as a reminder that nothing online is private.

View on HN

"a DMCA takedown of the dataset was a valid response by OKC."

No, this is an abuse of the DMCA. The DMCA only applies where there is copyright. Databases of facts are not copyrightable in the US. See Feist vs. Rural Telephone.[1]

Violation of the terms of service may be grounds for a civil suit, but it is not a criminal offense. That was settled a few years ago.[2]

[1] https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

[2] https://www.eff.org/deeplinks/2010/07/court-violating-terms-...

Doesn't profile data count as _creative works_ as opposed to facts, licensed by the users to OKC?

Justice O'Connor in Feist: "Many compilations consist of nothing but raw data -- i.e. wholly factual information not accompanied by any original expression. On what basis may one claim a copyright upon such work? Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The sine qua non of copyright is originality."

OKCupid is a collector, not an creator.

I think the point was that users are creators (of at least parts of their profiles that are not factual information) and OKCupid is their publishing platform of choice.

Even if there's originality, OKCupid isn't the creator, though. They at best have a non-exclusive license for the user's text. That doesn't give them DMCA rights.

Doesn't it give each individual user DMCA rights?

> http://yakamo.org/?p=112

"I setup 7 servers with mpi4py and it took 48 hours to scrape 3.2million profiles. Now i was ready to get the answer and find out how many profiles where public."

I think that works out to around 20 requests/sec. At a certain point it becomes borderline DoS?

20 requests per second per 7 servers so it comes out to 3 requests per second. Not even close to a DoS attack.

It's 20 rps -- not sure why you divided by 7. A DoS attack is measured by total, not per server.

Well if you are going to total might as well add in the traffic from all the other users. And in that grand scheme 20 requests per second won't matter.

I only divided by 7 because I thought he wasn't running all the 7 servers simultaneously, come to think of it, I was wrong.

I can't believe a company as large as okcupid has nothing in place to detect and stop this.

It's not fast enough to be a DOS attack by any stretch. But it's easy to detect that a given IP address has downloaded e.g. several hundred profiles in a few minutes, and OKC should have been throttling such obvious crawlers to around one per minute. If the behavior repeated, they should have blocked the address altogether for some period of time.

At the point where you are negatively affecting their servers availability and doing so with a malicious intent.

Too bad that OkCupid just doesn't publish data dump of public profiles or some API. People shouldn't need to resort to scraping to get the stuff that's public.

Just because someone does not do to your liking (for whatever reasons) you are allowed to do as you please? Is this the philosophy behind your comment?

Or at least anyones liking and anyones doing things non the less (to not become too personal here).

Let me clarify my position. Whatever can be downloaded programmatically will be downloaded programatically. You can either put "Don't" in your "Terms and conditions" (which in most countries in the world and for most internet users translates to "Pretty please") or be honest with yourself and your users that your site, whether you want it or not, is an API and build it like a good API. Side effect of acknowledging that might be that you actually will put in place some throttling mechanisms that will prevent guy with just a handful of machines from downloading the data of 2.8 million users in reasonable time: http://yakamo.org/?p=112

Reply on news.ycombinator.com