Comment by koolba | Hacker News Reader

Comment by koolba | Hacker News Reader

Parent Full thread

> You may scrape the website for the following reasons:

> Researchers may scrape public, non-personal information from GitHub for academic research purposes, only if any publications resulting from that research are open access.

> Archivists may scrape GitHub for public data for archival purposes.

> It is prohibited to scrape GitHub for spamming purposes, including for the purposes of selling GitHub users' personal information, such as to recruiters, headhunters, and job boards.

How about a level of indirection?

Alice the Archivist scrapes GitHub and compiles a compendium of names / emails. She then publishes this information in bulk for anybody who'd like to use. Alice's cousin Roger the Recruiter downloads the archive and does what he does best. Alice isn't selling anything, she's just providing the information bundled up. Roger isn't getting it from GitHub, he's getting it from the public feed (say via Bitorrent or a public data set posted somewhere).

I just don't think the whole "you can do X but only if you use the data in ways we want" is a good idea.

It's completely unenforceable, and there are millions of loopholes like the one you just said that will let them skate around the restrictions both legally and in action.

This seems like an addition because of the "GeekedIn leak"[0] of a bunch of scraped GitHub data from Nov of last year, but I just don't get what it's trying to do. Bad actors are just going to ignore it like normal (or find loopholes) and good actors weren't doing this in the first place.

Personally, I wish they left out the "what you can do with the data" and focused their ToS only on the "how you are allowed to scrape it". Give solid limits on amounts and speeds and other various aspects of the scraping, make them restrictive enough to prevent a service like that from working, then provide a point of contact so that an "archivist" can be allowed to do more on a case-by-case basis.

[0]https://www.troyhunt.com/8-million-github-profiles-were-leak...

I would imagine the idea is to have a legal recourse to go after a company built atop breaking the terms of service. An example of that would be Craigslist going after Padmapper. There's tons of scraping of Craiglist and I'm pretty sure it's a violation of the terms, but they'd reserve legal enforcement to specific circumstance.

I just think that there are better ways of limiting that (however I'll freely admit that I don't really have any experience in the area, especially the legal aspect, so I could be horribly off on this).

I've seen many very cool, creative, and useful tools pop up on top of GitHub and sites like it, and I just fear that changes like this will have the (perhaps unintended) side effect of shutting them down.

Things like CommitPrint [https://commitprint.com/], or Commits.io [https://commits.io/] come to mind.

"...up on top of GitHub..."

GitHub has an API. Use of an API is not 'scraping'. The act of scraping involves pulling full HTML pages and parsing out data. Because scraping and API usage are different, there are different ToS for each.

I suggest that useful tools are sticking to the API.

Maybe not enforceable (I dunno) but the academic scraping clause seems very good. You can't really hide that you scraped from GH if you want to publish it according to accepted standards (transparency, i.e. there will be a "data was collected this way" section). So if you don't open access your work you risk writing a worse paper if you want to hide where you got the data. At the very least this gives us a nice opportunity to shame people for not OAing their Github-data based papers. I like it :)

Intent matters.

If Alice's archive gets no traffic, and Roger's recruiting firm does 40% of their business using Alice's data, and Alice and Roger get dinner together every week, Alice seems to be doing something wrong.

Even on his own, Roger is still scraping Github. The defense "It wasn't me, it was the Python program that was in the wrong!" is just as ludicrous as trying to hide behind the fact that Alice's archive is scraping the site itself. They're putting limitations on the information, not limitations on how computers are allowed to talk to each other.

http://ansuz.sooke.bc.ca/entry/23

But how can GH establish intent? Legally it's unenforceable (good luck getting an answer from my lawyer), practically it's irrelevant (I will archive this data now, maybe in the future I will be convinced that selling them is ethical), etc.

Sure, in hindsight, it'll all be obvious..

The only time it'll be tested is in hindsight. If you scrape the data and put it on your laptop, they don't really care. If you then sell the data, they can come after you for breach of service.

They're not trying to keep you from scraping, they're trying to keep you from using the data in ways that would be distasteful to their users.

What if you're downloading just the git repository and scraping that. In theory you're not scraping github but the files github allows you to download.

That sounds like open source to me.

Yea it seems like they're possibly breaching AGPL on the AGPL licensed repositories.

This is why there are "derivitative" clauses on many licences. "You may do this, so long as what your produce/derive is under the same conditions". That would stop the problem you highlight

The legal precedent for data scraping and TOS really needs a rethink.

There is, unfortunately, precedent. So GitHub can say, "Only scrape if you do x, y, z."

But... You can't access the TOS without first requesting the data.

Consider it.

You cannot request and view data but in a way acceptable to the TOS.

You cannot view the TOS without first requesting and viewing the data.

what makes someone an archivist?

Archiving github history.

Like 4chan has archives of posts, same concept I think.

When you paid for winrar ?

Joke aside: Working in a library I guess ?

archive.org, etc...

Or software heritage

Reply on news.ycombinator.com