Hacker News Reader

Comment by simonw | Hacker News Reader

This is talking about The Stack. The poster says:

"I found two of my old Github repos in there. Both were deleted last year and both were private."

The Stack was constructed a while ago, so "deleted last year" wouldn't have an impact if it was constructed before then.

"Both were private" is the thing that needs to be unpacked here. Were these genuinely private repositories that had never been made public on GitHub?

https://huggingface.co/datasets/bigcode/the-stack-v2 talks about where the Stack comes from: "This dataset is derived from the Software Heritage archive, the largest public archive of software source code and accompanying development history"

You can search that here: https://archive.softwareheritage.org/browse/search/?q=simonw... - it would be interesting to know if the OP's "private" repos are included in that collection.

View on HN

Here's a reply from a GitHub staff member: https://mastodon.social/@correcthorse/112128192392083842

> I work for github. This post was shared in our slack, on a non work related channel. We don't think it's us.

> https://huggingface.co/datasets/bigcode/the-stack-v2

> They say they get data from SoftwareHeritage, a website that archives repos from github. If your repos were ever open, they might have been archived there even after you deleted from github.

> That's my best guess.

Depending on the license, how is this legal?

Which bit? The archiving on SoftwareHeritage, the gathering of that data into the Stack or the subsequent training of models?

that’s a good question… There seems to be two problems.

The definition of open source depends on a license existing in a repo. Without a license it’s not legal to copy and distribute.

Public vs Private repo is a platforms issue not the code maintainers.

If a public repo does not have a license, it does not mean it free to copy and distribute.

If a private repo has an open source license like MIT, then the crawler has a right to copy and distribute that repo. Regardless if it has authorization to access the repo or not.

Hopefully the crawler smart enough to properly handle edge cases...

e.x. if the repo has some sort of /used-licenses/ folder where the licenses for packages and the like are included, it could make a bad decision.

> Without a license it’s not legal to copy and distribute.

Yes it is. Due to both the terms you agree when you use GitHub and the general Implied License that covers everything public on the internet.

https://en.wikipedia.org/wiki/Field_v._Google,_Inc.

Looking at that ruling, it seems the case you linked to hinged on a fact not applicable with the Stack:

>Field had actual knowledge of the Googlebot. He also was aware of the ways to prevent Google from either listing his site at all or listing it but not providing a link to the cached version. Instead of opting out, however, he chose to allow Google to both index and provide a link to the cached version.

For the AI dataset, (A) did the person know their work was being collected by this group and for this purpose, and (B) did they know of a way to prevent that collection?

It is not clear to me if they are _only_ using GitHub as source. The Stack explicitly mentions they are using Software Heritage as source and Software Heritage definitely sources from repositories that are NOT stored in GitHub (and never have been).

I don’t think that “implied license” you’re referring to holds up in the courts.

> Without a license it’s not legal to copy and distribute.

Is this true? When you post anything publicly, from sticking a poster on the street to making artwork like banksy, isn’t the default set to “it’s legal to copy, unless explicitly stated otherwise”?

The default in the majority of the world is that most creative works (including software code) are by-default copyrighted by the author, and the author must explicitly license away those rights. Some jurisdictions (e.g. France) put limits on what rights the author is allowed to give up. I.e., the default is it is illegal to copy (subject to exemptions like “fair use”).

Note that this archive project is French.

Banksy apparently runs a licensing program. Their artwork is most definitely under copyright, and they rely on trademark protection as well.

There is also the practical issue that a lot of content is posted publicly without consent of the copyright owner. It's simply not true that just because someone else committed a copyright violation first, you can commit further violations without impunity based on that first violation.

> If a public repo does not have a license, it does not mean it free to copy and distribute.

Whether or not it is free to copy and distribute, it should be free to copy and distribute. (My opinion is that copyright is no good; if the file is public then you should be allowed to copy and distribute it.)

> If a private repo has an open source license like MIT, then the crawler has a right to copy and distribute that repo. Regardless if it has authorization to access the repo or not.

I should not think so. The license would only apply if you have a copy of it anyways. If you are not authorized to access it because it is private, then you would have to get a copy from somewhere else, and if nobody else is providing a copy, that shouldn't give you the right to unauthorized access. However, if it has been done, then it is done, so now there is a copy, and the license (if it is a license that allows copying it in this way) would authorize you to continue to use and distribute the copy that you have.

i’m not saying I agree or care about any of it. A sane company would never allow the use of source code from a third party without a license.

If repo is forked and the license is deleted the source code would need to be hashed to verify its the exact version of an open source repo. Mainly they don’t want copyleft or “malcious” license infecting their IP

If the hashes don’t match then it’s not technically the same code, so a company can’t safely use it without a license.

In the US: Fair use disregards licenses. Fair use can be found to apply, or found not apply, by a court of law. Archival of information is generally felt to be Fair use, as is Search indexing.

The real argument is when is AI re-use of copyrighted material a violation of copyright. That is a large grey area that will probably be determined in favor of large corporations and not in favor of individuals. (As in, Disney can use AI writers to copy you, but you won't be allowed to copy Disney)

Software Heritage aggressively insists on French law... which does not have fair use.

I have about fifty private github repos and hundreds more public github repos (half of them forks of other public repos). I've verified that none of my private repos are in the dataset, and all of my public repos are.

I was expecting to see at least one repo that shouldn't be there depending on when the dataset was put together. In 2015 I changed a repo from public to private, which I think might suggest that the dataset was built after 2015 since my now private repo isn't in the dataset?

I did the same and see none of my private repos. Which is quite a lot. I do see plenty of deleted repos but those were public.

Github publicly streams the global change log including all public repos. I have a public repo and noticed there were multiple clones daily after every commit I made. That's when I discovered the API: https://docs.github.com/en/rest/activity/events?apiVersion=2...

If the repo was public even for a single commit, that was likely cloned and replicated elsewhere.

> Were these genuinely private repositories that had never been made public on GitHub?

The poster responded to that:

https://hachyderm.io/@emenel@post.lurk.org/11212861313743638...

> I’m pretty sure the repos of mine were private. If it were only me i could be misremembering something, but i have heard from a number of people that they also found repos in this dataset that were private.

This is a really big claim. I think we need specifics on this - I'm inclined to think this is people not understanding the GitHub public/private repo model (or misremembering the history of their repos) over GitHub deliberately leaking private code to third parties.

I agree that might be the case. Since I do not personally know the author, I elected to use “may” in the submission title. Hopefully we get a definitive answer from those involved.

Still, the post continues to be useful for those who want to opt-out regardless.

I would use "claim" rather than may until it's substantiated.

Reads a bit weirder, but changed.

I would give it a 50:50 chance.

The other option is that the scraper got lucky with a tiny glitch (whatsoever).

On the one hand I bet github does everything to keep stuff secure, on the other hand I can't believe there wasn't a single glitch in the last few years.

And if there was a glitch then regular automated scrapers are a pretty likely siphon.

Heck I should check if any of my private repository show up.....

> a tiny glitch

That's not "a tiny glitch". What you described is "GitHub may show your private repo to strangers randomly" which is a even more serious issue than they appearing in an archive some time later.

Right: exposing private source code in this way is a showstopper bug for GitHub, and a major scandal if they've been covering it up.

A lot of companies pay GitHub a LOT of money to securely host their code. A breach like this is a way bigger story than just another "AI is training on your data!" thing.

I continue to doubt that private repos being exposed like this actually happened here.

I have personally experienced this. The person who did it didn’t realise the significance of the repo being “public only for a couple of hours”. I’m also inclined to believe it’s misremembering / misconfiguring.

I'd be interested in hearing from the cited number of people. If it's like 5 randos, then that's quite possibly a misremembering, or even conceivably the victims of some unrelated code theft. If it's a few dozen people, well that would have very different implications.

One more person:

https://news.ycombinator.com/item?id=39770776

Seems worth tracking publicly. Maybe I should create a GH rep.... er... hm...

Maybe worth posting leaked Windows, Apple, Intel, Steam, The Wicher codes on GH as private repos and then seeing who will sue whom first.

I went through my private repos and the only one I found was one where I had forked a project, called `documentation`, and my fork made it into the Software Heritage archive. Sometime between then and now I deleted my fork and cloned a private repo with the same name. I confirmed the Software Heritage archive only has the one that was public and not the private one. I feel like we'd be hearing much louder alarm bells if the Software Heritage archive included private code given how many people rely on private GitHub repos.

That person is unable to provide us with a repo name either.

None of my private repos are included in the software heritage project.

Nor mine. But if anyone in the year 10,000 needs to run a 4-digit LED segment display as a clock with a 1st-gen Raspberry Pi, they'll have the code to do it.

That’s really funny to think about. By then today’s internet may be the equivalent of the little NES machine you can buy with all the games on it.

He said year 10,000 not year 2052 tho

Considering private repos used to require a paid subscription misremembering seems likely.

It used to be public by default, and enough people got confused by this that AWS & Github used to specifically scan repo's for accidentally public AWS credentials.

For info, I checked and it looks like none of my AGPL licensed repos are included in The Stack. Neither are my private repos.

My GPL ones seems to be excluded.

I've got a couple that are intended to be GPL, but you wouldn't know unless you go to the GitHub issues to find the issue I raised about the licence file not being in the repo. They are included.

It's weird that they exclude GPL licensed repos, but not unlicensed repos. It seems like they would have even fewer rights with the unlicensed ones.

There's probably a ton more unlicensed repos than licensed. It isn't weird, its capitalism.

The software heritage sent me an email they were going to steal copies of my software. I think they assumed because it was published as a git repo they had unlimited rights to steal, reproduce, sell, etc my copyrighted work.

To quote the email they sent me:

"The mission of Software Heritage is to collect, preserve and share all the publicly available source code: https://www.softwareheritage.org"

So they're telling me their intent is to reproduce (share) my work, just because it was publicly available.

Upon my reply they did offer to cancel the request, but also told me they are facilitating the storage of my code for users private theft

" "Add forge now" requests are submitted by Software Heritage users who think that a forge is worth being archived.

After a careful examination of your arguments, we acknowledge that your forges may not be archived, so we won't process their ingestion, and close this add forge now request. However, we cannot prevent a publicly available from being archived by anyone using our "Save code now" feature

Steal your repos? So you don't have them anymore because they took them? Maybe ask nicely to have them back and with a bit of luck they'll give them back to you.

reproducing my work without license is theft.

I mean, this is a matter of law, and it isn't. It's a curious point to double down on.

Curiously, the people most vociferous about digital copying of IP not being “theft” tend to be those people who have produced the least amount of original IP.

Surprisingly, the people generating real, novel, IP like to buy groceries now and then.

Absurd, presumably anecdotal claim.

My own anecdotal experience is that it has no relation and I know many people who have violated the copyright on the IP they themselves produced.

People who produce IP either by work for hire or by selling their copyright rights.

I've had people "steal" IP by sending me a PDF of a paper they wrote whose copyright resides with their university. Not the typical image of a thief.

This is a sound point, even if it's being made a bit sarcastically. Copyright and intellectual property are an entirely different body of law than property rights, and people most often conflate them as an emotionally charged rhetorical technique that's generally rooted in a dissatisfaction that intellectual property and physical property are treated differently. The use of the word "steal" in this context is probably to incite readers to their cause, but ends up just sounding sophomoric to people that know anything about intellectual property law.

It's a shame because the GP had some valuable information to share about the emails software heritage was sending, but likely got downvoted because of this.

Problem is GP (me) is a dickhead about it.

but frankly, it's not the first time. Github (after purchase by microsoft) did the same thing to my code. They reproduced it and put it on ice in some place in norway. That was the guise right? I mean I am sure they did, but you can bet your ass they're also using my code to train their AI.

Most of it was not licensed. It was publicly available as a portfolio so people would see I write serious code and would hire me.

So then I took down my entire github after being informed I was enrolled, couldn't opt out, etc, and then some OTHER foundation is now scanning my self hosted git repos? That made my blood BOIL!

Do you happen to still have that email? I'd like to search my inbox for anything similar, and would love some snippets to search for.

Edit: I asked this before the parent commenter included the contents of the email. Thanks parent commenter!

Hello Calvin Morrison,

The mission of Software Heritage is to collect, preserve and share all the publicly available source code: https://www.softwareheritage.org

We have received a request to add the forge hosted at the URL below to the list of software origins that are archived, and it is our understanding that you are or know the contact person for this forge.

https://git.ceux.org/

In order to archive the forge contents, we will have to periodically pull the public repositories it contains and clone them into the Software Heritage archive. FAQs for our processes are available:

https://docs.softwareheritage.org/user/faq/#add-forge-now https://www.softwareheritage.org/faq/

Please let us know if there are any issues to consider before we launch the archival of the public repositories hosted on your infrastructure. Please use "Reply all" to ensure our system will process your answer properly.

In the absence of an answer to this message, we will start to archive your forge in the coming weeks. Only the publicly accessible repositories will be archived.

Thank you in advance for your help.

Kind regards, The Software Heritage team

-- bye, pabs

Have you figured out if there is a way to prevent them from doing that? The email I got, which I assume is similar, was suspiciously lacking a "just GTFO and leave my website alone" option.

They explicitly told me they could not blacklist me, but they did do a prompt takedown.

I blocked their IP address specifically. Contact me if you want it.

At least some of their software has unique user agents ("Software Heritage*"). I wish a popular FOSS host like Codeberg would block them.

> However, we cannot prevent a publicly available from being archived by anyone using our "Save code now" feature

This is not true. They just had to remove about 500 public repos to comply with my copyright.

It's true as in, that's what they told me when I asked.

They're lying.

Reply on news.ycombinator.com