Download all of your GitHub data | Hacker News Reader

Download all of your GitHub data

blog.github.com

151 pointsTimWolla38 comments

Now what I would really like to see is some tool for keeping my GitHub repos in sync with my GitLab ones.

I did a one-time migration from GH to GL, but my GH account is still open, primarily because some projects are GH-only, meaning I might continue to make forks, contributions, etc.

Some way of making sure that I can periodically get all of those over to my GL would be nice.

Not that I'm all that good at the more esoteric features of git, but you can overwrite the history with the correct flags, right? It seems like all you would need to do to sync them is have the most recent checkout of each, a way to determine which was newer, and push the newer branch over the older one (to be simplistic and destructive).

If you're always overwriting history of one to make it match the other, diffing the git logs of each against each other would likely tell you which had items the other didn't, and if it went just one direction or both (i.e. can you just overwrite one with the other or is some manual intervention required).

The git repo itself is the easy part. You just make a local clone with two remote repos, pull from one, and push to the other. The hard part is all the parts that aren't managed by git, the issue tracker, PRs, releases, etc.

I‘m using the setup linked below to sync a few repos to gitlab. It uses GL pipelines to do the sync, and a webhook to do it instantly (however you can trigger pipelines using curl, or on a timer in GL)

https://gist.github.com/cfstras/5e40ea264517552e29caf61bbd28...

Git is after all designed for this distributed scenario, so you'd think it would be possible, what about something like:

Setup:

    git add remote gitlab ...
    git add remote github ...

Then whenever you need to sync:

    git fetch --all
    git checkout -b gh --track github/master
    git checkout --track gitlab/master
    git merge/rebase gh
    git push gitlab

Could probably be done shorter but those steps should do it..

This has the issue that you have to add this to your Workflow and nice wrappers have to be put on all workstations.

A mirroring setup like provided in GitLab (see sibling post) takes that burden away.

Check out the `--mirror` flag.

Gitlab itself supports being a mirror of an upstream GitHub remote (or any remote). Last I checked it doesn't deal well with force pushes but I haven't checked in a long time.

Thanks for using GitLab! Please also see https://docs.gitlab.com/ee/workflow/repository_mirroring.htm... Pushing free to use, pulling is paid feature but public projects can use it for free too.

I want to be able to freeze this. Give me 30 days to OK a bulk data download. I don't want a single lost access token to have such an easy way for someone to get everything.

A single lost access token + the API is an easy way to get everything too. It doesn't take much code to get everything that way.

Well then I want that to be locked down too! I'm sick of the all-or-nothingness of many of these security measures.

Really the lesson is to use constrained API tokens, going so far as to make single-purpose users as necessary.

Then don't create an API token with full access?

A single lost access token that had global permissions already had the ability to get everything: GitHub’s API is already very full-featured.

The fix here is to use their scoping as intended: it should be amazingly rare to need an access token with such broad access.

When you request an archive of your data, we send the download link to your primary email address (the required token is not available via the web UI). Once you click that link, you'll be asked to re-enter your password. So for this particular feature, an attacker would need both your GitHub password (and your 2FA seed or an active session if 2FA is enabled) and access to your email.

There's also API (https://developer.github.com/v3/migrations/users/), which doesn't involve web UI or email.

The docs says it's "only available to authenticated account owners"; I hope it means you can't use a token for that, but I'm not sure.

Is this for GDPR compliance?

Almost certainly, but it's nice they're making it generally available. I've seen some sites' GDPR exports limited to EU customers, which seems like a dick move.

The thing is, sites cannot use IP geolocation to decide if their users are EU residents or not. I think in the end it’s more expensive to try to manage this than to let everyone do the export and deletion.

Don't you just need to provide access to the data within 30 days of asking for it? That would allow for not making it public but having it publicly available.

I’m not a GDPR lawyer, but AFAIK export has to be satisfied within 7 days and removal within 30 days. Not sure what are the delays for a correction.

I think the most expensive part could be the human time needed to make judgments on the residence of the requestor. Fully automated processes available to all the user based won’t have this extra cost.

Reminds me of California's labels in that way. Everyone mocks them, but some of them are legitimately useful e.g.:

https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-c...

Same thing here, because border specific stuff aren't worth the hassle, the lowest common denominator (in this case GDPR) becomes the new international standard.

Additionally some US politicians have proposed copying parts of the GDPR into US law which would pretty much make this a permanent staple.

> because border specific stuff aren't worth the hassle

For GitHub maybe, not true for all (or even most?) non-EU businesses. For most, the "hassle" of ignoring the region requiring GDPR is much easier/safer than the compliance/risk of conformance.

It wasn't clear to me from the blog post, but the data for events, issues, attachments, and so on is only available for repositories you own. Activity in repos belonging to organizations or other users isn't included.

Anyone know if it includes information on which repositories were given stars?

It does not. This is what it looks like for me (files with number > 1 omitted for brevity):

    -rw-r--r-- 1 timwolla timwolla  499 Dec 19 19:31 attachments_000001.json
    -rw-r--r-- 1 timwolla timwolla  52K Dec 19 19:31 commit_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla  72K Dec 19 19:31 issue_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla  35K Dec 19 19:31 issue_events_000001.json
    -rw-r--r-- 1 timwolla timwolla  34K Dec 19 19:31 issues_000001.json
    -rw-r--r-- 1 timwolla timwolla  356 Dec 19 19:31 milestones_000001.json
    -rw-r--r-- 1 timwolla timwolla  22K Dec 19 19:31 projects_000001.json
    -rw-r--r-- 1 timwolla timwolla 4.4K Dec 19 19:31 protected_branches_000001.json
    -rw-r--r-- 1 timwolla timwolla  25K Dec 19 19:31 pull_request_review_comments_000001.json
    -rw-r--r-- 1 timwolla timwolla 4.3K Dec 19 19:31 pull_request_reviews_000001.json
    -rw-r--r-- 1 timwolla timwolla  51K Dec 19 19:31 pull_requests_000001.json
    -rw-r--r-- 1 timwolla timwolla 9.0K Dec 19 19:31 releases_000001.json
    -rw-r--r-- 1 timwolla timwolla  80K Dec 19 19:30 repositories_000001.json
    -rw-r--r-- 1 timwolla timwolla   75 Dec 19 19:31 schema.json
    -rw-r--r-- 1 timwolla timwolla  27K Dec 19 19:30 users_000001.json

    attachments:
    total 12K
    drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:31 ./
    drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 c544f416-10c9-11e7-8337-82f88cac9fcd/

    releases:
    total 36K
    drwxr-xr-x 9 timwolla timwolla 4.0K Dec 19 19:31 ./
    drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 5aa3a138-9c86-11e5-99bc-79a6c3e509eb/
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 …
    drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 dba26980-d9f1-11e8-8357-3f2750c60e51/

    repositories:
    total 12K
    drwxr-xr-x  3 timwolla timwolla 4.0K Dec 19 19:30 ./
    drwxr-xr-x  5 timwolla timwolla 4.0K Dec 19 19:31 ../
    drwxr-xr-x 41 timwolla timwolla 4.0K Dec 19 19:31 TimWolla/

Glad to see more sites jumping on this party, I just wish more tools were out there that interpreted data from these dumps directly and made them more readable.

I've started the practice of keeping backups of somewhat recent "all my data" dumps from each site that offers them.

Legit question: what about the data stored about you, but that you have no longer access to? I have no clue if that is also covered by GDPR?

but I still cannot delete it...?

How would deleting work? They can't really anonymize any commit data without breaking git clones, so I guess deleting data would just delete anything non-git like PR text, issues, etc?

Yeah, I think deleting issues etc. should be possible.

There was a big discussion about this on the Git ML recently: https://public-inbox.org/git/20180417191549.GA1199@helen.PLA...

I.e. whether the argument that you're exempt from deleting data under the GDPR just because it's inconvenient for your storage format is going to fly in the EU. If not there's scary times ahead for Git.

Here's a good example of a 4 year old using (presumably) her full name committing to the kernel: https://github.com/torvalds/linux/commit/690b0543a813b0ecfc5...

If that kid grows up, moves to the EU, and wants Microsoft (GitHub) to stop publishing that content, will they be able to just say "no"?

How about if the kid has had their personal data embedded in the Bitcoin blockchain?

Interesting times indeed.

In addition to making the request difficult for systems like git, if inconvenienced storage formats are exempt then this could easily be a loophole to circumvent the GDPR entirely. Facefriend, Instatwit, etc could, e.g., store all user data in something like git (your messages are committed using your name, etc) and point to this convoluted system as an excuse to not comply.

You've been able to permanently delete your GitHub profile and all data associated with it for quite some time.

https://help.github.com/articles/deleting-your-user-account/

I've seen this 'ghost user' around on threads for years....

It seems odd for GitHub to be jumping through GDPR hoops - while at the same time RETROACTIVELY revealing all force-pushes [aka previously hidden and irretrievable by clone] in pull requests.

Of course, that data was compromised the second it was pushed, but, now they're parading the fact that people make mistakes...

What side are you on GitHub? Privacy? Or not?

Force pushing code doesn’t make your old code private.

That's an interesting point though, if you made the mistake of publishing a private key at some point and force-pushed your repo to remove it you expect it to be gone from the system, but it's not. I wonder if you could actually retrieve it from the cloned repo. Is it hidden somewhere in git?

EDIT: The cloned repos seems good, but following this post: https://stackoverflow.com/questions/3973994/how-can-i-recove... you can easily find some "lost" commit and retrieve secrets. I checked with one of my repo and I realized I had compromised information...

Reply on news.ycombinator.com