Hacker News Reader

Comment by avar | Hacker News Reader

I'm the person and Git developer (Ævar) quoted in the article. I didn't expect this to end up on LWN. I'm happy to answer any questions here that people might have.

I don't think the LWN article can be said to take anything out of context. But I think it's worth empathizing that this is a thread on the Git ML in response to a user who's asking if Git/SHA-256 is something "that users should start changing over to[?]".

I stand by the comments that I think the current state of Git is that we shouldn't be recommending to users that they use SHA-256 repositories without explaining some major caveats, mainly to do with third party software support, particularly the lack of support from the big online "forges".

But I don't think there's any disagreement in the Git development community (and certainly not from me) that Git should be moving towards migrating away from SHA-1.

View on HN

Have you considered moving over to a combined SHA-1, SHA-256 model where both hashes are calculated, with SHA-1 shown to the user and SHA-256 only used in the background to prevent collisions?

There is a compute cost for that, but it should be minimal relative to the security benefits?

Someone probably brought it up at some point, I can't remember. But I'm not aware of any known scenario where the SHA1DC library Git uses doesn't give you the benefits of that and more.

"And more" because to detect a collision with a background SHA-256 you'll need both objects, whereas SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions. So it won't pass along an object that collides with another one, even though it only has 1/2 objects.

That distinction is something that's generally considered important, e.g. there's been past exploits in Git where you could trick a client into doing something bad by e.g. a crafted .gitmodules file.

The fix has not only been to patch clients, but also to patch "git fsck" to detect and reject such bad contents, so that e.g. the forges can't be used to relay a repository exploit to users running older versions.

A viable hash collision exploit in the wild might likewise want to make use of such an attack scenarios, so having servers capable of detecting collisions without having both sides is preferable to doing so by re-hashing with SHA-256.

> SHA1DC detects attempts to spoof SHA1 in a way that leads to collisions

This is the kind of detail I would have loved to see quoted directly in the article. Sure enough, it's prominently displayed on the project's README::about section, but your very succinct explanation here made it clear in immediate context.

The idea of counter-cryptanalysis is eye-opening.

This is what I would have recommended... A transition period where both are used

Thanks for your work on Git!

> I'm happy to answer any questions here that people might have.

Is there any way to achieve a gradual, staged rollout of SHA256?

What's the impact of converting a repo to SHA256 - will old commit IDs become invalid? Would signed commits' signatures be invalidated?

The answer is somewhat hand-waivy, because this code doesn't exist as anything except out-of-tree WIP code (and even in that case, incomplete). But yes, the plan is definitely to support a gradual, hopefully mostly seamless rollout.

The design document for that is shipped as part of git.git, and available online. Here's the relevant part: https://git-scm.com/docs/hash-function-transition/#_translat...

Basically the idea is that you'd have a say a SHA-256 local repository, and talk to a SHA-1 upstream server. Each time you'd "pull" or "push" we'd "rehash" the content (which we do anyway, even when using just one hash).

The interop-specific magic (covered in that documentation) is that we'd use a translation table, so you could e.g. "git show" on a SHA-1 object ID, and we'd be able to serve up the locally packed SHA-256 content as a result.

But the hard parts of this still need to be worked out, and problems shaken out. E.g. for hosting providers what you get when you "git clone" is an already-hashed *.pack file that's mostly served up as-is from disk. For simultaneously serving clients of both hash formats you'd essentially need to double your storage space.

There's also been past in-person developer meet-up discussion (the last one being before Covid, the next one in fall this year) about the gritty details of how such a translation table will function exactly.

E.g. if linux.git switches they'd probably want a "flag day" where they'd transition 100% to SHA-256, but many clients would still probably want the SHA-1<->SHA-256 translation table kept around for older commits, to e.g. look up hash references from something like the mailing list archive, or old comments in ticketing systems.

Currently the answer to how that'll work exactly is that we'll see when someone submits completed patches for that sort of functionality, and doubtless issues & edge cases will emerge that we didn't or couldn't expect until the rubber hits the road.

Has anyone considered doing "add16" on the first character of the sha-256 hash, e.g., so the SHA-256 hash 1d06... becomes hd06... ? Then you could see, from the first character, if it is SHA-1 or SHA-256. Having a clear distinction on the first character would make it clear which hash is being used (without needing lots of chars).

I think this was discussed at some point early on, there's many outstanding issues with Git's SHA-1<->SHA-256 interop and potential unknowns, but figuring out which format a given abbreviated hash is in isn't really one of them.

For any internal part of Git the hash type is already known, e.g for the wire protocol dialog that "fetch" runs. Other commentators have pointed out that you can use the hash length to disambiguate the two, but the way it's implemented internally we could tell them apart even if we hypothetically had two hashes of the same length.

But that leaves abbreviated hashes, e.g. if you do "git show deadbeef" is that a SHA-1 or SHA-256 hash? We don't know.

It's forseen that in those cases we'll look up both, and in the case of ambiguity do the same thing as e.g. "git show dead" does now (try it on a non-trivially sized repo). Then just as you can do e.g. "git show dead^{commit}" now to disambiguate, you'd use the ^{sha1} or ^{sha256} peel syntax (still unimplemented).

If we changed the hash format from [0-9a-f]{4,40} to something that didn't match the first part of that regex there's even more downstream systems that would need adjusting. A lot of programs that work with Git's output use some variant of that regex. For those that don't limit themselves to "40" things usually Just Work.

So that's basically the reason, it's also less future-proof, as you'll run out of magical prefix characters faster than you'll run out of hash names. Although I'm hoping to be dead way before that small space would be exhausted :)

Has there been any feedback/communication with forges happening, on or off-list?

I'm curious how closely (if at all) they've been following this effort

The forges are keenly aware of this effort, e.g. the person who's by far done most of the work on the SHA-256 transition (brian m. carlson) has I believe mostly or entirely been done so on behalf of GitHub.

I myself do some work on upstream Git on behalf of GitLab, although none of it's been on anything related to the SHA-256 transition.

As to why no big forge has SHA-256 support, I think it's a bit of a chicken & egg problem (and these comments are entirely my own, and not on behalf of anyone).

I think it's safe to say that all of the forges are expecting the transition, e.g. I don't think there's anyone creating CHAR(40) database tables for Git hashes anymore (or if they are, someone is planning to deal with it).

Another is that for a successful transition for anything except entirely new repository networks (which already use SHA-1) you really need the "git" client to play along, see my other comment discussing hash interop plans. Some of that same code then needs to run on the server-side.

That code isn't part of git yet, and it's really needed for any sort of viable migration plan.

I mean, it's not really needed, at some point a lot of people reading this migrated from CVS and/or SVN to Git. But a full export/import with a lot of users is painful. We really want it to suck less for Git, to the point that it should Just Work for most or all users.

And a major one is the human factor. For things to happen in free software development someone needs to submit patches, brian m. carlson has been performing a heroic amount of effort over the year on the transition over the years. As he notes in the linked ML thread he's had life reasons for why he hasn't been able to work on it as actively recently as he did in the past.

brian moved from Texas to Canada, but mostly his employee, GitHub, is not priotizing the remaining sha256 patches. someone needs to finish up the transition patches, and forges need to double their disc space.

I've seen this before but why do forges need to double space?

They may or may not, it's a time-space trade-off, and I'm just guessing that they'll be more likely to go for eating disk over eating CPU.

There's some details on this in https://git-scm.com/docs/hash-function-transition/#_fetch; part of the expected hash transition document assumes that you could have say a SHA-1 *.pack, but maintain both a SHA-1 and SHA-256 index into its contents.

For blob objects you could serve them up as-is, but for the other types which refer to other objects (commits, trees and tags) you'd either need to store two copies and do something close to a sendfile(), or rewrite them on-the-fly as you stream them, using your SHA-1<->SHA-256 translation table.

So I got a bit ahead of myself there, but none of this interop code exists yet, so the trade-offs the forges will have to make are unknown at this point.

Don’t blobs dominate the total disk space? It sounds like some data would double but the overall disk space usage shouldn’t be quite so dramatic (except for really small repos where it probably doesn’t matter?)

that was my intuition which is why the 2x space seemed odd

for the old hash, plus the new hash, plus the transition tables. (new <-> old)

but I would expect the vast majority of storage to be metadata and blobs, not hashes

Reply on news.ycombinator.com