Hacker News Reader

Comment by adrianmsmith | Hacker News Reader

Another thing Git does not and cannot even attempt to do - file locking.

The assumption behind Git is, everyone develops on their machines and/or branches, and then things are merged. This only works for files which can be merged.

There are plenty of things pretty much any project wants to track which cannot be merged, for example Word documents (documentation), Photoshop files (source of graphics), PNGs (icons in webapps), and so on.

With a centralized system, that's easy, just go over to using file locks ("svn lock") for those files. With a distributed system, that's impossible.

View on HN

> Another thing Git does not and cannot even attempt to do - file locking.

That's a seriously hard problem for a DVCS if you're serious about the "D".

This topic turned into [the single longest thread in the history of the Fossil forum](https://www.fossil-scm.org/forum/forumpost/2afc32b1ab) because it drags in the CAP theorem and all of the problems people run into when they try to have all three of C, A, and P at the same time.

To the extent that Fossil based projects are usually more centralized than Git ones, Fossil has a better chance of solving this, but I'm still not holding my breath that Fossil will get what a person would naively understand as file locking any time soon.

> Word documents (documentation), Photoshop files (source of graphics), PNGs (icons in webapps), and so on.

You want to avoid putting such things into a VCS anyway, because it [bloats the repo size](https://fossil-scm.org/fossil/doc/trunk/www/image-format-vs-...). I wrote that article in the context of Fossil, but its key result would replicate just as well under Git or anything else that doesn't do some serious magic to avoid the key problem here.

Instead of Word files, check in Markdown or [FODT](https://en.wikipedia.org/wiki/OpenDocument_technical_specifi...). (Flat XML OpenDocument Text.) Or with Fossil, put the doc in the wiki.

Instead of PNG, check in BMP, uncompressed TIFF, etc., then "build" the PNG as part of your app's regular build process.

This has the side benefit that when you later change your mind on the parameters for the final delivered PNGs, you can just adjust the build script, not check in a whole new set of PNGs. My current web app has several such versions: 8-bit paletted versions from back before IE could handle 24-bit PNG, then matted 24-bit PNGs from the days when IE couldn't handle transparency in PNG, and finally the current alpha-blended 24-bit PNGs. It'd have been better if I'd checked in TIFF originals and built deliverable PNGs at each step.

> Instead of Word files, check in Markdown or [FODT]

Another fun option is to unzip the DOCX and check that in, since it is mostly a collection of XML files in a zip container. I built a tool to automate zipping/unzipping files like DOCX years ago as pre-commit/post-checkout/post-merge hooks. [1] It's an interesting way to source control some types of files if you can find a way to deconstruct them into smaller pieces that merge better. Admittedly, merging Office Open XML by hand is not a great experience (and dealing with subtly broken or corrupt internal contents is not fun, because programs like Word can fussy when things are even slightly wrong), but you get better diffs sometimes than you would expect.

[1] https://github.com/WorldMaker/musdex

Yes, I cover that option for Fossil at the end of the pointed-to document. I did it in terms of Makefiles rather than commit hooks, but whichever...

> You want to avoid putting such things into a VCS anyway, because it [bloats the repo size]

How do you suggest projects like games handle this, where data files are naturally linked to source files? Imagine trying to sort out an animation bug when you only have source level tracking and no idea which version of the animation data corresponds to the animation source files of the bug report. These data files are not 'built' from the 'build' step as they are the product of artists.

I’d guess not one user in 100.000 uses git decentralized (as in, doesn’t have a blessed “central” repo). It’s the disabling of locking that should be the special case! The big problem with git is that you can’t mark a repo as a master repo/blessed repo (which would be the one where lockfiles are stored). A lot of functionality would be helped if the commands could know which end is the important/central one.

> I’d guess not one user in 100.000 uses git decentralized

I understand your sentiment, but the denominator in that fraction is probably much lower than your guess.

Consider even simple cases like the disconnected laptop case. You may work at a small office with only local employees, and so you have one central "blessed" repo, but if one person locks a file and then goes off to lunch, working on the file while at the restaurant, you still have a CAP problem:

CA: Because the one guy with a laptop went off-network, you have no full quorum, so no one can use the repo at all until he gets back and rejoins the network. (No practical DVCS does this, but it's one of the options, so I list it.)

CP: When the one guy went off to lunch, we lost the ability to interact with his lock, and that will continue to be the case until he gets back from lunch. Also vice versa: if someone still at the office takes out a lock, the guy off at lunch doesn't realize there is lock, so he could do something bad with the "locked" file. (This is the mode DVCSes generally run in by default.)

AP: No locking at all, thus no consistency, thus your original problem that inspired the wish to have file locking.

Not sure I understand the problem. If I lock fileX and go to lunch, then I own the lock on that file while I’m out to lunch. It’s basically analogous me pushing the file fileX.lock to the repo next to fileX, with my user id as content. I can only do it if it isn’t there.

Everyone else will only see that lock if they fetch and if they don’t, they might edit their local copy of fileX too, but would be prevented from pushing their version to the blessed repository by the lock. They can push a copy under another name, or wait until I have removed the lock (but probably can’t resolve the conflict anyway because it’s likely a binary document). So they user will remember to never start editing without taking the lock in the future.

It’s not perfect by any stretch of the imagination but it’s all anyone asks for in terms of file locking. It’s what Subversion always did.

> Not sure I understand the problem. If I lock fileX and go to lunch, then I own the lock on that file while I’m out to lunch.

And if you go on vacation for two weeks instead?

Same thing obviously. But this is just a method of communication. It’s instead of emailing/shouting “don’t edit the background image today please” across the office.

An admin can remove the lock. Or you can allow force-pushing by anyone to replace it or whatever.

Not sure why this is seen as so complicated, version control systems have done it since forever. It’s not trying to solve some distributed lock system in a clever way. It’s dumb centralized mutex per file. And yet again this is all that’s needed (and it’s also added to git in git-LFS!).

You can set up custom merge drivers for different file types.

More importantly, if you've got stuff in your decentralised repo that shouldn't be decentralised, that's not the fault of the DVCS you're using, it's your fault. That everything looks like a nail does not speak against the value of a hammer.

> You can set up custom merge drivers for different file types

True, but there are inevitably some files which still cannot be merged, so the problem remains.

> More importantly, if you've got stuff in your decentralised repo that shouldn't be decentralised, that's not the fault of the DVCS you're using, it's your fault.

Indeed, if you want to store the history of your files - the whole software including the icons it uses, so that you can go back to any previous version and build it - and you chose a DVCS like Git, I would agree the fault was yours.

That's basically what I was arguing, that Git is the wrong choice if you have any binary assets like icons (even if those assets have small filesize) due to the lack of locking, sorry if I was unclear.

You should not be checking in binary files into git. That defeats the entire purpose and bloats the repo size enormously.

Git LFS should be used instead. Or storing a Sha256sum and putting the file elsewhere.

That is a solution but then it's no longer distributed.

Being distributive isn’t a feature for the binary files contents. I don’t want 100 historical versions of a huge game texture, just the latest one. The history is distributed however, so I can see who changed the texture even when disconnected. Centralized binaries like git LFS works like any package/asset manager.

There is no one that would want distributed binaries in git. But people also don’t want to switch from git to something else just because they have a 100GB or 10TB repo. Tooling (build tools, issue management) everywhere has decided that git is all that’s needed.

Not putting binaries in git isn’t a solution at all. Binaries are part of the source in many applications (e.g game assets, web site images...). Distributing every version of every binary to everyone is also not a solution.

Reply on news.ycombinator.com