Hacker News Reader

Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization.

These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data.

I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller problem domain than the one described in the article.

I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state.

Not all apps will work with that model, but more apps than you think would.

I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements a lot of custom application-specific sync with CRDTs and so forth.

Maybe you need the really fine-grained updates, or maybe you just need some Github-like site which allows coarse-grained collaboration.

In other words, there can be a network model between "e-mail a .PSD file" and Figma. I think this "in between" would scale to more applications than reinventing sync inside every app. Imagine audio editors, video editors, 3D modellers, etc. Rewriting all those in the style of Figma is prohibtive.

I would rather have open file formats and application-agnostic sync, like git. And I think it's a lot cheaper to develop, although current software business models don't really support its development.

“The point of git is that there isn't a single authoritative state.“

I think you’re missing the point of the post you refer to.

For git, there _technically_ isn’t a single authoritave state, but for many, if not most, git use cases, there _sociologically_ is. Projects typically have one repo that is _the_repo_: the repo most merges are done to, that releases get built from, whose url you give when you tell people where the project lives, etc.

All other clones of that repo, sociologically, are just caching the main repo and changes you make.

Then, humans have to decide when to flush what part(s) of the write cache to the main server, and when that results in conflicts, humans have to resolve them.

That’s, I think, what “which manages to push all the coherency issues onto the user” means.

You are correct, I meant that even in a peer-peer merge (which can even be just one person debugging the same program on both a Mac and a Linux machine), any merge conflicts are marked and left for the user to decide about.

The presence of useful forks means that there isn't a single authoritative state. For example, I've had a fork of bash-completion for a over a year, and I use it all the time.

How do I fork a Google doc? At best you can copy and modify, but here's no merge.

Merging is a feature, not a bug (maybe not for all applications, but 99% of them).

This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE.

Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.)

FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.

Not sure how well if fits your use-case, but I've been very happy with git-lfs in combination with a NAS I have at home. The NAS is just mounted as a normal network drive and available to use with LFS via lfs-filestore[0] and available on the go via the builtin DynDNS + VPC of the NAS.

I've been using it for a repo where I store all my university/academic stuff like lectures (recordings), PDFs of books and papers, Anki decks, etc., and it has now grown to be ~120GB big.

Biggest issue was that I'm always running low on disk space with my laptop, and git-lfs doesn't have a good built-in way to only checkout part of the files on your machine, so I built a small tool to make that easier[1]. Since I've been using that it's been a pretty smooth ride.

[0]: https://github.com/sinbad/lfs-folderstore

[1]: https://github.com/hobofan/lfs-unload

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes.

I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works means there'll be two copies of your data, which is not great for huge datasets.

Git-annex (https://git-annex.branchable.com/) is pretty great, and ticks almost every checkbox I want. Unlike git-lfs, it uses symlinks instead of pointer files (by default) and gives you a lot of control in managing multiple remote repositories. On the other hand, using it outside of Linux (e.g., MacOS) has always been a bit painful, specially when trying to collaborate with less technical users. I also get the impression that the main developer doesn't have much time for it (understandably - I don't think he makes any money off it, even if there were some early attempts).

My current solution is DVC (https://dvc.org/). It's explicitly made with ML in mind, and implements a bunch of stuff beyond binary versioning. It does lack a few of the features of git-annex, but has the ones I do care about most - namely, a fair amount of flexibility on how the remote storage is implemented. And the one thing I like the most is that it can work either like git-lfs (with pointer files), like git-annex (with soft- or hard-links), or -- my favorite -- using reflinks, when running on filesystems that support it (e.g. APFS, btrfs). It also is being actively developed by a team at a company, though so far there doesn't seem to be any paid features or services around it.

Pachyderm (https://www.pachyderm.com) also seems quite interesting, and pretty ideal for some workflows. Unfortunately it's also more opinionated, in that it requires using docker for the filesystem, as far as I can tell.

Edit: a rather different alternative I've resorted to in the past -- which of course lacks a lot of the features of "git for binary data" -- is simply to do regular backups of data to either borg or restic, which are pretty good deduplicating backup systems. Both allow you to mount past snapshots with FUSE, which is a nice way of accessing earlier versions of your data (read-only, of course). These days, this kind of thing can also be done with ZFS or btrfs as well, though.

+1 for DVC. Setting up the backing store can be some extra work if you are doing that yourself, but after that it's a breeze.

What do you use for the backing store?

Git-lfs has been a pain in my seat since my first use of it. Most of the issues stem from the pointer files that have to be filtered/smudged pre/post commit.

Haven't used git-annex myself, but I have heard from coworkers that cross-OS is a pain.

Mostly S3. I used to do SSH, but these days I can afford to keep the data in the cloud. I do appreciate the possibility of migrating to other stores if needed in the future, though - might have to soon, for $reasons.

Actually, a lot has updated about git-annex in the last few years. It actually does support git pointer files like git lfs, which makes it easier to when you want to modify binary files. In fact, it can even use git-lfs servers as one of its back-ends. However, I still prefer symlinks mode, because operations on them are faster because it bypasses the smudge filter.

Also, git-annex uses reflink copies whenever possible, on zfs, btrfs, or apfs. Also, since people were talking about p2p and git, git-annex does this amazing trick for syncing directly to other git-annex repos, even with the checked out branch. There is no need at all for a seperate server.

I have used git-annex for years on OSX, and have not found it be deficient in any way compared to linux.

Yeah, git-annex has a lot of cool features that I have yet to see in other systems. I still use it for some things. My main pain point on MacOS was that the symlink mode didn't work well with some apps that didn't understand symlinks. Obviously this is not git-annex's fault, but it still made it so I couldn't use it. I think I could try again at some point and see if I could get it to use reflinks -- maybe it's a version issue.

I also had weird conflicts with the line ending (the whole CR/LF annoyance) on some of the metadata files git-annex used which I couldn't fix, no matter how many .gitconfigs I tweaked. Again, this is not really git-annex's fault, I think.

Thanks for the response! I have heard of dvc and git annex, and it's probably time to give them another try :)

> It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue.

Well, you may already be aware, but Emacs is actually 'network-enabled' in this way through TRAMP, and to a lesser extent, the emacs client/server protocol.

There are also many issues other than file syncing that networking has to solve. And also, even for regular files, there are many ways to interact with them, and many protocols that existing systems can already speak.

I've been bouncing an idea around for a while, on how could I use git as a back-end for a filesharing/chat/collaboration suite -- I think it would work to have a git hook, pre-commit, that replaces all binary files / blacklisted file extensions, with a text file whose contents are the magnet address to download the large binary file over torrent - so the file name, permissions etc don't change, but a small text file is committed instead of the binary.

So, as a consumer of the repo, I would clone the repo, and then have a post-checkout hook check the blacklisted file extensions and grab their contents: the magnet link, all that's left is to find peers, download the file and replace the text file with a hardlink to the binary. Maybe this is all too laggy, asynchronous, and mismatched, but I think git+magnet could be a cool combination.

I found this repo [1] that generates the magnet link for you, I would just need to find away to use the repo to make all the contributors peers to each other, so we can download the binaries from whoever is nearest / highest bandwidth from us.

[1] https://github.com/casey/intermodal

What you are describing sounds a lot like git-annex[1] with the IPFS remote[2]. There is also a bittorrent remote[3] but it doesn't handle uploading content; you have to create the .torrent file or magnet link yourself and add it to the repo, at which point git-annex will download the content automatically on checkout.

[1] https://git-annex.branchable.com/ [2] https://git-annex.branchable.com/special_remotes/ipfs/ [3] https://git-annex.branchable.com/special_remotes/bittorrent/

I think that a version of this idea underpins the operation of the matrix messaging protocol. Rather than just sending the latest messages, the conversation is synced between clients to ensure that everyone sees the same history.

[1] https://matrix.org

There is git/github on top of Secure Scuttlebutt:

https://github.com/noffle/git-ssb-intro

I have searched something like this for backups, but have not found anything good.

But I also have conflicting wishes. I want it to store the history (so that a backup of directory A cannot be overridden by a backup of an unrelated directory B, and it can apply renames without copying large files again), and then I do not want it to store a history (so that large files that are removed from the main system can be permanently removed from the backup)

I agree that git is a great model, however it's often hard to explain to new users why merge conflict require time to resolve, and nothing saves you from the added work.

Life just isn't completely decentralizable... sorry.

git isn't completely centralized either. You can centralize it like Github, and that plays an important role in the ecosystem.

Decentralization is a spectrum, not the opposite of centralization.

----

One way to partially address to merge conflict problem is to explode application files into directory hierarchies.

For example, Word .doc files and Photoshop .PSD files are basically huge hierchical data structures inside a single file. I believe video formats also have significant hierarchical structure.

A lot of them even have immutable portions and mutable portions -- e.g. for storing an entire version history.

So if those were exploded into something that the OS (or tools like git/rsync) could understand, like a tree of files, then you would have a lot fewer merge conflicts.

That's how people tend structure their git repos too. If you have a frequently edited file that's hard to merge, then that's a smell, and you can fix it.

This won't solve every problem but again there needs to be something in between "rewrite Photoshop as Figma" and "email around a bunch of PSD files" (which is a very flaky distributed system on top of e-mail.)

> most of a whole Linux distro.

It's NixOS.

git has some robustness problem over slow, fragile connections, though. It will not corrupt data, but if your connection cannot stay up long enough to clone an entire repo, you need to resort to ugly workarounds.