Hacker News Reader

Comment by cfjgvjh | Hacker News Reader

I really really want to do version control for everything, but most of my data is binary so it really doesn't agree with git; I tried using LFS as well but it didn't work for my specific workflow. Hopefully something less text oriented comes along in the future.

Lore looked interesting for this purpose. https://github.com/EpicGames/lore

View on HN

Hash the files and commit the hashes. Store the large files somewhere convenient, keyed by the hash. Voila, you’ve invented an LFS.

That's just how plain git works anyway. Commits are just a tree of hashes which refer to the blobs in the objects store. Diffing is done on demand. If you can implement diff for binary files, then you've got git.

True. The main difference, like you pointed out, are the concept of a file tree and a commit tree which adds some structure along the time and space dimensions on top of the content-addressable blob storage.

That's exactly what git-annex does: https://git-annex.branchable.com/

LFS is kinda close, but you don't store the files "somewhere convenient": you need support on every git remote to handle the files.

But versionning everything is not really great idea. Probably 90% of your files are static. Archives, Media, LIBs. Not worth versioning. But for that 10%, having any (D)VFS is handy. Thats why I wrote DOT for myself, to handle that niche need to version every document I work with.

Its private for now, but here is printout of help: http://borg.uu3.net/~borg/?dot

why not version control those things? if they're static, there's no changes so version controlling them has little cost. maybe existing tools aren't adequate for this conception of the world, but I don't think that's a good reason not to work towards "version control everything"

Cost of keeping all data (up to +100%). If they dont change, no point of really versioning them. Backup of those is simple either, whatever syncing software (incremental).

Docs that often change, its completly different story. I love VCS like features where I can slam 'dot status' to see whats going on, commit when I feel I need snapshot, sync to whatever device I want. And because I run DVFS I dont even bother with backups because I have multiple copies of repo with entire history.

The bigger issue is that most of the files you're likely to capture in "version everything" are just noise.

Disk space is far from cheap when the value of what's being stored is basically zero.

You can version everything today with something like ZFS snapshoting - and very quickly realize hundreds of gigabytes of noise aren't worth keeping.

I guess it kinda depends on your use case. If you're versioning software artifact binaries and want to have support for rules for retention (keep n number of builds around), version numbering (you can only write a release artifact with a version number once), etc., using Git for text and Artifactory for artifact binaries has worked pretty well for me.

It's interesting you brought up Lore. I also wonder whether importing VCS tech from the games industry for versioning general agentic work would be a good idea.

Why not just fork git and make it better at binary files?

Problem with dealing with binary files with git (or source control in general) is that you want to be able to see diffs, which is very dependent on the type of binary file you have.

I think you can create git plugins to show diffs in different formats for certain file-types (or even open a 3rd party tool to let you visualize it), but it is a lot of work. Most productivity tools don't have anything of the sort.

If you just care about storing the file git LFS + manual text-changelog per binary file works well, but it is annoying to get everyone on your team to work in this workflow (heck, getting everyone and all automation scripts to install LFS is already a pain).

The problem is really starting g with a coders mindset that reducing a change to an explicit visual representation of a minimal delta is such a useful thing for everybody. It isn't. For example, for images, an A/B comparison is enough in a lot of cases. There are also no meaningful ways to "merge" changes for a lot of data outside source code, so the insistence that version control can't exist without merging is shortsighted.

Meaningful deltas and meaningful merges are actually a problem set that is totally and utterly disjointed from keeping the underlying change history. Deltas and merges can always be computed independently of the underlying storage format for versioned data. Git could have indicriminately used binary deltas for file storage and would still work the same on every higher layer.

In Git the deltas (or diffs) are derived data. They aren't central to what Git is. Natively and conceptually, Git works with repository snapshots.

You don't even need to fork it: git has something like a plugin system for exactly this use-cases. (I'm not exactly sure what their terminology is. But they thought about this use-case before.)

yes they are called git-merge drivers. https://git-scm.com/docs/gitattributes

I mentioned it before, see the following comment https://news.ycombinator.com/item?id=47749916

You can also use a custom difftool or mergetool that understand your formats. For example Beyond Compare https://www.scootersoftware.com/kb/moreformats

Reply on news.ycombinator.com