Further, I'm happy with git. I played with Mercurial years ago, long enough to work with it day-to-day, and just didn't find any relevant advantages versus git.
I love that people are still out there trying to improve things, and certainly don't want that to stop, but it's difficult for me to imagine switching at this point.
1) Better handling of large files than git-lfs. As in 10+ GB repos. This is needed for game development (currently they tend to use Perforce or PlasticSCM)
2) Sparse checkout via file system integration (like Eden has)
3) Build system integration, so unchanged files and modules don't even need to be fetched to be compiled, because cached builds can be fetched from a build server instead (requires proper modularization, so e.g. C++ macro expansion doesn't just prevent anything from being cacheable)
These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it.
Would this be a good fit for large machine learning data sets?
Every time a new model comes out that touts having been trained on something like a quarter of a billion images, I ask myself "how the heck does someone manage and version a pile like that?"
Potentially. It would depend on the image format and what metadata is (or can be) included in the file.
It would be a game changer if any SCM actually manages to achieve this without needing multiple developer teams just for maintenance.
That would probably break gits current dominance within a relatively short timeframe... But I don't think it'll happen within the foreseeable future,
More seriously, can you describe what "build system integration" would look like? Basically like what GNU make does with RCS? I.e. considering the SCM "files" as sources.
How would such a system build a dumb tarball snapshot?
In practice it means that if I clone the repo and build it doesn't ever invoke the compiler, so that it takes only a few minutes instead of an hour.
Bazel has a similar system but I haven't used it myself,
The GP talks about "build system integration" being "a game changer", and I can see that being a useful thing for e.g. "priming" your cache, but that's surely just a matter of say:
ccache-like-tool --prime $(git rev-parse HEAD:)
find -type f -name '*.[ch]' -exec cat {} \; | sha1sum | cut -d' ' -f1
Vertical integration, basically. There is value in these things working, and working well. As an example of the issues with ccache etc, I've yet to find a compilation cache that works well on windows for large projects.
The "automation" of our in-house system is what really makes the difference but then again we have a team of developers that focus on tooling so it's not so much automated as it is maintained...
Dud is meant to be a companion to SCM (e.g. Git) for large files. I was turned off of Git LFS after a couple failed attempts at using it for data science work. DVC[1] is an improvement in many ways, but it has some rough edges and serious performance issues[2].
With Dud I focused on speed and simplicity. To your three points above:
1) Dud can comfortably track datasets in the 100s of GBs. In practice, the bottleneck is your disk I/O speed.
2) Dud checks out binaries as links by default, so it's super fast to switch between commits.
3) Dud includes a means to build data pipelines -- think Makefiles with less footguns. Dud can detect when outputs are up to date and skip executing a pipeline stage.
I hope this helps, and I'd be happy to chat about it.
[0]: https://github.com/kevin-hanselman/dud
[1]: https://dvc.org
[2]: https://github.com/kevin-hanselman/dud#concrete-differences-...
Git LFS is intended more for a centralized "big repo" workflow, git annex's canonical usage is as a personal distributed backup system, but both can stretch into other domains.
In this case git-annex seems to have a feature that git LFS doesn't have that would be useful to you.
Because now you've got two versioning systems with different quirks and tools that behave slightly different that _must_ be kept in sync.
> I get that it's easier to keep everything together,
Easier is a bit of an understatement. Putting binary files outside of the project in another versioning system means you need two version control systems, _plus_ custom tooling to glue the two together. Also, the people who interact with these assets primarily are not technical at all, they're artists or designers, and they're the most likely to have issues with falling between the cracks.
> Better handling of large files than git-lfs.
PlasticSCM does really really well with binary files.
> Sparse checkout via file system integration
Windows only [0] but Plastic does this. I've been working through some issues but it's usable as a daily driver with a sane build system.
> Build system integration
UnrealBuildTool at the same time is both the coolest and most frustrating build systems I've ever used. It's _not_ a general purpose build system for anyone and everyone, it's tailored to Unreal, it's slow to run the actual build system, the implementation is shaky at times, but some of it's features are incredible. Two standout features are Unity Builds, and Adaptive Unity builds. Unity builds are common now [1], but adaptive unity is a game changer. It uses the Source control integrations to check what files are modified and removes them from the unity blobs, meaning that you're only ever rebuilding + relinking what's changed, and get the best of both worlds.
[0] https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha... [1] https://en.wikipedia.org/wiki/Unity_build
It can use the Git data store, so developers can in theory start using it without the whole team adopting it. Then it addresses the big problem with Git, the user interface:
https://github.com/martinvonz/jj
I'm not suggesting that "jj" in particular will disrupt Git, but I think eventually the a tool which supports the git data store with a better user interface could take hold.
So rather than use Git + Git Annex or something like that (maybe more), you’ll just use this alternative SCM.
(I keep hearing about how Git will eventually support big repos, but it’s still not on the horizon. Big repos with Git still seems to be limited to big corps who have the resources to make something bespoke. Personally I don’t use big repos so I have no skin in this game.)
- Protocol for more efficiently widening sparse checkouts.
- Virtual filesystem or other solution for automatically widening sparse checkouts to greatly improve the UX.
- Ideally changing sparse checkouts from a repo state to just some form of lazy loading. Otherwise as you touch more files your performance will slowly degrade.
One thing I recall was an effort from pijul to make blaming a file run in O(n log h) where n is the size of the file and h is the size of the (repo or file, I’m not sure) history. I wonder if Eden will also have improved blame performance. I noticed they mentioned large histories but maybe it is still linear in the history size of the file. (The way hg works gives I think O(nh) where h is the size of the file history, which he stores per-file rather than per-commit like git).
Similarly, I think Git(hub) has succeeded in open source because bundling more complete changes into PRs works well for unevenly distributed contributions.
You know the feeling of having to use SVN after using Git? This is what it feels like to use Git after getting used to Meta's Mercurial. I wish I could go into the details, but I don't know how much of it was ported back to Mercurial.
One possibility could be to use some kind of git ui. I only know about magit (which is built on/in Emacs) but I’m sure others exist.
If they are then I totally agree.