Git Submodules as a Package Manager
nesbitt.io
nesbitt.io
If I have a 20GB submodule I'd usually just do a `git clone <url> <path> && git submodule add <path>` and it will be treated the same by git. But now I can just delete it and it's purged. And it is more portable less fragile in some ways because it's not de-referencing a gitfile. I prefer my repos to be more bottom-heavy and to not clog my modules folder.
I have a rough script that's the inverse of `git submodule absorbgitdirs` but it's a bit fragile.
It would be cool if there was some plumbing to expose this a bit more.
If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos.
What does this even mean?
I'd argue that having the source to build the thing is more important than the artifacts. Release artifacts are more of a convenience. If the thing doesn't build from the source given, what use is it?
Sure you want the source available, otherwise it's closed source, but ideally you never need to look at it unless you're a contributor.
git has really conflated the two concepts.
Any system that can only consume manually built (and even in worst case, only tagged and globally pushed) dependencies is horrible for cross-module development, like large scale refactorings and the like.
To say nothing about hardcoded repo URLs (not resolvable identifiers), transient dependencies, URL schemes, and trying to pretend github is the only package source.
I'd argue the other extreme is less sane. Pip installs everything system wide and you can't have different versions of the same package without venv.
Even early NPM was much much better than the pre-uv Python situation. (TBH even with uv it still feels hacky at times.)
It’s telling that in Python there have been multiple package managers over the years to fix the broken status quo. Typescript/Javascript notwithstanding, in other languages and environments you get the one tool that’s been there for over a decade without competitors.
Very few languages have suffered from the same package manager issues as JS/TS and Python.
Also, gulp / grunt weren't package managers.
These aren't the only two options.
This
> Pip installs everything system wide
does not inherently mean
> you can't have different versions of the same package
For non-inert packages, such as applications installed via the language's dependency manager, that's a different use case and should be handled differently.
There's a whole umbrella of similar-ish utilities. The one that I think does almost the same thing is: https://github.com/mullvad/git-carbon
It's also "pointer" based, but like vendoring, it's up to you whether or not you want to keep files in the repository. It also seems to be intentionally built for the use case where you modify something from upstream, because if you do have local changes, it detects them and won't overwrite them without your permission.
Some things are annoying, like removing them. But in general they're extremely useful and designed basically exactly how you'd want: it's extremely consistent since to a good approximation, a submodule behaves the same as a normal git repo once you're inside it.
1. Git doesn't support them very well. Bugs are disappointing frequent. Try changing a submodule to a directory or vice versa and feel the pain.
2. They are often used to split up projects (especially commercially) but this makes testing much harder and cross-repo changes way way harder. Even changes that only touch the submodule become way more tedious because you have to always do another PR to update the pointer.
3. The submodules are referenced by URL, which means you can't easily move or copy projects that have them because they still point to the old submodule location. Sometimes they're even misconfiguration to always use ssh:// or whatever, breaking unauthenticated clones.
4. Switching branches where submodules have been added/removed is a mess.
5. Having to constantly remember `git submodule update --init --recursive` is a right pain.
6. When used as a crap dependency manager you easily end up with duplicates. A project I worked on had 12 copied of a common submodule via transitive dependencies.
IMO they're almost always the wrong solution.
I can't remember any "bug" in the submodule machinery. If you find one, report to the Git mailing list, they very receptive.
> Having to constantly remember `git submodule update --init --recursive`
Set this config: `git config submodule.recurse true`
> IMO they're almost always the wrong solution.
Yeah... but sometimes they are the only feasible solution, especially when the dependency needs to be shared across projects that use incompatible dependency managers and Git is the only common denominator between them...
Well that does save a flag at least so thanks! But I still have to run the command.
There is actually a flag to do submodule update automatically when you checkout or switch branches but DO NOT USE IT! It is known to be buggy and can break your .git directory in a way that is pretty much "delete everything and clone from scratch".
But if you work with larger projects I think you'll see how they are useful.
For example, some projects might commit their node_modules or composer vendor folder as submodules so that they have an exact snapshot of their dependencies that can be quickly cloned in place and referenced without mixing it with the parent repos source.
Submodules are certainly not the only way to do things with git. Sometimes monorepos or something else makes more sense.
But if you care about maintainability then what are you supposed to do when the majority of your commits are unrelated to your project because you're using git subtree or something similar? Or the majority of your files are also not directly relevant to your project but need to be there?
What about stuff like optional tests or maintenance scripts or even binary dependencies that you want to have closely associated with your project but don't want to clone every time?
My thinking is that the porcelain is incomplete but very open. The way submodules are represented can be added on to.
And the plumbing is consistent enough that you easily script around it.
You should check out the Datalad project to see some really clever usage of submodules.
And the url key can be annoying; I usually try to use relative paths in my .gitmodules folder when possible. You're actually free to use remote helpers and relative paths to get around hard a github url into it.
You can also embed and read arbitrary keys from your .gitmodules file with git-config -f which is really useful. You don't even have to use a .gitmodules file, in theory.
We need Git (or something else) to provide a third solution that actually works. I'm really hoping that JJ realises that this is a problem that needs to be solved but I'm not holding my breath.
I think submodules are good enough for what they do and there's room in Git's model for forward development and it's also flexible enough that third party tools can piggyback on it. Like Git in general it's a bit of a pain till you practice with it more.
Git can do some amazing stuff, I use it for data as well as code.
If someone adds a new submodule, you need to know to pull it or at least init it. You need to know the code is in a submodule and treat it differently, complicating integration procedures.
It certainly does not behave the same in that you certainly can't ignore whether it exists once it's set up.