Gittup, a Linux distro on top of Git
gittup.org
gittup.org
The documentation is a bit... eclectic.
Tup is a build system that keeps track of what files a command accesses. So if you run make from tup, it can look at every file your makefile reads from, and re-run make if any of those files change. Not that useful, in that example, but when you have it run gcc to make each source file instead, it can automatically rebuild just the files that need to be rebuilt.
Building on top of that, they've set up a git repo with the appropriate "tupfiles" (makefiles) to build linux and enough userspace to get by. It's not so much a linux distribution as a build-system that can build an entire linux distro from source.
The big difference between this and a normal build system is that all your source files can be in the same tree as git submodules, and `tup upd` will rebuild the entire thing, only recompiling code who's source file or shared libraries have changed.
I think it could be a pretty powerful system for a lot of projects, I'm particularly interested in using it for building android images, and firmware for ereaders.
Isn't a Linux distro largely defined by its packaging system, which in turn is largely defined by how software is built from source?
Presuming tup is good, of course. I played with it but I hated the fuse dependency and ended up switching to meson (though I despise the build-time python dependency, see [0]) for cross-platform projects and plain jane gmake or pmake Makefiles for others. I also hand code ninja build files for stuff that’s not too crazy and platform dependent, basically for things that gulp would normally be in control of.
With that said, I was curious how this project got around the 'install' problem, but it seems like they were largely able to just side-step the issue. They build an `initrd` from tup, and the `initrd` Tupfile copies all of the compiled executables into their proper folders before generating the `initrd`. So the projects themselves can't install themselves directly onto a system.
[0] https://groups.google.com/d/msg/tup-users/Kxw1nnXK12o/my4IYt...
I agree that the installation approach is a total copout, I would have hardlinked all the files from their installation destination to the build artifacts. (presuming state is intended to be thrown away on rebuild.)
Every time I have needed to clean a Makefile based project it has been due to imperfections of the build system, i.e. needing to do a full rebuild. In theory (I've never used tup in anger) you should never need to do that with tup because the dependecy gathering is so robust.
They're make a really clear distinction between a build system, and a build support system. The "clean" command falls into build support, and the Tup team would argue this is better served with techniques like "git clean -xdf"
The "make clean" command is not perfect. Even if someone writes this make phase correctly, it may break after you sync new makefile changes on a dirty project dir. What do you do in that case?
sqlite3 .tup/db 'select name from node where type=4' | xargs echo rm -f
[1]: https://github.com/gittup/tup/issues/120#issuecomment-64832104tup is in some ways brilliant, and i give it kudos for actually improving on make in many ways (unlike 99% of the make-replacement-hopefuls), but man oh man do i value and prefer software devs with a more user-focused philosophy.
(an excellent example of that would be homebrew, which arguably wasn't anything special from a pure-tech perspective, but had amazingly good product sense/user focus. imagine combining that attitude & ux skills w/ the technical brilliance of tup...)
And I agree that `make clean` isn't perfect, I think that is what makes the fact that `tup clean` doesn't exist even more annoying. A lot of people recommend including an auxiliary Makefile or script to do the cleaning, but it can never be as good as if `tup` just did it itself since it is aware of exactly what files it as created during the build process.
I would say the package dependency tree is probably the most important, defining feature of the package system. Second to that is delivery (and the fact it's so high on the list is a testament to its importance, as I can remember a time before it when we just had RPM and dpkg). Some package managers also build form source, but that's a quirk of hose those operate, not really a core requirement to delivering binaries to the end user.
Even FreeBSD's ports build system is largely defined be its dependency tree and the rules they have in place to make fetching and building easy. Whether binaries are delivered from it or source that's compiled locally is somewhat transparent to the end user, other than the extra time it takes.
Although repackaging upstream can be a fuzzy concept. E.g. Ubuntu's early days. They've always repackaged Debian packages, but I believe there was less differentiation in the beginning, before Ubuntu has many of its own initiatives and projects.
OSS maintainers don't have a unified mechanism to alert packagers of updates.
In place of a dirty hack to power IFTTT and Jenkins from fetchmail scripts when there is an announcement, we could have intelligence and transparent automation at the distro-level.
Maintainer patches a 0-day, and there's a mad scramble for everyone to patch, test, package and publish.
I would rather not wait on Debian to release a distribution or be forced to cherry-pick from Sid.
OpenSolaris and OpenBSD could have feature parity with Cygwin and Alpine.
We could have concurrent versioning, automation and container and filesystem-based deployments become a little less risky.
Less focus on Vendor, more on Developer.
That said, I don't think the vendor can be avoided unless you want to live on the edge. If you're on a released version of a distro (otherwise, you wouldn't have to cherry-pick from Sid), chances are you're not on the latest version of the software, and the patch has to be backported by the distro maintainer to version you are actually on.
Gentoo developers have http://euscan.gentooexperimental.org/ . Quite some automation is on the way to come soon.
In particular this idea is intriguing (and the honesty about not having tried it is appreciated):
> Bisection!
> Let's say your compy86 is running fine, and then you update to get the latest, and something breaks. You can use git bisect across your entire linux distribution to find the commit that broke it.
> Note: It may not work across config changes, since tup doesn't automatically run 'oldconfig' when necessary.
> Note: I've never actually tried this, so we'll just assume it works.
> ... you can't use the mark in a way that implies association with or endorsement by the Git project.
It might be worth checking with the Git project to see if they would approve this use or find it within their allowed uses.
See a similar discussion from a few days ago: https://news.ycombinator.com/item?id=15991785
[0]: https://git-scm.com/trademark [1]: https://public-inbox.org/git/20170202022655.2jwvudhvo4hmueaw...
> The USPTO initially rejected our application as confusingly similar to the existing trademark on GitHub, which was filed in 2008. While one might imagine where the "Git" in GitHub comes from, by the time we applied to the USPTO, both marks had been widely used in parallel for years. So we worked out an agreement with GitHub which basically says "we are mutually OK with the other trademark existing".
Upside: It really is the best of both worlds for binary/source distros. 99% of the time it just downloads an archive and unpacks it, so it's just as fast to install or upgrade as e.g. arch. On the other hand, patching software from upstream takes seconds to do (not counting the unavoidable build time).
Downside: In order to accomplish the upside, it completely changes the filesystem layout. "./configure; make; sudo make install" is basically guaranteed to not work. To install software not in nix, you will be writing a nix expression. There is a command to create an FHS compatible namespace that works for e.g. Steam, but applications installed in there are second-class citizens, so it's preferred to not do this for anything that can simply be installed via a Nix expression.
Hardcoding dependencies at build-time seems to me like a fundamental first step towards a stable system that has been absent from Linux since the late 90s.
I have always been curious and dumbfounded why he (Eelco) abandoned Maak (his build system, that I thought, would go together with nix). I think a proper, integrated build system (integrated with nix, the package manager) is a (or the?) missing piece to improve software development.
They go in opposite directions in replacing make:
redo boils down to "How can we capture 99% of the value of make with 1% of the complexity" and does it well. It helps that you don't have to figure out whether sh or make will be interpreting your (similar, yet different enough to trip you up) expressions; redo is "sh all the way down" It also manages to automatically have dependencies on build rules which Make could really use.
Tup is more like "How can we limit what make can do in order to make a better special purpose tool?" Tup can't handle files with spaces and implementing "make install" is flat out impossible. It also is very inflexible with your directory tree. However, it is the best tool for "Many files in, one file out, rebuild if any inputs change" and making this implementation tractable is a specific result of being less flexible than make.
[edit]
Just found the Maak paper:
1) Ports actually does unify building and deployment by using make for deployment
1a) This does require writing (or modifying) makefiles for every single package
2) Nix fits in the same space as Maak, as you use nix both for building and deployment; it just may farm out some of the work to a secondary build system such as make. If you consider the build-script to be the generator (rather than e.g. `cc`) then it maps closely to Maak.
2a) There is no reason I can think of why you couldn't have a nix expression for every single .o file, and have nix-build handle incremental builds for you.
2b) Farming out much of the work to existing build systems is a pragmatic approach to reduce the amount of work needed to create a nix expression. 99% of the time if cmake or autoconf are used, a minimal nix file will give you a mostly working package. This is much more reliable than trying to autoconvert Make (which is composed of two turing complete languages) to some other system.
3) One that the paper claims that only the generators can have complete dependency information (Make, Maak, and Nix all farm out specifying dependency information to the author of the build script; though Nix attempts to prevent you from omitting dependencies by sandboxing), but Tup sidesteps this by sitting between the filesystem and the generator.
https://github.com/ThomasDickey/ncurses-snapshots
This repo is generated by exporting from RCS(!):
http://invisible-island.net/personal/git-exports.html
There's also an unofficial git repo, which is more useful IMO:
1. uncomment the deb-src lines in /etc/apt/sources.list; if you do not have these, simply copy each deb line and have it begin with deb-src.
2. apt-get update
3. apt-get source <packagename>
4. cd <packagename>-<version>
5. (do modifications here)
6. dpkg-buildpackage -us -uc
7. install the .deb packages in the parent directory with dpkg -i
Do note, this does not properly embed your modifications in a patch or whatever, I haven't figured out how to do this and lose my mind at the same time.
I think you could avoid a lot of those bricking issues using btrfs snapshots.
On my desktop, I run into the same problem as you; I think my system partition is just a little small. On the laptops, I have not had any trouble with snapshots.
I do like the idea, though - if an update breaks anything, just roll back to the last working state. Windows did this 15 years ago, it was about time Linux distros started catching up.
Not so much hassle.
Way 1:
1. put your patch to /etc/portage/patches/sys-apps/coreutils/my-ls.patch
2. emerge coreutils
Way 2:
1. ebuild /usr/portage/sys-apps/coreutils/coreutils-8.28-r1.ebuild prepare
2. edit sources in /var/tmp/portage/sys-apps/coreutils
3. ebuild /usr/portage/sys-apps/coreutils/coreutils-8.28-r1.ebuild merge
Really nice to work this way if you debug compiled software a lot.
Explicitly inspired by the former, same build paradigm, doesn't use FUSE. Has some neat git integrations. Doesn't work on macOS/Windows yet though.
One funny feature that falls from the design is that you can just keep running & failing a fac build, and it'll nudge you toward the correct build.
Can anybody tell me when I should use gittup instead? Is it just a matter of size? Is rebuilding significantly faster with gittup due to the automatic rebuilding?
I mean to me it looks that gittup has some serious limitations in its current state (e.g. no 'make') while the others are full blown linux systems.
I'm also curious how else would you do it? The article make it sound unique but this seems like the obvious solution, at least conceptually. How do other distros manage and build their packages?
[1] I am not entirely certain what the situation on DragonflyBSD is, I have not touched it in a long time.
But why Linux kernel v2.6?
$ curl -Is http://gittup.org/gittup/ | grep ^Last-Modified:
Last-Modified: Fri, 23 Mar 2012 23:17:45 GMT