Upgrading what might be the oldest running Linux install | Hacker News Reader

Upgrading what might be the oldest running Linux install

theregister.com

"chiark’s skip-skip-cross-up-grade"[1] - blog post the article is about

HN discussion[2]

[1] https://diziet.dreamwidth.org/11840.html

[2] https://news.ycombinator.com/item?id=32256026

The Register's article contains some interesting information about the system being upgraded (it's where the Windows PuTTY client is hosted...) as well as about the original blog post's author, that I was not aware of:

> Jackson is highly qualified to perform such a complex upgrade: as well as being a former Debian project lead, he also wrote Debian's underlying package-management tool dpkg. Jackson stewarded the release of Debian 2.0, the first multi-architecture version of the distribution, and the first to use libc6

The history of dpkg is really interesting, it has its roots in a a biologist trying to avoid ripping his hair out with dependencies so he wrote "StopAlop" (http://www.verycomputer.com/195_d4343843af185e14_1.htm). A few months after he posted it others took on the task of porting it to a more robust shell script which was eventually re-written into C, and slowly evolving into what we have today.

What's more interesting is that a lot of package managers for programming language package managers use (or used) the fundamental concepts in dpkg. For example, for a long long time Composer (the thing that saved PHP from death) was a re-implementation of some parts of dpkg with a language-specific world view. I don't know for sure about others but I think this is also true for NPM, pip and ruby gems.

Something else that I think about a lot: the industry spends a lot of CPU cycles to compute the suitable dependencies, but there are several alternative ways to solve dependencies. But those alternatives don't generate interest because the StopAlop lineage is so built in to how we work on software it almost goes unnoticed as a natural law of the world like gravity, so there's no prompt to directly challenge it.

> Something else that I think about a lot: the industry spends a lot of CPU cycles to compute the suitable dependencies, but there are several alternative ways to solve dependencies. But those alternatives don't generate interest because the StopAlop lineage is so built in to how we work on software it almost goes unnoticed as a natural law of the world like gravity, so there's no prompt to directly challenge it.

If I recall correctly, Composer uses a SAT solver to do this, resulting in absurd amounts of CPU and memory time needed to create lockfiles in some situations. Dunno if they still do that.

They recently updated the algo to handle better performance but I don't know what it is now. Top-of-mind I think it was a 30% wall-time improvement.

I wish those people would work on emerge.

What are "absurd" amounts of CPU and memory? Calculating and pinning dependencies is just a very difficult problem which happens to be reducible to SAT. If you think a SAT solver takes a lot of time and memory, then all alternatives would probably be even slower.

Dependency solvers that obey semver are a really hard problem that have very unintuitive challenges. I think a very large number of people who use them will not have spent a lot of time to think about the issues in detail, just their experiences as a user of them. NPM has less complaints than composer for example, but node_modules has an interesting way out of the problem by making dependencies into a hierarchy but this brings a new bunch of challenges elsewhere (in the runtime, any object transiting between a mutual package which has two different package versions of the same package name in the lineage will be have "weird" behaviour).

  Your Project --depends-on-> Foo@1.2.3 --depends-on-> SomeLib@1.2.3
  Your Project --depends-on-> Bar@4.0.1 --depends-on-> SomeLib@4.5.6

If Your Project touches SomeLib from Bar and passes it into Foo things go sideways. It's easy to state that this violates best practices but it is a reality of how the runtime works. This category of issue is removed in PHP/Python by the package manager (not the language). The trade-off is the large computation that checks for conflicts in SomeLib.

> Dependency solvers that obey semver are a really hard problem that have very unintuitive challenges

throw in platform specific binary packages for which platform triplets can be wildcarded plus dependencies that may vary across these platforms plus specifying package sources for which dependencies may be on another source and you're in for a crazy world of hurt (well, at least I am, given I found some issues in bundler+rubygems and have been trying to solve them upstream for a couple of years)

Anything over "basically none" is absurd, but in this case the worst case really is very bad.

How would you solve it with very little computing resources? I've yet to see any reasonable solution.

There's a few novel techniques such as minimal version selection: https://research.swtch.com/vgo-mvs. I know I've seen a few others in the past years but I'm struggling to remember which ones are non-SAT solving approaches.

npm's direct ancestor is yinst, a Yahoo-internal package manager: https://manpages.org/npm-faq/7

I've done cross-grades on old ARM devices, as Debian gained repos built with more optimizations, and I've done in-place 'upgrades' between different Debian-based distros (e.g., Mint Debian -> Kubuntu).

As long as you have a busybox with a working dpkg in it, it's pretty easy to deal with any dpkg or apt errors you encounter as you see them. If you use these same strategies between Ubuntu releases, you can also 'push through' any brokenness or quirks that the normal wrappers for this process bail and reverse on.

If you run any Debian-based distro, just doing one or two upgrades like this will really liberate you from worrying much about your package manager getting 'broken', and spare you from ever doing clean reinstalld unless you really want to.

Reply on news.ycombinator.com