Hacker News Reader

Comment by hliyan | Hacker News Reader

I started my career in a world where we did everything using shell scripts running directly on bare metal servers, usually running Solaris, and later SuSe or RedHat. I never understood the "how would you reproduce your setup without Docker (or X, where X is some other technology)". The scripts were deterministic. The dependency versions were locked. The configurations were identical. The input arguments were identical. The order of execution was identical. It all ran on a deterministic computational device. How could it not be reproducible?

View on HN

Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times.

This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.

I tried to explain this topic in the article linked above. Not sure how far I succeeded.

Leaning into LTS is nice until you near EOL and have to migrate everything in an often Herculean effort to work with the next LTS release.

Like 12 years of life cycle is not enough for you to plan a transition?

You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.

I don't really think we're in disagreement here. The longer you wait, the harder the transition will be. LTS is a good foundation, and usually the right choice for "enterprise" or "business" settings, but you should not rely overmuch on any one LTS release's way of doing things, when the wider Linux ecosystem moves much faster.

The longer you wait the harder the pain. The less you wait the more frequent the pain. So it depends on the function that converts intensity and frequency to suffering :p But, most importantly, the fact that LTS gives you a choice is what I was highlighting.

For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.

It's not terrible in my experience of doing it several times now.

It is definitely less terrible than trying to unfuck tangles of terraform / terragrunt / yaml / bits of cloud infra.

I went through the migration from CentOS 6 to 7 and never want to do anything like that again. The good news, I guess, is that it never will happen again: CentOS is basically dead anyway, and it's not likely that so many core pieces of system software will change that drastically anymore.

I did CentOS 3 -> 4 -> 5 -> 6 -> 7 -> Debian. Very few problems.

(30 nodes)

I can't imagine you leaned into any one of those releases, then. That sequence involves major changes to the kernel, the init system, the configuration management tools, the core libraries, Apache, Python, Perl, etc. Any one of those alone could (and did, in my experience) trigger a major rewrite of configuration and/or code.

I'm glad it was painless for you. In my experience, it was not, and most of the reasons were beyond my control.

What does lean into mean here? A lot of software from 20 years ago compiles (if needed) and runs fine on the latest versions.

Every major release of every major distribution makes choices. These are choices about what software to include in the first place, what versions of that software to pin (especially for LTS releases), what default configuration to provide, recommendations about how to solve certain problems, etc. These choices are made based upon the experience and opinions of the distribution maintainers. However, those maintainers are (usually) not major contributors to the software they're distributing. This means distros can make "bad" choices, choosing for example to focus on software that eventually dies out, or recommending configurations that eventually get deprecated or removed, etc. Sometimes, these choices are even made in a way such that they exclude what will become the winning alternative, leaving no migration path except complete and total overhaul.

If all Linux is to you is a place to run some application software, these choices are mostly irrelevant. As long as the software you care about continues to run, the other things are just picayune details. If this comes off as derisive, I apologize, because I'm actually broadly endorsing that view of things, as much as it is possible to achieve. But if you start really taking advantage of the things which the distribution provides out of the box and recommends, especially around large-scale multi-system operation, you end up buying into the distibution's choices. When a large organization you're a part of does it too, now the sunk costs really start to mount. As the Linux ecosystem continues to evolve, especially in different directions than the distribution chose at the time, the cost of migrating to later releases grows. This is all a good reason to me to not marry oneself so tightly to those particular choices, but that isn't always feasible with deadlines and compliance requirements and so on bearing down on the sysadmin.

There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS, leaving lots of people hanging. In that case, I know some who started to pay Red Hat for RHEL, but most seem to have moved on to other distros, like Ubuntu. That kind of migration has a lot of the same issues, too, once again leaving me to recommend not to lean into the particulars too much.

> But if you start really taking advantage of the things which the distribution provides out of the box and recommends, especially around large-scale multi-system operation, you end up buying into the distibution's choices.

You mean management interfaces and repo mirroring stuff provided by the OS vendor, like cockpitd and Satellite and whatever?

Sure, that's part of it, if those tools are used. Daemons like the particular flavor of syslog and cron are also part of it. Patched kernels used to be more common, too. I listed a bunch of things that actually broke for me before in a sibling thread; sometimes it was down to e.g. the Python packages that were in EPEL vs. the Python packages that were actually being maintained by their original authors in PyPI, or various security tools configured around paths that changed, etc. There were usually workarounds or alternatives, but they were more difficult to set up than doing things the "native" way.

I see! Thanks for referring to your sibling post, that definitely made clearer what you're talking about.

And yeah if you package stuff against, e.g., the Python libs included in the distro (or EPEL), you essentially need to maintain a repo as a downstream repo of the distro, then rebuild the whole repo with whatever subsequent release as a new upstream when it's time to upgrade. That kind of thing is doable but it's substantial integration work, and if it's aomething you do once a decade nobody is ever going to be fluent in it when it's time to be done.

I think I'd rather just maintain two repos— one against the latest stable release and one against the upstream rolling release (Fedora Rawhide, Debian Unstable, openSUSE Factory or Tumbleweed, etc.)— and upgrade every 6 months or whatever than leap the wider chasms between LTS releases.

And yeah the Python and Python libs shipped in a distro are generally there for the distro's integration purposes, which may involve different goals and constraints than app developers usually have. Building against whatever a distro ships with is not always the best way, as your painful migrations demonstrated.

> There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS

If you are doing something serious you probably want to chose suppliers in such a way that you can demonstrate you have security and business continuity under control. That means you probably want to use RHEL, Suse or Ubuntu, distributions for which commercial support exists.

(Ubuntu is particularly interesting because you can start with an LTS release for free and activate commercial support if business goes well, without changing your processes.)

You can think about this beforehand or wait until customers require some kind of certification and the auditors ask you for your suppliers list + the business continuity plan, among other things. You will face this if you deliver to a regulated market or if your customers are large enough to self regulate this kind of thing.

LTS not good enough? Well, cloud native does not have LTS comittement and Pipy does not provide security fixes separated from logical changes.

Try to keep your Terraform code stable for two years in AWS, or try to understand the lifecycle of AWS Glue versions from the docs. Or trust that Google will not discontinue their offers :-)

I mean, maintaining software is never easy or effortless but I respect the effort done by LTS Linux providers - they sell stability and security for a fraction of what you pay for cloud native.

Using debian and bash and perl for setup and config. There is almost no work involved in the past decades; everything still works fine. I do not like busy work; trying to do things the hard way is not making me money or giving me happiness; running saas products (on non cloud cheap hardware that never dies) is and that's what I have done for the past 25 years.

There is no need to adapt things that work as they are already.

apache -> nginx. Python versions. postgres. All fine.

Did you crossgrade to Debian in-place?

It's a tradeoff. Doing a big effort once every 4 or 5 years, vs a hopefully smaller effort every year. Sometimes the intermediate smaller steps help you move forward, sometimes it just means more migrations. Sometimes the software/hardware you need means you can't use a LTS OS at all.

If possible, it's nicer to pick established, mature software for as much of your stack as you can, so that there's less of a difference in APIs over longer time frames. But it's not always possible.

What is it that people do that breaks so often due to lack of backwards compatibility from the OS?

IMO, the lure of an LTS is that you don't need to keep testing if your computer is still working every week when a set of updates come. Not that things that your software depends on the details remain frozen. If your software depends on the details of something, you should add it as a dependency.

The bigger problem IMO is not that things break, it's that if you depend on one LTS release too heavily, and you wait too long to migrate from one LTS to another, everything breaks all at once.

What should be a gradual migration as new things develop turns into a singular nightmare.

What are you depending on the OS that isn't extremely backwards compatible?

Once in a decade you get something like a breaking upgrade of nginx, or the glibc debacle of 2003. That may take a person-week to fix[1], what can hardly be called "herculean".

1 - If you go with 1 person * 1 week, if you try to go with 7 people * 1 day, it will suddenly cost 7 person-weeks. But the only way upgrading is such a hurry is if you borked a lot of things prior to it.

Off the top of my head, some of the things that have broken at an LTS transition that I've been involved with are out-of-tree kernel module builds, C code using OpenSSL, Puppet config, Salt config, RPM specfiles, Python code, Perl code, Apache configs, shell scripts, Java code, bootloader configs, bootstrap scripts, and init scripts/configs (esp. sysvinit to systemd). Any one of these things is not a problem in isolation, the problem is due to having to fix all of them all at once. Too much complexity put into any one of them (often arising from external requirements or rushed implementations) also makes migrating harder. Waiting until the 11th hour on the EOL clock just adds to the stress of the process.

Many of my bad experiences were because of corporate policies and lack of proper prioritization at levels above system administration. However, the sysadmin does have some choice in the matter, especially when greenfielding. You can turn stability into a vice if you're not careful.

You said it: Your versions were locked. Therefore it is not constantly up-to-date.

I was pinched myself: Security.

- With the cloud threats, everything needs to be constantly up-to-date. Docker images make it easier than permanent servers that need to be upgraded. We used to upgrade every week, now we’re upgraded by default. So yes, sometimes our images don’t start with the latest version of xyz. But this is rare, downgrade is easy with Docker, and reproduction on a dev engine easier.

- With the cloud threats, everything needs to be isolated. Docker makes it easy to have an Alpine with no other executable than strictly necessary, and only open ports to the required services.

I hate the cloud because 4GB/2CPU should be way enough to run extremely large workloads, but I had to admit that convenience made me switch.

What needs to be constant and up to date is reviewing the new patches and which ones can be released and not locked.

The versions that are not locked can be a test or dev environment that constantly updates and checks for errors.

Security threats are a thing, how we do and don't use technologies as well which ones can also factor in to how much is exposed.

A container is locking the whole OS, on this axis it's not an improvement either direction. You still need a way to update deps.

We did upgrades periodically, each time a conscious choice after reviewing the release notes of the dependency. Occasionally a script would need to be updated, but that was it.

To be fair there's real issues with this approach, too. For example, shell scripts aren't actually very portable. GNU awk vs nawk vs... multiply that by all your tools, and yeah those scripts don't run deterministically (they rely too much on the environment). This alone was a big reason why systemd exists today.

But there's a middle ground here too. To me there's a HUGE gap between Kubernetes distributed systems and shell script free for all.

reproducibility isn't just on your deployments, it's for development too. got old REAL fast when your fancy build doesn't work the same on every devs device or some one off issue with how your dev has setup their environment steals hours from everyone.

it was a big reason why we moved to containers at the bare minimum, because its quick and easy to spin up and destroy and you are guaranteed what runs locally runs on prod. no more "well it worked on my system!".

>reproducibility isn't just on your deployments, it's for development too

Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it.

You can achieve this with Docker as well, if the arrangement is not too complex.

The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.

Wouldnt there be slight differences in different Unix flavors so that the script couldnt run in all of them? If it only worked on Solaris, what would happen if Solaris retired? (Like what happened to Centos)

That's what POSIX was for. Keep your scripts and system calls POSIX compliant and you could move from something like AIX to Linux easily.

POSIX never specified things like disk partitioning or package management, so this still requires something else to give you a working system in the first place.

You know what happened when centos retired? Nothing for us. We still use centos 7 at work as we speak.

Depends on where you are in the ecosystem. If you're running your own service, the only flavors that matter and the ones you're using.

If all my machines are FreeBSD 4.11, I don't care if my scripts don't run on Linux or Solaris or SCO or even FreeBSD 4.8 or 14. I might care someday, but not today.

Maintenance scripts need to run on all the versions in the fleet (usually), but setup scripts can often be limited to the latest version, because why not use the latest OS if you're setting up a new machine.

If you're distributing software, yeah you've got to support a lot of variation. If you're at a shop that runs lots of different flavors, you have to support lots of variation. But a lot of people just pick a flavor and update the scripts as needed when the flavor of the day changes.

Trying to keep dependencies and running services as tight and small as possible helps a lot with keeping up to date on security. Don't need to update things that aren't installed, and may not need to update things that are installed but not running (but sometimes you do).

You will likely have to adapt your scripts for OS-specific or installation-specific tasks like package management and modifying filesystems. In the past I've used Nix (either via `nix run` and `nix shell` or templating in Nixpkgs' `writeScript` or similar) for this stuff to guarantee that I'm always running the same tools regardless of what's installed on the base system. This can free you up to use a different shell, rely on recent features of Bash, use GNUisms in coreutils, sed, grep, find, etc., fix a specific version of jq, use external templating tools, etc. For systemd-based distros, you can even use Nix to manually install system-wide services: just install a package to the default or system profile, and then symlink the included unit files from the profile (not the direct store path) into /etc/systemd. `systemctl daemon-reload` and you can manipulate them in all the usual ways one would with systemd.

Other Unix distros don't have first-class support with Nix so you may need to take some additional care when working out your script (especially the part of it that installs Nix), but if you don't need to set up services this way you can write portable scripts with few limitations that will work across all Linux distros, macOS, probably FreeBSD and maybe NetBSD.

I've never been so lucky as to work at a place that used any Unix flavors other than Linux and macOS, though.

Reply on news.ycombinator.com