Hacker News Reader

Comment by dmwilcox | Hacker News Reader

it's funny that many people mention boot time as a metric when for most servers, VMs and laptops the devices are either always on or sleeping. startup times do nothing in those cases.

personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs

systemd for me is solidly "at work" software, and not stuff I'm a big fan of either (journald, networkd and the silly dns resolver all have serious issues). so please, keep away from my alpine or I'll have to fork -again- :)

View on HN

> it's funny that many people mention boot time as a metric

For the majority of deployments of systemd, that's probably true. Note that this is a musl port, and that the existing musl port is from OpenEmbedded. It's embedded devices that are often using cut-down userspaces (e.g. using MUSL), and embedded devices often do care about startup time because they're not Always-On.

E.g. the VTech baby monitor I've just bought is generally pretty good, but spends >10 seconds at a splash screen when you turn it on. No idea if it's Linux under the hood, but an optimised boot time would be great.

In a previous work project, we need the Linux based software to be ready from low power state quickly (battery powered, only doing work occasionally). By getting the boot time down <1s, I could also avoid a load of sleep-state shenanigans - when we wanted to power save, we just turned off completely (there was already a supervisory micro to decide when to turn back on).

It's pretty trivial for an embedded device to trim down the series of startup tasks to only what's required, and then just write a C program that does those steps. I certainly wouldn't pick up such a gigantic, complex and potentially insecure package as systemd for an embedded project. I want less moving parts, not more

It depends on what you mean by embedded, I suppose. Having a service monitor to restart your jobs is pretty useful too, and it can be tricky enough to write a good one. Something like a baby monitor (to use an example mentioned elsewhere) might run almost as many services as a desktop Linux installation.

Good chance that baby monitor uses something like FreeRTOS, not Linux for doing it's stuff. If you are unlucky, it's Android and they did the bare minimum to have a functional product.

E: Also, Android uses the Linux kernel, but similarities with Linux based operating systems pretty much stop there.

Android is a Linux based operating system so saying it stops there doesn’t make any sense. There is a wild variety of Linux OS out there in embedded encompassing a lot of various patchsets, libc and userland. You just have a bad understanding of what are Linux OS if all you think about is the usual distros.

> Android is a Linux based operating system

Where are the posix utilities ? /s Where is man ? /s

No sarcasm, Bionic libc is a subset of POSIX, Musl strictly follows POSIX 2008 and Glibc is superset of POSIX. 99% of Glibc->Musl porting is dealing with non-POSIX features (a.k.a. GNU extensions)[1]. So in the end we could find that Android apps are more POSIX compliant than software from generic glibc-based distros.

[1] https://wiki.gentoo.org/wiki/Musl_porting_notes

Can you provide any specifics of the serious issues you’ve encountered with journald and networkd? There were some performance issues with journald early on, but I know a lot of work has gone into it over the years (a coworker of mine, for example, improved the performance of log retrieval by 10x). And networkd has always fulfilled my needs nicely, so I’m curious what sort of trouble you ran into.

Only issue I have with networkd is lack of adoption: network manager is pushed hard (Red Hat still clinging to nm for servers, ew) or want to add another abstraction around it (Canonical with netplan). It's an awesome, declarative way to manage networking, embrace it.

Yeah I really like networkd as well, switched my Debian servers to it. Netplan also seems fine.

Man, we fought hard at Red Hat to get them to continue using networkd with their latest container-focused OS. Ultimately, the organization just had too much institutional momentum behind NM. Frankly, I’m disappointed that they lacked the imagination and drive to push the industry forward.

Good on you. I hope they come around. IMO NM has no business on servers and systemd-networkd is relatively robust by now.

journald is still very, very slow. As in, getting the entries for a given process can take a minute or more, whereas just grepping a normal text file is nearly instant. It doesn't help that the default rotation settings are insane (like, why would using a certain percentage of my total space be fine, what if all programs thought that?).

Generally I don't really understand what journald brings to the table. I'm sure there are esoteric setups where the hash chaining thing makes sense, but meh? A lot of things needed fixing and systemd fixed many of them, but everything journals is a terrible user experience for me.

First, i'm a vocal systems appreciater. But. The one thing that frustrates me about the journal is that there is only one journal. If I have a chatty app, it can entirely take over the journal. Set a 1GB hour al, come back in a couple days, and nothing else is left in the journal other than it logging. I hate this so much.

If I could assign some apps to a secondary journal, or better quota individual services, that would make sure I can keep tabs on my entire system, even when one app is logging heavily.

Check out the LogNamespace option.

Why? Why must someone check out a new more complicated way to do something that was already simple by default before logs were turned into db rows?

Bro no. Logrotate shit was gross and gnarly. Every service has its own way of doing with logs. None of it worked the same. It never was simple it always had been crap.

I have my complaints but not replying on each service to do something special conpatible with each other log rotation system was not a complaint. Having this managed reasonably has been great.

This... is just not true. I use journald extensively on a 600Mhz Cortex-A9 daily and it's not slow. I don't know what kind of log load you have, but even when I have processes crashlooping and am sending logs over SSH over a VPN it's fine.

He's talking about reading logs not writing them.

So am I?

Hey everyone, it worked on his machine so it's actually not a problem and anyone who has a problem is lying or stupid. Glad that's cleared up.

That's not even remotely what I said.

The systemd journald test is the longest one on my 3 GHz Power9 system, nearly timing out:

    1475/1475 systemd:libsystemd / test-journal-verify                                    OK              28.44s

but on x86 it runs in under 1s. Not sure the deal there.

I'm not familiar enough with the POWER ISA or POWER9 systems to have any idea what's going on there, but that is surprising.

> like, why would using a certain percentage of my total space be fine, what if all programs thought that?

Because the journal’s role in a system is inherently different than most programs. It’s collecting logs for everything; not just itself. It seems reasonable to me that a system-wide service would use a percentage of the system’s resources.

Plus, if you don’t like it, reconfigure it. It’s only the default, after all.

> Plus, if you don’t like it, reconfigure it. It’s only the default, after all.

Or -- and I know this is crazy but hear me out here -- if you don't like it, and it brings nothing but problems to the table, just don't use it at all.

What problems?

For me, literally everything it does is a problem, including:

The file format is not text, and I use it under Linux, which is good at processing text files.

The command line arguments are insane.

Bad handling of log lines emitted at system crash.

Broke an entire prod network’s observability stack because ubuntu replaced syslogd with it in the middle of an lts support cycle.

Brings in a systemd dependency, which causes 100x more problems than this.

> For me, literally everything it does is a problem, including:

That makes it sound that your main problem is the fact that it exists.

> Broke an entire prod network’s observability stack because ubuntu replaced syslogd with it in the middle of an lts support cycle.

That's more ubuntu's fault I guess. I don't really remember this happening though.

> Brings in a systemd dependency, which causes 100x more problems than this.

Is it other made up problems or real things?

Tangentially I now have to remove systemd-resolved on each machine. It gets too complicated because I use dnsmasq on family machines and resolved is always trying to open ports to the internet (opensnitch knows).

Already run DNS on my router, so why have a second or third cache on each machine? Network manager is involved also.

Networkd is fine if you don't need vlans, bonding, and a bunch of semi-esoteric stuff. /e/n/i does fine there.

Journald is too monolithic IMO. Direct access to the files in a text format is standard Unix stuff. Forcing everyone through the journalctl tool is rough, performance being just one issue. The binary format has a habit of changing too, having written a journald log tailer.

Overall, like I said, systemd is work software for me. It's fine for the complexity in a company (where you have to balance time and business value), but for my own use I wouldn't choose it.

Hence, please don't touch Alpine ;). I was on KISS previously but got tired of compiling my own packages (and Firefox is nigh impossible to compile in a 'ragged' version package system)

> Networkd is fine if you don't need vlans, bonding, and a bunch of semi-esoteric stuff.

networkd does support vlans, bonding, bridges, and whatnot since quite some time now.

https://wiki.archlinux.org/title/VLAN#systemd-networkd

https://wiki.archlinux.org/title/Systemd-networkd#Bonding_a_...

What more semi-esoteric stuff would you be missing?

> Hence, please don't touch Alpine

I'm with you here though: I too appreciate Alpine being on OpenRC rather than systemd. And hope a musl port does not change the calculation there. You may also appreciate https://chimera-linux.org/ if you haven't checked it out already.

I recently found that systemd-networkd isn’t generating “up” events (or something similar, not posting from work) for bonded interfaces set up by cloud init which causes the network wait job to spin until it times out, adding about 60s to boot time for any debian-based box we run in our medium sized deploy, so maybe a few 1000. I noticed because my dev box took forever to come back after a kernel update and I was sweating that I’d broken something.

Unfortunately I was unable to allocate the time to figure out systemd at the layer where the event is expected but not generated to file a coherent bug. I ended up working around it several layers up by hard coding “bond0” as an argument in some config and moved on.

I don’t think openrc is perfect and I find broken shit in Alpine constantly but when I do it’s a whole lot easier to figure out what broke when you don’t have to dig through several layers of dbus event generators and consumers before figuring out what even happened.

I think I may be facing the exact same issue with bonded interfaces not being brought up properly (unrelated to cloud-init) and was hesitating if I should mention it as a caveat in the previous comment but maybe to your point I'm not 100% confident we're not missing anything in the configuration (:

For now an `ip link set up` hook has been a passable workaround here.

Yeah, the cloud init bit was from memory and might not even be right. What I found was that the dbus event doesn’t get generated for the bonded interface when it comes up.

This person blogged a bit and the details and fix don’t match my situation but the broad strokes do: https://www.thomas-krenn.com/en/wiki/Job_systemd-networkd-wa...

Networks is also fine if you like your PID 1 opening a network socket ... every ... time.

The biggest issue I ran into with OpenRC was the lack of metadata related to hot-plugging events.

There currently is no way to fire specific events based on vendor/device id like you would with systemd.

This forced me to write a shim script to enable USB passthrough with arbitrary k8s nodes. The shim was short but is very hacky and relies on 3p packages because the info needed isn't available in stock alpine.

Blog post about it here: https://stephentanner.com/home-assistant-on-k3s.html

Follow up post in the works based on additional improvements up streamed to OpenRC. The biggest issue was getting literally anyone from the project to comment on the PR to progress it forward to acceptance.

it's funny that many people mention boot time as a metric when for most servers, VMs and laptops the devices are either always on or sleeping. startup times do nothing in those cases.

People bring it up because it's usually the first thing they notice. Of course the faster boot times are not what makes systemd great, rather it's what makes those fast boot times possible.

Systemd is an indifferent OS with a great init system?

(I’m guessing this is supposed to be a joke in a similar vein as “Emacs is a great operating system that desperately needs a text editor”.)

You probably are looking for a different word, rather than “indifferent”. Indifference is a property of the observer/subject, not the observed/object. For example, if I don’t care about sports, I might say that I’m indifferent to the outcome of the game my friends are watching.

Perhaps “mediocre” is what you’re looking for?

See definition 2 here:

https://www.collinsdictionary.com/dictionary/english/indiffe...

There exists a world of devices that are off and need to power on within a few seconds. Think of your media console in your car.

Have you actually profiles that copying the 500mb rootfs is the bulk of your boot time? At around 150MiB/s, which is common for USB these days, I would expect that to only take ~4s, and copying from ram location to ram location is far faster. I would expect your boot time is a lot longer than that.

Bootloader most likely operates in much slower safe mode. Perhaps USB2, or even 1.1.

IDK; on my laptop, the boot time is dominated by opening the encrypted disk. From there it's well under 3 seconds to the Xfce login screen. (A T14, m.2 storage, runit as the init system.)

I find the same. Combine that with sleep being far more frequent that restart, and boot times are unimportant.

I think the need for faster boot comes from people who scale cloud servers or containers up and down a lot.

Yes. But a cloud VM instance can boot very quickly, because it has fixed hardware and does not have to probe for physical devices. Waiting for enough random events to happen to init the encryption used to be a major drain of startup time; likely not a problem any more.

Unlocking encrypted disk isn't I/O bound though.

The key derivation step is designed to take a set amount of CPU time (say, a second), so it's difficult to brute force.

Indeed, it ought to take time to afford protection. Optimizing the rest may be fun, but already changes little in a laptop.

(A cloud instance with a disk without encryption is a different business.)

I don't think there are any devices where "withing a few seconds" is a useful target for the startup sequence. Most devices should really be instant, which means hibernation / full memory snapshots and your real boot system becomes mostly irrelevent. Everything else is really in "meh" territory. That said, OpenRC is already quite fast and both my laptop and desktop boot in a few seconds using it - the EFI takes much longer to do whatever it does and hand control off to the kernel than it takes from kernel to desktop login screen.

Boot times are pretty important for serverless environments. Alpine, which uses musl, is very popular in those situations.

I thought serverless environments are containers and thus don't need to boot?

They still have a userspace to bring up.

And don't need systemd for that.

nowadays serverless environments are microvms. that is, a standard machine type that is explicitly virtual (so device drivers have no wait time for hw to init). however, they don't either need userspace to boot because the function/application/whatever is likely init itself or not far off.

so they need to boot, but only the kernel needs to boot.

With the musl/busybox/OpenRC stack, Alpine is top notch when it comes to battery use. Just by switching from Debian to Alpine i doubled the battery lifetime on my Acer A114 from 8 to 20 hours. This includes moderate use.

"Can leave my charger at home" moment for me.

Probably not a musl thing so much as a way less crap running in the background thing.

Systemd daemons and subsystems?

Could probably get similar improvements by using any distro with less "batteries included" than Debian. Though I'm not aware of any except Alpine which offer stable versions.

My Debian 12 VM has been up for 4 days, and Xorg clocked 5 hours of CPU activity in that time, but systemd clocked only 9.5 seconds, and its subsystems each less than that.

So I doubt if systemd and its subsystems are using much energy.

It doesn't seem likely to explain 20 vs 8 hours battery.

Its less the cpu usage, but more the frequency of wakeups.

I run a Void Linux Musl machine at home and cold boot time vs wake up is so trivial that it makes sense for me to shut down when I am finished. It takes about 10 seconds to boot to login, most of which is waiting for the POST operation to complete. In fact, I've has issues with sleep sometimes not waking up and the system hangs with a blank screen forcing a reset obliterating my session. In addition to that, sleep takes 20+ seconds to power down whereas a shutdown takes about 3 seconds.

systemd to me isn't useful at all for the desktop and is a big contraption for firing off VM's and containers in a highly automated server environment. I dont miss it which means I have zero need for it.

"personally I use a custom alpine live-usb that is immutable and I fully shutodwn and start-up all the time... the bulk of my startup time is copying my 500mb rootfs into a tmpfs"

If rootfs is immutable why copy to tmpfs.^1 Genuine question. Not Alpine or Linux-specific. For example, maybe it feels faster running binaries from tmpfs.

Immutable directories can be mounted from compressed files e.g., squashfs, etc., on the USB media. Alternatively one can copy only the binaries that one acually needs at a given time. Then delete them from tmpfs when not in use to free up memory.

1. I do this is so I can pull out the USB stick. But I use a much smaller rootfs that fits in the kernel ramdisk.

With the loss of S3 suspend, fast boot time is important again for laptop users.

What happened to S3 suspend?

Nothing. But many devices today do not support it anymore, instead they support S2Idle. S2Idle is more power-efficient than running-idle but less efficient than even Standby. Its huge advantage is waking up virtually immediately.

Linux has no problems running S2Idle but due to how S2Idle works, more components can wake up your system, therefore more components can misbehave, incl. devices attached by USB. Ugh.

Having my laptop wake up when it's in my bag (and the exhaust vents are blocked) is the last thing I want.

S2idle is also effectively how all smart phones and watches work. Note that not every wake up results in the same results though. Your phone wakes up all the time, but it doesn't necessarily do a lot of work when it wakes up. The same can be true for your laptop. It would be unlikely for it to wake up to the point where it needs active cooling. If it does the OS is doing something wrong.

https://news.ycombinator.com/item?id=41442490

Microsoft sabotaged it.

> personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs

I'm assuming this is for a personal use computer. Can you talk more about the setup here? Sounds interesting. If the OS is immutable, where do you store personal data and how do you perform updates?

"If the OS is immutable, where do you store personal data"

In a mounted folder.

"how do you perform updates?"

You use another image.

And thus a new mind was awoken.

This is patronizing. Especially since the response was tautological. You should not assume that I know nothing.

I was interested in a little more detail. Is the home directory on another USB, or maybe a laptop's built-in SSD? To add new software, is he creating a new custom image every time? Is this practical as a daily driver?

This not true nowadays when most "servers" are containers or VMs that are spun up on demand, and thus do need a fast (fraction of a second) boot time.

What do you do for persistence? Simply copying back tmpfs to rootfs on shutdown?

They said immutable OS, so I'd assume no persistence. Just power off instantly.

I don't want my fridge to take 5 minutes to boot. Seriously.

I'm always super annoyed at the current state of smart appliances that take ages to power on because they run some half-assed linux on an underpowered processor that would give the apollo guidance computer a run for its money. And that's why I only have dumb appliances.

How often does a fridge need to reboot? I’d hope for a nice steady power supply to a fridge.

Reply on news.ycombinator.com