Hacker News Reader

Comment by mxey | Hacker News Reader

I feel like these articles always do a disservice by misrepresenting things about Linux. Just some parts I stumbled over:

> [about jails] They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.

Yeah, of course, a single-node container tool is easier to run than a Kubernetes cluster.

> Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?

Why do I need to rebuild without Bluetooth support?

> When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism.

systemd starts services in dependency order. The author makes it sound like systemd just randomly shuffles the services each boot for fun.

View on HN

> systemd starts services in dependency order.

I'm not sure if this is still the case, because I no longer use Linux, and I may be misremembering it. But IIRC a good while back, there was no way to guarantee service start order at boot.

You could specify dependencies for some services, but even then, if there was more than one choice of dependency order that satisfied the requirements, systemd did not guarantee that the same one would be used on each startup. Otherwise, services with no dependency specs (or groups of services at the end of the same dependency chain) would get started in parallel. If you really wanted to start a sequence of services in order, doing so by specifying dependencies in separate unit files was really painful and basically impossible to debug, especially since dependencies weren't exactly easily described (various combinations of After=, Requires=, Wants=, BindsTo= that went both ways between targets and service links in the dependency chain).

Please note the "I may be misremembering it" part. My life is a lot happier now that I no longer need to care about Linux userspace stuff so I may be repressing some memories :-D.

Edit:

> Why do I need to rebuild without Bluetooth support?

The author chose a lazy example. Yeah, you don't usually need it. A better example: rebuild with `CONFIG_BFQ_GROUP_IOSCHED=y` to enable proportional weight division of disk bandwidth in CFQ.

If starting a service has dependencies, not explicitly spelling out those dependencies and relying on the init system is akin to relying on unspecified behaviour. So among other things, don't be surprised if that's unreliable.

> specifying dependencies in separate unit files was really painful and basically impossible to debug

This is actually quite easy to check and to claim otherwise is FUD

> This is actually quite easy to check and to claim otherwise is FUD

It's only easy to check if you have a small number of dependent services that mostly go one way. On the general case it absolutely isn't easy to check. If I had a dime every time I've seen non-trivial chains of serial operations implemented through a systemd target that just runs a shell script, I could erect an ice statue for Stephen Bourne and it would be made out of the frozen tears of everyone who's had to deal with both systemd and shell scripts.

Systemd still has no dependency order. You still use Requires=, After= ,etc. Otherwise it'll start when it starts.

I don't think dependency order is particularly valuable in the real world. Manually specifying order imposes a strictness upon reality that isn't there. Most apps are messy and starting a system tends to be a messy affair in of itself.

The much simpler solution to specifying services to start in order is to write a script and have one big service start and terminate them.

But I can't really imagine a real world scenario where that is somehow better than just giving each service the dependency definitions it needs.

edit: the simpler solution to compiling the kernel is just blacklisting the module. Then bluetooth won't be loaded obviously. Though not everything can be a module, in practise I've found it to be rare that I was looking for a kernel option to be enabled that wasn't in my distro (but then again, I run most of my personal stuff on archlinux, which ships a near default config of the kernel anyway).

> I don't think dependency order is particularly valuable in the real world. Manually specifying order imposes a strictness upon reality that isn't there.

I mean... sometimes you have to do it, because that strictness really is there.

> the simpler solution to compiling the kernel is just blacklisting the module.

What module? I'm not talking about removing a module, but changing a compile-time option for one.

Removing a module is certainly best achieved by just blacklisting it, that's why I mentioned this as a lazy example that the author picked. (But, FWIW, even that has unexpected consequences. A while back, for example, you could (sometimes significantly) improve the startup time of most embedded devices running Linux by removing the modules instead of blacklisting it because going through the list of blacklisted modules is surprisingly slow. That's probably not a relevant use case for what the author of this article is describing though)

> sometimes you have to do it, because that strictness really is there.

In that case, wouldn't it be more correct to specify the strictness explicity through service dependencies?

Service dependencies form a DAG. Sequential start of services can be modeled by a DAG in which every node except the first and the last one has exactly one dependency and one dependant. So systemd way of starting services is a superset of the traditional, sequential way.

That's sort of the model that systemd uses (used?). However, non-trivial setups that attempt to use it tend to degenerate into very complex beasts, for various reasons, including:

- Unit properties can be specified both ways. For example, an unit can specify both that it should start after another unit (After=) and that it should start before another unit (Before=). So if you have two units, B can start after A either because B's definition includes After=A, or because A's definition includes Before=B.

- Order and dependency is specified separately. If B depends on A (specified by Wants=A in B's definition) that does not, in fact, guarantee sequential start.

- Some properties only apply to some systemd units. For example, Before= dependencies don't (didn't?) apply to device targets.

> So if you have two units, B can start after A either because B's definition includes After=A, or because A's definition includes Before=B.

That's true. However, most software can be easily misconfigured if you don't know what you're doing. And when creating any kind of DAG, it's implied we need to avoid cycles. I suppose systemd could use a tool for visualizing service dependencies and order.

> Order and dependency is specified separately

Order is stricter than dependency, and it slows down the whole process. If things can run in parallel, they should. If they cannot, then we should specify it explicitly. Leaving out order would make systemd useless for situations when order matters, and leaving out dependency would slow the whole process down.

> Some properties only apply to some systemd units. For example, Before= dependencies don't (didn't?) apply to device targets.

I agree, the rules are pretty complex. I don't like that about systemd at all. Combined with the fact that there are 175 different options [0] you can use in a unit file makes systemd really unpleasant for writing any kinds of services. I have a couple of boilerplate unit files I've written long time ago and I just reuse them with changed parameters.

[0] check for yourself:

    cat <(man systemd.service) <(man systemd.unit) <(man systemd.exec) | grep '^       [A-Za-z]*=' | sed 's/=.*/=/' | sort -u

If it's really needed and there is no way around it, it'll be painful in almost any sane and modern init system (OpenRC isn't that much better at it than Systemd, neither is svcs from Solaris which sucks the most at configs).

A good init system IMO doesn't have deterministic behaviour built in but rather as emergent property. Your start order falls out of the service definition.

---

The bluetooth module. Linux is highly modular and you can blacklist modules you don't want. And you can even compile modules after the fact. If you need bluetooth but your distro didn't ship it, you can in fact compile just that module and load it.

Only some core options need a full recompile of the kernel but that isn't too terrible on most distros (apt source, Arch's ABS, etc.) either.

> The bluetooth module.

What bluetooth module? `CONFIG_BFQ_GROUP_IOSCHED=y` is a cgroup-related option.

It's not pretty, and not because of a lack of tooling (apt source, ABS, whatever). The problem is that, since there's no "standard" configuration that you can depend on (unlike FreeBSD), you can't just take a "standard" system and reliably change a compile-time setting (or compile a module, or blacklist a module). There are a lot more moving parts (other compile options enabled downstream, additional patches applied downstream, especially in LTS kernels, compile-time options for userland applications and so on) which vary from one distro to another.

I personally see the value of this approach. I think it's more flexible than what FreeBSD does. But it is far more time-consuming and, generally, pretty nerve wracking.

NixOS lets you do that kinda of stuff though. You specify that you want a kernel option enabled and it'll make sure that any kernel you install has it enabled or compiles one from the upstream source you set (or the default one).

Even upgrading your kernel will ensure the option is enabled. And it'll ensure that any downstream modifications done by wherever you pull the kernel from has them enabled too.

Any distribution can do that -- at the end of the day, you can always get e.g. the linux-kernel deb-src and set the option in Kconfig. That will not solve the rest of the problem -- interaction with other kernel config flags, or side-effects vs. applications that assume a specific kernel configuration and patch set.

No, NixOS solves that. Apps can require Kernel Config Flags to be set and the interactions between flags can be modelled in NixOS. And it'll all be side-effect free and the nixos rebuild command will tell you at build time if it's wrong.

> I mean... sometimes you have to do it, because that strictness really is there.

In which case you specify service dependencies and systemd will start them one after another, you know, you have the After= option in your units just for that.

In all other cases does it matter jack shit, to put it bluntly, if your bluetooth services start after your NFS share exports or before them? Or is it some sort of OCD speaking?

> In which case you specify service dependencies and systemd will start them one after another, you know, you have the After= option in your units just for that.

First off, After= specifies order, not functional dependency. This doesn't do what you want which, I think, kind of proves my point. The complaint in the original article (and mine) isn't that it's impossible to specify order between two services, which, yes, you can do just fine. The complaint is that doing so repeatably across boot times is impossible (because systemd literally and explicitly does not support it) and that debugging the "weak" version of it is really difficult.

It's really difficult because Aftter= is, literally, one twelfth of the story, possibly less. Order and dependency ae specified not just though After=, but also Before=, Wants=, Requires=, WantedBy=, RequiedBy=, Requisite=, RequisiteOf=, Triggers=, TriggeredBy=, PartOf= and ConsistsOf=.

Ordering and dependency are treated separately by systemd. I think that's a good design trade-off but it's not without consequences.

> guarantee service start order at boot

If the services boot after their dependencies, why do you care what order they boot in?

Plus, it's rare to find a CPU with fewer than four cores these days, so 'order' becomes harder to reason about.

First off, dependencies aren't just other services. You can have "when the network interfaces are up" as a dependency, for example, so this isn't just about how many programs run at once and in what order.

Second, you care about it in scenarios where predictability is required. If your customer says they want their smart TV or their POS or whatever to boot in no more than X seconds under normal operating conditions, it's really not cool if the system can take a slightly different way to multi-user.target. Dependencies across separate chains can contend for resources, for example.

"But these are embedded systems, you should just run rc.local" I hear you say? Right. These things run like fourty services, from telemetry to user interface and from firmware update to encrypted cloud backup, half of them written in Node.js, on top of a system that's basically built for systemd. It's not something you can extricate yourself out of.

I don't know if there are equivalent examples in the server world (not my field), but I wouldn't be surprised if there were.

Lennart wanted/wants devs to switch to "socket-based or bus-based activation", leaving explicitly defined dependencies behind; in that way, he kind of punted on the issue. That said, I find it fairly flexible, but, yes, complex (or not particularly transparent).

> Why do I need to rebuild without Bluetooth support?

Have you tried Bluetooth on FreeBSD? It's an unstable mess which does not support any newer device. Bluetooth always worked for me out-of-the box on any reasonable Linux-based OS. FreeBSD was like 20 years old memories of Linux Wi-Fi Cards. Remember 'ndiswrapper'? That's how bad Bluetooth on FreeBSD is. So the author probably assumes that Bluetooth is a piece of shite everywhere and want even Linux programs without a support or so.

Hah, so there's the problem right there. In Linux, you don't need to disable Bluetooth support, because it just freaking works.

Or your hardware doesn't have any Bluetooth and the relevant modules are never inserted.

Or you insert the module manually to send it a packet that will open a hole that... wait, to do that you need to have host and root access already.

My freaking god, when are fanboys going to grow a clue already?

damn, i do not know if I am happy for remembering or sad i remembered ndiswrapper...

So the advantage of FreeBSD over Linux is that a feature that is broken can be disabled?

> Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?

Why would you when you can just disable the kernel module, then nothing bluetooth works. Why rebuild ?

Good point.

I think determinism would be way better than a system with some pseudo random setup. I think hand waving away the specific language used, ie "determinism" without really talking about what it means or why it does (not) matter is dishonest.

Causality and determinism are different, and it honestly makes sense to have it be random to tease out bugs as early as possible.

For a prime example see hash maps in Go, which is specifically non-deterministic when iterating over them to prevent users from relying on any kind of implicit behavior.

> "When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next."

https://go.dev/blog/maps

> [...] prevent users from relying on any kind of implicit behavior.

Except now you just _know_ that somebody out there is using hash map iteration order as a source of entropy. ;)

Hush!

> "Is map iteration sufficiently random for randomly selecting keys?"

https://stackoverflow.com/questions/41019703/is-map-iteratio...

> systemd starts services in dependency order. The author makes it sound like systemd just randomly shuffles the services each boot for fun.

Systemd has a dependency graph which you can influence, but there is no guaranteed execution order and it does change between boots.

That's an acceptable thing or a bad thing depending on your use-case.

> Why do I need to rebuild without Bluetooth support?

I think the point here is that any code on your system is a potential security liability, regardless of whether the relevant feature is active. Removing linked in code removes potential security liabilities and as such it is always a good thing.

Regarding the original article's question Can you tell my how many steps . . . Ubuntu for example without Bluetooth support?, its not actually that bad

    apt-get source linux-image-$(uname -r)

edit your options and get compiling.

> I think the point here is that any code on your system is a potential security liability, regardless of whether the relevant feature is active.

zzz... Then I guess any binary distro that you don't compile your own kernel with absolute minimum requirement aren't good? Not sure how Red Hat customers are doing their business then.

Something working out well up to the present moment does not mean that it is the best possible solution. Most stuff is garbage and could be improved. If you want the most secure system, you minimize the amount of stuff that’s in it and you increase the quality of that stuff that is there.

Talk is easy. People weigh practicality over theoretical perfection.

> Regarding the original article's question Can you tell my how many steps . . . Ubuntu for example without Bluetooth support?, its not actually that bad

Finally. I'm surprised at how many posters here deflected this point, because last I checked it's actually really easy on major distros.

I guess that tells us something about how many Linux users on HN have ever compiled their kernel, that most were like "LOL why?" and not "sure, it's easy".

>Yeah, of course, a single-node container tool is easier to run than a Kubernetes cluster.

>>comparing even to plain Docker – not to even mention Kubernetes

>Why do I need to rebuild without Bluetooth support?

Because you don't need it? Or you don't want it, but cannot deactivate it in bios? Security and Kernel-size

>> services gonna start because it will be different each time

Please read and don't make your own interpretation like with docker -> kubernetes

> Because you don't need it? Or you don't want it, but cannot deactivate it in bios? Security and Kernel-size

Just blacklist the kernel modules, then. Or even plain delete them from disk.

You don't have to blacklist or delete anything if it is not there to begin with, which is the point of OP.

It's kind of crazy, people always say bloat here bloat there, but then come up with blacklisting....ahhh the forgotten "art" of just having the code you need on your machine ;)

I get the sentiment, but there is also something to be said for having a consistent and stable starting configuration. Yes, you might end up removing Bluetooth modules from servers, but you know exactly what the starting kernel config is. When you have completely custom kernels for each server, you’ll spend half of your time just figuring out what the starting configuration is.

Compiling kernels is really a lost art that many people have never done. It’s probably best to avoid too much of that in production until it’s absolutely necessary.

>It’s probably best to avoid too much of that in production until it’s absolutely necessary.

Well yes, being organized is always important, like a customer of mine wanted all Linux server with GUI and installed all of them as "Desktop-Role" aka preempt-kernel.

Gui-Server...bleh..but Customer is King, but i could at least change he's mind about the kernel so Gui-Server-Role it was.

If you blacklist it, it is also not there since it’s not loaded, and you get the added benefit of not having to recompile the kernel every time there’s an update.

It's there but not loaded.

A slimmed-down kernel make the difference from ~100MB (kernel) + 32MB (Modules) to 2MB (Kernel + Modules), faster load, or for smaller devices, and overall less potential attack surface.

Surely you don't reduce 132Mb of kernel+modules to 2Mb just by disabling bluetooth.

>>A slimmed-down kernel

The magic of reading....

I think you're missing my point.

By the time you want to slim down the kernel that much, Linux, OpenBSD, FreeBSD, they're not that different.

That was not the point of this thread, so back at ya:

> The magic of reading....

It should be clear that Bluetooth is not 90% of the kernel size...that's why i wrote "slimmed down" which means disable/not-compile everything not needed (like just one of 3 firewalls, filesystems, network-stacks etc).

>By the time you want to slim down the kernel that much, Linux, OpenBSD, FreeBSD, they're not that different.

Yes, but you don't do that by blacklisting modules right? Because that was your solution ;)

That was the solution to what this thread is about: easily disabling bluetooth.

No it was about why to build the kernel without some modules.

No. You do that by not enabling support for all that stuff that's available in the kernel but you don't have any idea what it even is.

I don't know why you people keep bringing up those subjects that have nothing to do with the thread.

This is about easily disabling support for one thing. The OP says it's easier in FreeBSD because you add a line to a text file and recompile the kernel. We're saying it's also very easy in Linux even without recompiling the kernel by simply blacklisting and/or deleting a module. Alternatively, recompiling the kernel is also quite easy but what's the point? To shave off 5kb from the kernel, or not even that, since the module is external anyway? My time is worth a lot more than those potential few kbs.

By the time you're fine-tuning your kernel to be the absolute minimum for a specific machine, it doesn't really matter if it's FreeBSD or Linux. It will take time and probably quite a bit of trial and error. And then again, if you have ever run `make menuconfig` on a Linux kernel tree you probably know it's pretty easy. In fact, the discoverability is excellent there, since the options are presented to you with explanations. Is that harder than manually editing a conf file, having to know beforehand exactly what lines to add/delete/edit and with what values? If you think it is, you can also edit the conf file for the Linux kernel.

I have nothing against FreeBSD, but this argument is total nonsense.

> I don't know why you people keep bringing up those subjects that have nothing to do with the thread.

Hint is was not about about Bluetooth ;)

But maybe everyone else is wrong, just not you.

If you need to rebuild the kernel to disable it, it means it was there to begin with.

Otherwise it's just an implementation choice: recompile kernel, or blacklist+delete modules.

> Security and Kernel-size

Oh yeah, since we care about security so much, let's have a whole fucking C compiler suite on every machine because what if we need to recompile the kernel to mitigate a potential problem that might not even be there!?

If you really care that much, you delete the modules and run depmod. It's fast and you don't need to have a compiler on hand.

You can configure "world" too, and you can compile your kernel/world and packages from a jail or vm, with or without poudriere.

But why so aggressive? No one attacks your substitute religion.

I know, when you run out of any arguments of merit, you call the other side "aggressive". But the truth is, you have nothing to say.

Okay, you can set up a whole container so you can recompile your kernel to your heart's delight. Fine. How does it scale? How do you distribute your new shiny kernel to hundreds of machines and keep track which is running which? What if you make a mistake, is it easy to roll back? How easy is it?

Also, need I remind you that a world where you don't have to (but can if you really really want to) compile things to get the results you want is infinitely better than a world where you have to compile things?

>How do you distribute your new shiny kernel to hundreds

Like you distribute a iso or a image, is that really so impossible for you?

>How does it scale?

As much as you want.

>What if you make a mistake, is it easy to roll back?

See you don't even know zfs or be's..or hell version's??...your a evangelist and i don't like them.

>I know, when you run out of any arguments of merit

Seams like you run out of arguments, again you can create your own image of freebsd without compiler, nfs, zfs whatever you want. But your are probably happy having the cruise-ship for the fishing side trip. Have fun with your religion.

You can disable building the "whole fucking C compiler suite" using WITHOUT_CLANG as documented in https://www.freebsd.org/cgi/man.cgi?query=src.conf

By the way, as an example, you can also disable IPv4 and IPv6 in the kernel, I'm not sure how to do it in Linux.

Conflating a C compiler with security... Let me guess, you are some kind of old school script kiddie who believes security is improved by making it harder to compile a 0day? :-D

Yeah, totally clueless BSD fanboy. I have seen many during the time I was a heavy BSD user.

> Yeah, totally clueless BSD fanboy. I have seen many during the time I was a heavy BSD user.

He works at Kyndryl do you know what it is/was?

He might be, but adding some levity to this, it might be that he is just deeply ironic (even if some here complain of things incorrect in his analysis.) :D

Reply on news.ycombinator.com