Hacker News Reader

Comment by abcrawf | Hacker News Reader

Can you provide any specifics of the serious issues you’ve encountered with journald and networkd? There were some performance issues with journald early on, but I know a lot of work has gone into it over the years (a coworker of mine, for example, improved the performance of log retrieval by 10x). And networkd has always fulfilled my needs nicely, so I’m curious what sort of trouble you ran into.

View on HN

Only issue I have with networkd is lack of adoption: network manager is pushed hard (Red Hat still clinging to nm for servers, ew) or want to add another abstraction around it (Canonical with netplan). It's an awesome, declarative way to manage networking, embrace it.

Yeah I really like networkd as well, switched my Debian servers to it. Netplan also seems fine.

Man, we fought hard at Red Hat to get them to continue using networkd with their latest container-focused OS. Ultimately, the organization just had too much institutional momentum behind NM. Frankly, I’m disappointed that they lacked the imagination and drive to push the industry forward.

Good on you. I hope they come around. IMO NM has no business on servers and systemd-networkd is relatively robust by now.

Networkd is fine if you don't need vlans, bonding, and a bunch of semi-esoteric stuff. /e/n/i does fine there.

Journald is too monolithic IMO. Direct access to the files in a text format is standard Unix stuff. Forcing everyone through the journalctl tool is rough, performance being just one issue. The binary format has a habit of changing too, having written a journald log tailer.

Overall, like I said, systemd is work software for me. It's fine for the complexity in a company (where you have to balance time and business value), but for my own use I wouldn't choose it.

Hence, please don't touch Alpine ;). I was on KISS previously but got tired of compiling my own packages (and Firefox is nigh impossible to compile in a 'ragged' version package system)

> Networkd is fine if you don't need vlans, bonding, and a bunch of semi-esoteric stuff.

networkd does support vlans, bonding, bridges, and whatnot since quite some time now.

https://wiki.archlinux.org/title/VLAN#systemd-networkd

https://wiki.archlinux.org/title/Systemd-networkd#Bonding_a_...

What more semi-esoteric stuff would you be missing?

> Hence, please don't touch Alpine

I'm with you here though: I too appreciate Alpine being on OpenRC rather than systemd. And hope a musl port does not change the calculation there. You may also appreciate https://chimera-linux.org/ if you haven't checked it out already.

I recently found that systemd-networkd isn’t generating “up” events (or something similar, not posting from work) for bonded interfaces set up by cloud init which causes the network wait job to spin until it times out, adding about 60s to boot time for any debian-based box we run in our medium sized deploy, so maybe a few 1000. I noticed because my dev box took forever to come back after a kernel update and I was sweating that I’d broken something.

Unfortunately I was unable to allocate the time to figure out systemd at the layer where the event is expected but not generated to file a coherent bug. I ended up working around it several layers up by hard coding “bond0” as an argument in some config and moved on.

I don’t think openrc is perfect and I find broken shit in Alpine constantly but when I do it’s a whole lot easier to figure out what broke when you don’t have to dig through several layers of dbus event generators and consumers before figuring out what even happened.

I think I may be facing the exact same issue with bonded interfaces not being brought up properly (unrelated to cloud-init) and was hesitating if I should mention it as a caveat in the previous comment but maybe to your point I'm not 100% confident we're not missing anything in the configuration (:

For now an `ip link set up` hook has been a passable workaround here.

Yeah, the cloud init bit was from memory and might not even be right. What I found was that the dbus event doesn’t get generated for the bonded interface when it comes up.

This person blogged a bit and the details and fix don’t match my situation but the broad strokes do: https://www.thomas-krenn.com/en/wiki/Job_systemd-networkd-wa...

Networks is also fine if you like your PID 1 opening a network socket ... every ... time.

journald is still very, very slow. As in, getting the entries for a given process can take a minute or more, whereas just grepping a normal text file is nearly instant. It doesn't help that the default rotation settings are insane (like, why would using a certain percentage of my total space be fine, what if all programs thought that?).

Generally I don't really understand what journald brings to the table. I'm sure there are esoteric setups where the hash chaining thing makes sense, but meh? A lot of things needed fixing and systemd fixed many of them, but everything journals is a terrible user experience for me.

First, i'm a vocal systems appreciater. But. The one thing that frustrates me about the journal is that there is only one journal. If I have a chatty app, it can entirely take over the journal. Set a 1GB hour al, come back in a couple days, and nothing else is left in the journal other than it logging. I hate this so much.

If I could assign some apps to a secondary journal, or better quota individual services, that would make sure I can keep tabs on my entire system, even when one app is logging heavily.

Check out the LogNamespace option.

Why? Why must someone check out a new more complicated way to do something that was already simple by default before logs were turned into db rows?

Bro no. Logrotate shit was gross and gnarly. Every service has its own way of doing with logs. None of it worked the same. It never was simple it always had been crap.

I have my complaints but not replying on each service to do something special conpatible with each other log rotation system was not a complaint. Having this managed reasonably has been great.

This... is just not true. I use journald extensively on a 600Mhz Cortex-A9 daily and it's not slow. I don't know what kind of log load you have, but even when I have processes crashlooping and am sending logs over SSH over a VPN it's fine.

He's talking about reading logs not writing them.

So am I?

Hey everyone, it worked on his machine so it's actually not a problem and anyone who has a problem is lying or stupid. Glad that's cleared up.

That's not even remotely what I said.

The systemd journald test is the longest one on my 3 GHz Power9 system, nearly timing out:

    1475/1475 systemd:libsystemd / test-journal-verify                                    OK              28.44s

but on x86 it runs in under 1s. Not sure the deal there.

I'm not familiar enough with the POWER ISA or POWER9 systems to have any idea what's going on there, but that is surprising.

> like, why would using a certain percentage of my total space be fine, what if all programs thought that?

Because the journal’s role in a system is inherently different than most programs. It’s collecting logs for everything; not just itself. It seems reasonable to me that a system-wide service would use a percentage of the system’s resources.

Plus, if you don’t like it, reconfigure it. It’s only the default, after all.

> Plus, if you don’t like it, reconfigure it. It’s only the default, after all.

Or -- and I know this is crazy but hear me out here -- if you don't like it, and it brings nothing but problems to the table, just don't use it at all.

What problems?

For me, literally everything it does is a problem, including:

The file format is not text, and I use it under Linux, which is good at processing text files.

The command line arguments are insane.

Bad handling of log lines emitted at system crash.

Broke an entire prod network’s observability stack because ubuntu replaced syslogd with it in the middle of an lts support cycle.

Brings in a systemd dependency, which causes 100x more problems than this.

> For me, literally everything it does is a problem, including:

That makes it sound that your main problem is the fact that it exists.

> Broke an entire prod network’s observability stack because ubuntu replaced syslogd with it in the middle of an lts support cycle.

That's more ubuntu's fault I guess. I don't really remember this happening though.

> Brings in a systemd dependency, which causes 100x more problems than this.

Is it other made up problems or real things?

Tangentially I now have to remove systemd-resolved on each machine. It gets too complicated because I use dnsmasq on family machines and resolved is always trying to open ports to the internet (opensnitch knows).

Already run DNS on my router, so why have a second or third cache on each machine? Network manager is involved also.

Reply on news.ycombinator.com