Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance

> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.

It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...

[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.

[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...

I’m not sure Red Hat ever had a lot of database design expertise internally and that probably explains the design and outcome. As I said earlier, the design was not subject to public scrutiny before it was implemented.

You’re lucky. The original syslog protocol was fire-and-forget UDP (which I believe is still the default, though it’s been ages and I could be wrong) and the daemon was single threaded. I/O or CPU starvation could easily lead to dropped logs.

Well I think we found the reason for journald's complexity right there, systemd devs and reinventing the wheel (plus breaking backwards compat in the process) is a match made in heaven

Had a process quietly in a crash loop for a solid month on my workstation until I figured out what was causing my random network outages.

Im sure this can be automated, but I want to see what Im disabling instead of going bulk all.

But you’re generally right. I think that’s why most databases have the notion of a WAL, which is carefully append-only. But the non-WAL data files in most DBs I’ve used are accessed via mmap.

A long time ago I had this over-optimistic idea: x86 hardware (and probably most other hardware) has these cool hardware-managed dirty page bits. So you would write to a mapped page, not even take a page fault, and the hardware would record that it's dirty. Later on the kernel would notice and flush. Excellent performance.

Hahaha. It's much much much more complex. For various reasons (maybe good, maybe bad -- see below), Linux barely uses the real hardware dirty bit. Instead, when you map a file as shared-writable, at first it might not really be mapped at all. If you read it, it gets faulted in and becomes readable. When you first try to write to it, a page fault is generated, and, on non-FRED x86, the page fault itself is very slow. The kernel will do things, including calling into the FS and updating atime [0], to make the page logically writable. It updates the page tables so that the CPU knows it's writable, and it sets the dirty bit right then (after all, this is a bit faster than letting the CPU set it immediately thereafter when you retry the faulting write).

Okay, now it's writable. Writes are essentially free until the kernel decides to write the data back to the disk. The kernel will mark the page non-writable (because it wants to get notified the next time you try to write to it) and flush the TLB (which is extremely expensive, especially on x86 systems that aren't the latest AMD CPUs). And it will write the page back, more or less as if you had used normal syscalls to write it.

There's more fun, though. Some filesystems and/or backing stores need "stable pages" -- they need the page cache pages that are being written to not be modified while being written back. btrfs, for example, wants to checksum the data and then write the data and the checksum out consistently, and if something changes the data while it's being DMAed, then this can't happen. So special locks might be taken to delay future writes to the page until writeback is done, and that includes blocking the "make writable" page fault handler. Oops, there goes performance.

Could the kernel do better? Probably. Will it? Unlikely in the near future. I've contemplated a special mechanism to map a "fast write" window onto a file that would be permanently writable and use the hardware dirty bit to tell the kernel when to transfer the data out. Even if anyone ever implemented this, it would be a very specialized thing, it would incur polling overhead, and it would be utterly silly to use it for something like syslog.

Just use pwrite or io_uring unless you have actual evidence that mmap is better.

mmap read is a different story, of course.

[0] I think that updating atime at make-writable time instead of at writeback time is both non-performant and semantically incorrect. I've never convinced the maintainers well enough, though.

$ man systemd.exec

Though reading the question again, I should have probably linked to the equivalent of

$ man systemd-system.conf

as well, that's where you can set the default behavior across systemd, not per-service as the first man page is.

If network egress fails and logs are pushed in real time over a connection with no local backing, you face an ugly tradeoff: either drop log data silently (loss of visibility during the very network partition you need to debug) or apply backpressure to services (potentially hanging applications when logging buffers saturate).

A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.

Also, in regards to your original comment:

> This issue report feels like it ought to be accompanied by a fix.

This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!

So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.

[0] AIUI, it came out of Google, but my understanding might be incorrect.

[1] ...regardless of whether or not that effect is intentional...

Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.