Comment by otterley | Hacker News Reader

Comment by otterley | Hacker News Reader

Parent Full thread

Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):

"""

The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.

"""

See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.

Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.

[0] <https://news.ycombinator.com/item?id=49291376>

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc.

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)

> complaints about logs being lost during a power failure or kernel panic

I recall reading somewhere about a thing which persists data in a particular region of RAM that is guaranteed to be left alone by the kernel, and therefore will persist across a reboot. Buffering in such a region could persist data when the kernel panics but it obviously wouldn't survive loss of power.

Wouldn't opening the file O_APPEND (maybe O_DIRECT also?) and using fdatasync be better? That way we've basically implemented a WAL and skipped all the other database parts we don't need or care about.

Is that true even with PRAGMA journal_mode = WAL?

https://www.sqlite.org/wal.html

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.

With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.

journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.

VictoriaLogs could be a good alternative: it stores logs in column-oriented compressed format, so they occupy small amounts of disk space, and it consists of a single small executable without external dependencies, which stores the logs into a single directory on a local filesystem.

That would make for a fun experiment for a syslog service. I encourage anyone who wants to give it a shot!

Clickhouse seems to be a popular logstore these days. And is column-oriented.

Isn't it at least somewhat heavy weight though?

Yes it's true even with journal_mode=WAL. The same pages are touched (minimum 1 full page plus one per index), only the atomicity layer is different.

Yes, WAL by definition means it writes the data at least twice.

Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.

[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.

[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...

The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions.

This optimally solves the problem of the space taken by logs on disk.

The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.

Witty very large files using some form of indexing (graylog etc) is sensible.

If decompression is a pain though, change your logrotate so it doesn’t compress. Obviously costs more in disk space and less in compute.

You can also write directly compressed and flush (without resetting state) after every line. Let the compression state reset on reboot, it's not that important to preserve it.

I would argue the described format is exactly the origin of the problem.

It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.

In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.

Reply on news.ycombinator.com