Hacker News Reader

Comment by throw0101a | Hacker News Reader

I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc.

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)

Is that true even with PRAGMA journal_mode = WAL?

https://www.sqlite.org/wal.html

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.

With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.

journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.

That would make for a fun experiment for a syslog service. I encourage anyone who wants to give it a shot!

Clickhouse seems to be a popular logstore these days. And is column-oriented.

Isn't it at least somewhat heavy weight though?

VictoriaLogs could be a good alternative: it stores logs in column-oriented compressed format, so they occupy small amounts of disk space, and it consists of a single small executable without external dependencies, which stores the logs into a single directory on a local filesystem.

Yes, WAL by definition means it writes the data at least twice.

Yes it's true even with journal_mode=WAL. The same pages are touched (minimum 1 full page plus one per index), only the atomicity layer is different.

> complaints about logs being lost during a power failure or kernel panic

I recall reading somewhere about a thing which persists data in a particular region of RAM that is guaranteed to be left alone by the kernel, and therefore will persist across a reboot. Buffering in such a region could persist data when the kernel panics but it obviously wouldn't survive loss of power.

Wouldn't opening the file O_APPEND (maybe O_DIRECT also?) and using fdatasync be better? That way we've basically implemented a WAL and skipped all the other database parts we don't need or care about.

Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.

[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.

[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...

Reply on news.ycombinator.com