Comment by amluto | Hacker News Reader

Comment by amluto | Hacker News Reader

Parent Full thread

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.

SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work.

An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.)

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.

Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.

I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?

Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?

On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.

Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.

I want my logs readable on a completely read-only disk. Sure, I can start a daemon, but that’s mildly annoying.

Also, the syslog daemon should be extremely reliable, and throwing giant table scans at it makes this more complex.

Reply on news.ycombinator.com