Hacker News Reader

Comment by amluto | Hacker News Reader

Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.

View on HN

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

Or, you could write a plain text file.

Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.

If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.

In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.

Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone.

Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log aggregation layer.

You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.

My personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.

SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work.

An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.)

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.

Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.

I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?

Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?

On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.

Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.

I want my logs readable on a completely read-only disk. Sure, I can start a daemon, but that’s mildly annoying.

Also, the syslog daemon should be extremely reliable, and throwing giant table scans at it makes this more complex.

Reply on news.ycombinator.com