A log/event processing pipeline you can't have (2019)

From my experience if it comes to log shipping from hosts rsyslog + relp + disk assited in-memory asynchronous queues are preferred, most of the time you just only have network i/o as logs would not touch disk.

The idea is to ship logs off the device ASAP as well as destination acts as a sink server capable to handle most of the spikes withouth stressing local source. All done via rsyslog which also wraps actual logs into json format locally. The glue could be syslog-tag.

At the other end you could have ELK stack and logstash using json_lines codec input (pretty fast) structuring data further to your likings.

Just looking into metrics now the avg time for logs showing in ELK is 7-200ms (the latency comes mostly from specific reads happening against the ES cluster).

As ELK is always the slowest component, dropping logs compressed in-memory directly onto disk is also an option.

One thing to note is that RELP can produce extra duplicates which are easily handled by inserting into Elasticsearch using specific document ID (some performance penalty) which could be some unique hash computed on (log content, timestamp, host) etc. With this in place you can also easily "replay" stream of logs to fill potential gaps.

This type of setup scales really good as well.

Edit: typos

syslog-ng, logstash or fluentd on the host to collect and aggregate logs. (logstash/fluentd can parse text messages with regex and handle a hundred different things like s3/kafka/http but they are much more resource intensive).

kibana or graylog to centralize logs and search, the storage is elasticsearch.

A simple syslog-ng on the devices could probably do the job. Little known fact about syslog, it can reliably forward messages over TCP, logs are numbered, have retries and syslog-ng can do DNS load balancing.

...

> The kernel notices that a previous dmesg buffer is already in that spot in RAM (because of a valid signature or checksum or whatever) and decides to append to that buffer instead of starting fresh.

This sounds like it should be very unreliable. Perhaps it works in practice but I couldn't see myself relying on such a mechanism.

I don't know about anyone else, but I have this inherent hatred of company marketing material disguised as blog posts.

If you are going to write a decent blog post, then write a decent blog post. If people are curious about the author they can look them up (and their affiliation). Don't turn it into a sales pitch.